The Adolescents 360 (A360) project ran between 2016 and 2020 to increase adolescent girls’ access to and demand for modern contraception in Nigeria, Ethiopia, and Tanzania.
As COVID-19 swept across the world in Spring 2020, the evaluation team was gearing up to conduct its final round of data collection. This would usually involve our teams spending weeks in A360 communities collecting qualitative data in person – through interviews, focus groups, participatory research and observations.
What we did
In 2020, we had to replace all in-person data collection with remote methods, including interviews with adolescent girls. In total, we conducted 273 remote interviews across the three countries over the phone or using Microsoft Teams. We shortened our topic guides, developed phone-based consent procedures and incorporated public health messages into our interview scripts. One of the biggest challenges we faced was in reaching adolescent girls, who often do not own their own mobile phones.
Our approach to data collection
In Tanzania, COVID-19 had completely halted all A360 activities during the data collection period. Girls were therefore recruited and interviewed entirely remotely, using numbers provided by project participants stored on A360 Tanzania’s ‘call center database’ (where girls had given consent to be contacted by the project). We also worked with peer mobilisers – who had their own phones – who recruited friends and acquaintances who also had phones.
In Ethiopia and Nigeria, service provision was ongoing and so it was possible to conduct interviews with the support of implementers in-country. This enabled the process evaluation team to interview girls who did not have access to a phone. Health Extension Workers (HEWs) in Ethiopia and paid mobilisers in Nigeria were trained on the mobilisation process and provided with personal protective equipment. The HEWs and mobilisers invited respondents in line with process evaluation sampling criteria; and scheduled interviews appointments with them to be conducted in health facilities using the HEW’s or mobilisers’ phones, following a strict physical distancing and sanitisation protocol designed to protect the girls’ health.
What were the advantages of remote data collection?
Catherine Kahabuka and Isabellah Luhanga, Tanzania:
“We managed to reach a larger number of people compared to if we were there in person, after factoring in travelling and so on. You could finish an interview in an hour and then do another one straight away.
“It felt like the girls were a bit freer on the phone because they didn’t see the person interviewing, as compared to when you interviewed them face to face. Usually, they would be less outspoken than this.
“We found it easier with government officials than if we were there in person. In the past, when visiting their offices, we would often have to wait for an hour or two to speak to them, or someone might come in, but the phone experience was different – every time we called they would have an allocated time.”
Morenike Ukpong and Toyin Alaba, Nigeria:
“Before, we might be in a location for a specific number of days. But this time, we were in our own offices conducting the interviews, so we were not limited by the number of days in the field. That’s something that worked well.
“Working remotely provides advantages to the researcher, who is more relaxed in the comfort of their familiar environment, than when working in another country. Also, with the girls, the way we planned the interviews with the mentors worked well. Usually, we would be waiting for the girl, but this time, girls would arrive in time for the interview – they were always there before the time, and appointments were kept. This enabled us to make better use of time.
“The [A360] mobilisers arranged and communicated with the girls. This seemed to make a difference – mobilisers didn’t want their own time to be wasted, so the schedule was perfect.”
What were the key challenges?
Morenike Ukpong and Toyin Alaba, Nigeria:
- Sampling and recruitment
“We would usually be physically present working with A360 implementers to sort through the list of potential participants based on the target audience for the evaluation. We still had to sort out our participants, but we had to do it virtually making us hundred percent dependent on the A360 team. This created some friction with the staff in-country because it introduced new work for their already busy schedule worsened by the pandemic. This prolonged the process because they couldn’t find enough time for us.
“We gave A360 a list of characteristics to select the girls, but during the interviews, we realised that some of the girls hadn’t been selected according to those characteristics. Remote data collection means you don’t have the opportunity to self-correct in the field. For example, the sample of contraceptive users we interviewed was small compared to the non-users.”
- Ensuring privacy and confidentiality
“Our researcher in the North was concerned about privacy. The instructions we gave were to let the girl use a private space. However, sometimes the researcher was able to hear background noises even though she was told the girls were in a private place. This could have interfered with girls’ ability to speak freely.
The researcher also had a feeling that the girls were being trained in terms of what to respond to because the girls kept giving similar comments consistently. If we were there physically, we could have ensured privacy.”
- Non-verbal cues
“We missed the ability to probe for non-verbal cues in phone interviews. Our cultural context makes non-verbal communication a major means of communication. Our inability to observe body language meant we could not probe for non-verbal cues that we otherwise very often identify when we hold face-to-face interviews.”
- Getting hold of stakeholders
“Apart from the girls, interviews with other stakeholders were challenging. Having an appointment time and sticking to it was hard. When you are physically present, they will try to make the time for you even if they are really busy.”
Catherine Kahabuka and Isabellah Luhanga, Tanzania:
“Our experience was not so different from Nigeria – in previous rounds, we would be there for participant recruitment. We would go to the regions a few days before and supervise the recruitment process to make sure we got the right candidates for interviews to minimise the risk of bias.”
- Recruiting participants
“Before, we would work very closely with local leaders and community health workers, who were very attached to the programme so we could avoid depending too much on programme employees to recruit participants. But this time we had to use them to recruit participants, especially teachers, parents, government officials, service providers and the girls.
“We used a snowball technique1 to reach girls in one region. Nigeria did better in the sense that they managed to capture girls who didn’t own mobile phones – in Tanzania the approach prioritised girls with phones.
“We used A360’s database of girls who were registered in the programme and had given consent to be reached, and used the number they gave to the programme to reach them. This limited us to a group of girls who might have been better off because they owned phones. However, some of the girls were using other family members’ phones.
“Contacts of providers were mainly obtained through the programme and that was a potential bias.”
Challenges with data collection in rural Ethiopia
- Network connection issues
“The network connection was not fully reliable to conduct remote phone interviews. In some interviews, the interviewers tried to call again and again to complete the interview. In rare cases, the interviewers were forced to reschedule the interview.
“It was challenging to include girls and other community stakeholders who live in a remote village in the interviews as the phones there didn’t have a signal. We believe there was also a selection bias for the girls who own phones – we didn’t select girls who lived in faraway villages, where network signal is very weak.”
- The position of Health Extension Workers
“Health Extension Workers were also very busy as they are assigned to create COVID-19 awareness among the community. In some scenarios, it was challenging for them to recruit the girls and the community stakeholders for the interview as some of the community were very resistant to talk to Health Extension Workers, as they suspected they could contract the virus from them.”
- Getting hold of stakeholders
“It was challenging to schedule interviews with government stakeholders as they were very busy with Covid-19 related work. It was challenging to interview husbands as it was a peak harvesting season in Ethiopia. At the Woreda2 level, some of the interviewees had a financial expectation to take part in the interview.”
- Other challenges
“One of the data collection sites was a pastoralist community, therefore, it was challenging to schedule interviews with the girls as it was a rainy season when most of the community moves in search of water for their cattle.”
What lessons did we learn about successful data collection?
Serawit Omer, Ethiopia:
- Send a reminder text to interviewees a few hours ahead of the interview
- Use recruitment scripts to make introductory calls, to introduce the purpose of the interview and to build rapport with the respondents.
- Remember to ask the interview participants that their phone is properly charged, they are alone and in a quiet place.
- Inform the interview participants that the interview will be recorded during the introductory call and remember to ask for their consent.
- Use Health Extension Workers to recruit adolescent girls and other community stakeholders. They are close to the community.
Morenike Ukpong and Toyin Alaba, Nigeria:
- Put aside enough time for preparation. This way you can support partners by doing some of the background work and help with the selection process.
- Make sure you can use video calls (e.g via WhatsApp). This can help pick up on some of the non-verbal cues.
Catherine Kahabuka and Isabellah Luhanga, Tanzania:
- Make sure you give recruiters the selection criteria. This will improve the screening process.
- Work more with local leaders and community health workers for recruitment, rather than just A360 staff.
A steep learning curve
While moving data collection remotely involved a steep learning curve and posed many challenges, we ultimately managed to collect a large amount of high-quality qualitative data across the three A360 countries, thanks to the dedication and commitment of our partners and the support of A360 staff.
The final process evaluation report is due to be released in the next month – so watch this space!
