Vitamin A Supplementation Intervention Report
This is a supporting document for our analysis of Helen Keller International, which builds on the work done by highly respected charity analysis organisation GiveWell on Helen Keller International and VAS.
Contents
Evidence on Vitamin A Supplementation (VAS)
a) Summary of evidence - Is there strong evidence that the program is effective?
b) DEVTA trial
c) Are there any negative or offsetting impacts?
Evidence on Vitamin A Supplementation (VAS)
This documents explores three key questions around the evidence on VAS:
Is there strong evidence that the programVitamin A Supplementation is effective at reducing child mortality, and how should we interpret the results from the Cochrane review on VAS?
Should we be concerned that the results from a large, recent VAS trial, DEVTA, do not show a statistically significant effect on child mortality?
Are there any substantial negative or offsetting effects of VAS?
a) Summary of evidence - Is there strong evidence that the program is effective?
A large number of randomised control trials (RCTs) of VAS that were conducted in the 1980s and 1990s found that VAS greatly reduces child mortality.
A 2017 Cochrane review was conducted with a meta-analysis of 19 randomised controlled trials for the effects of VAS on all-cause mortality, involving over 1.2 million children. Its fixed-effect meta-analysis finds that VAS causes a 12% reduction in child mortality (95% confidence interval: 7% to 17% reduction). However, its random-effects meta-analysis was more favourable, finding that VAS causes a 24% reduction in child mortality (95% confidence interval: 17% to 31% reduction).
A fixed-effect model answers the question - what is the best estimate of the intervention effect if we assume that heterogeneity between trial results is purely random (i.e. the true effect of the intervention is the same in every trial)? A random-effects model answers the question - what is the average intervention effect if we assume that there may be heterogeneity in the effectiveness of the intervention across trials? The Cochrane handbook provides a more detailed description of the differences between fixed-effect and random-effects meta-analyses. The difference in results between the two models may be explained by the fact that a random-effects model gives relatively more weight to smaller trials. In other words, the less favourable results of DEVTA (see section "DEVTA" below), a large trial, are given less weight in the random-effects model.
Some systematic differences between trials could include variations in locations, populations (e.g. diet), doses, and time periods that lead to meaningful differences in VAS effectiveness. One possible explanation is that differences in vitamin A deficiency (VAD) prevalenceexplain differences in VAS effectiveness between studies. GiveWell finds that VAD probably varies quite widely between trials found in the meta-analysis. However, there does not appear to be much of a correlation between VAD prevalence and the mortality risk ratio (where a risk ratio <1 represents a mortality reduction from VAS), although it is difficult to source data on VAD prevalence that is both accurate and comparable. As we explore below (see "DEVTA"), baseline mortality rates also differed between trials. Another possibility is that VAS is more effective for populations that have not been vaccinated. Differences in age, sex, or continent are unlikely to drive heterogeneity in effect sizes, as the Cochrane review did not find any statistically significant difference across groups for these factors.
SoGive has reviewed an unpublished work that suggests the random-effects model performs substantially better when there is heterogeneity between studies. The paper shows statistically that there is underlying heterogeneity between the Cochrane review's VAS studies and therefore supports the use of the random-effects model for VAS.
We conclude that the random-effects model is most likely to reflect the true average effect of VAS on child mortality across past trials in different contexts. We therefore use the 24% reduction in child mortality in our cost-effectiveness model, but apply an 85% internal validity adjustment to account for uncertainty regarding use of the random-effects or fixed-effect model.
b) DEVTA trial
A 1999-2004 trial in India with more participants (around one million) than all previous studies combined (the Deworming and Enhanced Vitamin A, or DEVTA, trial) did not find a statistically significant effect on mortality. DEVTA estimates that VAS reduced child mortality by just 4%, and the 95% confidence interval includes the possibility of negative effects (3% mortality increase to 11% mortality reduction).
There is no clear explanation as to why earlier trials and DEVTA found such different results. Some potential explanations include:
The population treated by DEVTA had lower baseline child mortality rates and may have had better overall health than many previously studied populations. Deaths usually averted thanks to VAS may have already been averted through other means (e.g. increased vaccination rates) in the DEVTA population. Two of the six most highly weighted studies in the meta-analysis (including DEVTA) had lower baseline (control group) mortality rates, and also a low mortality reduction after VAS in the treatment group. Differences in overall health or baseline mortality rates could therefore explain differences in VAS effectiveness between studies. In contrast, a review (p.68) of eight earlier VAS trials before 1993 showed no correlation between mortality reduction due to VAS and the baseline mortality rate. The authors of the DEVTA trial also argue that most of the deaths in the DEVTA study were due to preventable infections (e.g. measles) that VAS would have prevented if VAS was effective. This implies that the lower baseline mortality rate may not be driving the differing results between DEVTA and other studies.
Some researchers not involved in the study have pointed to evidence suggesting that DEVTA may have failed to achieve as high a coverage rate as it reported (86%). Others argue that the biological efficacy of VAS had already been established, and that the DEVTA study should have been a programme evaluation, for example investigating bottlenecks to delivery and use. However, the coverage rate appears unlikely to have been the cause of DEVTA's surprising results. If DEVTA's real mortality reduction was in fact similar to other studies (say a mortality reduction of 17%, the lowest mortality reduction within the 95% confidence interval of the meta-analysis without DEVTA), then the DEVTA coverage rate would have to have been as low as 24% to achieve the reported mortality reduction of 4% (i.e. 0.04 / 0.17 = 0.24, assuming that the mortality risk reduction scales linearly with the coverage rate). It is highly unlikely the coverage rate was so low.
DEVTA may have had methodological weaknesses (such as reporting biases and under-reporting) that caused it to fail to detect a statistically significant mortality effect, even if VAS had a real effect on mortality rates in the population studied. DEVTA did not target children in remote areas, who are more likely to suffer from vitamin A deficiency (VAD). Some control group members even received one dose of vitamin A (details here) from the organisation Pulse Polio. However, the treatment group received around 9.5 doses (see this presentation, 18:36 mins), and so Pulse Polio is unlikely to explain the difference in results between DEVTA and previous trials, especially since the WHO (p.4) recommends regular doses for each child. The authors of DEVTA respond to criticisms by arguing that the trial passes two key tests for trial reliability. Firstly, there was good treatment compliance, and secondly, there was no material difference in the number of missed child deaths between the two treatment groups. Some scholars, in support of DEVTA, suggest the intense scrutiny of DEVTA may have been influenced by "belief disconfirmation bias", because the trial found no evidence of significant effect.
Another potential explanation is that VAS has become less effective over time, since DEVTA is one of the two most recent trials (alongside Fisker 2014), both of which found statistically insignificant effects of VAS on mortality. While we take into account the reduction in moderate to high VAD prevalence since the Cochrane review trials (see section "External validity adjustment" in our Helen Keller International report), this may be an imperfect measure of how VAS effectiveness has changed over time. For example, if therecent vitamin A food fortification carried out in recent years has reduced cases of extremely high VAD compared to cases of moderately high VAD, this could reduce the number of lives saved by future VAS programmes, even after considering changes in moderate to high VAD prevalence. However, we have no evidence as to why VAS would have become less effective over time, so we ultimately assume that this is not the main mechanism driving the DEVTA results.
We conclude that the DEVTA trial likely had some methodological weaknesses, but received an unusually large level of scrutiny due to its size and surprising results. A low coverage rate was unlikely to have caused the insignificant effect. For this reason, we do not believe that the evidence justifies placing less weight on DEVTA in a meta-analysis. In contrast, there may have been factors related to the delivery of VAS and the populations involved that caused differing effectiveness between DEVTA and other Cochrane review trials. It is highly unlikely that differences between the results of DEVTA and those of previous large trials were due to random chance alone, if there were no systematic differences between DEVTA and previous large trials (p-value for heterogeneity = 0.001). This further justifies use of the random-effects model, which is explained in section "Summary of evidence".
c) Are there any negative or offsetting impacts?
This section considers factors that could offset the impact of VAS campaigns, either by reducing their effectiveness or contributing to negative outcomes.
Potential VAS interaction with vaccines and increased mortality in some groups:
GiveWell's review of the literature suggests that interactions between VAS and vaccines are not likely to be a concern. Benn et al. 2009re-analysed data from an earlier VAS trial in Ghana to test the hypothesis that VAS reduces mortality in children whose most recent vaccine was a live vaccine (e.g. measles), but could increase mortality in children (particularly girls) whose most recent vaccine was an inactivated vaccine (e.g. DTP). The data re-analysis found that VAS was associated with non-significant increases in mortality among girls who had received measles vaccinations but were missing one dose of DTP. The authors of Fisker et al. 2014, a 2007-2010 VAS trial in Guinea-Bissau, intended to test for interactions between VAS, live or inactivated vaccines, and gender. The trial found no significant effect on mortality overall, a non-significant increase in mortality for boys, and no evidence of a differential effect based on receiving live or inactivated vaccines. Based on the results of Fisker et al. 2014, it does not appear that increased mortality following VAS and live or inactivated vaccinations for boys or girls is a substantial concern. GiveWell has also completed research on the biological plausibility of interactions between VAS, vaccines, and sex, and did not find reasons to believe that harmful impacts are highly plausible.
Adverse side effects of vitamin A supplements:
GiveWell reports that some preschool-aged children experience side effects after taking vitamin A supplements, including loose stools, headache, irritability, fever, nausea, and vomiting. WHO cites an estimate of the prevalence of these types of side effects of 1.5% to 7%.
Potential vitamin A overdose:
GiveWell reports that chronic excessive vitamin A intake can cause a serious condition called vitamin A toxicity (also known as hypervitaminosis A). Cases of vitamin A toxicity occur very rarely and are not known to occur as a result of public health interventions such as VAS programmes.
Diversion of skilled labour:
It is possible that VAS mass campaigns could hinder the ability of staff who participate in VAS programmes, such as Ministry of Health staff, nurses, and other health workers, to complete other duties. The observations of GiveWell staff suggest that VAS campaigns usually take between a few days and a few weeks to complete.
In conclusion, the offsetting/negative effects appear to be minimal.