Invited commentary |
Comment on contributions regarding the impact of the Bangladesh Integrated Nutrition Project
Operations Evaluation Department, World Bank, Washington DC, USA and Institute of Development Studies, University of Sussex, UK
Correspondence: Howard White, Operations Evaluation Department, World Bank, 1818 H Street NW, Washington D.C., 20433, UK. Tel: +1 202 473 1435; E-mail: hwhite{at}worldbank.org
 |
Introduction |
|---|
 Top Introduction EndnotesEvidence regarding impact on...Reasons for low impactHow then is improved...The successes of the...What is to be...Biography and disclaimerReferences |
|---|
The Bangladesh Integrated Nutrition Project (BINP), started in 1995, sought to reduce child malnutrition through growth monitoring and supplementary feeding of children aged 6–24 months, and supplementary feeding for pregnant women so as to increase pregnancy weight gain and, hence, reduce the incidence of low birth weight. The paper by Hossain et al. (2005)
in the January issue of this journal laid out the workings of the programme more fully, and presented evidence of limited impact based on a survey undertaken by Save the Children UK (SC). Two letters received in response to that paper, published in this issue, draw on other data sources to argue that the BINP has in fact been a success to be replicated. But the need to address the issue of malnutrition in Bangladesh, and the undoubted commitment of the NGO workers who are implementing the BINP at field level, should not deflect us from the need to make a rigorous assessment of the available evidence as to how well the programme is working.
The BINP has been heavily studied. In addition to the analysis by Save the Children UK (2003), which is summarized by Hossain et al., and the mid-term and endline evaluations produced on behalf of the project (data from both surveys are reported in Karim et al. 2003, referred to here as the BINP evaluation), there has been an evaluation by the Implementation Monitoring and Evaluation Department (IMED) of Bangladesh's Ministry of Planning (Haider et al. 2004), research undertaken by two Ph.D. students at the University of Cambridge, various operational research studies undertaken by the International Centre for Diarrhoeal Disease Research, Bangladesh (ICDDR,B), data analysis (as yet unpublished) based on data collected by the Nutritional Surveillance Project (NSP) of Helen Keller International (HKI), and the study, with which I was involved, by the Operations Evaluation Department (OED) of the World Bank (World Bank 2005a). In addition, World Bank operational staff have produced an Implementation Completion Report (ICR) (World Bank 2002) and OED a Project Performance Assessment Report (PPAR) (World Bank 2005b).1
What conclusions can be drawn from the weight of the evidence from these different studies? Does this evidence justify the scaling-up of the project to national-level and, if so, what changes in project design appear desirable? This note reviews the evidence presented by the three sets of contributors and from these other analyses of the BINP.
|
Evidence regarding impact on nutritional outcomes |
|---|
|
TopIntroductionEndnotesEvidence regarding impact on...Reasons for low impactHow then is improved...The successes of the...What is to be...Biography and disclaimerReferences |
|---|
The strongest conclusion that can be drawn from reviewing all available evidence is that the project has had, at best, a very modest overall impact on malnutrition. Both SC and IMED calculated ex-post single difference estimates of nutritional impact comparing project and control thanas. Neither study found any significant difference in either moderate or severe malnutrition.2 The problems with using ex-post single differences3 are acknowledged by Hossain et al., and stressed by Sack et al. (2005)
, though the consistency of the results is striking, especially as the results are not that dissimilar from the studies of the BINP using more rigorous methods. The BINP project evaluation undertook three surveys: baseline, mid-term and endline. However, there are problems of comparability on account of differences in survey design, the survey taking place at different times of year and the small size of the control group, especially at the baseline. Single difference estimates between project and control groups using these data find a significant effect on weight for age (WAZ) at mid-term, but not height for age (HAZ). But by the endline there is a significant effect on height for age, but not weight for age, which is difficult to interpret. The double difference estimates between mid-term and endline show little impact on severe malnutrition – less than 2% on HAZ and close to zero for WAZ – and a deterioration relative to the control for moderate malnutrition, which is to be expected if the project is successful in changing severe to moderate malnutrition but does not affect the moderately malnourished.
All of these results compare thana means, and thus conflate participation and treatment effects: if mothers do not enrol in the programme then it is less likely that results will be observed. The OED study used the BINP evaluation data to calculate both thana-level effects and effects on the treated only. Moreover, the analysis used not the control area data, but a statistically constructed control from HKI's NSP data, using propensity score matching. Single difference estimates find positive impacts on both WAZ and HAZ at mid-term and endline, with the effect on WAZ being the larger of the two. These results are more internally consistent than those from the BINP evaluation data, suggesting that the method of constructing the control has produced more reliable results. But the impact found is very small: the project is found to have reduced malnutrition by less than 2%.
As Levinsohn and Rohde (2005) point out, there has been an impressive reduction in the incidence of low birth weight in BINP areas. But what they fail to mention is that an even greater improvement has been observed in the control thanas. Low birth weight incidence was 25.6% in project areas at mid-term, falling to 15.9% by endline, a reduction of 9.7%. But it fell by 12.3% in the control thanas over the same period. Taking this into account, a double difference estimate would suggest that the project increased low birth weight by 2.6%. Similar results are found with respect to pregnancy weight gain.
|
Reasons for low impact |
|---|
|
TopIntroductionEndnotesEvidence regarding impact on...Reasons for low impactHow then is improved...The successes of the...What is to be...Biography and disclaimerReferences |
|---|
How can this low impact be explained? It cannot be discounted by referring to spill-over effects, as claimed by both Sack et al. and Levinson and Rohde. First, the spill-over would have to be nationwide to be picked up in all studies, especially that of OED which drew on national data to construct the control group. Secondly, both the SC and the BINP evaluation data do show significant differences in nutritional knowledge between the project and control after the intervention. Since changes in knowledge are the channel through which spill-over effects would operate, they cannot explain why there is little nutritional difference between project and control areas.
Since the project has been successful in changing nutritional knowledge, why has it not had a larger impact on outcomes as intended by the project's designers? Three main reasons may be advanced. First, as mentioned by Hossian et al., structured questionnaires of beliefs may not be reliable if respondents report the ‘right answer’, even if they remain sceptical. Secondly, the project targeted nutritional education at mothers, but as is well known, and quantified in the OED study using data from the Bangladesh Demographic and Health Survey (DHS), mothers are rarely the sole decision-makers regarding child health and nutrition in Bangladesh. Husbands and mothers-in-law play an important part. Hence, social pressures may prevent the adoption of different practices. Finally, women are constrained from putting knowledge into practice by time or resource constraints. The OED analysis showed that women living with elderly male relatives or who undertook farm work were less likely to rest during pregnancy. Poorer women were less likely to eat more during pregnancy.
Contrary to the claim of Levinson and Rohde, the BINP's inter-sectoral nutrition component did not successfully tackle these constraints. This component was expanded at the behest of government on the basis of the recommendation of an inter-ministerial committee (World Bank 2005b). But once the project began, neither the project office nor the parent ministry (the Ministry of Health and Family Welfare) took much interest; some planned activities did not take place at all. At the field level, activities were implemented by different NGOs than those carrying out the nutrition component. This arrangement resulted in poor coordination and consequent weak targeting, so, as shown by SC's data, the majority of participants in the inter-sectoral activities were not amongst the BINP's target group. The ICR prepared by World Bank operations staff responsible for the project concludes that that the level of inputs was insufficient to bring about any improvement in the nutritional status of children in beneficiary households (World Bank 2002).
In addition to nutritional counselling, the project provided supplementary feeding. This alone may have been expected to yield nutritional gains, but apparently did not. The reasons include weaknesses in the design and implementation of targeting. Growth faltering children were included regardless of their nutritional level, despite the fact that growth faltering is common in healthy, well-nourished children. Data from SC and the Cambridge studies show errors in targeting, with a high proportion of children receiving supplementary feeding not being eligible, pointing to difficulties community workers have in correctly using the growth charts. This difficulty was confirmed in OED's fieldwork. In addition, supplementary feeding may not be supplementary, since, as Hossain et al. point out, there is either leakage (it is fed to people other than the intended beneficiary, which is especially the case for food given to women) or substitution (other foodstuffs are reduced). Evidence of both leakage and substitution is reported in the OED study, which also shows both to be possible, since the requirement that the food be consumed at the Community Nutrition Centre is frequently not enforced.
|
How then is improved nutrition in Bangladesh, in both project and control areas, to be explained? |
|---|
|
TopIntroductionEndnotesEvidence regarding impact on...Reasons for low impactHow then is improved...The successes of the...What is to be...Biography and disclaimerReferences |
|---|
The main cause of nutritional improvement in Bangladesh is most plausibly the substantial rise in daily energy supply which took place in the late 1990s. Food and Agriculture Organization (FAO) data show that the average calories available per person have risen by over 10%, from under 2000 in 1994 to close to 2200 by the year 2000.4 This improvement has been fuelled by increased food availability from rising rice production, on account of the adoption of new technologies. As a result, the real average rice price fell by 14% from the early to late 1990s, and a further 15% from 1999 to 2002, whilst incomes have been growing at a respectable rate.
|
The successes of the BINP |
|---|
|
TopIntroductionEndnotesEvidence regarding impact on...Reasons for low impactHow then is improved...The successes of the...What is to be...Biography and disclaimerReferences |
|---|
The evidence of limited impact does not mean that the BINP is not without its successes. First, the project has raised awareness of nutrition issues amongst both government and non-government organizations. Secondly, there has been successful mobilization at the community level, with high levels of participation in growth monitoring. This success was despite early concerns that social restrictions would limit coverage of growth monitoring sessions. The coordination of growth monitoring with the delivery of antenatal and immunization services has indeed meant that, as pointed out by Levinson and Rohde, coverage of these other activities is greater in BINP areas than elsewhere. Finally, as Sack et al. demonstrate, there have been benefits to some target groups. They present data to show that supplementary feeding for pregnant women has a beneficial impact after enrolment of between 100 and 130 days (though it is not clear what controls, if any, were used, nor whether adjustments were made to allow for selection bias). OED's analysis similarly finds that supplementary feeding for children has a larger impact for children attending on a regular basis, with a still larger impact for the most malnourished children. OED's analysis of DHS data found a similar result with respect to less educated mothers. As reported by Levinson and Rohde, there is also a significant project impact on low birth weight for children born to low-income women.
|
What is to be done? |
|---|
|
TopIntroductionEndnotesEvidence regarding impact on...Reasons for low impactHow then is improved...The successes of the...What is to be...Biography and disclaimerReferences |
|---|
The letters received by this journal in response to the paper by Hossain et al. suggest that, by criticizing the BINP, Save the Children UK are undermining efforts to tackle malnutrition. But this charge can better be laid at the doors of those who would expend scarce resources on activities with at best low impact. At present the BINP is being scaled up to the national level under the National Nutrition Project (NNP) with a largely unchanged approach, though there have been improvements in the targeting criteria for supplementary feeding for children and management of the inter-sectoral nutrition component. Such scaling up will be very costly. It is reassuring to note that the World Bank team responsible for the NNP are undertaking a serious, in-depth review of the available evidence to consider possible redesign of the project. Two areas for improving the cost-effectiveness of the interventions suggest themselves. First, the thrust of the project is behaviour change through nutritional counselling, carried out in the context of growth monitoring. Yet, as Levinson and Rohde suggest, there are many channels for communicating nutritional information, including formal and informal education, media and other NGO projects. Hence, this costly infrastructure may not be necessary, and resources better used in addressing the constraints on utilizing nutritional information. Supplementary feeding is one way of doing this. Secondly, the evidence of significant impact only in certain sub-groups points to the benefits of tighter targeting of these programmes.
|
Biography and disclaimer |
|---|
|
TopIntroductionEndnotesEvidence regarding impact on...Reasons for low impactHow then is improved...The successes of the...What is to be...Biography and disclaimerReferences |
|---|
Howard White is a Fellow of the Institute of Development Studies, University of Sussex, currently on leave of absence at the Operations Evaluation Department of the World Bank. He was task manager and principal author of the report Maintaining Momentum to 2015? (World Bank 2005a
), which included a review of the BINP. He was also a member of the advisory panel for the final report of the BINP project evaluation. The findings, interpretations and conclusions expressed in this paper are entirely those of the author. They do not necessarily represent the view of the World Bank, its Executive Directors or the countries they represent. |
Endnotes |
|---|
|
TopIntroductionEndnotesEvidence regarding impact on...Reasons for low impactHow then is improved...The successes of the...What is to be...Biography and disclaimerReferences |
|---|
1An ICR is a self-assessment produced for all completed World Bank projects. A PPAR is an independent review by OED, produced for a purposive sample of one-quarter of completed projects.
2These results for WAZ (the measure used by the BINP to monitor growth) are as follows. Moderate malnutrition: SC: 35% in BINP and 36% in non-BINP; IMED: 27% in BINP and 27% in non-BINP. Severe malnutrition: SC: 11% in BINP and 12% in non-BINP; IMED: 14% in BINP and 14% in non-BINP.
3A single difference estimate is the difference in an indicator between project and control areas. Double difference is the difference in the change in the indicator over time between project and control.
4This comparison overstates the long-run increase, as daily energy supply had reached close to 2100 after a period of slow and erratic growth since the mid-1970s, but then fallen.
|
References |
|---|
|
TopIntroductionEndnotesEvidence regarding impact on...Reasons for low impactHow then is improved...The successes of the...What is to be...Biography and disclaimerReferences |
|---|
Haider SJ, Hussain D, Nayer I et al. 2004. Impact evaluation of Bangladesh Integrated Nutrition Project. Dhaka: IMED and Read.
Hossain SM, Duffield A, Taylor A. 2005. An evaluation of the impact of a US$60 million nutrition programme in Bangladesh. Health Policy and Planning 20: 35–40.
Karim R, Lamstein SA, Akhtaruzzaman M, Rahman KM, Alam N. 2003. The Bangladesh Integrated Nutrition Project: Endline Evaluation of the Community Based Nutrition Component. Dhaka: The Institute of Nutrition and Food Sciences, University of Dhaka, and Boston, MA: The Friedman School of Nutrition Science and Policy, Tufts University.
Levinson FJ, Rohde JE. 2005. Letter to the Editor. Health Policy and Planning 20: this issue.
Sack DA, Roy SK, Ahmed T, Fuchs G. 2005. Letter to the Editor. Health Policy and Planning 20: this issue.
Save the Children. 2003. Thin on the Ground. Questioning the evidence behind World Bank-funded community nutrition projects in Bangladesh, Ethiopia and Uganda. London: Save the Children UK.
World Bank. 2002. Bangladesh Integrated Nutrition Project: Implementation Completion Report. Report No. 25183. Washington, DC: World Bank.
World Bank. 2005a. Maintaining Momentum to 2015? An impact evaluation of interventions to improve maternal and child health and nutrition outcomes in Bangladesh. Washington, DC: Operations Evaluation Department, World Bank.
World Bank. 2005b. Bangladesh Integrated Nutrition Project: Project Performance Assessment Report. Report No. 32563. Washington, DC: World Bank.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  D. Sridhar Hungry for Change: The World Bank in India South Asia Research, July 1, 2008; 28(2): 147 - 168. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||