Health Policy and Planning Advance Access published online on August 7, 2008
Health Policy and Planning, doi:10.1093/heapol/czn017
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
An integrated method for evaluating community-based safe water programmes and an application in rural Mexico
1 School of Public Health, University of California at Berkeley, USA.
2 Energy and Resources Group, University of California at Berkeley, USA.
3 Unidad de Investigación Epidemiologica y en Servicios de Salud, Morelos, Instituto Mexicano del Seguro Social (IMSS), Mexico.
* Corresponding author. Assistant Professor, Energy and Resources Group, University of California at Berkeley, 310 Barrows Hall #3050, Berkeley, CA 94707, USA. Tel: +1 510 642 1640. Fax: +1 510 642 1085. E-mail: isharay{at}berkeley.edu
 |
Abstract |
|---|
 Top Abstract IntroductionBackgroundFramework for integrated...Application of evaluation...Integrated programme evaluation...ConclusionsEndnotesReferences |
|---|
The burden of diarrhoeal disease remains high in the developing world. Community-based safe drinking water programmes are being promoted as cost-effective interventions that will help reduce this illness burden. However, the effectiveness of these programmes remains under-investigated. The primary argument of this paper is that the biological exposure reductions underlying safe water interventions vary tremendously over space and time, and studies that only report results of intent-to-treat analyses cannot reveal why such programmes succeed or fail. The paper develops a stepwise evaluation framework to characterize, and so analyse, the technical, financial, social and behavioural factors that underlie exposure and mediate the impact of safe water investments. Relevant factors include physical performance of the water system, community capacity to maintain and manage the systems, and the time and budget constraints of households participating in the programme. The approach draws on the public health, community-based resource management, and household choice literatures to identify modifiable points of failure along the causal pathway to programme impact. The evaluation framework is used to assess the performance and impact of UVWaterworks, a community-based water purification system in rural Mexico, 5 years after the programme began. No impact on diarrhoea incidence was found in this case. The assessment method revealed that (a) household priorities and preferences were a key factor in maintaining exposure to safe drinking water sources, and therefore (b) user convenience was a primary leverage point for programme improvement. The findings indicate that a comprehensive examination of the many factors that influence the performance and impact of safe water programmes is necessary to elucidate why these programmes fail or succeed.
Key Words: Access, water, community participation, developing countries, evaluation
KEY MESSAGES
|
|
Introduction |
|---|
|
TopAbstractIntroductionBackgroundFramework for integrated...Application of evaluation...Integrated programme evaluation...ConclusionsEndnotesReferences |
|---|
Diarrhoeal disease remains a major cause of morbidity and mortality in the developing world, responsible for the deaths of an estimated 4000–6000 children each day (WHO 2005
). Although oral rehydration therapy has led to reductions in mortality, there is tremendous need for affordable water, sanitation and hygiene programmes for the 1.1 billion people who lack access to improved drinking water (Hutton and Haller 2004). The Millennium Development Goals emphasize this need with a specific target to halve the proportion of people without sustainable access to water and sanitation by 2015 (UNDP 2006).
During the past 20 years, water investments in the developing world have largely focused on community-level interventions that provide clean drinking water (Clark and Gundry 2004; Harvey and Reed 2006). The World Bank, for instance, invested US$5.5 billion on rural water and sanitation from 1978–2003 and devoted 95% of that funding to community-level interventions such as hand pumps, source water protection and treatments to community tanks (Iyer et al. 2006). Many of these interventions are delivered through decentralized programmes in which communities are responsible for ensuring that their pumps and pipes are maintained well enough to deliver safe water consistently (Bryce et al. 2005; Victora et al. 2005).
Whether existing community-level safe water technologies and programmes are performing well enough to sustain health benefits is an under-investigated question (Moe and Rheingans 2006). The public health and development literatures suggest many possible causes of programme failure, such as recontamination prior to use (Wright et al. 2004) or poor system maintenance (Parker and Skytta 2000). These communities are increasingly calling for rigorous impact evaluations of safe water programmes which examine health impacts, identify how health outcomes can be improved in existing, underperforming programmes, and improve our understanding of what drives variation in programme performance over space and time. Although considerable progress has been made in evaluating the overall health impacts of safe water interventions and programmes (e.g. Poulos et al. 2006; Kremer et al. 2008), there are no established methods for systematically assessing the integrity of the entire causal chain from the targeting of the intervention to its intended health impacts. We remain limited in our ability to connect variability in performance over time and space with the underlying drivers of these variations. This disconnect has resulted in a fragmented picture of how safe water programmes perform over time, and how to identify modifiable programme features in order to improve health impacts.
The purpose of this paper is to present a novel evaluation framework that enables a step-by-step analysis of where things go wrong (or right) in existing safe water programmes. First we examine the contributions and limitations of the public health literature for understanding the performance and impact of such programmes. We then present insights from the literatures on household choice and community-based natural resource management (CBNRM), which can be used to understand the technical as well as behavioural mediators of performance. Combining these literatures, we introduce our integrated evaluation framework, which is specifically designed to help identify modifiable points of failure within existing safe water programmes. Finally, we apply this framework to a community-based water purification programme in Guerrero, Mexico, to illustrate its usefulness in characterizing the causal pathways that result in successful or failed safe water programmes over time.
|
Background |
|---|
|
TopAbstractIntroductionBackgroundFramework for integrated...Application of evaluation...Integrated programme evaluation...ConclusionsEndnotesReferences |
|---|
The public health literature has demonstrated that improvements to drinking water quality can effectively reduce diarrhoea when diarrhoea is caused primarily by waterborne pathogens (Arnold and Colford 2007
; Eisenberg et al. 2007; Luby 2007). Health benefits are greatest when there is low population exposure to unsafe water (e.g. when clean water is used exclusively) and are diminished when exposure to unsafe water is increased (Clasen 2007; Kremer et al. 2008). This exposure-response relationship has long been the underlying rationale, whether implicit or explicit, for programmes that aim to improve health through access to safe water (Clark and Gundry 2004). Figure 1 illustrates how safe water programmes can reduce population exposure to enteric pathogens, and can deliver a beneficial health impact, when each component of the following causal pathway is in place:- Drinking water is a principal route of exposure to enteric pathogens in the population;
- The intervention is operated and maintained to deliver adequate amounts of safe water consistently and reliably;
- The population drinks the safe water consistently and reliably;
- The water is kept free of microbiological contamination until point-of-use.
; Zwane and Kremer 2007). First, targeting of safe water programmes to areas where diarrhoea is primarily caused by poor drinking water quality is imperfect. Many systems are located based on assessments of poor water quality with no additional hygiene data, or based on household surveys that contain no baseline information on domestic hygiene practices (Lenton 2005). Secondly, the technical functionality of interventions is consistently shown to be difficult to sustain. A WHO assessment found that 70% of rural water supplies in Africa were functional at any given time (where ‘functional’ was defined as operating 70% of the time with no more than 2 consecutive weeks of non-functioning) (WHO and UNICEF 2000). Thirdly, usage patterns from any one water source shift over time if the original source declines in quality, or more water sources become available, or both. Lastly, household drinking water supplies can be re-contaminated between treatment and consumption when storage containers are improperly washed or not covered.
|
Given these difficulties, it is perhaps not surprising that there are only a few documented cases of safe water programmes that have achieved their intended health benefits over the long-term (Blum et al. 1990; Hoque et al. 1996
). Of the 33 trials for household- and community-level water treatment systems included in a recent meta-analysis, only three studies examined impacts after 3 years and only one study examined impacts after 5 years (Clasen 2007). These and numerous other studies have provided valuable information on the health impacts of various safe water interventions in diverse settings. All 33 studies discussed in Clasen (2007), however, used intervention/non-intervention groupings to identify health impacts. They did not report exposure variations within the intervention group caused by, for example, system malfunctioning or inconsistent use. This lack of information on the resulting exposure variability limits our understanding of why a health impact was or was not found, and prevents robust comparisons across studies that report only intent-to-treat analyses. Few safe water evaluations measure performance at each stage along the intervention's causal path, nor do they analyse the social and institutional factors that drive technical performance, compliance and household storage practices. Evaluations that rely on an exposed/unexposed dichotomy to examine short-term impact are not appropriate for examining the processes that drive exposure and impact over time in programmatic settings. Studies that aim to understand the impacts of existing interventions need further specification of factors that underlie biological exposures. The relevant question here is not just ‘Did the programme produce an impact?’ but also ‘Why did it do so, or not do so?’
The literatures on community-based natural resource management (CBNRM) and household choice offer many insights into the community- and household-level processes that drive technical performance and system utilization. This study proposes an evaluation approach that integrates analyses of key community- and household-level characteristics into a performance/process assessment of ongoing safe water programmes. The evaluation includes programme targeting, technology functionality and utilization, water storage practices, as well as social and economic factors that mediate these. Underperformance anywhere along the causal path (in one place or time) can result in a programme that is ineffective at reducing exposure and therefore unlikely to deliver a health impact (in that place or time).
This approach helps to identify points of failure and points of leverage within programmes and sheds light on how these pieces fit together to mediate impact on diarrhoea rates. By bringing together the public health and CBNRM literatures, a better understanding of the causal pathways to long-term programme success and failure should emerge.
Our focus is on community-based water treatment, although our general approach could be applied to evaluations of other existing health programmes. We first develop the integrated evaluation framework, we then show how the framework can be applied through a case study of a safe water programme in rural Mexico, and we conclude with a discussion of the utility of the integrated evaluation approach. Our specific goals in the evaluation are (1) to assess the impact of UVWaterworks (UVW), a community-based water purification programme, on diarrhoea case rates in rural Mexico over the first 5 years, and (2) to assess the performance of the programme at each step along the causal path 5 years after the programme began.
|
Framework for integrated programme evaluation |
|---|
|
TopAbstractIntroductionBackgroundFramework for integrated...Application of evaluation...Integrated programme evaluation...ConclusionsEndnotesReferences |
|---|
Our programme evaluation is derived from the stepwise evaluation approach proposed by Habicht et al. (1999
) and integrates a health impact analysis with a process-performance analysis of a safe water programme. In this study, a longitudinal intervention-control plausibility design is used to assess the overall health impact of the programme, and a mixed cross-sectional and longitudinal performance evaluation is used to examine the causal pathway. As each step in the causal pathway is evaluated, the factors known to mediate completion of that step are also identified and assessed. We describe here the steps for the performance evaluation.
The first step in the performance evaluation is to assess the targeting of the programme. Water quality interventions may have minimal health impacts in areas where sanitation and hygiene levels are poor (Gundry et al. 2004). Recent modelling work by Eisenberg et al. (2007) suggests that water quality interventions will reduce diarrhoeal risk in a community when waterborne transmission is non-zero provided that community and household transmission is low. For this reason, it is necessary to assess the sanitation and hygiene levels of the communities.1
The next step is to assess whether the systems deliver safe water consistently and what drives the technical performance (Figure 1). Many rural drinking water systems, particularly community-level infrastructures, are in poor physical repair or are abandoned after 3 to 5 years (Davis and Iyer 2002). For community-managed programmes to deliver safe water reliably over time, the communities must have the capacity either to maintain and operate the systems or to delegate their operation. Not all communities are so capable; a multi-nation evaluation of water systems by Sara and Katz (1997) reported that 50% of the study communities did not have the necessary capacity or the spare parts. Numerous CBNRM studies have shown that programme sustainability depends not only on the technology, but also on the financial, organizational and physical resources of community members (Ostrom 1990; Morgan 2001; Harvey and Reed 2006). Norms and incentives should be in place to encourage the community to maintain and regulate use of the systems and not have them fall prey to the ‘free rider’ problem (Baland and Platteau 1996; Isham and Kahkonen 2001). Therefore, the functional integrity of the systems and the capacity of communities to maintain and manage the systems need to be evaluated.
The third step in the performance evaluation is to assess whether populations consistently drink safe water from the systems and what drives usage patterns (Figure 1). Household preferences2 and values form one such driver of safe water consumption patterns. The frequency with which people drink water from a safe water system depends on, amongst other things, the perceived value of its output (in terms of understood health benefits, taste and aesthetics) (Foltz 1999; Abrahams et al. 2000). Low usage has been attributed to a low preference for safe drinking water because of low awareness of the connection between diarrhoeal diseases and water quality (Gadgil 1998; Jalan et al. 2003). Similarly, higher maternal education is correlated with clean drinking water in the home (Sanchez-Perez et al. 2002; Jalan et al. 2003).
Valuing clean water does not mean that community members will use the safe water system. Many households obtain drinking water from various sources and these sources change over time (van Koppen et al. 2006; Kremer et al. 2008). In limited-resource environments, factors such as cost, time and effort can force trade-offs and compromises on what sources of water to use and for what purpose. Women may prefer more expensive but convenient sources of water because of the disutility of fetching (Whittington et al. 1990) or because of revenues forgone due to work-time lost to collecting water (James et al. 2002). Therefore, to understand programme adoption, we must determine if households understand and value the benefits of safe water, and evaluate how the availability and costs of alternatives affect the utilization of the safe water system.
The fourth and final step is to assess whether the water delivered from a safe source is likely to become re-contaminated in the process of being transported to the home or while stored in the home prior to use. While there is some debate about the relative importance of source water quality versus in-home storage conditions, observational as well as experimental studies in low-income communities around the world have found that improperly transported and stored water significantly attenuates the effectiveness of a safe water technology (Jensen et al. 2002; Gundry et al. 2004). Therefore, testing water quality at the point of delivery, in transit, and at the point of use is essential.
In summary, the performance evaluation consists of four steps: assessing programme targeting, evaluating technical performance through analysis of community management capacity and system functionality, evaluating population usage through analysis of community knowledge of programme benefits and the availability of alternate water sources, and assessing the extent of recontamination through transport and in-home water storage.3 This comprehensive approach addresses the primary links of the causal pathway (Figure 1) and, when paired with a health impact assessment, permits a structured, in-depth examination of what leads to programme success or failure over time.
|
Application of evaluation framework in rural Mexico |
|---|
|
TopAbstractIntroductionBackgroundFramework for integrated...Application of evaluation...Integrated programme evaluation...ConclusionsEndnotesReferences |
|---|
Study site and programme
The State of Guerrero is located on the southern Pacific coast of Mexico. With an annual per capita income that is approximately half the national average (US$2838 vs. US$5051 in 2003) (PNUD 2004), Guerrero is one of Mexico's poorest states. Mortality rates from diarrhoea in Mexico have fallen since the 1980s (PAHO 1998; Lopez et al. 2006), but Guerrero's high mortality rates during the cholera epidemic in the early 1990s indicate that its population continues to be at elevated risk (Cifuentes et al. 1998). While considerable progress has been made in increasing access to safe water, the 2000 census reported that 29% of households in Guerrero still do not have access to piped water (compared with 16% nationally) (INEGI 2000).
In 1998, the Guerrero Department of Health (henceforth, DoH) purchased 60 community-level water treatment systems that used the UVWaterworks technology from Water Health International.4 Comprised of two tanks, a series of filters and an ultraviolet light for disinfection (Figure 2), the UVW systems were designed to disinfect and filter 10 L per person per day for up to 2500 people.5 The systems were installed in rural communities that, in 1998, were known not to have access to safe water. The exact selection criteria are unclear, but the official in charge of the Water and Sanitation Division of the State of Guerrero DoH indicated that the population of these communities was less than the 2500 that one UVW system could fully serve, that electricity was available to power the UV light, and cases of cholera were reported during the epidemic.
|
A community-based management programme was developed around the systems to ensure that the UVW technology was regularly maintained and utilized. The systems were installed in centralized locations such as health clinics and schools, and households were expected to fetch the water from these points. Communities were responsible for the routine maintenance of the systems, as well as for paying for the replacement parts and the electricity that the systems required. Each community determined its own strategy for cost recovery and system maintenance. The government did not seek to recover capital costs. As part of the programme, the DoH contracted with an engineer to visit each UVW village four times per year to make technical repairs and deliver replacement consumables. The UVWaterworks programme was not developed with a process or impact evaluation in mind; neither the DoH nor any other evaluation team conducted comprehensive baseline or periodic surveys. Our integrated evaluation, which took place 5 years after the initiation of the programme, thus evaluates the intervention in a programmatic setting.
Study population and sample size
Health impact
Of the original 60 communities, we identified 31 as study communities based on population size and the existence of both official UVW installation records and weekly health outcome data from 1999–2004.6 In line with the plausibility study design, we selected 1–2 comparison villages for each UVW village, based on shared municipal services, similar proximity to roads and larger towns, and comparable population size (see Table 1). A total of 44 communities without UVW systems were identified as comparison villages.7 This sample of 75 villages is considerably larger than most other community-level evaluations to date, and we expected that it would be large enough to account for intra-class correlation and still detect a difference in diarrhoea case rates (Zwane and Kremer 2007).
|
Programme performance
To evaluate the process-related factors in the performance of the UVW installations, 23 communities were selected for site visits. We identified these villages using a stratified cluster sampling design: all communities with UWV installations were divided into geographical clusters and four clusters were chosen randomly for visits. All but two UVW villages located in the selected clusters were visited and included in this study (Figure 3).8
|
Data collection and analysis
Health impact
Routinely collected data on diarrhoea morbidity trends in UVW and comparison villages were obtained from the Ministry of Health (MOH) in Chilpancingo, Guerrero. The data consisted of age-specific weekly gastrointestinal (GI) illness incidence reports from each of the 75 health clinics for 1998–2004.9 The data from the health clinics were assumed to be a reasonable approximation of incidence in the village. However, the actual incidence of disease might differ from reported incidence because a clinic might serve more than one community, a clinic might not always be open, and not all cases might be brought to the clinic. An underestimate of the true incidence of GI illness was likely due to this selection bias.10
We obtained data on confounding factors such as sanitation levels, maternal education and socio-economic status in UVW and comparison villages from the 2000 national census (INEGI 2000), and figures on village-level enrolment in the poverty alleviation programme Oportunidades from the Government of Mexico. No information was available on hygiene practices.
We conducted an ecological analysis to examine the effect of the UVW programme on community-level incidence of GI illness in children under five between late 1999 and 2004. Multivariate generalized estimating equations (GEE) models were used to analyse the repeated observations of GI cases over time in all 75 villages.11 The impact of the UVW programme on childhood GI illness was modelled using average weekly GI incidence rates from late-1999 to mid-2004 as the primary outcome measure. Other outcome measures included cumulative annual GI incidence and number of weeks when high levels of GI illness were experienced (>5/1000 children). Participation in the UVW programme was the exposure metric for villages since the UVW programme was designed to provide clean water to the entire community. Our hypothesis was that the incidence of GI illness in children under five would be significantly lower over the programme period in UVW villages than in comparison villages. Presence of a certified piped water system in the village, household participation rate in the Oportunidades programme, percentage of the households with a latrine, and average weekly GI rate in 1998 were included in the GEE model as potential confounders and the resulting relative risks (RRs) and confidence intervals were examined. A dichotomous exposure variable was used exclusively because there was insufficient data for dose-response analyses even after the performance evaluation.
Programme performance
Data for this portion of the study were comprised of informational interviews, maintenance record reviews and technical inspections during site visits to the 21 selected villages. The process/performance evaluation was conducted in 2004 with no consistent year-to-year data from 1998–2004.
We assessed the functional integrity of the UVW systems by visual inspection and by testing for the presence of coliform bacteria using Idexx Colilert tests. Historical data on the performance of the 21 UVW systems in 2002–03 were obtained from the DoH in the form of reports that the contracting engineer submitted after each site visit. From the records it was possible to determine whether or not each system functioned technically at the time of the engineer's visit,12 but not to determine the functionality of the system between visits or the use of the system by the community.
We collected data on the organizational, physical and financial capacity of communities to manage the UVW systems in each village through interviews with health clinic staff and community leaders. These data revealed how the routine tasks of UVW system management were organized, the presence or absence of community organizations, the ability of the community to perform routine maintenance tasks, and how capital was raised for maintenance and repairs.
Current and historical UVW water use patterns were assessed through informant interviews with clinic workers and water committee members. Information was also collected on the availability of other water sources within the community, their associated levels of usage and when these alternatives had become available. The National Water Commission (CNA) provided records on the presence of certified piped water systems in communities. No data were collected on the storage and quality of UVW water at the point-of-use because it was found that usage of the UVW systems was extremely low and that nearly all households used water from multiple sources.
For the programme performance study, each step of the programmatic causal pathway (Figure 1) was evaluated. To the extent that the social, financial and institutional components of each step were in place, the system performance was considered adequate and not attenuating the expected health impacts of the UVW programme. These analyses illustrate sources of variation in exposure that ultimately mediate the health impact of the programme.
|
Integrated programme evaluation findings |
|---|
|
TopAbstractIntroductionBackgroundFramework for integrated...Application of evaluation...Integrated programme evaluation...ConclusionsEndnotesReferences |
|---|
Health impact
There was no detectable impact of the UVW programme on GI morbidity by any measure. In the 5 years following UVW system installation, all study villages reported, on average, 25% fewer cases of diarrhoea per week in children under five than were reported in the 2 years prior to installation (Figure 4). The declines did not differ in UVW and non-UVW villages: UVW villages reported 26% fewer cases and non-UVW villages reported 24% fewer (P = 0.76). In the multivariate GEE analyses, no difference was found between the average weekly GI incidence rates in UVW and non-UVW villages over the study period (RR = 1.01, P = 0.77). UVW villages averaged 4.5 cases/1000 children and non-UVW villages averaged 4.4 cases/1000 children. None of the covariates included in the regression model was found to be significantly associated with GI rates. The number of weeks with high numbers of cases did not differ between UVW and non-UVW villages over the programme period. The lack of difference in the GI morbidity between UVW and non-UVW villages by every measure suggests either:
- that there was no effect on GI morbidity due to the presence of a UVW system in the villages, or
- that, given the limitations of our data, in particular our inability to account for changes over time (for instance in education, sanitation or hygiene practices), we were unable to detect a significant effect.
|
Programme performance
Had our health outcome analysis shown a significant and positive reduction in diarrhoeal disease from the UVW programme, we might plausibly have concluded that the programme was successful. Given that we were unable to detect any impact, it is important for health policy purposes to understand why. While one possible reason could be data limitations, a persistent feature of the real world, we also need to examine the performance of the programme itself. We now present the findings for the process-performance study of the 21 villages visited. For each village, each step of the causal pathway (except for the fourth step, for the reasons given above) was analysed. We then determined for which villages the entire pathway was completed (Figure 5).
|
System is installed in area where diarrhoea is caused by waterborne pathogens
In 12 of the 21 villages we visited, more than half of the households reported using latrines in the 2000 national census (INEGI 2000). In these areas, community-wide transmission through poor sanitation was likely to be low enough for water quality improvements to have a health impact, provided household-level transmission was also low. No information was available on domestic hygiene practices, but with 48% of these households participating in Oportunidades, we found the awareness of the importance of hygiene in the population to be high. We expect that at least some diarrhoea in these villages was due to waterborne pathogens and could be reduced through water quality improvements. In the other 9 villages, fewer than half the households used latrines. In these villages, water quality improvements alone may not have been able to reduce the risk of diarrhoea (see Figure 5).
System is maintained such that it delivers safe water consistently
A system is considered adequately maintained when all of its parts are in good working condition and the community has the organizational, physical and technical capacity to maintain the system, or to delegate its maintenance. Based on inspections during site visits, we classified the technical status of the 21 UVW systems into four categories: functioning, requires simple repairs, disassembled, and requires technical assistance (Table 2).
|
Three dimensions of community capacity were investigated: organizational, physical and financial. Information was collected for each of the 21 communities on how the UVW system tasks were performed or delegated, the presence or absence of other community organizations, and how capital was raised to pay for maintenance and repairs. These data were analysed to determine if there were any correlations between community capacity and the technical status of the systems.
Organizational capacity: Each community developed its own system for management of the UVW installation. When the system was located at the health clinic and was technically functioning, the doctor or nurse was usually in charge of UVW maintenance.
The presence of active committees that carry out various community-oriented tasks can be used as an indicator of whether or not the community has organizational capacity. At least one functioning committee was present in most communities in the study, such as local school, water or health committees. Even where the UVW systems were not in use because they needed simple repairs, some committees continued to wash the UVW tanks and to collect contributions from the community to pay maintenance costs. Interviews indicated that free-rider problems were not prominent in any of the 21 villages that were visited.
The presence of a community committee, however, did not always lead to an operable UVW system. In the village of La Dicha, for example, the UVW system was dismantled during the construction of a new health clinic and a piped system was installed that conveyed unpurified river water directly to household lots. Due to insufficient pressure, water could only be supplied to different sectors every four days. As a result, each day, one water committee member was responsible for turning on and off the valves that divided the community into zones. That the piped water system was fully functional indicates that the community has the capacity to manage complex infrastructure. Nonetheless, after completion of the clinic and the water system, the UVW system was not reassembled, even though the river water is not fit for human consumption. Based on this and other site visits, we conclude that the non-functioning of the UVW system is not correlated with the community's organizational capacity.
Physical capacity: Although an engineer is paid by the DoH to provide technical assistance on a quarterly basis, the community is responsible for routine maintenance tasks such as checking the system for UV lamp functionality, broken pipes, leaks and loose seals. In the seven communities where the UVW system was non-functioning but required simple repairs, it was not the physical capacity of the community to perform the maintenance that prevented it from working. The repairs required could have been made with a small investment of labour and capital. For example, a nut and washer had been missing for months in one community and were in fact replaced shortly after our visit. Moreover, in at least two communities the water committee continued to clean the UVW water tanks and to wash the filters, even when the systems were not functioning.
Financial capacity: The exact cost of electricity needed to operate the UVW systems is unknown, as few communities have separate meters for the system. In two communities, families were asked to contribute 5 pesos per month to cover the electrical costs. The cost of replacing the three filters four times a year and replacing the UV lamp once a year is US$460 or approximately 5088 pesos per year.13 However, several communities received free filters from the DoH and thus had even lower annual costs. The cost of replacing a few metres of plastic tubing, a washer or a faucet is low, on the order of a few hundred pesos. Thus for a community of 217 families (the mean number for UVW villages) the estimated cost per family per year to operate the UVW system would be approximately 84 pesos or US$7.55. Given that US$7.55 represents less than 0.3% of the annual average household income in the state, it appears that the UVW system costs do not place an extraordinary burden on a typical community.
Overall, we found no apparent patterns linking capacity characteristics to the technical status of the system. We conclude that the non-functioning or non-use of the UVW systems could not be attributed primarily to community incapacity, or to free-rider problems that often cause community-based resource systems to fail.
Population consistently uses the system
The presence of technically functioning systems does not mean that the system is benefiting the communities.14 Of the 21 villages visited, only eight reported that community members obtained drinking water from the UVW systems (see Table 3 for details). Thus, 5 years after installation, the UVW systems were providing safe water to only a small fraction of the target population.
|
One reason for non-use could be the lack of awareness of the importance of clean water (see, for example, Gadgil 1998
; Jalan et al. 2003). We found the importance of clean water and its relationship with health to be well understood by community members in the visited villages. In 1997, the Mexican Government instituted a conditional-cash transfer programme entitled PROGRESA (now Oportunidades) for low-income families (Braine 2006). Participants in the programme are required to attend several educational workshops a year, many of which focus on health. As a result, everyone interviewed for this research and all focus group participants understood the benefit of using purified water (boiled, chlorinated or filtered) for drinking and cooking. Even in the poorest study communities some of the women interviewed reported boiling their water.15 Furthermore, several of the interviewees commented that UVW water was safe and had a better flavour than chlorinated water from the community's piped water system. We thus conclude that under-valuing the role of clean water in safeguarding health was not the primary reason for low utilization of the UVW systems. Another reason for non-use could be that UVW water is not the preferred source for many households. There are multiple mechanisms through which these households can obtain their drinking water (see Table 4). More than half of the communities have water piped directly into household lots, which in some cases were constructed after the UVW systems had been installed. Several of the piped systems are chlorinated and this water is safe and convenient, although the residual smell of chlorine is unpleasant to several users. Even piped non-chlorinated water must be considered as an alternative to the UVW system as household members may choose to treat water at home rather than haul purified water to their house. Additionally, the availability of bottled water has increased dramatically in Mexico in recent years.16 By 2004, many of the study households could purchase bulk bottled water at a price of 8–20 pesos or US$0.72–1.81 per 19 litre garafón. Bulk bottled water was available in 10 of the 21 UVW villages, especially those that were well connected via main roads to larger towns, and the home delivery of expensive garafónes is preferred by many to free UVW water that has to be fetched.
|
The various literatures that seek to explain why safe water systems are ineffective often jump to the conclusion that community capacity, undervaluing of safe water by users or the physical systems were at fault. In this case we conclude that household preferences, choices and constraints, which are relatively neglected in evaluations of drinking water systems, largely determined how the UVW water was used, by whom it was used, and whether it was used at all.
Discussion of UVW evaluation
Our findings indicate there is no evidence to conclude that the UVW programme in the state of Guerrero has been effective at reducing diarrhoeal disease in children under five. The health impact assessment finds no detectable difference in the weekly rates of GI illness between the UVW and non-UVW villages from 1999–2004. Given the absence of baseline information from the intervention and comparison villages and an inability to fully account for all confounding factors, we cannot be confident about the effectiveness or non-effectiveness of the UVW systems based on an analysis of GI illness alone.
However, the process-performance evaluation reveals that it is unlikely the UVW systems had any effect on reducing GI illness because the pathways of the impact model frequently did not connect. Only two out of 21 communities met all the requirements of effective programme performance. Overall, technical performance of the systems did not appear to be the factor limiting programme effectiveness. As witnessed by other community activities and an analysis of the relative costs of maintaining the system, the study communities had the capacity to use and to maintain the systems.
The process-performance assessment shows that the primary cause of poor UVW programme performance is low levels of demand and thus low usage of the purification system. In the majority of the intervention villages visited, UVW purified water is not being consumed even though community members know the value of safe drinking water and believe that UVW treated water is safe to drink and acceptable in taste. Households decide whether or not to use the programme based on the costs (time, labour and money) of participating in the programme and the alternatives available to them. Water sources that are more convenient are preferred to the UVW systems. The systems that were regularly used were located in communities where there were few other options. Low utilization of the UVW systems, which could not have been detected through the health outcome analysis, helps to explain the indeterminate impact of the intervention on diarrhoeal infections.
In this case study, the primary leverage point for programme improvement is user convenience. International standards require a drinking water source to be no more than 1 km away from the dwelling for the source to count as ‘accessible’ to the household. Yet many households located well within that distance from the UVW systems preferred to treat water at home or to buy bottled water rather than to fetch free and safe UVW water. Encouraging local entrepreneurs to develop a delivery business at modest cost to the households might have increased participation in the UVW programme.17 More generally, these evaluation results implicitly support the critique that community-based, decentralized and ‘participatory’ development programmes too often overlook the value of poor peoples’ time and effort (Cleaver 1998; Chambers 2001).
|
Conclusions |
|---|
|
TopAbstractIntroductionBackgroundFramework for integrated...Application of evaluation...Integrated programme evaluation...ConclusionsEndnotesReferences |
|---|
In this paper, we present a new framework for evaluating the long-term performance of existing safe water programmes. This framework allows for the explicit examination of key community- and household-level characteristics that mediate programme-related reductions in exposure to unsafe water and health impact. We applied this framework to a community-based safe water programme in rural Mexico to understand why a safe water programme did or did not deliver health impacts.
Our primary argument is that the exposure reductions underlying a safe water programme can be undermined for a range of reasons, in the short term and over time, in programmatic conditions. A programme's ability to reduce exposure to waterborne pathogens is determined by the technical performance and utilization of the system, both of which are mediated by behavioural/economic and social/institutional factors. These factors change over time with economic development and its associated social effects, resulting in further spatial and temporal variations in performance and utilization.18 Thus, exposure to waterborne pathogens will vary across even originally comparable communities with the same safe water programme. Studies that report only intention-to-treat analyses of safe water interventions, without sufficient attention to their underlying biological exposures, miss these variations entirely. Comparisons across such studies cannot be robust if there is unmeasured or unreported variation in how an ‘intervention’ was delivered and taken-up by the study population.
Our evaluation framework assesses the separate programmatic pieces that must be in place for exposure reductions to actually occur and for health impact to follow. The framework centres on an explicit model of the causal pathway and helps organize thinking and data collection around the modifiable programme factors that drive exposure reductions. Without details on the use and performance of existing safe water programmes, few points of leverage can be identified for improving programme performance. Our proposed approach allows us to learn from safe water interventions that are already on the ground and that were installed without an accompanying research effort—something that randomized evaluation approaches cannot do.
Using standard indicators of provision, functioning and coverage, the UVW programme in Guerrero would appear to have failed because the systems were largely non-functional and therefore under-utilized. Many assessments of rural water programmes implicitly attribute programme failure to the poor maintenance of the water systems (e.g. Sara and Katz 1997; Parker and Skytta 2000). But our assessment reveals that, in reality, many of the systems were under-utilized and therefore became non-functional over time. While both the standard and integrated evaluations would recommend increasing utilization, our integrated evaluation specifically points to making the systems more convenient for the intended users. From a broader perspective, this approach illuminates the shifting dynamics that underlie programme performance. Such information contextualizes evaluation findings, increasing transparency and helping readers to understand the relevance of those findings to their contexts (Rychetnik et al. 2002; Victora and Schellenberg 2005).
Finally, our integrated approach suggests that researchers and practitioners in the safe drinking water arena re-think current notions of ‘success’ and ‘failure’. Even though the UVW systems appeared to be ineffective in the prevention of diarrhoeal diseases, failure because of serious technical flaws or inadequate community capacity is entirely different from failure because of the presence of alternative and preferred supplies. Furthermore, in some cases the UVW purification systems in Guerrero continue to be used by a subset of the community, for example the local school, the primary health clinic or the poorest residents. These backup uses are of value to the community, and such systems are arguably not failures even though the served populations are much smaller than the original targets. Rural drinking water systems are usually considered sustainable if they continue to provide safe water to the majority of their target populations over several years. But deconstructing the ‘failure’ of the Guerrero UVW systems, a process enabled by the stepwise evaluation approach taken in this paper, shows that success and failure may have to be redefined in a landscape of economic development and changing choices.
|
Acknowledgements |
|---|
We are grateful to Ben Arnold for helpful comments on an earlier draft of this paper. We also thank Ashok Gadgil for initiating our involvement in the research that made this paper possible. We particularly thank the residents of the several villages we visited in Guerrero for generously sharing their time and knowledge with us. This research was funded by grants from COR (University of California) and from the UC MexUS program.
|
Endnotes |
|---|
|
TopAbstractIntroductionBackgroundFramework for integrated...Application of evaluation...Integrated programme evaluation...ConclusionsEndnotesReferences |
|---|
1 For safe water programmes to be effectively located, the sanitation levels and hygiene practices in target communities should be evaluated as part of a baseline study (Poulos et al. 2006
). Whether or not such baseline information exists, these factors should be re-assessed in the evaluation of ongoing programmes. 
2 We use the term ‘preference’ in the economist's sense, meaning ‘what consumers want’. These preferences are revealed by the consumers’ choices, given resource limitations (http://www.economist.com/research/Economics/).
3 This fourth step, assessment of re-contamination in transport and storage, was not carried out systematically for our case study for reasons explained in the section ‘Data collection and analysis’, under sub-section ‘Programme performance’.
4 Details of the technology can be found at http://www.waterhealth.com/water-solutions/.
5 The World Health Organization and the United Nations consider 20 litres per person per day to be the minimum amount of potable water needed (WHO and UNICEF 2000). Ten litres is adequate, however, for drinking and cooking.
6 From the original 60, 21 communities were removed because the populations served by the clinic containing the UVW system had grown to over 2500; four were removed for lack of adequate maintenance records; another four were removed because of the inadequacy of the GI data reported from their local clinics.
7 To the extent that the treatment and comparison villages are not perfectly matched (see Table 1), the UVW villages appear to be somewhat better off than the non-UVW villages. Better-off communities are less likely to under-report children's illnesses, but are also likely to have lower rates of GI illness.
8 Two villages are not included in the study. One was not visited because of logistical constraints; the other was visited but the UVW system was inaccessible and no useful data came from the visit.
9 Weekly health incidence data were not available for all communities for all years. Missing data were reported for no more than two villages in any given year. Variation in the number of villages included in the health impact analysis each year was accounted for in the analyses.
10 Our study implicitly assumes that the extent of under-reporting was not likely to be different between UVW and non-UVW villages. Both UVW and non-UVW communities are mostly populated by poor rural residents, and the government-run local clinics are free, accessible to most and are usually the only health care option.
11 Model specifications included an exchangeable correlation structure, robust estimates for variance, and a negative binomial family. Exposure significance, confounding and effect modification were assessed for each outcome, using the hierarchical backward elimination strategy.
12 For each visit, the engineer filed a report with a checklist indicating if: the storage tanks were full of untreated water; the valve was in place; the 30, 10 and 5 micron filters were in place; the carbon filter was in place; the UV lamp worked; the distribution/dispersal tanks were filled with clean water; the pump was working; and the voltage was adequate.
13 In 2006, US$1 = 11.06 pesos.
14 This is a common assumption in epidemiological studies for community-level sources (Clasen 2007) and in indicators that measure access to safe water.
15 We have no evidence of boiling water being the norm in these villages, or of the extent to which these respondents actually purified their water, but their responses show that they were aware of its importance for health.
16 In 1999–2000, the sales of bottled water in Mexico rose by 11.4%, and in 2000–01 they rose again by 14.9%. The sales growth in 2006–07 was 19.3% (http://www.cronica.com.mx/nota.php?id_nota=291821).
17 This point was raised by the official currently in charge of the Water and Sanitation Division of the State of Guerrero DoH in a discussion on how the UVW programme could have been made more effective. Water Health International has now initiated such delivery models for their current projects in Andhra Pradesh, India.
18 Policy-level indicators of ‘access’ or ‘coverage’, used both within nations and for international comparisons, are thus generally misleading because presence of a ‘safe’ source does not mean that it is actually reducing exposure to pathogens.
|
References |
|---|
|
TopAbstractIntroductionBackgroundFramework for integrated...Application of evaluation...Integrated programme evaluation...ConclusionsEndnotesReferences |
|---|
Abrahams NA, Hubbell BJ, Jordan J. Joint production and averting expenditure measures of willingness to pay: Do water expenditures really measure avoidance costs? American Journal of Agricultural Economics (2000) 82:427–37.[CrossRef][Web of Science]
Arnold BF, Colford JM Jr. Treating water with chlorine at point-of-use to improve water quality and reduce child diarrhea in developing countries: A systematic review and meta-analysis. The American Journal of Tropical Medicine and Hygiene (2007) 76:354–64.
Baland J-M, Platteau J-P. Halting degradation of natural resources: Is there a role for rural communities? (1996) Rome: Food and Agricultural Organization of the United Nations.
Blum D, Feachem RG. Measuring the impact of water supply and sanitation investments on diarrhoeal diseases: Problems of methodology. International Journal of Epidemiology (1983) 12:357–65.
Braine T. News: Reaching Mexico's poorest. Bulletin of the World Health Organization (2006) 84:592–3. Online at: http://www.who.int/bulletin/volumes/84/8/news10806/en/index.html.[Web of Science][Medline]
Bryce J, Victora CG, Habicht JP, Black RE, Scherpbier RW. Programmatic pathways to child survival: Results of a multi-country evaluation of integrated management of childhood illness. Health Policy and Planning (2005) 20(Suppl 1):i5–i17.[Abstract]
Chambers R. Qualitative approaches: Self criticism and what can be learned from quantitative approaches. Qualitative and quantitative poverty appraisal: Complementarities, tensions and the way forward—Kanbur R, ed. (2001) accessed June 2007. 22–25. Online at: http://www.q-squared.ca/papers01.html.
Cifuentes E, Hernandez JE, Venczel L, Hurtado M. Panorama of acute diarrhoeal diseases in Mexico. Health & Place (1999) 5:247–55.[CrossRef][Web of Science][Medline]
Clark R, Gundry S. The prominence of health in donor policy for water supply and sanitation: A review. Journal of Water and Health (2004) 2:157–69.[Medline]
Clasen T, Schmidt WP, Rabie T, Roberts I, Cairncross S. Interventions to improve water quality for preventing diarrhoea: Systematic review and meta-analysis. British Medical Journal (2007) 334:782.
Cleaver F. Choice, complexity and change: Gendered livelihoods and the management of water. Agriculture and Human Values (1998) 15:293–9.[CrossRef]
Davis J, Iyer P. Taking sustainable rural water supply services to scale: A discussion paper (2002) Washington, DC: World Bank.
Eisenberg J, Scott JC, Porco T. Integrating disease control strategies: Balancing water sanitation and hygiene interventions to reduce diarrheal disease burden. American Journal of Public Health (2007) 97:846–52.
Foltz F. Science, pollution, and clean drinking water: Choosing between tap water, bottled water, and home purification. Bulletin of Science, Technology & Society (1999) 19:300–9.
Gadgil A. Drinking water in developing countries. Annual Reviews of Energy Environment (1998) 23:253–86.[CrossRef]
Gundry S, Wright J, Conroy R. A systematic review of the health outcomes related to household water quality in developing countries. Journal of Water and Health (2004) 2:1–13.[Medline]
Habicht JP, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. International Journal of Epidemiology (1999) 28:10–18.
Harvey PA, Reed RA. Community-managed water supplies in Africa: Sustainable or dispensable? Community Development Journal (2006) 42:365–78.[CrossRef]
Hoque BA, Juncker T, Sack RB, Ali M, Aziz KM. Sustainability of a water, sanitation and hygiene education project in rural Bangladesh: A 5-year follow-up. Bulletin of the World Health Organization (1996) 74:431–7.[Web of Science][Medline]
Hutton G, Haller L. Evaluation of the costs and benefits of water and sanitation improvements at the global level. (2004) Geneva: World Health Organization. Online at: http://www.who.int/water_sanitation_health/en/wsh0404.pdf.
INEGI. XII Censo General de Población y Vivienda. (2000) Aguascalientes, Mexico: Instituto Nacional de Estadística, Geografía e Informática.
Isham J, Kahkonen S. Institutional determinants of the impact of community-based water services: Evidence from Sri Lanka and India. Economic Development and Cultural Change (2002) 50:667–91.[CrossRef][Web of Science]
Iyer P, Davis J, Yavuz E. Rural water supply, sanitation, and hygiene: A review of 25 years of World Bank lending (1978–2003). In: Water Supply and Sanitation Working Notes, Note #10. Washington, DC: The World Bank. (2006) Online at: http://www.worldbank.org/watsan.
Jalan J, Somanathan E, Chaudhuri S. Awareness and the demand for environmental quality: Drinking water in urban India. In: Discussion Paper 03–05 (2003) New Delhi: Indian Statistical Institute, Delhi Planning Unit.
James AJ, Verhagen J, van Wijk C, et al. Transforming time into money using water: A participatory study of economics and gender in rural India. Natural Resources Forum (2002) 26:205–17.[CrossRef]
Jensen PK, Ensink JHJ, Jayasinghe G, et al. Domestic transmission route of pathogens: The problem of in-house contamination of drinking water during storage in developing countries. Tropical Medicine and International Health (2002) 7:604–09.[CrossRef]
Kremer MJ, Leino J, Miguel E, Zwane A. Spring cleaning: Rural water impacts, valuation, and institutions. (2008) accessed 3 December 2008. Online at: http://www.econ.berkeley.edu/~emiguel/research2.shtml.
Lenton R. Health, dignity, and development: What will it take? (2005) London: UNDP.
Lopez AD, Mathers CD, Ezzati M, Jamison DT, Murray CL. Global burden of disease and risk factors (2006) Washington, DC: World Bank Publications.
Luby SP. Quality of drinking water. British Medical Journal (2007) 334:755–6.
Moe CL, Rheingans RD. Global challenges in water, sanitation and health. Journal of Water and Health (2006) 4(Suppl 1):41–57.
Morgan LM. Community participation in health: Perpetual allure, persistent challenge. Health Policy and Planning (2001) 16:221–30.
Ostrom E. Governing the Commons: The evolution of institution for collective action. (1990) New York: Cambridge University Press.
PAHO. [Health in the Americas. 1998 Edition]. (1998) Washington, DC: Pan American Health Organization.
Parker R, Skytta T. Rural water projects: Lessons from OED evaluations. World Bank Operations Evaluations Department, Working Paper Series 3 (2000) Washington, DC: World Bank.
Poulos C, Pattanayak S, Jones K. A guide to water and sanitation sector impact evaluations. Doing Impact Analysis Series 4 (2006) Washington, DC: World Bank.
Programa de las Naciones Unidos para el Desarrollo (PNUD). [Human Development Report, Mexico 2004]. (2005) Mexico.
Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health interventions. Journal of Epidemiology and Community Health (2002) 56:119–27.
Sara J, Katz T. Making rural water supply sustainable: Report on the impact of project rules. UNDP World Bank Water and Sanitation Program (1997) Washington, DC: The World Bank.
UNDP. Human Development Report 2006: Beyond scarcity: Power, poverty and the global water crisis. (2006) New York: United Nations Development Program.
van Koppen B, Moriarty P, Boelee E. Multiple-use water services to advance the Millennium Development Goals. In: Research Report 98 (2006) Colombo, Sri Lanka: International Water Management Institute.
Victora C, Schellenberg J, Huicho L, et al. Context matters: Interpreting impact findings in child survival evaluations. Health Policy and Planning (2005) 20(Suppl 1):i18–i31.[Abstract]
Whittington D, Lauria D, XM Mu. A study of water vending and willingness to pay for water in Onitsha, Nigeria. World Development (1991) 19:179–98.[CrossRef][Web of Science]
WHO. Water for life: Making it happen (2005) Geneva: World Health Organization.
WHO, UNICEF. Global Water Supply and Sanitation Assessment 2000 Report (2000) Geneva: World Health Organization.
Wright J, Gundry S, Conroy R. Household drinking water in developing countries: A systematic review of microbiological contamination between source and point-of-use. Tropical Medicine and International Health (2004) 9:106–17.[CrossRef]
Zwane AP, Kremer M. What works in fighting diarrheal diseases in developing countries? A critical review. CID Working Paper No. 140 (2007) Cambridge, MA: Harvard University Center for International Development.
Accepted for publication 12 May 2008.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||