Health Policy and Planning Advance Access originally published online on May 2, 2007
Health Policy and Planning 2007 22(4):234-245; doi:10.1093/heapol/czm013
Getting results used: evidence from reproductive health programmatic research in Guatemala
1Director, Research Center, Graduate School of Public Administration and Public Policies, Tecnológico de Monterrey, México City. At the time this article was written: Program Associate, Frontiers in Reproductive Health Program (FRONTIERS), Population Council, Washington DC, USA.
2Senior Consulting Associate, FRONTIERS, Population Council, Washington DC, USA.
3Research and Evaluation Officer, NetMark, Academy for Educational Development, Washington DC, USA. At the time this article was written: Evaluation Associate. FRONTIERS, Tulane University, New Orleans, USA.
4Director, Center for Communication Programs, Johns Hopkins University, Baltimore, MD, USA. Previously Professor at Tulane University, New Orleans, USA.
* Corresponding author. Escuela de Graduados en Administración Pública y Políticas Públicas, Instituto Tecnológico de Montrerrey, Campus Ciudad de México, Calle del Puente 222, Tlalpan, México, D.F. 14380, México. E-mail: carlos.brambila{at}itesm.mx.
 |
Abstract |
|---|
 Top Abstract IntroductionTheoretical perspectives on...The Population Council's...Evidence of utilization of...ConclusionsAppendixEndnotesReferences |
|---|
This article reviews 44 operations research projects aiming to improve reproductive health services in Guatemala, conducted by the Population Council from 1988 to 2001. It documents the experience of the research programme, traces the extent to which research results are identifiable in existing programmes, and analyses factors influencing utilization.
Utilization of research results occurs as a gradual process of information sharing, where researchers influence decision-makers through a continual stream of information rather than a single set of findings. Utilization depends on leadership, collaborative planning and implementation, close monitoring, and feasible research designs, among other factors. To influence policy formulation, organizations should form enduring links among institutions and develop critical research skills among personnel who collaborate with or manage service programmes. To understand how operations research affects policy and programme change, one must consider not just individual projects, but rather the synergistic impact of multiple projects on a broad range of themes over time.
Key Words: Guatemala, operations research, research utilization, family planning, contraception
KEY MESSAGES
|
|
Introduction |
|---|
|
TopAbstractIntroductionTheoretical perspectives on...The Population Council's...Evidence of utilization of...ConclusionsAppendixEndnotesReferences |
|---|
This article draws on an a posteriori appraisal of 44 operations research projects conducted by the Population Council in Guatemala from 1988 to 2001. It examines the influence of research on programme improvement in reproductive health, and how the cumulative and incremental effect of the research contributed to large-scale programmatic change. The studies reviewed cover a wide range of topics, target groups, collaborating institutions and researchers.
This article does not prove a causal relationship between research and programmatic decision-making in a strict sense. However, it provides evidence of links between research and decision making. This article will argue that, although making attribution to a single initiative may be difficult, operations research conducted in Guatemala shows evidence of the cumulative effect of a research programme on service delivery.
Utilization of results is relevant to decision-makers responsible for reproductive health programmes, to individual researchers and research institutions interested in increasing the societal impact of their research, to national and international donors interested in increasing the impact of their funding, and to the public health community interested in improving access to and quality of health services.
This article is relevant to an ongoing debate on whether to dismiss operations research in favour of an approach based on macro-level organizational change theory or systems analysis (Caldwell et al. 2002
; Simmons et al. 2002). It shows that, contrary to becoming obsolete, the operations research approach to solving specific problems within managers’ control through a series of discrete, small-scale projects is particularly well suited to introducing new concepts and services and taking them to scale. Critics argue that operations research should not be done in a piecemeal fashion, with researchers jumping to do a study whenever money becomes available; that it is important to examine the health infrastructure in a particular context and develop a research strategy so that various efforts complement and build on one another. Operations research—or indeed, any research—cannot expect to produce radical changes in entire health systems, but operations research provides the evidence base for improvements essential for health systems to achieve any measure of success. We will demonstrate that the Population Council's research programme in Guatemala combined operations research with technical assistance and capacity-building activities in a broad programme that progressed from working with a single non-governmental organization (NGO) willing to provide family planning, to collaborating with numerous, often indigenous, NGOs and district-level public sector facilities in offering an expanded constellation of services, and ultimately helping the Ministry of Health provide pre-service training to service providers.
The second section of this article explores theoretical perspectives on the relationship between research, programme improvement and policy making. In the third section, we describe the evolution of the Population Council's research programme in Guatemala, main research topics and results achieved. The fourth section focuses on an impact evaluation of a subset of 22 projects and examines factors that facilitated or hindered the uptake of results. In the final section we present conclusions.
|
Theoretical perspectives on research and practice |
|---|
|
TopAbstractIntroductionTheoretical perspectives on...The Population Council's...Evidence of utilization of...ConclusionsAppendixEndnotesReferences |
|---|
The concept of knowledge utilization has many meanings (Weiss 1979
) and continues to evolve. Social scientists from a broad range of fields have attempted to describe the role of research in the policy process, often conceiving of it as a linear process, whereby policy makers adopt evidence-based practices through a process of ‘enlightenment’ (Sundquist 1978; Weiss 1979), rational choice (such as the ‘evidence-based policy approach’, as discussed by Stone et al. 2001) or bounded rationality (following Herbert Simon). Alternatively, Lindblom (1980) views policy making as a series of steps in which policies are gradually modified in an incremental manner. Lack of use of research-based evidence is explained in terms of a ‘cultural gap’ between researchers and policy makers.
More recent literature has questioned some of the assumptions of the linear and incremental views. In their annotated bibliography of over 100 documents on the research-policy process, de Vibe et al. (2002) cite three fundamental shortcomings of earlier models: they assume ‘that research influences policy in a one-way process; ... that there is a clear divide between researchers and policy-makers and ... that the production of knowledge is confined to a set of specific findings’.
A number of authors have written overviews of the research-policy literature or summarized international meetings on this topic in recent years (Sutton 1999; Neilson 2001; RAWOO 2001; Stone et al. 2001; Askew et al. 2002; Nutley et al. 2002; Lindquist 2003). There is a growing consensus that the relationship between research and policy is an interactive two-way process, embedded in political and social factors affecting both sides of the equation. One approach that has proven successful in incorporating research experience and results into health services, in particular reproductive health services, is operations research.
The discipline of operations research is unique in that an operations research study can be considered successful only if its results are used to improve programmes or policy, regardless of the contribution it may make to the behavioural science literature or body of knowledge. Operations research is a form of action research that has proven effective to improve health services; it continues a 30-year tradition of research aiming not only to understand better health processes, but also to improve service delivery (Seidman and Horn 1991; Foreit and Frejka 1998). It is similar to impact-oriented research in other fields such as poverty alleviation, human development, growth and livelihoods, social, political and environmental change, and communications (de Vibe et al. 2002; Court and Young 2004). The common denominator of impact-oriented research is to make better use of research-based evidence in development policy and practice to save lives, reduce poverty and improve quality of life.
This research approach involves a process of identifying programme and policy problems and then designing, testing and evaluating alternative solutions. Interventions are designed with the assumption that, if successful, they will be scaled up and institutionalized.
Published case studies of operations research projects reflect the extent to which each adheres to these steps; in addition, they often describe factors believed to facilitate or hinder utilization. A recent report of 58 operations research projects in the field of reproductive health (Marin et al. 2004) concludes that the determinants of operations research utilization can be divided into two groups: those related to the research process, and those related to individuals, organizations or the innovation itself (see also Reid 2003).
One of the research-specific determinants most often mentioned is the relevance of the research topic. Decision-makers are more likely to utilize research that reflects their own priorities and whose recommendations are timely (Agarwala-Rogers 1977; Frenk 1992; Hegazi 1997; Solo et al. 1998; IFPRI 2001). Programme planning cycles offer researchers a window of opportunity to influence decisions; results that become available too late may no longer be relevant.
Additional research-specific factors include skills of local researchers or research institutions, perceived quality of the research, dissemination of findings, interaction and communication between researchers and decision-makers, and the nature of the recommendations. While testing an intervention on a large scale may create momentum for its adoption (Corwin and Louis 1982; Haaga and Maru 1996), in general, recommendations for incremental changes are more likely to be implemented than those that call for more sweeping changes, particularly in large public sector programmes (Koenig et al. 1991).
The second category of factors considered to influence utilization are those that are not specific to the research process but relate to organizational culture, availability of funding and continued technical assistance, and the larger social and political context. Results that are consonant with an organization's values and mandate, or national policies, are more likely to be adopted (Haaga and Maru 1996), whereas those that are not are more likely to be resisted (Weiss and Bucuvalas 1980; Siegel and Tuckel 1985). Continued support is essential, both in terms of funding and an ongoing relationship between researchers and programme practitioners who can translate results into actionable recommendations (Davis and Howden-Chapman 1996; Solo et al. 1998; Anderson et al. 1999; Askew et al. 2001). Conversely, changes in leadership and high levels of staff turnover affect institutional memory and inhibit utilization.
One shortcoming of the literature is that it generally uses the research project as the unit of analysis, and as such is forced into adopting a linear, rational model for analysis. The Population Council's operations research programme in Guatemala offers a different perspective: its 44 studies spread over a decade and a half allow us to examine the synergistic impact of multiple complementary studies on the evolution of reproductive health programmes and policy in Guatemala. We argue that research utilization should be a process of organizational learning, both for policy-making institutions and for service organizations. While individual research projects are rarely convincing or comprehensive enough to exercise a determining impact on policy making, accumulated research findings gradually influence decision-makers’ perceptions of both the causes of problems and the likely effects of policy interventions (Restivo and Loughlin 1987; Singer 1990). Policy makers tend to trust knowledge organizations and accept advice and recommendations of those they perceive as producing high-quality research without a political agenda.
|
The Population Council's research programme in Guatemala |
|---|
|
TopAbstractIntroductionTheoretical perspectives on...The Population Council's...Evidence of utilization of...ConclusionsAppendixEndnotesReferences |
|---|
Background
Guatemala has a unique socio-political history that colours all aspects of contemporary life, including its reproductive health programme (Santiso-Galvez and Bertrand 2004). Seventy-five per cent of the population is rural, and roughly half of Guatemalans are ‘indigenous’ (Mayan Indians that speak one of 22 different languages).
The devastating history of civil war and violence against indigenous peoples that pervaded the rural highlands during much of the 1960s to the mid-1990s led to deeply rooted mistrust of all government initiatives and to the development of a well-organized civil society sector. In this context, many perceived contraception and family planning programmes as part of a war-connected genocide.
The Ministry of Health and the Guatemalan Social Security Institute (IGSS) have been the primary source of health care in Guatemala, but, primarily for political and religious reasons, the Guatemalan government had been reluctant to offer family planning services. Moreover, existing services did not serve the Mayan population well, either in language or style. Reproductive health services were thus not available to most women, and APROFAM (the International Planned Parenthood Federation's affiliate) represented the primary source of reproductive services during the years when this operations research was conducted (Santiso-Galvez and Bertrand 2004).
The 1996 Peace Accords between the Guatemalan Government and the guerrillas (UN/MINUGUA) ended 35 years of civil war and established a national reconstruction plan. The Accords included health sector reform and an explicit commitment by the government to provide services to indigenous populations traditionally marginalized from them, which took form with the establishment of the Integrated Health Care System (SIAS). To address the virtual absence of access for rural populations, SIAS leveraged existing indigenous NGOs and assigned operational responsibility and funding to them. This strategy still exists as an atypical and constructive (albeit not problem-free) interaction between the government and NGOs.
In January 2000, the Ministry of Health established its first ever Reproductive Health Program to provide a normative framework for reproductive health service delivery in public hospitals and health centres.
In the following sections we describe the Population Council's operations research programme in Guatemala and highlight results that contributed to changes in programme operations. Projects described in this section, categorized by type of research, are summaries of published documents and institutional records (see Table 1).
|
Operations research programme in Guatemala, 1988–2001
The strategic objective of the operations research effort was to test innovative strategies to improve reproductive health services, in particular in the Mayan highlands, through:
- Testing strategies for improving quality, access and coverage of integrated reproductive health care in the public sector;
- Partnering with the more flexible private sector to test strategies to improve its response to the needs of the Mayan population; and
- Disseminating results and institutionalizing and expanding successful strategies.
The Population Council's operations research portfolio in Guatemala from 1988–2001 included the following nine types of research.
Acceptability and accessibility research
Guatemala's predominantly Mayan rural population is traditional and closed; discussion of reproductive health is considered strictly personal and therefore taboo. Women and couples often exhibit fatalistic attitudes (‘God's will’) regarding the number and spacing of their children. Family planning users generally do not admit use, even within close circles. A series of diagnostic studies was conducted to understand community and cultural barriers related to reproductive health and the results were applied to developing respectful, culturally appropriate and acceptable services.
Diagnostic studies to assess widely held opinions
A series of diagnostic studies explored potential barriers to the provision of reproductive health services. The results often indicated that ‘prevailing opinions’ were not necessarily true or at least were amenable to change. Two examples follow.
- ‘Indigenous populations are not interested in knowing about or practicing family planning’
).
- ‘Mayan men have low reproductive health knowledge and oppose the use of contraceptive methods by their wives’
), and another among contraceptive users and non-users in Quetzaltenango (Méndez-Domínguez 1995), confirmed that men had very low knowledge of contraception but were interested in learning how to plan their families.
An anthropological study of cultural norms and symbols of the Mayan fertility and reproduction cosmovision determined that reproductive health services were not acceptable to most Mayans in the form offered at that time. It recommended that reproductive health programmes take into account the indigenous culture and lifestyle (Mendez-Dominguez 1995).
Intervention studies to improve APROFAM's community-based systems
Men were perceived to be a barrier to couples’ use of family planning. Using results from the above diagnostic studies, APROFAM tested alternative approaches for reaching men with reproductive health education in El Quiché province (Fernandez et al. 1997a). This study showed that self-standing health education talks were not effective in reaching men, but that men were willing to learn about reproductive health when attending recreational and athletic events. APROFAM initiated a community-based programme to include these types of activities to attract men.
APROFAM experimented with different means of reaching its target audience. Specifically, APROFAM tested alternative on-the-job versus clinic-based training for its community health promoters. The latter proved to be more expensive but more effective in increasing referrals for long-term methods to APROFAM clinics (Figueroa et al. 1998). In addition, APROFAM compared two approaches for conducting household visits (Figueroa et al. 2001). In one, promoters conducting house visits encouraged the women to discuss family planning with their husbands and, if desired, they could ask the promoter to return to speak with both about spacing methods. In the other approach, promoters discussed family planning methods in the context of prevalent health problems. The first proved more effective in encouraging couples to purchase contraceptives during the promoter's re-visit, and the promoter's presence appeared to facilitate spousal discussions and prompt decision-making on contraceptive initiation.
Efforts to incorporate family planning and reproductive health into non-health NGO programmes
It is estimated that over 1 million Guatemalans—mostly Mayans from the Western highlands—either live on or seasonally migrate yearly to coffee and sugar plantations on the Pacific coast. Vernon and Schenkel (1992) studied self-financed family planning services for migrant and resident farm workers in rural coffee fincas (large landholdings), finding an unmet demand for reproductive health services. They showed that services could be provided on-site by trained health promoters, with referrals for clinical methods to APROFAM. A 2001 evaluation of the portfolio of operations research in Guatemala, discussed in more detail below (Bertrand and Marin 2001), confirmed that reproductive health services, particularly counselling, continued to be offered at finca health posts several years later, although personnel working in the programme did not necessarily link the existence of these services to the earlier operations research study.
An attempt to incorporate reproductive health service delivery into an indigenous NGO working in non-health development areas via links with APROFAM's community programme failed due to deep-felt distrust from the participating women's groups (Escalante et al. 1996). However, this project provided valuable lessons for later attempts to strengthen the capacity of NGOs to provide reproductive health services. The pioneer project suggested strategies for improving inter-NGO collaboration and identified the ideal community worker profile.
Public-private sector collaboration and service organization
Until the mid-1990s, the public and private health sectors in Guatemala were separate and often antagonistic. Due to historical distrust between indigenous and non-indigenous, or ladino, populations, few NGOs were willing to collaborate with the predominantly ladino APROFAM and Ministry of Health providers. Several studies tested alternative mechanisms of collaboration among NGOs and between NGOs and the public sector.
Research on service delivery resulted in extensive re-engineering of services in public (Burkhart and Solórzano 1999; Solórzano and Burkhart 1999) and private (Figueroa et al. 1997) settings. Studies showed that when public service providers were empowered to identify problems and develop feasible solutions, medical and non-medical health centre personnel were willing to work harder to improve quality; moreover, provider and client satisfaction increased. The 2001 follow-up evaluation of operations research in Guatemala found that additional health centres and at least one other health area (Sololá) subsequently adopted this methodology.
Research into action: the NGO Strengthening Project
The NGO Strengthening Project, initiated in 1997, was developed to respond to the considerable unmet need for reproductive health services among rural indigenous populations and underutilized resources in NGOs interested in satisfying the health needs of the population. The NGO Strengthening Project proposed to test: (1) whether NGOs working with Mayan populations could be technically strengthened to provide accessible, acceptable and quality reproductive and child health services in the highlands, and (2) whether these NGOs could be strengthened managerially to ensure the sustainability of their reproductive health programmes.
Several NGOs interested in participating were identified from a database developed during 1994–96. Challenges during the first year included establishing working relationships and building trust with partner NGOs, training them to conduct a baseline survey, developing tools and materials, training trainers in this large group of diverse NGOs with virtually no experience working in the field of reproductive and child health (RCH), and strengthening capacity to deliver these services. Though NGO administrators were enthusiastic, a subsequent training needs assessment and situation analysis indicated large deficiencies in the capacity of these NGOs to integrate family planning/reproductive health services into their ongoing programmes.
In 2001, the NGO Strengthening Project conducted an end-of-project assessment among nine NGOs, in addition to a population-based impact evaluation of four NGOs, using a standardized questionnaire applied to mothers of children under 2 years of age. One of the most dramatic findings was the increased use of birth spacing methods, most notably injectables, which tripled during the project period. Prenatal, birthing and postpartum care also improved during project implementation: the percentage of pregnant women seeking prenatal care and receiving tetanus toxoid vaccines increased significantly; furthermore, the proportion of women delivering with trained attendants skilled in normal deliveries and obstetric emergency management increased, representing an important shift away from the use of traditional birth attendants (TBAs).
Research to improve provider technical competence
Certain Ministry of Health service providers had never received focused training on reproductive health, were practising ad-hoc rather than evidence-based medicine, and needed upgrading. Public service managers were anxious to correct this deficiency and requested assistance from the Population Council. Together they trained over 1500 service providers in contraceptive technology and integrated maternal and child health (MCH). In the process, they established close ties with district and area level providers throughout the Guatemalan highlands (Burkhart et al. 1999a). An evaluation 18 months after the training demonstrated that providers trained in integrated services continued to practice what they had learned, an important achievement in skill maintenance after training.
To ensure that providers would be trained pre-service, the Population Council tested innovative distance education methodologies in a school for auxiliary and professional nurses (Arriaga et al. 1999). As a result of positive outcomes, the distance education course was institutionalized within the nursing school and is currently offered to new auxiliary nurses entering the system (Bertrand and Marin 2001).
To enable programme managers at a local level to offer quality reproductive health and family planning services in an efficient manner, Health District teams of providers participated in distance education, resulting in improved provider knowledge and skills, and subsequently in the following quality improvements (Brambila et al. 2001):
- Reduction in client waiting times and unnecessary contacts with administrative and support personnel;
- More efficient and effective client-provider exchanges;
- Provision of more complete, individually tailored services;
- Improved access to reproductive health services for women attending health facilities for other reasons; and
- Increased response to women's contraceptive needs.
Research to increase appropriate contraceptive choices
For many years, the Government of Guatemala actively opposed family planning (Santiso-Galvez and Bertrand 2004) and restricted the range of contraceptives available to only two or three methods. Culturally preferred contraceptive methods such as injectables were not available. Accurate information about natural methods such as periodic abstinence was conspicuously absent, especially at the community level.
Beginning in 1997, the Population Council supported several initiatives to improve access to a wider range of contraceptive choices, including:
- Provision of injectables at the community level as well as in clinics (Fernandez et al. 1997b);
- Expansion of the use of a necklace as a memory aid for fertility awareness-based methods, introduced in Guatemala in 1988 (Vernon and Schenkel 1992) and reintroduced 10 years later (Mazariegos et al. 1999);
- Introduction of NORPLANT (Taracena et al. 2001a) and emergency contraception (Taracena et al. 2001b); and
- Introduction of emergency contraception for rape victims: the Population Council trained personnel of the judiciary in basic information on emergency contraception and developed educational materials for distribution to women in rape cases.
As shown in Table 2, between 1987 and 2002, total contraceptive use increased from 23.2 to 43.3%; use of modern methods nearly doubled to 34.4% (MSPAS et al. 2002). Use of the injectable increased from 0.1 to 9.0% and use of periodic abstinence increased from 2.8 to 6.3%. During 2001, NORPLANT use, at 14%, was second only to the injectable at IGSS and APROFAM.
|
The Population Council and its partners were key in the introduction and initial training related to the above methods, but do not claim sole responsibility for advances in contraceptive use. Other contributing factors include improved procurement, growing consolidation of public and private reproductive health programmes, and political, social and cultural changes (see Santiso-Galvez and Bertrand 2004
for a complete discussion). Improvement may be due to a synergism of all the above and other factors rather than to a single factor or actor. In asserting that the Population Council contributed to these increases in contraceptive prevalence rate, we are limited to employing the concept of plausible attribution, which requires that: (1) the change in service delivery takes place after the intervention, and (2) the noted change is consistent with the results and recommendations of the operations research studies. Research on missed opportunities in service integration
Traditionally, most family planning services were not located at a reasonable geographic distance for rural Guatemalan women. Providers were neither able to communicate in the local language nor trained in counselling techniques, and their knowledge and skills needed updating. Despite a significant unmet need for family planning, diagnostic studies revealed scarce demand for existing services and ineffective and costly vertical programmes (Brambila and Solórzano 1995; Vernon et al. 1997). They also showed many missed opportunities for providing services to women who attended health facilities for other reasons (e.g. seeking care for a child). Women generally received only the service they requested and left with unmet reproductive health needs.
To solve these problems, the Population Council and the Ministry of Health introduced and tested the use of an algorithm to systematically identify unmet needs and offer appropriate reproductive health services during the same clinic visit. They tested the job aid in a small number of public clinics (Vernon et al. 1997). In the last 9 months of 1996, the clinics using the algorithm had an increase of 124% new family planning clients, compared with an increase of 21% in control group facilities. Control group clinics experienced a 64% decrease in couple years of protection (CYP, a widely used indicator to monitor family planning programmes) while experimental group clinics increased CYP by 41%. The introduction of injectables in the intervention clinics accounted for a large part of these increases. The changes observed in other reproductive health services covered by the algorithm (prenatal, post-natal and well baby care) were not as consistent as those observed for family planning.
The above results demonstrating the benefits of a simple job aid to identify unmet needs and reduce missed opportunities were used to implement the strategy at the national level by the Ministry of Health (Solórzano and Burkhart 1998; Burkhart et al. 1999b).
|
Evidence of utilization of results |
|---|
|
TopAbstractIntroductionTheoretical perspectives on...The Population Council's...Evidence of utilization of...ConclusionsAppendixEndnotesReferences |
|---|
Follow-up assessment conducted by Tulane University
In April-May 2001, two researchers from Tulane University conducted a review of operations research activities completed by the Population Council under various funding cycles from 1988 to 2001 (Bertrand and Marin 2001). The purpose of the follow-up was two-fold: to assess process and to evaluate ‘impact’ (the extent to which change occurred following the dissemination of the operations research). The evaluation team relied on three primary sources of data: key informant interviews, document review, and site visits to health centres and NGOs that were implementing operations research interventions. Based on this information, the Tulane team scored each of 22 projects on 25 process and impact indicators. Process indicators assessed participation by partner organizations, barriers to research implementation, acceptability to local stakeholders, research quality and dissemination. Impact indicators assessed the extent to which the study contributed to programme improvement, policy change or increased funding, or was scaled up or replicated. The researchers based their selection of impact indicators in large part on their own field-level experience of what can occur in-country as a result of an operations research project. The researchers assigned one of three possible scores (1, 2, 3) to rate the potential or expected impact of each intervention.1 The Appendix shows a complete listing of indicators used.
As mentioned earlier, operations research is successful to the extent that it triggers change in policy or in the provision of services. This follow-up assessment by Tulane researchers represents one of the few systematic attempts to track the extent of changes following the conduct and dissemination of operations research studies on a number of pre-established outcomes (italicized herein). According to evaluation results, in 13 of 20 studies the intervention under study proved effective in bringing about improved results.2 Four of the studies yielded mixed results, and three interventions were judged not effective. In 14 of the 20 studies, the implementing agency acted on the results, continuing the intervention if successful or abandoning it if ineffective. Successful interventions were scaled up within the same organization in 9 of 17 projects; the intervention was introduced in other facilities or health districts, or expanded at the original site, for example by training additional staff to offer improved services to more clients. Another in-country organization adopted successful interventions in 5 of 17 projects. Indeed, two USAID-funded projects in Guatemala—Calidad en Salud and the NGO Strengthening Project—were expected to build on the lessons learned and materials and tools developed within operations research projects. Research findings did not generate substantial new funding to sustain or expand successful interventions, and project activities were seldom replicated in other countries (presumably because this was not a priority activity, nor were funds dedicated to doing so). However, the evaluation concludes that a number of studies did lead to policy changes, primarily at the programme level.
Facilitating factors for research implementation and utilization of results
Many of the factors identified in the Tulane report as most important in conducting successful operations research activities and scaling them up are consistent with those identified by Simmons et al. (2002) and others in the literature on research utilization and scaling up. Characteristics considered to contribute to successful utilization are the following.
Charismatic leadership: often described as a ‘champion’, a person from either the research or implementing organization who is passionate about the project inspires others and works tirelessly to achieve a desired outcome. Some activities require long periods of advocacy before stakeholders come to an agreement on a study proposal, while others deal with controversial topics and require the support of hesitant policy makers. Beyond simply being an advocate, the ‘charismatic leader’ can inspire the trust of these individuals to take risks and try unconventional solutions in the context of operations research.
Collaborative planning and intervention design: the intervention should respond to a perceived need of the implementing organization and the solution should be feasible in the long term, both of which are most likely when the organization in question participates in planning the study.
Continued involvement and close monitoring and supervision of the intervention by the research group: this helps to maintain a high quality of services provided as well as to ensure that the intervention is being applied consistently and according to plan, facilitating replication.
Simple, easy to use materials: for many people, using job aids is a difficult change when practices are established. The more user-friendly and simple the materials, the more likely they are to be adopted.
A feasible design: the intervention (change) must be within the capabilities of project staff and not cause an undue burden on the organization.
A good match between the intervention and the implementing organization: the intervention should be consistent with the values, goals and culture of the organization.
Compelling results: observable improvements are communicated immediately and comprehensively to programme managers and policy makers.
Provider motivation: providers and others directly implementing the intervention must want to maintain the short-term changes introduced during the research project (e.g. because the increased quality of care or client-provider interaction gives them greater job satisfaction).
Continuing technical assistance and institutional support beyond the end of the project help the transition from an intensive small-scale activity to business-as-usual on a larger scale, and can be gradually phased out as organizations gain more experience and become more self-sufficient and independent.
Fortuitous timing: results become available at the correct time in the decision-making cycle (sometimes described as a ‘policy window’ or ‘window of opportunity’) or have the right idea at the right time in terms of community demand for change (Bertrand and Marin 2001).
Challenges for research implementation and utilization of results
The Tulane assessment revealed two general areas that, if improved, could increase the perceived quality of study results and participation by the end user, both of which are generally acknowledged as important for achieving utilization. First, several studies had too many objectives (making it difficult to fully achieve them) or inappropriate objectives (that described activities to be carried out, not results to be achieved). Secondly, most organizations did not build sufficient technical capacity to enable them to conduct subsequent operations research projects without substantial external technical assistance. The following observation emerged from the evaluation:
'(t)his finding underscores a dilemma for the Population Council: on one hand, they want to foster maximum skills-building and ownership of results in the implementing agencies; on the other, they are responsible to the donor agency for ensuring quality control at each phase of the research process. As such, they often ‘step in’ to ensure a quality product, but in doing so they may defeat their own efforts at capacity building.’ (Bertrand and Marin 2001
The field experience of the Population Council in Guatemala and the Tulane assessment point to two main areas where a programme should concentrate its efforts to maximize the utilization of results: (1) building and maintaining inter-personal and inter-organizational relationships with those working in both the public and private sectors, and (2) developing the skills and ability of individuals to solve problems systematically, using evidence to guide programming. In Guatemala, the Population Council made both of these important parts of its work, which we believe was largely responsible for many of the changes in access to and quality of reproductive health services over the past decade and a half.
Institutional relationships always begin as inter-personal relationships between individuals from collaborating organizations; if based on common organizational goals, they can expand and endure beyond the tenure of these individuals. One problem many researchers in Latin America face is that no sooner do they develop a rapport with someone in the position to make the desired changes than that person is replaced. This is particularly true in the public sector, where staff may turn over multiple times in the life of a single research project. Researchers change as well: in the 9 years that the Population Council had a Guatemala office, the director changed three times. Forming strong institutional relationships can minimize the negative impact of such changes. As research and programme partners work together, they gain trust in one another, a trust based on greater understanding of each other's goals, values and operational practices. Without this trust, even the most rigorous, conclusive study would have difficulty attracting the attention of decision-makers, much less convincing them to act.
The projects discussed in this article were collaborative and inter-institutional. The Population Council partnered with key reproductive health service providers (Ministry of Health, APROFAM, IGSS) on numerous projects, and, on many others, with members of the NGO network it supported under the Cooperative Agreement, educational institutions and other branches of government. While staff turnover remains high, the Guatemalan reproductive health community has had the good fortune that people leaving one of these partner organizations often move to another.
|
Conclusions |
|---|
|
TopAbstractIntroductionTheoretical perspectives on...The Population Council's...Evidence of utilization of...ConclusionsAppendixEndnotesReferences |
|---|
The operations research portfolio in Guatemala reflected a long-term vision but responded to practical issues that emerged over time. Diagnostic studies had an important role in identifying programmatic problems and feasible solutions. In several instances, interventions built the momentum necessary for governmental and private organizations to decide to scale-up evidence-based best practices. Through incremental interventions, the Population Council research achieved a demonstrable cumulative impact in several programme areas. The process of project development, implementation and extension was often iterative: for example, diagnostic studies needed to be widely disseminated before service organizations became interested in reproductive health service provision. Often, reluctant public sector officers became interested in programmatic interventions after project results were successfully disseminated.
International experts embracing a structural perspective to research utilization generally assume it is possible to establish a ‘master plan’—a set progression or list of priority activities to be followed to improve the health sector in a systematic manner. They draw strategic maps outlining the expected ways to achieve structural change along a rational process of sector reform.
However, the experience of the Population Council in Guatemala suggests an alternative path. Rarely do knowledge organizations know where all the key problems or solutions lie, and seldom do service delivery organizations change service delivery practices without considerable deliberation. Moreover, research often reaches dead ends that force both researchers and programme managers to reconsider and change paths. Thus, the strategy of the Population Council consisted of conducting a sequence of operations research projects that built on each other, moving from diagnostic research to better understand the cultural context for reproductive health service delivery in Guatemala (especially among the Mayan population) to the testing of interventions on a small scale that—where successful—could be scaled up.
Such a strategy must be flexible. Where the research shows an approach to be ineffective, researchers and practitioners must review the evidence and develop alternatives. Where unforeseen opportunities arise, researchers and practitioners must remain open to taking advantage of them rather than rigidly adhering to some ‘master plan’. No study, single-handedly, created widespread change. Rather, the cumulative effect of the research provided practitioners with a body of evidence in support of needed change, and it provided the time needed to build a trusting, collaborative relationship between researchers and practitioners. Specifically, it created an environment for evidence-based decision-making.
Context is also an important factor. It is relatively easy to work on a large scale with national programmes if the government is supportive and committed. However, where the government is less enthusiastic, it is necessary to start with interested persons and organizations. Official recognition, as in the case of Guatemala, may come only after several years of a sustained effort involving multiple researchers and actors.
The experience of the Population Council in Guatemala was somewhat unique, in that it entailed a long-term, problem-based strategy that allowed for multiple reinforcing research studies to build influence over time. Rarely do university researchers enjoy this luxury, unless they are successful at obtaining funding for recurrent cycles of their research, such as has been the case in the Matlab and Extension projects in Bangladesh, and the Navrongo and Community-based Health Planning and Services (CHPS) projects in Ghana. These geographical areas have served as field laboratories for reproductive health operations research projects conducted over multiple decades (since 1978 and 1988 in the two sites, respectively) and both were financed by multiple international donors. These two experimental mega-projects have had a profound impact on national health policy, donor priorities and public health action.
The Guatemala experience reinforces the value of programmatic research that runs over an extended period of time in order for it to have the desired effect on programmes. This experience benefited from several 5-year cycles of funding, which allowed for the accumulation of evidence in addition to the development of collaborative ties between researchers and practitioners, which ultimately resulted in changes to the service delivery environment.
|
Appendix |
|---|
|
TopAbstractIntroductionTheoretical perspectives on...The Population Council's...Evidence of utilization of...ConclusionsAppendixEndnotesReferences |
|---|
Indicators for assessing the process and impact of operations research (OR)
I. Process Indicators
P-1: Did the implementing/collaborating organization(s) actively participate in the design of the OR project?
P-2: Did the implementing/collaborating organization(s) actively participate in the implementation of the OR project?
P-3: Did the implementing/collaborating organization(s) participate in developing programmatic recommendations?
P-4: Did the study accomplish its research objectives?
P-5: Was the intervention implemented as planned (or with some modifications)?
P-6: Was the study completed without delays (or other adjustments to the timeline) that would compromise the validity of the research design?
P-7: Was continuity in key personnel maintained over the life of the OR project?
P-8: Was the study design methodologically sound (free of flaws that could have affected the final results)?
P-9: Was the research design feasible in the local context?
P-10: Did the implementing/collaborating organizations judge the OR technical assistance to be useful and provided in a collegial manner?
P-11: Were results of the OR study judged to be credible/valid in the local context?
P-12: Was the research relevant for the national programme?
P-13: Were the results disseminated to key audiences, including policy makers, programme managers, service providers, and donors?
P-14: Are the results readily available in written form?
II. Impact Indicators
I-1: Did the results indicate that the intervention was effective (i.e. that it improved service delivery in areas identified by the OR study)?
I-2: Did the implementing/collaborating organization(s) ‘act on’ the results (i.e. continue to implement the activities tested in the OR study after its completion if effective or not implement/discontinue this activity if ineffective)?
I-3: (If the intervention was effective and continued after the study) Were the activities tested under the intervention still observable 24 months post-implementation?
I-4: If the intervention was effective and continued after the study, was the intervention scaled up by the original implementing/collaborating organization in the same country?
I-5: If the intervention was effective and continued after the study, was the intervention adopted by another organization within the same country?
I-6: Was the intervention replicated in another country?
I-7: Was there a change in policy that can be linked to the OR project?
I-8: Did the implementing/collaborating organization conduct subsequent OR studies?
I-9: Did the implementing/collaborating organization conduct subsequent OR studies without the Population Council?
I-10: Did the original donor fund new programme activities based on the results of the OR study?
I-11: Did other donors provide new or expanded funding based on results of the OR study?
III. Contextual Factors
C-1: Were there other factors (not mentioned above) that facilitated the conduct of the research project?
C-2: Were there other factors (not mentioned above) that facilitated the utilization of results from this OR project?
C-3: Were there other factors (not mentioned above) that hindered the conduct of the research project?
C-4: Were there other factors (not mentioned above) that hindered the utilization of results from this OR project?
C-5: Did USAID use the data from the OR study for a specific purpose? (Explain)
C-6: Did the study include an assessment of the costs of the intervention?
|
Acknowledgements |
|---|
The research described in this article was the collective effort of approximately 73 persons, including research directors, collaborators, officers, programme managers, assistants, support personnel, service providers and donors, over a 14-year time span. The authors have only put in context their individual contributions, and no individual person may claim authorship of the collective achievement. This article is an acknowledgement and tribute to their participation. Research projects analysed in this article were funded by USAID through different contracts awarded to the Population Council over a 12-year period. USAID's support is gratefully acknowledged.
|
Endnotes |
|---|
|
TopAbstractIntroductionTheoretical perspectives on...The Population Council's...Evidence of utilization of...ConclusionsAppendixEndnotesReferences |
|---|
1 The Tulane team had previously developed the assessment methodology through a process where they reviewed literature reviews available at the time the research was conducted and interviewed many practitioners in the field to determine what made the difference between successful and unsuccessful projects in terms of implementation. Based on these discussions, the research team developed a set of process indicators.
2 The denominators in this section vary, depending on the number of studies that were applicable to the question. For example, some questions on impact were valid for all 22 studies, whereas others were relevant only for the 17 studies in which the intervention was effective and continued after the study.
|
References |
|---|
|
TopAbstractIntroductionTheoretical perspectives on...The Population Council's...Evidence of utilization of...ConclusionsAppendixEndnotesReferences |
|---|
Agarwala-Rogers R. Why is evaluation research not utilized? In: Evaluation Studies Review Annual.—Guttentag M, ed. (1977) 2. Beverly Hills, CA: Sage. 327–33.
Anderson M, Cosby J, Swan B, Moore H, Broekhoven M. The use of research in local health service agencies. Social Science and Medicine (1999) 49:1007–19.[CrossRef]
Arriaga D, Guzmán D, Burkhart M. Testing a continuing education system in reproductive health for nurse auxiliaries. In: Population Council, FRONTIERS project report (1999) Guatemala City.
Askew I, Miller R, Solo J. Using operations research for producing and communicating knowledge about reproductive health services in sub-Saharan Africa. (2001) Paper presented at the IUSSP seminar on ‘The production and circulation of population knowledge’, March 2001: Brown University, Providence, USA.
Askew I, Matthews Z, Partridge R. Going beyond research. (2002) A key issues paper raising discussion points related to dissemination, utilization and impact of reproductive and sexual health research. Southampton, UK: University of Southampton.
Bertrand JT, Marin C. Assessment of the process and impact of operations research in Guatemala: 1988–2000. (2001) Washington, DC. FRONTIERS/Tulane University project report.
Brambila C, Solórzano J. Cost analysis of reproductive health services provided by the Ministry of Health of Guatemala. (1995) Guatemala City. Population Council, FRONTIERS project report.
Brambila C, Lopez F, García-Colindres J, Marcos V. A distance learning program on management of reproductive health program for health district directors and NGOs. (2001) Guatemala City. Population Council, FRONTIERS project report.
Burkhart M, Solórzano J. Testing a tripartite strategy in San Marcos. (1999) Guatemala City. Population Council, FRONTIERS project report.
Burkhart M, Solórzano J, Taracena B, López F. Training for MSPAS and NGOs in counseling and family planning methods. (1999a) Guatemala City. Population Council, FRONTIERS project report.
Burkhart M, Solórzano J, Taracena B. Evaluation of two training programs: training in the algorithm for integrated MCH services and training in counseling and family planning. (1999b) Guatemala City. Population Council, FRONTIERS project report.
Caldwell JC, Phillips JF, Khuda B, eds. Special Issue: Family planning programs in the twenty-first century. Studies in Family Planning (2002) 33((1)).
Corwin RG, Louis KS. Organizational barriers to the utilization of research. Administrative Science Quarterly (1982) 27:623–40.[CrossRef][Web of Science]
Court J, Young J. Bridging research and policy in international development: an analytical and practical framework. (2004) Overseas Development Institute Briefing Paper, RAPID Programme. London: Overseas Development Institute.
Davis P, Howden-Chapman P. Translating research findings into health policy. Social Science and Medicine (1996) 43:865–72.[CrossRef][Web of Science]
de Vibe M, Hovland I, Young J. Bridging research and policy: an annotated bibliography (2002) ODI Working Paper no. 174: London: Overseas Development Institute.
Escalante SL, Enge K, Ottolenghi E. Testing the collaboration between two NGOs, ATI and APROFAM, in the delivery of family planning services. (1996) Guatemala City. Population Council project report.
Fernandez VH, Mazariegos L, Abaj P, et al. Designing and testing appropriate health education strategies for men in four health districts in the department of El Quiché. (1997a) Guatemala City. Population Council project report.
Fernandez VH, Montufar E, Ottolenghi E, Enge K. Injectable contraceptive service delivery provided by volunteer community promoters. (1997b) Guatemala City. Population Council project report.
Figueroa W, Enge K, Lopez R, Burkhart M. Re-engineering the community-based distribution program of APROFAM. (1997) Guatemala City. Population Council project report.
Figueroa W, Burkhart M, Coy G. Mejorando las referencias que los promotores voluntarios hacen hacia las clínicas de APROFAM [Improving referrals of health promoters to APROFAM clinics]. (1998) Guatemala City. Population Council project report.
Figueroa W, Yax D, Burkhart M, Brambila C. Testing promotional activities by promoters, APROFAM. (2001) Guatemala City. Population Council project report.
Foreit J, Frejka T, eds. Family planning operations research: a book of readings. (1998) New York: Population Council.
Frenk J. Balancing relevance and excellence: organizational response to link research with decision making. Social Science and Medicine (1992) 35:1397–404.[CrossRef][Medline]
Haaga JG, Maru RM. The effect of operations research on program changes in Bangladesh. Studies in Family Planning (1996) 27:76–87.[CrossRef][Web of Science][Medline]
Hegazi S. Utilization of operations research in Egypt. Special report. (1997) Cairo: The Population Council.
IFPRI. Impact evaluation: assessing the impact of policy-oriented social science research. (2001) Washington, DC: International Food Policy Research Institute.
Koenig MA, Hossain MB, Whittaker M. The influence of quality of care upon contraceptive use in rural Bangladesh. Studies in Family Planning (1997) 28:278–89.[CrossRef][Web of Science][Medline]
Lindblom C. The policy-making process. (1980) New York: Prentice Hall.
Lindquist E. Discerning policy influence: framework for a strategic evaluation of IDRC-supported research. (2003) Ottawa: International Development Research Centre.
Marin MC, Gage A, Khan S. Frontiers in reproductive health: Tulane University final report. (2004) New Orleans, LA: Tulane University, School of Public Health and Tropical Medicine. Online at: http://www.dec.org/pdf_docs/PDACA970.pdf.
Mazariegos L, Salazar S, Burkhart M. Testing the ‘Blanket rule’ rhythm method among indigenous Guatemalans. (1999) Guatemala City. Population Council project report.
Méndez-Domínguez A. Study of cognition and speech patterns of urban and rural indigenous community residents about reproductive health. (1995) Guatemala City. Population Council project report.
Ministerio de Salud Pública y Asistencia Social (MSPAS). Instituto Nacional de Estadística (INE), Centers for Disease Control (CDC). In: Guatemala: Encuesta Nacional de Salud Materno Infantil 2002. Informe Final. (2002) Guatemala City. MSPAS–INE–CDC.
Neilson S. Knowledge utilisation and public policy processes: a literature review. (2001) Ottawa: Evaluation Unit, International Development Research Centre.
Nutley S, Walter I, Davies H. From knowing to doing: a framework for understanding the evidence-into-practice agenda. (2002) Discussion Paper 1. St. Andrews, UK: Research Unit for Research Utilisation, University of St Andrews.
Pineda A, Mazariegos L, Gould B, Hoskins MA, Enge K. The Population Council INOPAL II sub-project report. (1995) Baseline study of reproductive health beliefs and attitudes of males in four health districts in the Department of El Quiche, implemented by: Asociacion Pro-Bienestar de la Familia de Guatemala (APROFAM), 14 May 1994 to 31 January 1995. Unpublished.
RAWOO. Utilization of research for development cooperation, linking knowledge production to development policy and practice. (2001) Publication no. 21. The Hague: Netherlands Development Assistance Research Council.
Reid F. Evidence-based policy: where is the evidence for it? In: School for Policy Studies Working Paper Series. (2003) Paper no. 3. Bristol, UK: University of Bristol.
Restivo S, Loughlin J. Critical sociology of science and scientific validity. Knowledge: Creation, Diffusion, Utilization (1987) 8:486–508.[Web of Science]
Santiso-Galvez R, Bertrand JT. The delayed contraceptive revolution in Guatemala. Human Organization (2004) 63:57–67.[Web of Science]
Seidman M, Horn M, eds. Operations research: helping family planning programs work better. (1991) New Cork: John Wiley-Liss.
Siegel K, Tuckel P. The utilization of evaluation research. Evaluation Review (1985) 9:307–28.
Simmons R, Brown J, Diaz M. Facilitating large-scale transitions to quality of care: an idea whose time has come. Studies in Family Planning (2002) 33:61–75.[CrossRef][Web of Science][Medline]
Singer B. The interface between genetics, demography, and epidemiology: examples and future directions 1990. In: Convergent issues in genetics and demography.—Adams J, Lam DA, Hermalin AI, Smouse PE, eds. (1990) New York/Oxford: Oxford University Press. 257–74.
Solo J, Cerulli A, Miller R, Askew I, Pearlman E. Strengthening the utilization of family planning operations research: findings from case studies in Africa. (1998) Nairobi: The Population Council.
Solórzano J, Burkhart M. Extensión del uso del algoritmo en el Altiplano de Guatemala. (1998) Guatemala City. Population Council project report.
Solórzano J, Burkhart M. Reporte del componente de reingeniería del modelo de atención. (1999) Guatemala City. Population Council project report.
Stone D, Maxwell S, Keating M. Bridging Research and Policy: An International Workshop funded by the UK Department of International Development. (2001) Warwick University. 16–17. July 2001.
Sundquist JL. Research brokerage: the weak link. In: Knowledge and policy: the uncertain connection.—Lynn LE, ed. (1978) Washington, DC: National Academy of Sciences.
Sutton R. The policy process: an overview (1999) ODI Working Paper 118: London: Overseas Development Institute.
Taracena B, Burkhart M, Brambila C. Pre-introductory study of NORPLANT: Guatemala. (2001a) Guatemala City. Population Council project report.
Taracena B, Contenti M, Burkhart M, Brambila C. Promoting reproductive health in victims of violence or rape. (2001b) Guatemala City. Population Council project report.
Vernon R, Schenkel P. Self-financed incorporation of family planning in rural fincas in Guatemala. (1992) Guatemala City. INOPAL project report.
Vernon R, Ottolenghi E, Solórzano J, et al. Systematic offering of family planning and reproductive health services in Guatemala. (1997) Guatemala City. INOPAL project report.
Weiss CH. The many meanings of research utilization. Public Administration Review (1979) 39:426–31.[CrossRef][Web of Science]
Weiss CH, Bucuvalas MJ. Truth tests and utility tests: decision-makers’ frames of reference for social science research. American Sociological Review (1980) 45:302–13.[CrossRef][Web of Science]
Accepted for publication 20 February 2007.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||