MONITORING & EVALUATION
|
Youth
celebrate the AYA program in Uganda. |
Background
In adolescent sexual and reproductive health (ASRH) programs, a clear
need exists particularly in developing countries for well-documented
evaluations (process and impact) linking program intervention, changes in
youth behavior and health outcomes. Evidence or program impact is mixed in
literature reviews of HIV/AIDS programs, with evaluations suffering from inadequacies
in study design, analytic techniques, outcome indicators and reporting of
statistics (UNAIDS, 1997). Commonly, programs lack a focused conceptual
framework showing intersecting processes through which program strategies
aim to achieve impact. Without this program theory, the choice of intermediate
and impact indicators (the basis of evaluation) is haphazard (Shepard, 2001).
For the ASRH field, with little evidence of best practices, the current
focus should be the testing of innovations; yet few projects include evaluation
and those that do are too new to allow assessment of quantitative outcomes
(Senderowitz, 1994). Scaled-up youth programs, widely acknowledged as important
to reach youth in need, are still rare. Other research identifies two basic
reasons evaluations
have only demonstrated limited results: many have been short term and
therefore unable to show results, and most programs that have been evaluated
apply a limited set of strategies that are insufficient to change behavior
(Adamchak, et al, 2000). In this regard, evaluation of AYA its design,
approach, strategies and impact is critical to
-
Build
evidence to support the relationship between program
intervention, changes in youth behavior and health
outcomes
-
Evaluate
and document the innovation of partnerships and
the alliance model
-
Identify
components of successful scaled up interventions
- Better
understand through
process evaluations why
programs do/do not achieve desired outcomes
The
AYA Monitoring and Evaluation Mandate
The African Youth Alliance (AYA) program seeks to contribute to the improved
sexual and reproductive health of young people aged 10-24 years old, with particular
emphasis on those 10-19 years old. Specifically, AYAs goals include reduction
in HIV and other STIs, unwanted pregnancy, unsafe abortion, and forced/coerced
sex; and increased condom use, age of first sex and contraceptive
prevalence rate. Program objectives, individually and collectively, contribute
to achievement of the program goal. Intermediate results identify essential
steps to achieving program objectives. Indicators will be used in the assessment
of progress (monitoring) and successes (evaluation) at the goal and program
objective level.
AYAs underlying logic is based on
-
the
well established relationship of condom use, age
of first sex and number of sexual partners on HIV/AIDS,
sexually transmitted infections, pregnancy (AGI,
2002, UNAIDS, 2000, 2002) and by deduction, abortions.
AYA seeks to achieve ASRH heath outcomes by promoting
selected behavioral outcomes: increased condom use,
abstinence, use of modern contraceptives, reductions
in the number of sexual partners, and elimination
of first sex that is forced/coerced.
- the
integration of six program areas to achieve the objectives:
| Policy
and Advocacy |
An
improved, enabling and supportive environment
for ASRH |
| Behavior
Change Communication |
Increased
knowledge, skills, norms and positive
attitudes towards adoption of safer sexual practices |
| Youth-Friendly
Services |
Increased
access to and use of quality, youth-friendly
ASRH services |
| Livelihood
Development Programs |
Integration
of ASRH into livelihood programs |
| Coordination
and Dissemination |
Established
and/or strengthened coordination and dissemination
mechanisms for improved ASRH partnerships |
| Institutional
Capacity Building |
Strengthened
implementing partner institutional
capacity to sustain ASRH outcomes |
| Partnership |
Establish
internal and external partnerships for effective
and integrated program design and implementation |
| Youth
Participation |
Maintain
active participation of youth in program
design and implementation |
| Gender
and Sexuality |
Design
AYA programs to reflect and address issues of
gender equity and sexuality (including ASRH rights) |
| Sustainability |
Increase
sustainability of ASRH programs |
| Scaling
Up |
Increase
number of youth reached in a broader geographic
area by institutionalizing effective programs |
AYA
will aggressively seek these results (goals and program
objectives) and practice results-oriented
management of its initiatives and activities. Monitoring and evaluation
will be an on-going process to regularly assess and report on progress towards
achievement of these results. Successes, best practices and lessons learned,
as identified by this process, will be disseminated through various
knowledge sharing mechanisms. AYA works to ensure processes of monitoring,
evaluation, documentation and dissemination occur sequentially. Dissemination
serves both a knowledge sharing and partnership strengthening function with
various stakeholders, including other donors, implementing partners, NGOs,
host communities, host governments and youth.
Program
Design
AYA recognizes that its ability to monitor and document the program in
a high quality and meaningful way is inextricably linked to the design and
implementation of the program itself. In recognition of this, AYA has identified
and defined what the program aims to do (goal and program objectives)
and how AYA will accomplish its aim (strategies and activities).
AYA will also show the progress of these strategies and activities
towards the goal and program objectives (indicators). Program design is evidence-based,
drawing on documented successes and lessons learned and applying state-of-the-art
approaches, to the extent possible, from ASRH evaluations.
Monitoring
and Evaluation Challenges
Partnerships like AYA play an important role in the achievement of positive
sexual and reproductive health outcomes. But such partnerships also face unique
challenges that have an effect on and may even shape the results-oriented
programming approach
-
The
involvement and management of multiple partners
at different operating levels.
-
As
an approach, partnerships are an innovation for
which little documented evidence exists to show
that they work (evaluations, lessons learned, success
stories, best practices. Can partnerships achieve
stated outcomes? Can outcomes be sustained? Can
partnerships reduce the cost of initiatives? Are
they the best choice for scaling up?
-
Differences
in implementing partners capacity to implement,
monitor and evaluate ASRH activities with
the result that the intensity and quality of program
implementation, monitoring and evaluation will vary.
-
Significant
reliance on implementing partners to conduct monitoring
and evaluation activities.
-
Differences
in partners geographic coverage. Some implementing
partners will be national organizations, while others
will be community-based with the result that
the scope of program implementation will vary significantly.
-
Comparability
issues when implementing and evaluating a program
that is regional and spans countries with different
characteristics.
-
Attribution
issues due to the high level of involvement of other
donors in AYA countries who are implementing similar
programs.
-
Varying
degrees of political commitment of host countries,
whose participation is critical to implementation,
monitoring and evaluation.
Monitoring
and Evaluation Approach
Characteristics of the AYA monitoring and evaluation process include
Design
Baseline and endline surveys, a mid-term assessment, annual school surveys,
as well as qualitative and formative research to inform program design and to
support and enrich population-based quantitative data.
A dose response evaluation of AYA implementation and control sites (as
part of the endline evaluation), based on variances in time, intensity, quality,
and levels of integration and partnership attained, and their subsequent effect
on the program goal and objectives.
Responsibility
Specific partner responsibility for monitoring and evaluation of respective
program areas according to program objectives.
Monitoring
On-going monitoring to ensure that all activities in all sites are implemented
as designed and according to joint workplans.
Documentation
Process evaluation and documentation of the partnership/alliance management
process to show how a multi-site, multi-approach program is coordinated
to go to scale.
Research
Operations research studies that will be determined based on maximizing
opportunities in which AYA is best suited to contribute to the body of ASRH
knowledge. Primarily, AYA is not an operations research project and therefore,
to the extent possible, will leverage external resources to achieve
monitoring and evaluation objectives through external partnerships and
technical assistance.
Structure
Technical advisory committees at the global and country level to advise
AYA on development and implementation of a monitoring and evaluation plan.
An internal advisory and technical group made up of partner agency monitoring
and evaluation staff, to provide technical assistance for practical implementation
and support.
Process
Participatory, consultative and inclusive approaches that ensure the involvement
of stakeholders at the different operating levels.
On-going capacity building for AYA implementing partners and staff. As
appropriate, maximize opportunities to use and/or strengthen existing monitoring
and evaluation systems (e.g. research and data collection systems).
Dissemination
Dissemination of evaluation findings, identification of best practices
and lessons learned and knowledge/information sharing using existing systems,
with ASRH stakeholders. Promotion of the dissemination effort through various
fora at global, national and district levels.
|