Luciane Veeck - Training impacts evaluation using the Success Case Method

We are a Team of authors: 

Lu Veeck (WMO), Barbara Bourdelles (Meteo France), Pat Parrish (Independent Consultant), Daniela Schroeter (Brinkerhoff Institute), Alan Bol (The COMET Program), Vieri Tarchiani (IBE), Marina Baldi (IBE), Moussa Waongo (AGRYMET), and Leah Yeddes (WMO).

Some information about the capacity development project that was evaluated

In 2016, WMO and the RTCs of Italy (IBE) and Niger (AGRYHMET) began a 4-year capacity development project (PACC-RRC) to help improve climate services for the agricultural sector in 17 countries of West Africa. The project included four blended-learning courses and one fully online course hosted by the RTCs, as well as a Networking Event attended primarily by Directors of the NMHSs of the target countries. Around 80 people attended one or more courses offered by the PACC-RRC project. As a closing project activity, it was decided to investigate the impacts of the training on the job performance and outcomes of the participants who attended the training events.

The impacts evaluation method we used

A traditional recommendation for training evaluation suggests that, when possible, it is important to evaluate training at four levels.  

1. Reaction: Perceptions of value by course participants after a course 
   
2. Learning: The increase in knowledge and skills from the training 
   
3. Application: The use of new knowledge and skills on the job by learners 
   
4. Results of Impacts: The improvements on individual performance that in turn result in improvements in organizational performance 

Levels 1 and 2 were evaluated as part of PACC-RRC project, so for this closing project activity we addressed levels 3 and 4. The Success Case Method (SCM) was chosen for this evaluation as it focuses on the impacts recognized by the practitioners themselves. In addition, rather than looking at average performance, the method strives to examine high and low impact cases.  This maximizes learning about the types and scopes of impacts that have been achieved while also helping to identify obstacles to success.

The steps we followed for this evaluation were: 

• Meetings of the evaluation team and stakeholders to identify potential success indicators to inform the next steps. 
  
• Development of a survey, which was distributed to 80 program course participants, 60 of whom responded, for a 75% response rate. 
  
• Based on the survey analysis, eleven participants were invited for interviews. The evaluation team sought to interview success cases and also those who appeared to have barriers to success.
  
• After analysis of the interviews, a comprehensive report was developed utilizing and triangulating all findings. This was published by WMO ETR and it is available at the WMO E-Library as ETR-28
  

Sharing experiences about training impacts evaluations

The evaluation team appreciated the value of direct contact with a number of participants through interviews, which created the opportunity to gather genuine stories of successful application and potential barriers to success. Through this combination of interviews and the 75% response rate on the survey, the team feels they have a very good picture of the impacts of the PACC-RRC project.

Similar methodologies that take an appreciative approach might also bring similar benefits.

Have you been evaluating the impacts of training? We would like to take this opportunity to hear about your experiences too!

We will have a live session on Thursday 30 September at 06:00 UTC. During the live session we will explore a bit more about the experience we had using SCM for this training impacts evaluation. But we do not want this to be just about our experience. 

We invite you to join us in this conversation about ‘evaluating the impacts of training’ using the Forum and participating in the live discussions!

References

Meanwhile, if you want to learn more about impact evaluation and the Success Case Method, these are some resources to consult: 

1. Peersman, G. (2015) Impact evaluation. BetterEvaluation. http://www.betterevaluation.org/themes/impact_evaluation 
   
2. Jonathan Deller (2019): Brinkerhoff Model 101: Methodology and Goals
3. Using Brinkerhoff’s Success Case Method to evaluate the impact of training (2021): A conference from Robert Brinkerhoff and Daniela Schröter, available on WMO - OMM Vimeo channel, see below.