WMO-No. 1205 distinguishes between competency assessment methods, or general activities for gathering evidence, and competency assessment tools, specific instructions, questions, simulations or protocols for carrying out those activities. This section
will outline the key methods, while a number of brief examples of specific tools are offered as guidance in the Toolkits on this site. The methods and tools provided are intended to help you in creating your own assessment programme. The five methods
discussed here are Direct Observation,
Experiential Questions, Quiz Items, Simulation, Portfolio and Third-party Feedback, which generally correspond to the methods described in WMO-No. 1205.
Direct Observation is done by following the work of an individual in their workplace, during one or more operational shifts. The assessor may ask the individual to describe their work activities and decisions, using a "think-aloud" protocol to reveal their thought processes, while the work is conducted. For example, the individual can be asked to describe why the chosen forecast products or tools are being used and how this supports the decision-making process. But the assessor should be careful to ensure that any questions by and discussion with the assessor
does not excessively slow the natural forecast process when the time available to issue forecasts is limited, such as during severe weather.
Experiential Questions are asked during an interview with the assessed individual. (This method is included in the Questioning method in WMO-No. 1205.) Experiential Questions are particularly
helpful when it is not practical to directly observe work processes during rare and severe events, or for seasonal phenomena that do not coincide with the assessment schedule. These questions may be structure by asking questions such as, "Describe
a time when you had to forecast during a (specific weather) situation. What data and other resources did you consult and what steps did you take to make your forecast decisions? What was most positive about your forecast process, and
what could have been done better?" If it is unlikely that the forecaster would have experienced such an event, the questions could be hypothetical: "If you were forecasting during a (specific severe weather event), what data sources would you require to make the best forecast? What would you do if (specific data) was unavailable? What steps would you follow to make a good forecast?"
Quiz Items are difficult to design to assess complex tasks, but they can help to identify the skills and knowledge available to the individual to complete their forecasting process (such
as factual information, process steps, and analysis of data). More complex Quiz Items that assess more complex skills are possible, but because they take more time to design, these are
useful only for very large audiences. (Experiential Questions or Simulations might be better choices.) Quiz items have the advantage of being consistent across all individuals being assessed, and taking less time to administer. Ideally, they
would be used not as a primary assessment method, but as a supporting method in a larger assessment process that also includes other methods.
Simulation is a good choice for assessment competency to complete a complex task like forecasting, but it is even better when combined with Direct Observation to ensure validity. High-fidelity Simulations with large data sets and tools similar to those in the workplace take a large amount to time to develop, but when the number of individuals to be assessed is larger, this can
be a wise investment. Low-fidelity simulations, those that use primarily paper copies of data, can also be valuable tools, but have more limited validity. Role play simulations, asking an individual to assume a role in a hypothetical scenario,
can be surprisingly effective using low-fidelity materials, and may be effective to assess communication and coordination skills. For a useful simulation tool, see the EUMETSAT Simulator User Guide.
Portfolio is a method that can provide access to evidence of competency that might be missed if only other methods are used. A portfolio is a collection of materials prepared by the individual being assessed that they feel demonstrates their competency, such as a work diary, prepared case studies, reports and research. It may also include a structured self-assessment, which can reveal strengths and self-awareness that is not visible elsewhere. While training completion does not equate to competency, a Portfolio might include evidence of successful participation in training events to provide a larger picture of professional growth. If only the Portfolio method is used, the time commitment for assessors can be greatly increased, so it is best used as support to a more complete assessment programme using a variety of methods.
Third-party Feedback is another valuable source of evidence of competency, but to avoid bias, it should be used only in combination with other methods. Third-party Feedback can include interviews or documentation from supervisors, peers, and customers who have interacted with the individual. This method will not be included in the Toolkits, but you may wish to include this form of evidence in your total assessment programme.
