Book Evaluating Training Programs

The Ten-Step Evaluation Process

You need the ability to evaluate training programs to determine their effectiveness. Upper-level executives and training managers must be able to make decisions based on whether or not a program worked. To end up with a solid training program, consider these 10 factors when you are planning and implementing training:

• Determine needs - Using interviews or a survey, ask potential participants, their bosses and others familiar with the job what they need to know. Test the participants, or analyze their performance appraisal forms. With a survey, participants can quickly indicate their level of need for training on a particular subject. Compile these sums and create a weighted score for each category. Use this data to help you decide what training programs to establish. An advisory committee can help you decide what to cover. Keep them posted on evaluation results to deepen their knowledge and ability to help you.
• Set objectives - Once you know what the needs are, establish objectives for the program according to the results you want, such as improvements in production, sales, quality, turnover, absenteeism, morale, quality of work life, profits, or return on investment. Determine what knowledge, skills and attitudes you want participants to learn and what new behavior you want them to adopt as a result.
• Determine the subject - You or the trainer should decide what topics to present to meet the stated needs and to accomplish the objectives. Based on the subject and on your objectives, determine how to provide the training.
• Select the participants - To choose participants, ask four key questions. 1) Who will benefit? 2) What training programs do various regulations require you to provide? 3) Should training be voluntary or compulsory? 4) Should you offer the training to participants at different levels of the organization, or combine organizational levels in the same training sessions?
• Determine the schedule - Consider the convenience of the trainees and their bosses, and the best conditions for learning. Consider whether to set up a concentrated program - such as a solid week or weekend of training - or, to spread training over several weeks or months as an ongoing program. Generally, an ongoing program - such as a monthly three-hour session - is preferable. A three-hour session provides enough time to include various kinds of instruction, group participation, and videotapes or other materials.
• Select the facilities - Choose facilities that are appropriate, comfortable, and convenient. Avoid rooms that are too small, noisy, or stuffy, or have uncomfortable furniture. Avoid places that are too hot or cold, or that have inconvenient locations.
• Select the instructors - Instructors should know the subject, enjoy teaching, communicate well, and be effective at getting people to participate. Seek "learner-oriented" instructors who are focused on meeting the needs of your trainees.
• Select and prepare audio-visual aids - Audio-visual aids help participants stay interested and encourage communication.
• Coordinate the program - In some cases, an outside trainer will handle the coordination details and will teach. In other cases, you or others in your organization will assist with the details of coordination, such as arranging for meals and for materials such as flip charts, handouts, and reaction surveys.
• Evaluate the program - Plan your approach to evaluating the program’s effectiveness.

Reasons to Evaluate Programs

You want to evaluate training programs to determine how effective they have been and how you might further improve them. Other reasons for evaluating programs might include determining whether or not to continue a program and to assess the importance (or continuation) of a training department by showing how it contributes to the company’s objectives and goals.

“By demonstrating to top management that training has tangible, positive results, trainers will find that their job is more secure, even if and when downsizing occurs.”

To improve a training program, focus on eight key areas when you conduct an evaluation:

1. How well does the subject matter meets the needs of attendees?
2. Is the current leader the best-qualified person to teach the program?
3. Does the leader use the most effective methods for maintaining interest and teaching the content you want taught?
4. Are the facilities satisfactory?
5. Is the schedule appropriate for participants?
6. Are the teaching aids (audio-visuals, etc.) effective in holding participants’ interest and improving communication?
7. Was the program was coordinated effectively?
8. What can you or the trainer do to improve the program?

Four Levels of Evaluation

The four levels of evaluation represent four ways to assess programs, starting with the most fundamental level and progressing to a deeper analysis. Start with the first level and then look at each subsequent level, since each on is important and has an impact on the next. The four levels are: Reaction, Learning, Behavior and Results.

• Level One: Reaction. This is essentially a measure of "customer satisfaction" in that it measures how program participants respond to a program. You are hoping for a positive reaction, since participants generally must respond favorably to be in a frame of mind in which they to want to learn. If they react to a program negatively, they are less likely to learn from it.
• Level Two: Learning. Measure the extent to which participants changed their attitudes or gained greater knowledge or skills as a result of the training program. Some programs are set up to change knowledge, skills, and attitudes; others focus on one particular element. For instance, programs on diversity are designed primarily to change attitudes; technical programs are designed to improve skills, and programs on leadership, motivation, or communication are designed to cover all three objectives.
• Level Three: Behavior. Did the participants’ behavior change due to the program? Four conditions must be met for behavioral change to occur. The person being trained must want to change, must know what to do, must know how to do it, and must work in a climate that rewards change. These last two conditions are critical, since even if a person seeks change and has been taught what to do and how to do it, behavioral change won’t occur without a favorable climate and a supportive boss. The workplace climate should encourage change, not prevent it, discourage it, require it, or be neutral about it. An encouraging climate offers participants intrinsic or extrinsic rewards for changing. An intrinsic reward might be a sense of pride or achievement; an extrinsic reward could be praise, recognition, or even extra money.
• Level Four: Results. Look at the final results of the training from the organizational perspective. You might measure increased production, improved quality, decreased costs, reduced frequency or severity of accidents, increased sales, reduced turnover, heightened regulatory compliance, or higher profits. Often these results are why your company wanted training programs in the first place, so you state training’s final objectives in these terms, based on corporate priorities. These results can be more difficult to measure, since many other internal (i.e. turnover, etc.) and external influences (i.e. market pressures, etc.) can affect them. However, link what happened in training to these broader organizational goals as much as you can.

Evaluating Reaction

Most trainers use reaction sheets, which come in dozens of formats. Determine what you want to find out and design a survey form and scoring sheet to quantify reactions specifically along those lines. Encourage participants to write down their comments and suggestions. Strive for a 100% immediate response. Encourage an honest response by telling participants not to write their names on their evaluation forms. You can develop acceptable standards several ways, such as tabulating responses to get a baseline rating. Then, measure your reactions against these standards, and act accordingly to improve the program if necessary. Finally, communicate these reactions as appropriate to the program trainer or top management, so the program can be modified or continued based on these results.

Evaluating Learning

Develop measures to assess what knowledge was learned, what skills were developed or improved, and what attitudes were changed. You can evaluate the levels of knowledge, skills, or attitudes before training, and again afterwards. If practical, use a control group of people who have not yet received training so you can measure changes in the people who participated in the training (called the "experimental group"), compared to those who did not. You can use a paper and pencil test, such as a multiple-choice test, to measure knowledge and attitudes. However, when you measure skills, use a performance test to see how well the participants perform after the training compared to their performance before training. Again, seek a 100% response and then use the results to take the appropriate action.

Evaluating Behavior

First, you need to allow some time after training for the behavior to take place. You can’t predict exactly when a change in behavior will occur, since a trainee may not immediately need to apply the learning he or she has gained. It could take some time for the new behavior to occur, or it may never happen. However, the way to promote behavioral change is to provide the trainee with help, encouragement, and either intrinsic or extrinsic rewards. If feasible, evaluate the behavior before and after the training, and use a control group for comparison. You can also survey or interview trainees, their immediate supervisors, their subordinates, or anyone else who observes their behavior. Get a 100% response if you can, or use sampling. Repeat the evaluation at appropriate intervals, considering the costs of the program compared to the benefits. Sample interviews or survey questionnaires might be helpful when you design your evaluation.

Evaluating Results

Use a control group and measure before and after the program if you can. Allow time for participants to achieve results. Consider the cost of the training program versus its specific benefits for your organization. You may not be able to prove the results of the training positively, since the results you are measuring (such as profits or turnover) are affected by so many other factors. In that case, try to obtain the best possible evidence of the results in making your assessment.

Effective Evaluation

This four-level approach has proved effective for many different companies, thus providing case studies to help you design your own program. These include programs on: 1) performance appraisal and coaching, 2) leadership training, 3) creative management, 4) problem solving, 5) corporate performance improvement, 6) outdoor-based training, 7) safety training, and 8) stress management. You can apply this evaluation approach to any type of training program.