Here’s a general overview of the requirements to implement a Kirkpatrick Level 3 evaluation for online learning.

Level 3 evaluations measure whether the there was an actual transfer of learning to the actual work setting. This level of evaluation will increase the visibility of learning and development efforts, because successful implementation of Level 3 evaluations affects learners, manager, project stakeholders, subject matter experts (SMEs), and line supervisors. It may also relate to mentoring programs, annual performance reviews and development planning activities.

Consider the following situation:

Your boss needs you to start demonstrating the impact that online learning solutions are having on the company, whether the skills can be transferred to the job, and eventually, if the e-learning programs contribute to the company’s bottom line results. Enter Level 3 evaluations.

You understand that Kirkpatrick Level 3 evaluations include the verification that Level 1 and Level 2 have been met, and that the learner is able to demonstrate the new knowledge, skills, or attitudes (KSAs) in the work setting. In short, did the transfer of KSAs from online learning to the work setting occur? Now you feel paralysis setting in along with realization that translating this concept into actual practice within your organizational culture may not be a simple task.

Cultural Beliefs

Begin with how your organizational culture views e-learning. Put on your consultant hat and answer the following questions on behalf of your organization.

·         Managers and supervisors are willing to complete surveys for direct reports; surveys will evaluate the transfer knowledge, skills, and attitudes (KSAs) learned via online learning.

·         SMEs are willing to identify the key performance measures as observable, measurable behaviors that will result from the e-learning materials.

·         Existing tools and processes are in place regarding the evaluation of the transfer of KSAs through e-learning

o        annual performance review (form/process)

o        career development (form/process)

o        mentoring

o        coaching

o        learning management system (registration, completion, pre/post assessment, surveys & survey results).

·         Learners will be provided opportunities to demonstrate any new knowledge, skills, and attitudes (KSAs) that result from online learning activities.

·         Rewards, such as financial compensation, will be used to reinforce the desired behavior.

If you answer “No” to any of these questions then you may have several additional people/process areas to address. For example, managers may not be able to complete Level 3 evaluation surveys because your organization has not addressed how to sufficiently motivate them.

Selecting Courses for Level 3 Outcomes

Once you have addressed people, process, and culture issues, it is necessary to identify which courses are suitable candidates for measuring Level 3 outcomes.

Indeed, not every course requires a Level 3 evaluation; therefore, it’s necessary to choose your battles wisely.

The first step is to identify the business need that drives Level 3 evaluation. I suggest that you ask your customers where they have already identified performance discrepancies, as well as how they determine and track the discrepancy indicators. General areas may include the following:

• cycle time; the time it takes to complete work
• cost; the cost associated with work/work processes
• defects; poorly constructed work products resulting in scrap or rework
• damage/injury; damage or injury caused by accidents, processes, or procedures. 

In addition, specific topics may include:

·         increase productivity

·         reduced scrap rates (i.e., less wasted resources)

·         increased sales

·         reduced absenteeism

·         reduced accidents.

For example, safety is a very high priority in my current company, and I have targeted a fall protection safety course as a candidate for Level 3 evaluation.

In choosing the topic, the SME provided the following information:

Business need: An employee not following proper fall protection procedures while working at heights receives a write-up and dismissal because this behavior can result in a serious injury or death. In the event of injury, the company incurs losses in productivity, medical expenses, and possible lawsuits—not to mention the effect an injury can have on the employee’s family or morale on the job site.

Defining Performance Outcomes

Performance outcomes represent the desired behavior or outcome, and they should be defined as both observable and measurable behaviors. Let’s continue with the safety example.

Because we are looking at employees that climb and work on tall structures, we define performance outcomes for employees while working at heights. Several outcomes provided by the SME include

·         appropriately used required safety equipment

·         followed safety protocol regarding unsecured edges

·         followed proper protocol for raising and lowering materials

·         followed proper protocol for climbing.

Each of the performance outcomes has specific rules, measurements, costs, and consequences. In addition, we confirmed with the job site supervisors that they are willing to review each of these outcomes.

The next step is mapping these performance outcomes to the learning objectives.

Don’t forget: you still have a course to build. After learners take the course, the performance outcomes can be used to develop a performance evaluation to identify knowledge transfer. Before you develop the course, I suggest you create a design methodology for collecting and analyzing results.

The Design Methodology

You should become familiar with the following concepts:

• Assessment: Use pre- and post-assessments that will help you measure the degree of comprehension regarding newly gained KSAs.
• Time: Allow time for the behavior change to take place.
• Surveys: Survey learners, managers, and others who often observe the performance outcome/desired behavior.
• Control groups: If feasible, use a control group to help measure the differences in performance by measuring and comparing a group that has not participated in the online learning.

Remember: The success of your design depends on people who actually observe the performance outcomes; their input is required to effectively measure Level 3 outcomes.

One-group design. In a one-group design, it is necessary to establish a baseline measurement of current learner KSAs. This requires both pre- and post-assessments. Surveys and questionnaires document performance outcomes by asking learners, managers, and co-workers to observe the degree of desired behavior.

Control group design. If it is feasible to use a control group, you may apply a control group design. This design also requires both pre- and post-assessments. The goal is to identify if the control group performed better, worse, or no different from the group that received e-learning.

Choose a design that works for you and modify it as necessary to meet the needs of your environment. Then, take a snapshot of the situation: define, measure, and document the design potential outcomes. You can use this as your baseline for measuring actual performance outcomes. Now, that you have completed the preliminary groundwork you are ready to build the course.

Troubleshooting

Level 3 performance outcomes for online learning can break down for two primary reasons: 1) ineffective learning materials or 2) discrepancies in work conditions.

Cause	Symptom	Possible Solution(s)
learning materials	learner doesn’t understand how the KSAs should be used	·         online learning should reflect the actual work environment o        practice using actual examples o        provide performance feedback  ·         online learning should explain how the KSAs can be applied
work conditions	learner is not demonstrating the KSAs	·         provide opportunities to demonstrate KSAs ·         provide performance feedback to encourage the desired behavior ·         check compensation and rewards process;  does it encourage the desired behavior?

Sounds simple, right? Keep in mind that this is relatively high-level approach to implementing a Kirpatrick Level 3 evaluation, and you will probably encounter a variety of problems. The following table provides possible solutions to problems faced during several phases of your level three evaluation process.

Problem Solution Table

Problem	Solution
performance objectives are not linked to learning objectives	Work with your SME to outline how performance outcomes should link to the learning objectives, and focus on observable measurable behaviors can be identified.
objectives do not effectively measure for Level 3 results	Talk to managers before creating a mock-up of the evaluation data. Examine the mock-up and identify whether it will give managers the results needed to evaluate your efforts. You may also want to re-evaluate the course as a candidate for a Level 3 evaluation.
mangers don’t want to complete questionnaires	Establish and communicate the Level 3 evaluation process requirements with your customers early in the process. Tie in with existing process tools career development activities, and so forth. Identify the manager’s motivation.
management doesn’t believe the results	Document your process. Create a process map and make your data sources and reports clear and easy to understand. Ask managers their opinion about what data they think will validate your findings.

Bottom Line

If Level 3 evaluations are conducted as part of existing career development and performance review activities, then it represents another tool to improve, explain, control, and predict performance.

As you move forward, keep in mind the following ideas:

• make learning more than an event; Level 3 makes learning a process
• use business requirements to indicate whether a course requires Level 3 evaluation
• ensure that stakeholders are concerned enough about the outcomes to support the extra effort and costs
• ask SMEs to help develop the performance outcomes and evaluation criteria
• determine whether managers are willing to observe, document, and evaluate the desired behavior
• develop a process that is practical—and as painless as possible
  • control groups may not be feasible
  • managers will not have much time for observation and follow-up
  • you typically do not get 100 percent response rates on surveys/questionnaires.
• remind leaders that companies get the behaviors they reward, which are not always the behaviors they want.

Well, that should give you enough to get started. If all goes well, you will also want to have a communication strategy to explain your success and encourage others to participate.