Do happy learners make performance improvements?
I think it goes without saying that a happy workforce will in fact drive better business results but what about when you focus on the workforce when they have just engaged in piece of learning. In other words they are no longer just workers but learners.
Level 1 evaluations - AKA 'Happy Sheets'
When ever I ask the question...
"do you evaluate the your training?"
I will more than likely get an answer of "yes" followed shortly by a statement something like...
"we send out evaluation forms with every course we run"
These level 1 evaluations or 'happy sheets' usually provide information on the venue of the training, the facilitator, the course content, course administration, potential indications of the knowledge / skills gained, etc. What they fail to do is identify any actual real world implementation of the learning, performance improvement or business results. So why is it that the majority of organisations never look beyond the 'happy sheet'.
Is it the fact that by performing any evaluation, albeit at a very high level, is enough to put a tick in a box and satisfy senior management? Is there an acknowledgement of the fact that more detailed and worthwhile evaluation data requires a much greater amount of resource and that the time and effort to acquire the data cannot be justified? Or perhaps, more worryingly, do people think that level 1 evaluation data provides enough of an indicator of potential improvement that any other evaluation is unnecessary?
It is this final question that I seek to answer in this post. Can level 1 evaluation data indicate the eventual performance improvement?
The raw data
Given our broad client base and the fact that we pass roughly 700,000 learners through our systems each year, we have a wealth of data relating to level 1 evaluations. As you would imagine there are much fewer instances of level 3 evaluations and this data is required to validate a real performance improvement. The final data sample I have used consists of entries from 4 recent years and across all vertical markets, thereby reducing the effect of any potential outside influences.
The graph below shows the spread of data when level 1 scores are matched to their level 3 counterparts. The level 1 scores relate to immediate feedback in relation to the training they participated in. This will include admin, course content, facilitation, etc. The level 3 data is a score based upon the extent to which the learning has contributed to their performance improvement.
What is clear is that there is little to no correlation in the data. What is interesting about this fact is that prior to me sifting through the data I myself expected there to be more correlation than there is. I anticipated that those learners who came away from the training enthused and energised about their experience were more likely to put into practice the learning covered on the course. It startled me to find that immediate enjoyment of the learning and even its content has very little bearing on improvement in role.
After observing the lack of correlation I decided to do a deeper dive into the data, the often startling results of which are covered in the sections that follow.
Obviously the answer has to be NO. We have to remember that these responses are taken close to, if not on, the training programme itself and as such there will likely be an abundance of positive emotions. Things that invoke a positive reaction may include:
This positive slant effectively means that trying to find any correlation between level 1 scores and performance impact becomes much harder. For example, take the following relatively standard 5 point scale:
If the overall question for a piece of training reads something like...
94% of individuals are scoring at either "Agree" or "Strongly Agree". This does not make for insightful data analysis relating to potential performance improvement. However it is a great gauge of learner satisfaction immediately post event.
The graph below shows the spread of data when level 1 scores are matched to their level 3 counterparts. The level 1 scores relate to immediate feedback in relation to the training they participated in. This will include admin, course content, facilitation, etc. The level 3 data is a score based upon the extent to which the learning has contributed to their performance improvement.
What is clear is that there is little to no correlation in the data. What is interesting about this fact is that prior to me sifting through the data I myself expected there to be more correlation than there is. I anticipated that those learners who came away from the training enthused and energised about their experience were more likely to put into practice the learning covered on the course. It startled me to find that immediate enjoyment of the learning and even its content has very little bearing on improvement in role.
After observing the lack of correlation I decided to do a deeper dive into the data, the often startling results of which are covered in the sections that follow.
Level 1 evaluations - 70% satisfaction guarantee
Unsurprisingly, level 1 evaluations lean towards awarding full marks more often than not. What is surprising is the lack of any significant outlier data. 94% of level 1 responses come from individuals scoring training at 70% or more. In fact, 49% of all responses were scored at 90% or better. With only 6% of respondents scoring less than 70% in terms of course satisfaction, does this mean that all training is worthwhile, enjoyable or relevant?Obviously the answer has to be NO. We have to remember that these responses are taken close to, if not on, the training programme itself and as such there will likely be an abundance of positive emotions. Things that invoke a positive reaction may include:
- Learning something new that may impact their abilities
- Having a fun / engaging exercise prior to completion of the evaluation
- Spending 2 days out of the office
- Meeting colleagues they have not seen in some time
- Actually just attending training rather than the usual day to day work
- etc.
This positive slant effectively means that trying to find any correlation between level 1 scores and performance impact becomes much harder. For example, take the following relatively standard 5 point scale:
If the overall question for a piece of training reads something like...
"Did the training deliver against your development needs?"
94% of individuals are scoring at either "Agree" or "Strongly Agree". This does not make for insightful data analysis relating to potential performance improvement. However it is a great gauge of learner satisfaction immediately post event.
What exactly are you measuring?
Throughout this post I have mentioned that level 1 evaluation measures general satisfaction with the training but perhaps more accurately it measures an subjective opinion or feeling in the moment. This opinion needs no justification or evidence to support it and as such can be easily influenced by outside factors.
Level 3 evaluations on the other hand are based upon more considered thinking. Whilst these are often opinions they are based upon information gathered over a period of time and then evaluated by the learner. The following question is a spin on one we use in a large proportion of our level 3 evaluations:
"To what extent has the course been directly responsible for your performance improvement?"
This question seeks to establish a tangible link between the learning and performance. Level 1 evaluations do not focus on creating any meaningful link, and in fact cannot, given their completion immediately after the learning. Instead they focus on how suitable the content of the learning was or how much new information / knowledge you have gained, all of which is pure speculation.
The crux of it is that level 1 evaluations don't ask the right questions to be able to accurately predict eventual performance improvement.
The crux of it is that level 1 evaluations don't ask the right questions to be able to accurately predict eventual performance improvement.
Dissatisfied learners won't improve!
So I have already discussed the fact that 94% of responses come from people scoring more than 70% in level 1 evaluations and that this has little to no bearing on performance improvement. However, there is a trend when it comes to the remaining 6% of respondents. In other words those that answer below 70% on their level 1 evaluations.
Those that respond with 60% satisfaction at level 1 tend to see a reduced amount performance improvement that can be related to the training. In fact 89% of respondents that scored 60% or less at level 1 attributed less than 50% of their performance improvement to the training they received. Is this surprising?
In many ways this statistic makes complete sense. If the training doesn't meet the needs of the individual or map well to their role then it would stand to reason that any performance improvements wouldn't map to the training. It must also be considered that even if the training is relevant, if the individual has a negative learning experience then they are less likely to attribute any improvements to the learning.
The difficulty with making decisions based upon this information is that it is such a small proportion of the data available. Discontinuing or changing training based upon this amount of information could not be recommended. Without level 3 data relating to the success of the actual content, making such suggestions would not be wise.
Conclusion
To answer my original question... There is no guarantee that happy learners make performance improvements. This is not to say that level 1 evaluations are not relevant. They provide meaningful data as long as it is understood what this data measures. What our data shows us is that you cannot use level 1 data to predict performance improvement. If your goal is to truly understand the effectiveness of training then time and effort has to be invested into proper detailed evaluation methods.
0 comments: