Analytics Deep Dive: Knowledge Gaps & Lifts Analytics Panel

Welcome to week four of our OttoLearn analytics email series! 

Last week, we took a look at the Assignment Engagement analytics panel and how you can use it to track Module assignments and learner engagement. 

This week, we’ll be exploring the Knowledge Gaps and Lifts analytics panel, where you can learn more about learner proficiency.

What is proficiency?

Every time a learner completes an Activity, OttoLearn’s algorithm tracks several factors, such as accuracy, confidence, duration, past performance, etc. This data is used to determine if the learner was proficient (knew the answer) or not proficient (didn’t know the answer or likely guessed).

The OttoLearn analytics page, showing three analytics panels. The Knowledge Gaps and Lifts panel is the center of the three, and it is circled with a blue box. The panels to the left and to the right of it are blurred out.

The Knowledge Gaps and Lifts analytics panel has two tabs: Knowledge Gap and Knowledge Lift.

Close-up of the first row on the Knowledge Gaps and Lifts panel, showing two blocks: one with the knowledge gap percentage (22%) and the knowledge lift percentage (0.08%), both in the last 30 days.

The metric on the left, Knowledge Gap, represents the percentage of Activities not completed proficiently in the past 30 days. 

Looking at the screenshot above, you can see that 22% of my Activities were not completed proficiently in the past 30 days. The small red number shows the knowledge gap is 0.77 percentage points higher than the previous 30-day period.  

A knowledge gap represents the probability that a learner will not complete an Activity proficiently.

You want your Knowledge Gap metric to drop over time. As learners reach their target mastery strength in each Concept, they should answer fewer Activities incorrectly.

When you add new content or assign training to new learners, this number will increase before dropping again.

The Knowledge Lift metric shows the average improvement in proficiency for Activities performed in the past 30 days. 

Looking back at the screenshot, you can see that proficiency has increased by 0.08% — so not very much!  The small red number shows this is one percentage point less than the previous 30-day period.  

A knowledge lift represents your learners' improvement in proficiency.

Ideally, your Knowledge Lift metric should increase over time as learners become more proficient in answering your Activities. My metric is relatively small, but that can be attributed to the fact that most of my learners have reached their target mastery strength.

Generally, when you see a high knowledge gap value, you can expect to see a large knowledge lift around the same time. Proficiency should improve as learners get more familiar with your content.

Each metric represents a tab you can click to view more information on gaps or lifts. 

Let’s look at the Knowledge Gaps tab first. 

Knowledge Gaps Tab

The first metric shows the knowledge gap for learners based on their mastery strength in a Concept at the time they answered an associated Activity (an Activity within that Concept). 

The second row down the panel shows a bar chart of knowledge gap by mastery level. The x-axis goes from 0 to level 5. The 0 level shoes the highest knowledge gap at around 28%, with it gradually decreasing as you move up in mastery level. Level 4 and 5 are the lowest gaps, both around 10%. The bar at level 0 is red, the bar at level 1 is orange, and the remaining levels 2-5 are blue.

From the screenshot above, you can see that when my learners are at mastery level 0 or 1, they are more likely to answer an Activity not proficiently. Put another way, the lower a learner’s mastery strength when they answer an Activity, the larger the knowledge gap.  

When looking at this metric, you want to see a downward trend. As learners raise their mastery strength (moving towards level 5), they should have a smaller knowledge gap (the bars should get shorter)!

Throughout this section, each bar’s color reflects the size of the knowledge gap.

A bar that shows the range of colors that represent the knowledge gaps, from 0% to 100%. Gaps between between 0% to 20% are blue. At 21% the blue turns to orange, which then turns to red as it hits 100%.

The next metric in the Knowledge Gaps tab shows the average knowledge gap when learners answer an Activity with a mastery strength of 0. It provides insight into how your novice learners are performing.

Knowledge gap by Module bar chart, for assignments at mastery level 0. It shows a list of all available Modules in the OttoLearn account along the y-axis and the percentage knowledge gap along the x-axis.

As you can see, my learners have the highest gap in the Science of Learning Module, answering 34% of Activities not proficiently when they are at mastery level 0. At the other end of the spectrum, my learners have the lowest gap in the Leadership 101 Module, answering 18% of Activities not proficiently when they are at mastery level 0. 

To get these specific percentages, I hovered over each bar. For even more details, you can click each bar to see how learners perform in the Topics and Concepts associated with each Module. 

By looking at the Knowledge Gap in each Module, you can see what content learners might struggle with (larger gap) or find too easy (small gap). This insight can help inform your decisions when setting the mastery goal for each Module, deciding where to add new content, or delivering supplemental training.

The final metric, Module Knowledge Gap by Mastery Level, is presented in a heat map. Here you can see a detailed breakdown of how learners perform in each Module based on their mastery strength at the time they performed Activities. 

Heat map displaying Module knowledge gap by mastery level. The chart is a 7x7 grid of squares. Each square has a percent (0-100%) and a color (red, orange, blue), and correlates to a Module (y-axis) and a mastery level (x-axis). The grid is mostly blue squares with low numbers, under 15%, around the center and moving to the right of the grid, indicating the expected behavior: fewer knowledge gaps the higher the mastery level.

Looking at the screenshot above, you can see that as my learners' mastery strength increases, their knowledge gaps decrease. 

For example, looking at the bottom row, learners at mastery level 0 tend to answer 34% of Activities in The Science of Learning Module not proficiently. When learners answer these same Activities at mastery level 3, this drops to 15%. When they are at level 5, this drops to 13%.

Similar to the previous metric, you can click each block within the map to see how learners perform in the Topics and Concepts associated with each Module. Drilling into the Concept level is a great way to see  where content may need to be expanded upon or explained better.

As learners increase their mastery strength, their knowledge gaps should decrease (they should answer fewer Activities, not proficiently).

By looking at the All column, you can get an at-a-glance performance summary across learners at all mastery strengths.

Knowledge Lifts Tab

If you click the Knowledge Lift metric at the top of the panel, all the data will update to show information on knowledge lifts. 

Recall that a knowledge lift represents learners’ improvement in proficiency.

To learn more about how we calculate knowledge lift, view our Knowledge Lift Calculation help page.

Equation: ( Overall proficiency - Baseline proficiency / Baseline proficiency ) x 100

The first metric you’ll see is the knowledge lift for learners based on their mastery strength in a Concept when they answered an associated Activity. Each value represents the improvement in proficiency compared to when learners answered at a lower mastery level. 

A simple bar graph showing knowledge lift by mastery level. The y-axis goes from 0% up to 30%, and the x-axis shows mastery levels 1-5. The level 1 bar starts at around 18% and gradually increases to about 26% at level 5.

In the screenshot above, you can see that when my learners are at mastery level 1, they tend to answer Activities 18% more proficiently. When they are at level 5, they tend to answer 27% more proficiently. 

In general, this metric should always show an upward trend. As learners increase their mastery strength, they answer Activities more proficiently.

Typically, you will see the most improvement between mastery levels 1-3. After that, the values tend to taper off.

The Knowledge Lift by Module metric shows how proficiency has improved in each Module. This improvement is based on performance when learners were at mastery level 0.

Bar chart showing knowledge lift by Module, proficiency improvement compared to mastery level 0. It shows a list of all available Modules in the OttoLearn account along the y-axis and the percentage knowledge lift along the x-axis.

As you can see, my learners have the largest improvement in proficiency in the Smart Goals Module (28% improvement) and the smallest improvement in the Leadership 101 Module (7% improvement).

You can hover over each bar to see specific values and click to drill in to view information on Topics and Concepts.

When looking at this graph alongside the Knowledge Gap by Module graph, you will often see a trend where Modules with the largest gaps also show the largest lift.

For example, if you recall from the Knowledge Gaps section above, my Leadership 101 Module had the smallest gap. This Module also shows the smallest lift because learners have less room for improvement.

The last metric is another heat map. It shows specific information on the average knowledge lift in each Module based on learners’ mastery strength at the time they performed Activities.

Heat map displaying Module knowledge lift  by mastery level. The chart is a 7x7 grid of squares. Each square has a percent(0-100%) and a color (light blue to dark blue), and correlates to a Module (y-axis) and a mastery level (x-axis). The grid is mostly dark blue squares with higher numbers, between 20% and 35%, around the center of the heat map.

Based on this screenshot, as my learners’ mastery strength increases, so does their knowledge lift. Put another way, as my learners’ mastery strength increases, their improvement in proficiency also increases. 

For example, looking at the top row, learners at mastery level 1 showed a 13% improvement in proficiency in the Creating Effective Content Module. Learners at level 5 showed a 23% improvement in the same Module. 

Like the knowledge gaps heat map, you can click each block to drill in to view information on Topics and Concepts. 

As learners increase their mastery strength, they should also have a knowledge lift (improvement in proficiency).

Suppose learners show a large improvement early on. This trend might indicate that they are already familiar with the content or that your Activities are too easy.

As another example, imagine one of your Modules, Topics, or Concepts shows a small knowledge lift over time and a large knowledge gap. This trend suggests that content in that area needs to be improved or expanded.

Well, that was a lot of information to absorb! Hopefully, you now better understand your knowledge gap and lift analytics and how to interpret them. If you do have questions, our Support Team is happy to help. 

Join me next week to learn about the Knowledge Card analytics panel.

  • For learning content to enter and remain in a learner’s long-term memory, the learner needs multiple exposures to the content. Long-term encoding “needs opportunities for rehearsal and repetition,” Jan Breckwoldt et al. wrote in a study on mass vs. spaced learning.
  • Repeated exposures alone are not as helpful as spaced repetitions that ask learners to recall and apply information — and especially when learners have to use that information in different ways, many studies have found (for example Rohrer, Lin et al., and Bjork and Bjork).
  • The ability to remember information depends on the number of times a learner encounters it and the interval between repetitions, according to Tabibian et al.

Intrinsic

Extrinsic

Access to knowledge or performance support tools

Achieving a worthwhile or meaningful goal

Achieving a reward — a grade, a badge, points, a prize

Receiving an unexpected reward

Contributing to improving a project or a product

Wanting to be perceived as a team player, wanting to be liked

Improving performance or effectiveness relative to own past performance

Improving performance or effectiveness relative to coworkers; “winning” or being the best

Knowing enough to avoid making mistakes and do better work

Losing status or levels within a gamified framework as the result of making a mistake

Feeling of completing a task, accomplishing a goal, finishing a project

Doing the “right” thing — following rules or norms, being ethical

  1. Is the corporation’s compliance training program well designed?

    Prosecutors will look at whether the training is designed to prevent and detect wrongdoing and whether management is enforcing the program by means of training, incentives and discipline.

  2. Is the program being applied earnestly and in good faith? In other words, is the program being implemented effectively?

    Prosecutors are expected to directly investigate whether a program is merely a “paper program” or a sincere effort. Evidence of a company-wide commitment to ethics and compliance, promoted by senior and middle management, is needed.

  3. Does the corporation’s compliance training program work in practice?

    Good intentions and training don’t count if they don’t work; in assessing whether the program “works in practice,” prosecutors will look at how the suspected misconduct was detected, what the company’s investigation process is and how the company is trying to correct the problem.

Microlearning delivers small, narrowly focused bits of information.

Adaptive microlearning tailors that content to each learner’s knowledge gaps and learning goals, ensuring the training is relevant.

Continuous adaptive microlearning conditions each learner to engage with relevant training every day — for just a few minutes.

Current Rank

Previous Rank

Technology

1

1

Personalization/adaptive delivery

2

3

Artificial intelligence

3

new

Learning analytics

4

2

Collaborative/social learning

5

5

Micro learning

6

new

Learning experience platforms

7

7

Virtual and augmented reality

8

10

Mobile delivery

9

4

Consulting more deeply with the business

10

6

Showing value

11

new

Performance support

12

11

Neuroscience/cognitive science

13

13

Video

14

9

Curation

15

12

Developing the L&D function

No items found.

When people have a question or don’t know how to do something, what do they do?

Whip out a smartphone and look for information. What they don’t do is sign up for a 1-hour seminar.

Microlearning brings corporate eLearning into the modern paradigm. Microlearning describes eLearning content that is:

  • Narrowly focused
  • Short
  • Available on demand
  • Mobile-first or mobile-friendly

It must answer a question, meet an immediate need, or help the learner solve a problem.

In the City of BigTown, there was held a conference,
One of training professionals — those making a difference.
A difference to company ROI by delivering training,
From many perspectives — like Manufacturing.
And, too, there were call centers, colleges, corporate sectors,
Each chiming in about outcomes and metrics.
All shipped their training through an LMS platform,
But were desperately seeking true training reform.

Antonio

One was Antonio, who hated the manuals —
For his product revisions and updates, they were annual.
Plus his printing costs? Oh, they were crazy!
And he truly believed that franchisors were hazy.

None knew how to train in an effective way,
"There’s too much to read, to do!” they’d all say.
For there were many levels of training to assign,
From the top at head office, down to those on the front-line.

Trainers Helen and Abinash nodded, “We agree!”
Said Feng, "Paper and handbooks? Just another dead tree.
On the job, not everyone will have the info they need,
Because the content changes and updates they never did read.
They never learned the content added along the way
That may apply to their region or division today.
Plus, in the field with team members in many locations,
Mobile-first training would make a stronger foundation!

Said Sales trainer Jane of her PDFs stored online,
“They’re rarely revisited after onboarding time.
I need content delivered in snack-sized bites,
And the ability to test them until they get it right.”
Ursula
chimed in, "Onboarding’s a pain for new hires,
With most feeling like their hair is on fire!
Plus, promoted reps must refresh what they know
To be properly prepared to perform their new role."

"I deal with compliance," sighed Manal the Banker.
Abinash nodded, Frank turned to thank her,
For she’d raised the ugliest concern of them all —
That certifications aren’t based on year-long recall.
“To maintain the standards and follow each rule,
We need more than one test that comes out of the blue.
When it comes to things like health & safety, it's a game-changer
Because if their training is lacking, they could be in danger.”

Antonio

Continuing he asked, “Could training be location-specific?
As learners move through the plant, alerts would be terrific!”
Helen asked who used traditional classroom training
Combined with online to keep interest from waning.
Did they have workshops, seminars, or events,
The kind that take workers away from their desk?
"They learn at that moment, then likely forget —
is there a way to get long-term retainment?”

Rachel had been quiet, she’d said not a word,
When suddenly she leaned in so her voice would be heard.
"We solved these concerns after ditching binders and books —
We use daily drip training and our learners are hooked!
When we update our content, it gets to them faster,
And metrics and KPIs reveal the content 'masters.'
We use OttoLearn for microlearning and we’ve been thrilled,
for all of our training needs — and more — are fulfilled."

So ends our tale of the nine trainers complaining
about the problems they had delivering training.
Training that mattered, with metrics and firm ROI,
Based on data analysis of prime KPIs.
Many problems they shared, with no clear resolution,
Found Agile Microlearning with Otto was the solution!
Microlearning both adaptive and agile saved them from disaster,
Making trainers and trainees learn happily ever after!

  • Arabic
  • Chinese
  • Dutch
  • English
  • Filipino
  • French
  • German
  • Hebrew
  • Hindi
  • Indonesian
  • Italian
  • Japanese
  • Korean
  • Persian (Farsi)
  • Portuguese
  • Romanian
  • Russian
  • Slovak
  • Spanish
  • Thai

Content Studio

  • Combining the question and activity tabs
  • New WYSIWYG editor which is “inline” with the text
  • Ability to include media (images, video, audio) within activities (question, answers and feedback)
  • Icons to indicate correct answer, position locking, whether or not the answer is visible to learners (active), and override feedback

Recently Released

  • Learner password reset
  • Streamlined data entry into the content studio, by being able to quickly add
  • Numerous small updates and bug fixes
  • Check out our most recent updates and add yourself to be automatically notified when we push updates

Option

Pros

Cons

Seats

  • Super easy to understand
  • Very predictable cost, if you have a specific number of users (eg: employees)
  • Doesn’t differentiate between users that have different volumes.
  • Have to purchase seats for your maximum number of users.‍

Active Users
(Typically the number of users that log in during a month)

  • You don’t need a license for every specific user, you can often only license half of your users (since perhaps only half ever log in during a month)
  • Typically there is a large cost for going over your licensed number of users, which can be incredibly expensive (eg: 5-10x more than your licensed cost)
  • You often have to “play games” as an administrator, not wanting to do a mass course enrollment if you have only have your users licensed in a month

Trend

What It Means

Why It Matters

Adaptive Learning

An algorithm determines each learner’s knowledge gaps and feeds them practice activities to close those gaps.

Efficiency. Learners learn the material faster because they spend less time on what they already know.

Personalized Learning

Learners can follow a scaffolded learner path or self-direct their learning.

Learners are inquisitive. We all Google for information when we need it, so why lock learners into a particular learning path?

Learners engage the most when they are allowed to deviate from a set path and explore available content.

At the end of the day, as long as each learner reaches their mastery goal, the particular path they took to reach there is unimportant.

Microlearning

Delivering content to the learner in smaller chunks.

Chunking content is important only if it is paired with the ability to search for and find specific content chunks “on demand” and the ability to consume just the chunks a learner needs. With these features, training doubles as a performance support.

Learning Experience (LX) Design

Using science and art to create experiences that help learners fulfill the learning outcomes they desire, in a user-centered and goal-directed way.1

Have you used Google? If so, then you have benefitted from Experience Design (XD): When you search for something, you rarely have to go past the first result.

With good XD, you don’t think about the design;  it “just works.”

With poor XD, your learners will disengage. They’ll say they “don’t have time.” What they are really saying is that they “don’t have time for the poor experience.”

Artificial Intelligence

Typically, when used in relation to L&D, AI actually means “machine learning.”

Machine learning algorithms learn from data and “get smarter” over time.

Have you used Netflix or Amazon recommendations? They are based on machine learning.

The algorithms look at a ton of data, including your past choices and choices made by others who are similar to you, to make predictions as to what you will want to watch or buy.

In L&D, machine learning principles are being integrated in much the same way: to provide recommended content for a learner to consume.

This reduces the burden on training administrators to try to predict or guess what is relevant for each learner. It also provides a more personalized experience for each learner.

Imagine that you are a salesperson, and your training mix subtly and automatically shifts, based on the nature of opportunities in your sales pipeline. You are offered training only on available products that you have not already mastered. That would be a training program that is driven by machine learning.

Learning Analytics

An algorithm determines each learner’s knowledge gaps and feeds them practice activities to close those gaps.

Use learning analytics to make better decisions by converting data into insights.
The true value is not just in providing more data, more charts, and more graphs. The value is in leveraging AI to search for and surface insights that you’d never think to look for.

Combine the analytics from learners’ performance with key KPIs for the outcomes you desire, and have the analytics engine generate predictions such as, “Learners who reach mastery in the Objection Handling module will close 3.4 percent more deals.”

Now that’s actionable intel.

1 learningexperiencedesign.com

14% of organizations are experimenting with artificial intelligence tools such as machine learning and live chat (up from 6% in 2016)
30% of organizations are using games and simulations (up from 20% in 2016)
Fastest growth segments include continuous learning, Artificial Intelligence (AI), Augmented Reality (AR)
  • English
  • Chinese
  • French
  • German
  • Italian
  • Portuguese
  • Romanian
  • Slovak
  • Spanish
The Cognitive Science Behind OttoLearn - OttoLearn Adaptive MicrolearningThe Cognitive Science Behind OttoLearn - OttoLearn Microlearning

Hi Josh,

About a week before I began getting my Ottolearn Mastery Moments, I had a popup window from Adobe appear on my screen as I was working on another project, prompting me to update my version of Flash. We do use Flash, so like an idiot, I clicked on the popup and asked it to start the update—and only then noticed that the url was not an adobe address. Of course, I closed the popup window using the X in the upper corner, which didn’t solve anything. Our IT guys did the best they could for me, but my computer is still compromised, and is being replaced.

Fast forward to Ottolearn and your Online Security for Employees course. After completing several mastery moments, I have now learned what to do with popups like that. This morning, as I restarted my computer again, that same Adobe popup appeared and this time I was ready! I opened task manager and killed that little $%^&^ dead in its tracks.

I know the point of letting us try out OttoLearn as participants was for us to experience the power of this platform from the learner’s point of view. I can tell you that I personally am very grateful for the training you provided to me, and the fact that I was able to let others in my company know how to kill off those nasty virus-carrying popups. Yes, it works. Yes, it’s fun! And yes, I have a true feeling of accomplishment.

I can’t wait for the point at which we can talk more about developing courses for our clients.

Thank you!

Experience

Exp.

High

New accounting rules

Workplace violence & harassment prevention

Low

Framing a basement

Changing a tire

High

Low

Perceived Relevance

No items found.

Retrieval practice is the key to retention.

Your brain wants to be as efficient as possible. Why would it try to encode information for long term storage if it thinks you don’t need it? You need to actually practice retrieving memories (information) in order to have your brain store it in long-term memory.

Spaced retrieval radically improves learning efficiency.

You not only need to practice retrieving information from memory, but you need to wait until you’re on the edge of forgetting it. This is why cramming is so ineffective at generating long-term retention.

Interleaved learning feels strange at first, but dramatically improves retention and skill.

Interleaved learning—mixing up material while learning and practicing, such as mixing up practice activities while learning WHMIS and supervisory skills, will improve your retention of both.

Option

Pros

Cons

User-based
(Seats)

  • Cost predictability. Each seat costs you $x/month
  • Typically more expensive than a usage-based license

Usage-based

  • Typically less expensive than a seats license
  • Cost variability tempered by pre-purchasing usage credits that never expire and consume them over time

Engagement Factors

Experience

Exp.

High

  • Best possible quadrant for engagement 
  • Will overcome learning obstacles
  • Will find a way to learn, even if materials are poor  
  • Won’t need nudging or incentives
  • Text is great
  • Can easily learn something
  • May need to work up the energy to engage in low quality materials  
  • May procrastinate, so incentives can help motivate.
  • Text is great

Low

  • Wants to learn
  • Has little experience so can benefit from more instructional quality
  • Greatest benefit of video and other rich media
  • Worst possible quadrant  
  • May not have experience in the topic
  • May not really care about it
  • Will require a lot of motivation to see engagement
  • Video can help

High

Low

Perceived Relevance

Neovation Learning Solutions Team - OttoLearn Microlearning is a product of Neovation Learning Solutions

Want the benefits of gamified eLearning at your organization?

Improve your learning culture with gamification features designed to motivate!

eLearning Learning

Want to stay updated?

Subscribe to the OttoLearn blog mailing list and never miss another post!

Get the latest Otto news!

Stay current—read all about the latest OttoLearn release updates in our release notes.

View the Release Notes

Extra! Extra! Read all about it:

Want even more? Check out the full blog archive.