Review: Knowledge Factor Confidence-Based Learning

Learning professionals often talk about the need to teach adult learners only what they need to know. Well, Knowledge Factor’s Confidence-Based Learning suite and its tool, Confidence-Based Assessment *, do exactly that.

Knowledge Factor is a little out of the ordinary when it comes to online learning authoring tools, and that’s what makes it interesting. Knowledge Factor’s web-delivered product is built on “confidence-based assessments,” or CBA, a two-dimensional testing method that requires learners to choose not only what they think is the correct answer but also their confidence in the answer: Sure, Partially Sure, or Don’t Know.

The two-dimensional testing concept is based on research by Dr. James Bruno, a professor of education at the University of California at Los Angeles. For Knowledge Factor, the assessment is where the learning begins.

Using the assessment

Learners first take the assessment. Items are pulled from a pool of questions (roughly 35 to 50 per module) so that the instruction is teaching to the knowledge, not the test. Each question is linked to a learning objective so that knowledge is mapped to specific competencies. Based on their results, learners are given a learning track that targets their knowledge gaps. Learners review the questions they miss—their knowledge gaps—and then retake the assessment.

The knowledge gaps fall into four categories, or quadrants: misinformed, uninformed, doubt, and mastery. Each quadrant has implications for on-the-job performance: Misinformed means that the learner answered the question incorrectly but marked “Sure,” meaning he confidently holds incorrect information. This state is also known as cognitive dissonance because it creates conflict with what learners hold to be true. Misinformed employees are prone to mistakes.

Uninformed means that the learner answered the question “I Don’t Know” and therefore understands that he doesn’t have the knowledge to answer the question. The likely work outcome is paralysis.

Doubt is associated with the response, “I Am Partially Sure.” With this answer, the learner selects two of three possible answers and believes that one of them is correct. Doubt usually results in hesitation or inaction on the job, even if the knowledge is correct.

Mastery means that the learner answered the question correctly and marked “I Am Sure.” This state leads to smart and appropriate action on the job.

The goal is for learners to attain Mastery, which means they answer all or a high percentage of the questions confidently and correctly. Knowledge Factor bills this as the only productive, actionable state of knowledge. Take, for instance, a salesperson who must sell a particular set of products to a potential client. If she isn’t completely confident of her product knowledge, she likely won’t sell the products she doesn’t know or will make mistakes trying to sell them. Both situations cost the company money. The consequences are more dire when dealing with situations such as patient safety.

Confidence-Based Learning significantly reduces guessing. Guessing, or claiming “I’m Sure” and being incorrect, has negative scoring implications. If a learner answers confidently and incorrectly, the algorithms in Knowledge Factor’s system dock her more points than if she answered “I Don’t Know.”

Extensive reporting

Knowledge Factor’s excellent reporting functionality offers three categories of reports: User Progress, Performance, and User Activity. Within each of these categories, the following reports are available:

User Progress Reports

Individual Progress, All Iterations: The detail of every iteration for each assessment a learner has taken.
Individual Progress: The quadrant scores, percentage of total questions answered, and CBA Score for each learner in each module, based on the last time a learner saw each question, regardless of iteration.
Individual Progress, First and Last Time Iterated: Learners’ individual progress determined by comparing the quadrant scores, CBA Score, and confidence index for the first and last iteration by learner by module within a domain.

Performance Reports

Mastery, User by Module: The number of iterations taken, number of questions in the module, number of questions seen by the learner, number of questions mastered, and mastery percentage, by module by learner.
Misinformation Red Alert: The percentage of incorrect answers by learner based on the last time the learner answered the question, regardless of iteration.
Course Performance: The percentage of questions in each quadrant at the course level. This report also points out key areas of misunderstanding. Results are based on the last time each learner answered each question.
Knowledge Quality by Competency: Quadrant scores and CBA scores by question at the module level for the last time learners answered the question.

User Activity Reports

Learner Iterations by Module: A count of the number of times a learner took each assessment/module within a domain.
User Iterations: The total number of assessments completed by a learner. This report shows iteration counts at the domain level, not divided by module.
Last Date Assessment Taken: The last time a learner took an assessment within a module.

The reports are also available at the organizational level. For example, you can generate reports that reveal, by learning objective, where pockets of doubt, misinformation, or lack of information exist in an entire department or enterprise. These “dashboard reports” go beyond just reporting how many questions employees answered correctly on an assessment, which is where most training programs stop.

Knowledge Factor, by definition, gives you not only the baseline data of learners’ knowledge on their first iteration, but it also gives you the Kirkpatrick Level 2 so you can measure the learning that occurred in each module. Lastly, Knowledge Factor can also incorporate Level 1 (reaction) surveys into each course.

Major benefits

Because the Knowledge Factor tool is delivered on the web, you don’t have to learn a specialized interface. Basically, an instructional designer creates questions in Knowledge Factor’s Microsoft Word template, and they are uploaded into the system.

The Knowledge Factor CBA tool has two major benefits. First, it allows learners to test out of knowledge they already know so they can focus on what they need to know. This targeted learning allows instructional designers to prove through online testing environment that their learners have the requisite knowledge. This is important not only because instructional designers can demonstrate knowledge but also because it might lead to another reason why learners are not performing at optimal level.

Second, the tool significantly reduces guessing because learners must answer not only what they think is the correct answer but also if they are confident in it. The beauty of that is a learner confident about what turns out to be a wrong answer is docked more points than if he marked “I Don’t Know.” Assessment items are randomized so each learner never sees exactly the same assessment.

As a company, Knowledge Factor is very customer oriented. They will work with you from beginning to end to work out anything you need, even if it’s not in their standard offering. And, if you don’t have instructional designers on staff, they offer those services too.

Drawbacks and caveats

In my experience, there aren’t any major drawbacks to the Knowledge Factor system. From an instructional designer’s viewpoint, it is a shift in the way course are developed. Having developed many assessments, I thought it wouldn’t take as long as it did to create the questions for the system. It takes time because each question must have a set of links to back up the correct and incorrect answers, but less time than it takes to develop a traditional online course. This is just a matter of having the links, but as any instructional designer worth her salt knows, gathering information and links can be a huge task. (It’s kind of like trying to get subject matter experts to review course content!)

One other word of warning. As with any tool, what you get out of it is equal to what you put into it [x = f(y)]. I suggest creating scenario-based questions that mirror your learners’ job context so the leap from course to application is smaller.

Also, even though learners can test out of a course, you are wise to test learners’ ability to apply the information in their job context. This is not a limitation of Knowledge Factor’s system but rather a limitation of most online learning. I recommend helping learners apply their knowledge on the job with a part of your curriculum.

Technical requirements

Knowledge Factor operates as a learning system, similar to an LCMS, and its reporting function is similar to an LMS’s. Because questions are linked to learning objectives, each set of questions is an individual piece of content and can be turned on or off, depending on need and relevancy. This capability is especially useful if you have content that needs to be delivered to multiple audiences but may need customization by audience.

Knowledge Factor also integrates with LMSs so that information such as completion data and log-in information can be communicated between systems. My understanding is that this communication piece has to be created for each client, but the creation time is relatively short—a couple of weeks or so.

Pricing

Knowledge Factor uses an approach to pricing similar to a wireless phone plan: system usage buckets by users. Instead of minutes per month, the company charges a rate for the number of users per month. The rate varies by volume and the size and the complexity of the program; the floor rate is $50 per user. Registered users have unlimited access to the program. Knowledge Factor charges a development and service fee for creating the initial program with the client and teaching the client how to create future programs more independently.

Recommendation

Knowledge Factor is a definite “buy.” Not only is their tool cutting edge—their customer service is superb. Knowledge Factor strives to be a company that builds tools that their customers need based on what their customers tell them they need rather than on their idea of what customers need. The tool goes above and beyond the current trend of “rapid e-learning development” because it accurately measures the knowledge learners gained while in the module and course. As with any tool, though, the course is only as instructionally sound as the methodology used to get there. Simply dumping content into great tool doesn’t make the course instructionally sound.

Product Ratings

Knowledge Factor Confidence-based Learning