Skill Assessment Technology

Explore the technology that scientifically calculates skill scores

Try Demo Contact Sales
Test your team across multiple cloud platforms and technologies

What Is a skill and a skill score?

A skill is a topic that you want to master. The skills that Cloud Academy measures include a wide variety of software knowledge, from general domains and concepts (like Cloud Fundamentals or Disaster Recovery) to specific platforms and software packages (such as Microsoft Azure or Git).

Cloud Academy calculates a skill score for you based on your performance on most quizzes, exams, and hands-on labs related to the skill. Playground labs and quizzes in study mode do not affect your skill score, and usually attending a course does not affect your score, unless the course includes a quiz at the end to test your understanding.

The skill score is a number between 0 and 1,000, where 0 indicates a user with no knowledge of the skill, and 1,000 indicates a user who is fully proficient in that skill.

Learn how to build practical skills with labs →
What is a skill score?

How does task difficulty affect my skill score?

A skill score is not like the scores you got on tests in school, where your teacher simply counted your correct answers. Instead, the skill score reflects your ability to correctly answer questions and complete tasks of a certain difficulty level more often than not. We call the difficulty level of the question or task the task difficulty. Similarly to the skill score, the task difficulty is a number between 0 and 1,000, where 0 indicates an easy task almost everyone is able to successfully complete, while 1,000 indicates a task only fully proficient users are expected to correctly accomplish.

For example, imagine you take an exam about the skill Disaster Recovery. If you usually answer questions with a task difficulty less than 500 right, but you usually don’t answer those with a task difficulty of over 600 right, your skill score on Disaster Recovery is probably above 500 but not quite to 600 yet. We derived this scoring approach, in part, from Item Response Theory (IRT), one of the reference theories Cloud Academy uses.

 

Learn about Natural Language Processing (NLP) →
Assessments Track Skill Growth

Why didn’t my score go up more when I got a question right?

Even if you answer a question correctly or complete a hands-on lab successfully, your score might not increase. That is because the skill assessment algorithm updates your skill score based on your performance AND the task difficulty.

If your skill score is already high, for example 800, and you successfully complete a task with a difficulty of 400, then your score might not go up at all. On the other hand, if your skill score starts at 250 and you successfully complete the same task, you will see a big increase in your score.

Learn more about assessments →
Cloud Academy Skill Profile

How is the task difficulty measured?

The content author initially sets the difficulty of a question or task when creating it, but over time the scoring algorithm adjusts the task difficulty value based on how learners perform on it. For example, the content author might have thought that a question was an intermediate-level difficulty, but if even beginner-level learners consistently get the question right over time, the algorithm adjusts the task difficulty down.

Difficulty is probably the most intuitive property of a question or task, but there are other properties that the scoring algorithm tracks and uses. Another important property is discrimination — how well the task differentiates between beginner and proficient students. For example, you expect beginners to fail a very difficult task, but if both beginners and highly proficient students tend to fail the task, then the task does not have good discrimination and isn’t as effective at measuring the learner’s skill. The algorithm flags these tasks for the author to revisit.

Learn more about assessments with multiple-choice questions →
View real-time insights about your organization’s capabilities

How are skills organized?

The skills that Cloud Academy teaches relate to each other in a hierarchical list, or taxonomy. Currently, there are several hundred skills in the list, and the list is always growing as technology — and our training — evolves.

Behind the scenes, the algorithm is based on a Knowledge Graph whose nodes represent the skills and whose edges model the skills’ semantic relationships to each other. As you can imagine, a graph with hundreds of nodes is too complex for a human to read easily.

To simplify the experience for users, the skills appear as a multi-level tree in Cloud Academy. When you navigate to your Skill Profile, you see the top-level of skills called Overall. You can click a skill at this level, for example Storage for AWS, to drill down into the next level of the hierarchy. You can continue drilling down through the sub-skills until you reach the bottom, for example, Amazon S3.

Assessments Track Skill Growth

Can a task or question affect multiple skill scores?

The content authors choose one or more relevant skills when creating the tasks and questions in the Cloud Academy library. Because of the hierarchical relationships between skills, choosing a skill that is lower in the hierarchy implies that the task or question also affects the skills that are above it in the hierarchy. For example, if a question affects the skill Cloud Bigtable, the question must also affect its parent Storage for Google as well as its grandparent, Google Cloud Platform.

However, scoring well on a task about Cloud Bigtable does not have an equal effect on the parent and grandparent skills. After all, the grandparent skill, Google Cloud Platform, covers a lot more subskills than just Cloud Bigtable. In general, the score for Google Cloud Platform is the average of its subskills, with the algorithm estimating any missing subskill scores through existing skill interconnections.

Learn how we calibrate new test questions →
Asses - Accurately pinpoint baseline skills

Create custom assessments with
Content Engine™

Learn more