How We Calculate Scores

NOTE: SCORING METHOD CHANGED 7/1/2018. THE FOLLOWING ENTRY IS NO LONGER VALID. PLEASE REFER TO THE NEW METHOD BLOG ENTRY. 

This post is divided into two sections,

• Individual Competency Score Calculations
• Overall Score Calculation

All scores are presented on a 0-100 scale.

Keep in mind that Overall Score is calculated to be essentially a Normal Distribution with a mean of 50 and a standard deviation of 21.  It is not a percentile.

To get the overall score, we must first obtain scores for each individual competency. We then combine these scores in order to arrive at the overall score.

Step 1: Individual Competency Score Calculation

1. For cognitive modules, we calculate the score as the percent of available points the test-taker achieved. 
2. For AIMS, we sum the points from the indiv items (likert scale), apply a lookup table to convert to 0-100.
3. For Biodata, we use the average points per item and then apply a lookup table to convert to 0-100.
4. For skills/knowledge tests, we calculate the percent of available points.

Overall Score Calculation

The overall raw score is the weighted average of the individual competency scores. The overall final score is a standardized, then transformed version of the raw score.

The basic steps are:

1. Determine the score value to use for each competency. In many cases it's the competency score, but not all.
2. Set the weights for each competency using a standard formula.
3. Adjust weights if needed.
4. Apply ONET to the weights.
5. Calculate raw score as weighted average of individual competency scores
6. Standardize the raw score to a z score (mean=0, standard deviation=1)
   
7. Transform the z score to a scale with a mean of 50 and a standard deviation of 21.06

Step 1: Determine the score value to use for each competency.

Some competencies, such as behavioral history and some attitudes, interests, and motivations are set to use the assigned color category rather than the numeric value for that competency. This is because the more desirable regions of the score scale may be in different parts of the scale. For instance, Develops Relationships is best in the middle of the scale rather than at the high end. Or, Past Behaviors may be skewed highly to the high end of the scale.

Step 2. Set standard weights for each competency such that:
  - total  weights for all Cognitive Ability competencies = 1
  - total  weights for all Skills/Knowledge competencies = 0.8
  - total weights for all AIMS competencies  = 0.7
  - Total weights for all Biodata competencies  = 0.4

For individual competencies within a category, the weights are spread evenly (i.e. 1/n each).

Step 2. For some jobs, we add some weight to specific competencies. For instance, we increase the weight on the Expressive AIMS factor for sales people. This weight will override the weight determined in step 1.

Step 3. Next, apply ONET importance ratings to each competency. Since not all competencies have an onet importance value, those that don't are assigned an onet importance of 4.2 (80% on a 1-5 ONET importance scale). This is to prevent these from getting extra weight just because they are not in ONET. Most ONET weights are below this value and the average is about 2.5. The ONET value is converted to a fraction and applied to the weight from steps 1 and 2.  Weight fm step 1 * ONET as fraction = final weight.

Step 4. The overall raw score is the weighted average of the individual competency scores.

Step 5. An overall z score is calculated using a mean/std derived from existing data (if there are over 100 test events in dbms)  for that test or "all tests" if there is not enough data.  The derived mean measured on about 100 events after changing the above is 54.6, std=9.15  These are the values I'm using.

Step 6. The overall z-score is transformed to a score using a mean of 50 and a std of 21.06.  These values are the values used for a Normal Curve Equivalent Scale (NCE) used in various achievement tests.  If the score is outside the range of 0-100, it is set to 0 or 100 as appropriate.

Notes:

1. Key values are the mean and standard deviation of the overall raw score (the weighted average), which is used to create the z score. These values are based on test results for this test. If there are not enough completed results for a given test, overall mean and standard deviation values are used.

2. Sometimes the overall score will look higher than it should be, or lower than it should be. This is because the overall score is not just a raw weighted average, which most people assume. It's a reflection of the number of standard deviations from the mean that the person scored. So, if their overall weighted average is 40, but the average overall weighted average is 80 and the standard deviation is 20, then they are two standard deviations below the mean. So when converted to NCE (mean 50 and std 21.06), they will have an overall score of 50 - 2*21.06 = 7.88. So, the overall score will say 8.
 
Note: For information on the Normal Curve Equivalent Scale (NCE) go to:
http://en.wikipedia.org/wiki/Normal_curve_equivalent