Introduction
HR Avatar updated its scoring methodology in June 2018. The following blog entry describes the new method.
What changed?
HR Avatar has implemented test-specific norm-referenced scoring for most competencies. This data-driven approach means competency scores are consistent between tests and between scales and may be more easily compared. Color-shaded areas are now consistent between tests and have been shifted to match the normalized distribution parameters.
HR Avatar also revised the method of combining competency scores to be consistent with the new approach. The overall score is now consistent between tests and is more directly related to individual competency scores. Similarly to competency scores, color-shaded areas have been shifted to match the normalized distribution parameters.
The New Method
Each HR Avatar test is scored by first calculating a score for each individual competency and then combining the competency scores into an overall score.
For each competency a raw score and a scaled score is calculated. For the overall score, a raw overall score and a scaled overall score are calculated. Only scaled scores are presented to the employer or test taker.
Step 1: Competency Raw Scores
For each competency, the raw score is a normalized "Z" score. A Z score has a distribution where the mean is 0 and the standard deviation is 1. This is calculated by first computing or estimating the mean and standard deviation of the raw total score value, which is typically the sum of points assigned to the individual responses made for each item that contributes to the competency. These statistics can then be used to calculate the Z score. Please contact us for more information on how we calculate each competency mean and standard deviation.
Step 2: Competency Scaled Scores
The competency scaled score is calculated by transforming the Z score to a distribution with a mean of 65 and a standard deviation of 15. The mean of 65 was selected to more closely mirror typical score distributions on academic tests. The standard deviation of 15 was selected to allow for at least 2 standard deviations of variation both above and below the mean when scores are restricted to a 0-100 scale.
Step 3: Overall Raw Scores
The overall raw score is the weighted average of the competency Z scores, with some adjustments due to nonlinear competencies. Weights are computed using references to the US Federal Government's O*Net database which catalogs the importance of various competencies to about 1000 different jobs. Since some competencies are set up so that the "good" scores are in the middle rather than the high end of the scale, these scores are adjusted to be linear before used in the combination by using -1* the absolute value of the competency raw score rather than the competency raw score directly.
Step 3a: Overall Z Score
Convert the weighted average for competency Z scores calculated in Step 3 above to a Z score. The mean and standard deviation used are calculated using previous test results. Please contact us for more information on how we calculate these values if necessary.
Step 5: Overall Scaled Score
The overall scaled score is calculated by transforming the Overall Z score to a distribution with a mean of 65 and a standard deviation of 15. The mean of 65 was selected to more closely mirror typical score distributions on academic tests. The standard deviation of 15 was selected to allow for at least 2 standard deviations of variation both above and below the mean when scores are restricted to a 0-100 scale.
The Resulting Scales and Color Ranges
The results of the process above are that competency scores and the overall score are consistent with each other and between tests. In most cases, the data used to calculate the individual competency scores is test-specific.
For the overall score and linear competency scores where the best score is 100:
Red: More than 2 SD's below the mean
Red-Yellow 1-2 SD's below the mean
Yellow 0-1 SD below the mean
Yellow-Green: 0-1 SD above the mean
Green: More than 1 SD above the mean
For non-linear competency scores where the best scores are at the mean:
Red: More than 2 SD's below or above the mean
Yellow: 1-2 SD's below or above the mean
Green: 0-1 SD below or above the mean
FAQ:
Did the tests change?
No. There are no changes to the tests themselves.
How much will the scores change?
Our analysis shows that scores can vary by about 10 percent compared to their value prior to the change. Individually, the overall score should be more clearly related to the individual competency scores - taking into account the weights applied to each competency score.
When will the change take place?
For most of our tests, the scoring will change over the next two weeks.
Can I rescore old tests using the new method?
In selected cases we can rescore tests previously administered to help you evaluate the degree of change. However, this is fairly time consuming. Please contact us directly if you would like an analysis.
Why was this done now?
After several years of operations, HR Avatar now has sufficient live candidate data to enable test-specific, norm-referenced scoring of competencies. The biggest advantage of this approach is that scores are now more directly comparable to one another and are more directly referenced to job applicants in similar or related roles, rather than indifferent non-role-specific volunteers.
