HR Avatar Overall and Competency Score Calculation Method

Introduction

HR Avatar updated its scoring methodology in June 2018.  The following blog entry describes the new method.

What changed?

HR Avatar has implemented test-specific norm-referenced scoring for most competencies. This data-driven approach means competency scores are consistent between tests and between scales and may be more easily compared. Color-shaded areas are now consistent between tests and have been shifted to match the normalized distribution parameters.

HR Avatar also revised the method of combining competency scores to be consistent with the new approach. The overall score is now consistent between tests and is more directly related to individual competency scores. Similarly to competency scores, color-shaded areas have been shifted to match the normalized distribution parameters. 

The New Method

Each HR Avatar test is scored by first calculating a score for each individual competency and then combining the competency scores into an overall score.

For each competency a raw score and a scaled score is calculated. For the overall score, a raw overall score and a scaled overall score are calculated. Only scaled scores are presented to the employer or test taker. 

Step 1: Competency Raw Scores

For each competency, the raw score is a normalized "Z" score. A Z score has a distribution where the mean is 0 and the standard deviation is 1. This is calculated by first computing or estimating the mean and standard deviation of the raw total score value, which is typically the sum of points assigned to the individual responses made for each item that contributes to the competency. These statistics can then be used to calculate the Z score. Please contact us for more information on how we calculate each competency mean and standard deviation.

Step 2: Competency Scaled Scores

The competency scaled score is calculated by transforming the Z score to a distribution with a mean of 65 and a standard deviation of 15. The mean of 65 was selected to more closely mirror typical score distributions on academic tests. The standard deviation of 15 was selected to allow for at least 2 standard deviations of variation both above and below the mean when scores are restricted to a 0-100 scale. 

Step 3: Overall Raw Scores

The overall raw score is the weighted average of the competency Z scores, with some adjustments due to nonlinear competencies. Weights are computed using references to the US Federal Government's O*Net database which catalogs the importance of various competencies to about 1000 different jobs. Since some competencies are set up so that the "good" scores are in the middle rather than the high end of the scale, these scores are adjusted to be linear before used in the combination by using -1* the absolute value of the competency raw score rather than the competency raw score directly. 

Step 3a: Overall Z Score

Convert the weighted average for competency Z scores calculated in Step 3 above to a Z score. The mean and standard deviation used are calculated using previous test results.  Please contact us for more information on how we calculate these values if necessary.

Step 5: Overall Scaled Score

The overall scaled score is calculated by transforming the Overall Z score to a distribution with a mean of 65 and a standard deviation of 15. The mean of 65 was selected to more closely mirror typical score distributions on academic tests. The standard deviation of 15 was selected to allow for at least 2 standard deviations of variation both above and below the mean when scores are restricted to a 0-100 scale. 

The Resulting Scales and Color Ranges

The results of the process above are that competency scores and the overall score are consistent with each other and between tests. In most cases, the data used to calculate the individual competency scores is test-specific. 

For the overall score and linear competency scores where the best score is 100:

Red: More than 2 SD's below the mean

Red-Yellow 1-2 SD's below the mean

Yellow 0-1 SD below the mean

Yellow-Green: 0-1 SD above the mean

Green: More than 1 SD above the mean

For non-linear competency scores where the best scores are at the mean:

Red: More than 2 SD's below or above the mean

Yellow: 1-2 SD's below or above the mean

Green: 0-1 SD below or above the mean

FAQ:

Did the tests change?

No. There are no changes to the tests themselves.

How much will the scores change?

Our analysis shows that scores can vary by about 10 percent compared to their value prior to the change.  Individually, the overall score should be more clearly related to the individual competency scores - taking into account the weights applied to each competency score.

When will the change take place?

For most of our tests, the scoring will change over the next two weeks.  

Can I rescore old tests using the new method?

In selected cases we can rescore tests previously administered to help you evaluate the degree of change.  However, this is fairly time consuming. Please contact us directly if you would like an analysis.

Why was this done now?

After several years of operations, HR Avatar now has sufficient live candidate data to enable test-specific, norm-referenced scoring of competencies. The biggest advantage of this approach is that scores are now more directly comparable to one another and are more directly referenced to job applicants in similar or related roles, rather than indifferent non-role-specific volunteers.

Converting NCE Score to Weighted Average or to an older NCE score

Recall that scores are calculated as follows:

Raw Score=weighted average -> Converted to a Z score (using mean M and std STD) -> Transformed to an NCE Score (mean 50, std 21.06)

So, to convert a raw score to an NCE score you need a mean (M) and standard deviation (STD).

Sometimes we change the mean and std for a specific test based on test results for that test.

Before we have enough test results, we use the standard values which are:

M=54.29
STD=10.59

These scores were based on the first ~1,000 job specific tests completed on our system (computed across all job specific tests).

However, most tests now use a different M and STD based on results of that specific test.

Sometimes a client wants to go back.

To go back, you must convert from NCE back to Z and then back to raw using the old M and STD and then go forward using the new M and STD.

The values:

S1 = New Overall NCE Score we have now.  (uses M1 and STD1)
S2 = New Overall NCE Score we want to calculate. (uses M2 and STD2)
Z1 = old Z score
Z2 = new Z score

The Calculation - to get S2 from S1

So, give a score S1 created with mean M1 and std STD1, if we want to convert to a score with M2 and STD2 we do this:

Z1 =(S1 - 50)/21.06

Raw = Z1*STD1 + M1

Z2=(Raw - M2)/STD2

S2 = Z2*21.06 + 50

HR Avatar "Profile" Functionality

HR Avatar has a line of assessments sold off-the-shelf. However, many customers want to customize various aspects of the scoring and report.

To start, we can name any test a special name for a specific customer. [HR Avatar Note: Do this by creating a Product Record for the Sim using the new customized name, then create in a specific product group for that client only].

Next, a client may want to do one of the following:

A. Show a "Blue" Baseline Line over the overall score graph and the individual competency graphs (in report, in emails, in online reporting).
B. Use customized weights to calculate the overall score.
C. Customize the red-yellow-green sections of the overall score graph or individual competency graphs (in report, in emails, in online reporting).
D. Evaluate the competencies measured in a test against a second job, or a third or a fourth, etc.
E. Evaluate the overall scores in a Test Battery against a different set of weights (create an alternate battery score).
F. Evaluate the competency scores for all tests in a Test Battery against a set of weights to create an alternate Battery Score based on a weighted combination of competency scores across all tests in the battery rather than just the overall scores. 
G. Change the colors used in the red-orange-yellow-light green-green scheme.

We can do any of these things now using a "Profile" in our system.

A Profile is like an overlay that goes on a Sim or on a Test Battery. If can be configured to do any of the above things.

[HR Avatar Note:  We do this by creating and configuring a Profile, and attaching it the Org that needs it and mapping it to the Product record for the Sim or Test Battery.]

Profiles need to be configured by HR Avatar staff. The cost, once a spec is ready should be minimal.

 

Internet Domains Used by HR Avatar

These public domains are needed in order to fully access HR Avatar services:


All Functions
https://vi.hravatar.com
https://cdn.hravatar.com
http://www.hravatar.com
https://www.hravatar.com
https://test.hravatar.com
http://test.hravatar.com
https://s3.amazonaws.com
http://cfmedia.clicflic.com

Testing Only:
https://cdn.hravatar.com
https://test.hravatar.com
http://test.hravatar.com
https://s3.amazonaws.com


Video Interviewing Only:
https://www.hravatar.com
https://cdn.hravatar.com
https://s3.amazonaws.com
Admin Website Only:
http://www.hravatar.com
https://www.hravatar.com
https://s3.amazonaws.com
http://cfmedia.clicflic.com


Custom Test Authoring:
https://sim.hravatar.com
https://imo.hravatar.com
https://s3.amazonaws.com
http://cfmedia.clicflic.com


Certain customers may need to open these domains on their firewall in order to permit access.

Expired Test Keys and Credits

Credits

Credits are charged to a test taker when they START a test. This means they did something to start the test - typically clicking on a test key link provided by a company and they advance through the name/email or demographic screens to start the test.

Test Key Expiration Dates

All test keys have expiration dates. This is to keep the database clean and avoid having too many test keys that are in limbo, which could degrade the testing experience for legitimate test takers.

Whenever you create a test key - you assign an expiration date. By default this is 3 months after the test key is created. For single-use test keys, it can be changed in the advanced settings when creating the test key or when reviewing the detail for incomplete test keys. For multi-use test keys, the expiration is always 3 months after the  test taker starts the test.

When do Test Keys Expire

1. If they have not started the test before the expiration date, the test key will expire on the expiration date and they will not be able to restart the test.

2. If they have started the test before the expiration date, the test key will expire 7 days after the expiration date.

Credits and Expiration

If a test key is expired before it is started by the candidate (expire date), no credits are charged.

If a test key is expired after it was started by a candidate (expire date + 60 days), credits are charged and are not recoverable.

Emergency Contact Information

In the event of a system outage, please call us in the following order from the top down:

Mike Russiello  Mobile: +1 571 213-5677
                           Home:   +1 703 938 1615
                           Skype:    mikerussiello

Shoa Appelman   Mobile: 703 966 2080
                            Skype: shoaonskype

Bill Lake:            Office: 703-687 6212
                            Mobile: 202 701 8166
                            Home:  703 266-1427
                            Skype:  williamhlake

Call anytime, day or night if there is an issue.
 




             

Information Regarding Translation and Localization

Translation Overview

At the highest level, the basic steps for localizing HR Avatar tests are:

1. Translate text from inside the test
2. Regenerate images that include text from the test
3. Translate native text for IMO operation
4. Translate text keys for test administration system
5. Translate text keys for test scoring, report generation and distribution system
6. Identify text from tests that is used in animation script and record audio voiceovers.
7. Re-animate the animations used in the test module using the new voiceovers. 
8. Post all text changes. 
9. Check for quality assurance.

The HR Avatar authoring system has an online capability to support all text translation tasks. An HR Avatar administrator creates each translation task and assigns it to an individual translator. The translator then logs on and clicks the translation tab to view all assigned translation tasks. Then the translator clicks on the task they want to work on and begins filling in translated text in the text boxes as appropriate, saving work occasionally.

Because of the modular and template-driven nature of the HR Avatar simulation authoring system, the translation of the first test requires the translation of a set of templates and short test modules. All of these templates and modules are then re-used in all additional test translations so later translations are much easier. The same is true for the HR Avatar test administration, IMO, and report generation/distribution systems. These are needed to support the first test but are all re-used for future tests.

A Collection of Modules

Each HR Avatar Job-Specific test is really a collection of one or more of the following test modules:

15 cognitive simulation modules
2  Personality modules
2 biodata inventory modules
1 essay module
1 typing speed/accuracy module
35+ skills and knowledge modules

Typically, a job specific test uses one cognitive simulation module, one personality module, the essay module, one biodata module, and 0-2 skills/knowledge modules.

Additionally, the test uses several modules that define the beginning, end, and transitions between other modules, or specify templates for interactions and the player look/feel. These modules are used across all tests.

A list of which tests use which module is available on request. 

Note that only the cognitive simulation modules and several of the template modules use animations. All of the other modules are text-based.

In order to complete a test in another language, all modules that are used by that test must be translated.

Planning your first translation

Step 1:  Choose the test or tests that you want to translate. This should be a short list. HR Avatar will then set up the necessary translation tasks to support these tests.
Step 2: Complete all translation jobs.
Step 3: Obtain any images that must be converted to the target language.
Step 4:  Obtain a list of voiceovers that must be recorded for different characters from HR Avatar.
Step 5: Record voiceovers and send to HR Avatar.

Estimating workload:

Templates:
Autogeneration Sim:6000 words
Simlet Template (interaction patterns):  2200 words
Sim Template: 1000 words
Splash In:     400 words, 1 voiceover
Splash Out:  100 words, 1 voiceover
Transition: 25 words
Pre-Personality (intro): 140 words, 1 voiceover
Pre-Biodata (intro): 50 words, 1 voiceover
Pre-Essay (intro): 200 words, 1 voiceover
Pre-Knowledge (intro): 50 words, 1 voiceover

Total templates:  10,200 words, 6 voiceovers

Software Keys:
IMO Operations Keys:  1500 words
Test Administration Keys:  4500 words
Report Generation Keys:   3500 words

Total software keys: 9500 words

Test Modules:
Typical Cognitive Module:  4500 words, 25 voiceovers (30 secs each)
Personality Module:  4500 words
Biodata module:  1000 words
Knowledge/Skills Module: 800 words

Total for first full test modules: 10,800 words, 25 voiceovers

Total for first full test: 30,500 words, 31 voiceovers

After first test complete, maximum for another test is 10,800 words and 25 voiceovers. However, could be no effort also, as long as all the modules are already translated.

Remember, there are only 15 cognitive modules - each test uses one of these. We are creating a few more. Also, there are only two personality and two biodata modules and one essay module.

About voiceovers:

Typically we hire non-union voice actors to come to our office for voice sessions. We have a setup with a good microphone and baffle boards that works with our computer. The 30 voice scripts are probably from about 5 different voice actors. Each person will probably take an hour or so to record.

We can provide additional coaching on creating a voiceover setup. There are also online resources.