Check the Scores for Consistency

Most companies who monitor calls and do Quality Assessment (QA) have some form of calibration. This usually follows some pattern of listening to a call with a group of people and analyzing it together to find out where you agree and disagree on an evaluation. It can be a painful process.

You can often learn just as much, if not more, by simply comparing a large sample of evaluations scored by different analysts.

For example, I was recently pouring over a comparison of quarterly data for one of our clients. I have a member of my team run the raw data on a regular basis and provide me with a comparison report. In this particular project, we have four analysts scoring a couplehundred calls per month. There are roughly 65 different behavioral elements we analyze that are rolled up into 14 corresponding attributes. There is a score for each attributeand a corresponding Overall Service score on a scale from 0 to 100.

First, I compared the average Overall Service score for each of the four analysts. The four analysts were within a half-point of the overallaverage for the group. This told me that I didn't have anyone who was particularly lenient or harsh in their analysis. Our overall service numbers were very similar. If an analyst had an overall service score that was much higher or much lower, it would have motivated me to dig into the underlying data to find out why. So far, so good.

Next, I compared the average scores for each of the 14 Attributes. Because some Attributes rarely apply, there are much higher deviations. Keeping this in mind, I focused on the attributes that apply most often and have the greatest impact on the Overall Service score. Once again, our scores were very similar.

Finally, I looked at the average number of times each analyst marked a behavior "yes," "no," or "not applicable" for each of the behavioral elements. In this instance, I found one of my new analysts who had marked a particular behavior as applicable on 100% of the calls analyzed while the other analysts had it applicable on less than half. Because it's an element within an attribute that the client normally scores very well – it didn't show up in the corresponding score. Nevertheless, it could eventually make a difference, and we were clearly not calibrated in scoring this particular behavior. By looking at the data, I was able to address the issue with the analyst and coach them on how to more accurately measure that particular behavior. From this point forward, we should be more closely calibrated.

Sometimes, you've got to let the data show you the way.

  3 comments for “Check the Scores for Consistency

  1. May 3, 2009 at 3:55 pm

    Consistency is the real key to all customer service interactions.
    To be the master of consistency would be to be the master of customer service.
    Very easy to say. So hard to do.
    I wish there was an easier way. I wish that small business could get it right and that big business would get it wrong. But often this gets reversed.
    So come on small business – it’s time to realise that you really can make the difference and that you really have the ability to get the edge over Corporate America.

  2. Skytte
    May 4, 2009 at 10:58 am

    how you are able to effectively monitor hundreds of calls with 65 behavioral elements?

  3. May 4, 2009 at 3:59 pm

    Great comments, Jonty! As a small business owner, I applaud your sentiments!
    Skytte, it can actually be done effectively and efficiently if your scorecard elements are well defined, your analysts are well trained and your methodology is sound. Of course, average call time is a factor. This particular client averages just over three minutes per call. Our analysts can effectively analyze 5-6 calls an hour, on average, for this client!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: