5 Tips for Integrating Rating Scales Into Your Surveys

For my masters work, I'm currently focusing on creating a framework for classifying computers users as experts or novices. I was very surprised that such a framework or template didn't all ready exist. In most cases, researchers will choose a single attribute and use that to classify users as "experts" or "novices". For example, the most common selection criteria I've seen is to use Web developers as "internet experts", and computer science students and professors as "computer experts". As anyone in the field can tell you, these simple criteria rarely match with actual expertise, due to domain specific knowledge and experience.

As part of this framework, I've been researching both common criteria used for expert and novice classification, as well as more generic practices to remove bias from the survey. One excellent paper I've found by Friedman and Amoo provides guidelines for removing bias from rating scales. Some highlights from the paper include:

1. If questions regard "concern" towards an idea, it should be prefixed with a more general question asking if there are concerns at all. Otherwise, respondents are more likely to indicate concern where there is none.
2. It is best to ensure that no questions force an opinion. Otherwise, respondents are likely to give incorrect information. The important part is that this applies to agree / disagree Likert-style questions. If there isn't an "undecided", then respondents are likely to indicate a neutral opinion when there is none. There is a subtle, but important difference between "I don't care", and "I don't know".
3. Thinking along the same lines, a "don't know" response for factual questions is also important. For example, in the framework I'm developing the user is asked about the operating system they run on their computer. Providing "I don't know" responses is really important to be able to capture the true novice users!
4. Ensure that your respondents aren't "primed" by questions in the survey. If you start your survey by asking respondents about how they secure entrances into their house, and follow up by asking how concerned they are about their physical security, a bias is likely to be introduced.
5. Many academic surveys will ask respondents to rate their own skills on a given subject or task. If you ask how skilled they are, responses will tend towards the positive. However, if you ask users how much they can improve, they will tend towards the negative. To help account for this, ask both types of questions at different points in the survey.

Follow these steps, and bias in your surveys should be dramatically reduced. Anyone interested in collaborating on expert and novice user classification should contact me, and I can introduce you to others in the research group who are also involved.