Can You Spot an Outlier?

If you’re asked to describe camouflage, your first thought may be of an animal blending into its surroundings. Something that is otherwise hard to spot without sudden movement. Assessment writers assume the arduous task of camouflaging the correct answer, or key, into a set of incorrect choices, or distractors. On formative or summative assessments, a poorly camouflaged answer choice is an outlier. Your students will spot these and gravitate toward them like the school bus at the last bell.

In our previous blog, Are You Writing Strong Assessments?, we defined a valid assessment item as meeting two criteria:

  1. Students who have mastered the relevant content will answer correctly.
  2. Students who have not mastered the relevant content will answer incorrectly.

Poor distractors—the wrong answers of a multiple-choice item—are the problem that most commonly invalidates an item in our experience, and the most common kind of poor distractor is the outlier. 

Can you identify the outliers in these four items?


Implausible Outliers

Item one has an example of an implausible outlier. Can you spot it?

Regardless of how much—or little—students know about coal, they will almost certainly know it does not form from dead aliens. Distractor A is implausible and thus easy to eliminate. Depending on grade level, distractor C may also be implausible: No geologist would talk about “dried lava”—lava hardens or solidifies. To a younger student, this may not matter, but middle- or high-schoolers may sense the problem—“dried lava” sounds “off,” even if they can’t put a finger on why. If your students can eliminate a distractor for a reason unrelated to the content being assessed—in this case, how coal forms—the distractor is an outlier and the item is invalid.

Note that changing distractor C to “solidified lava” solves another possible problem (assuming we also replace “dead aliens”!): Currently, three of four distractors begin with the letter “d.” As a result, “pressed diamonds” stands out.  Does it stand out enough to be an outlier? Perhaps not. But why risk creating a problem that is so easily avoided? Thinking in this way about each distractor can only improve your items.


Dissimilar Outliers

Number two has an example of a dissimilar outlier.

Of all the pronouns you might use for distractors, only “I” is capitalized. It thus stands out from the other three options—a problem that may be exacerbated by the language of the stem, which asks about case. (Capital “I” is an uppercase letter.) If this concern seems trivial, consider further that in many fonts (including this one), capital “I” is indistinguishable from lowercase “L”. If a student mistakes—however briefly—the vowel for the consonant, they will be distracted from the task of solving the item.

Does this mean that capital “I” should never be used as a distractor? Surely not. But again, why risk confusing students when there are clearer pronouns—for example, “we”—to choose from?

Number three has a more obviously dissimilar outlier.

Answer choice A (πr2) is the only option that includes the symbol π. (Alternatively, it is the only option that does not include the letters b and h.) In this case, however, the outlier is the key: The formula for calculating a circle’s area is πr2. We might change the key, but then we would also have to change the stem. The easier fix is probably to change the distractors. Note that we do not have to change all three—consider this revision:

Because two answer choices include π and r and two include b and h, none stands out unnecessarily. How else are these options balanced? Two include parentheses and two do not; two include fractions and two do not; two include superscripts or subscripts and two do not; and all four include the number 2. (Most importantly, each is an actual geometric formula that students will likely have studied—under no circumstances should such an item have distractors that are not formulas!) Of course, achieving such balance takes time, but the more you practice looking for outliers, the easier it will be to spot them.


Nonparallel Outliers

Number four has an example of a nonparallel outlier.

The first three options—democracy, monarchy, theocracy—are all nouns; in contrast, option D—totalitarian—is an adjective. (D is also the only option that does not end in -y, so it is dissimilar in this respect as well.) This contrast might lead students down several stray paths: They might guess D is the only correct answer, or they might guess D is not correct and select only A, B, and C. (In fact, all four answers are correct.) To fix this problem, why not change “totalitarian” to “oligarchy”?


Ungrammatical Outliers

A glaring example of non-parallelism is the ungrammatical outlier. Consider this item:

Only a noun can complete the sentence; here, option D not only stands out from the other choices, it cannot be correct. Be prepared to receive calls from parents who notice these types of glaring issues, especially if the issue is with the key!   


Let’s try one more...

Have you ever written an item with pairs of opposing answer choices? Outliers help to explain why this is not generally considered a best practice. Consider this example:

As you can see, options A and B form a pair of opposites; options C and D form another pair. On the surface, this may not seem problematic: Students who select the correct answer, C, have demonstrated they understand the effects—and limits—of the Sun’s gravity. Right?

The problem is that distractor A is an impossible outlier: Stars outside the solar system cannot “stop” orbiting the Sun because they do not orbit the Sun to begin with. Savvy students who know this fact can eliminate A regardless of whether they know anything about gravity.

Why might someone use an impossible claim for a distractor? Here, it was necessitated by the writer’s desire to form a pair of opposites with distractor B. In our experience, such a desire is a fertile source of outliers. The first pair of opposites comes easily enough, but the writer gets stuck on the second pair—it can be intensely difficult to think of statements that are both contradictory and equally possible! Rather than find another approach to the item, the writer settles for an outlier.

Ultimately, that word “settle” explains a lot. Assessment items are deceptively complex, and writing good ones takes time. Assuming an item is “good enough” may lead you to overlook an outlier that seems inconsequential yet, upon closer inspection, invalidates the whole thing. The good news is that fixing outliers is often quite simple. The crucial step is thinking to look for them.


Helpful Resources

Writing Good Multiple-Choice Test Questions

Characteristics of Good and Bad Multiple-Choice Questions

Good, Better, Best: Multiple-Choice Exam Construction


gotto's picture

Submitted by gotto

information on opposing outliers was particularily useful