Is Your Bias Showing?

What comes to mind when you hear the word bias? Do you think of a person who is judgmental or unfair? Or someone who believes their own opinion is better than the opinions of others? Have you ever considered the bias that might be present in your student assessments? 

In a previous blog, we defined a valid assessment item as meeting two criteria:

1. Students who have mastered the relevant content will answer correctly.
2. Students who have not mastered the relevant content will answer incorrectly.

A common, and often overlooked, issue that may invalidate an assessment revolves around bias and sensitivity. Assessments are considered biased when the results show an advantage or disadvantage to a certain student group. Items that include hidden or unconscious bias, or lack of sensitivity to students from different backgrounds, can lead to assessments that do not provide valid data about student knowledge. 

Can you identify the potential bias in these items?

Socioeconomic Bias

Item one may show socioeconomic bias. Can you determine why?

Unless students have currently been studying sports and sports equipment, a correct answer depends on a student’s background. If students come from middle- to upper-class families, they may be familiar with the game of golf and how a putter is used in that game. On the other hand, students with a lower socioeconomic status may not have the same advantage. Furthermore, the word bat has more than one meaning, making the question more difficult without a full understanding of the first relationship. An unbiased question would rely on information a student has learned through schoolwork, and does not draw on what a student may or may not have learned at home. Examples of socioeconomic bias may be seen commonly in items related to careers, purchases, home-ownership, and recreational activities. 

Gender Bias

Item two shows an example of gender bias. Can you spot it?

Notice the answer choices include three female names and one male name. And the male student is the one with the correct answer! A better item would include even numbers of each gender or identify the students as Student 1, Student 2, and so on. However, it is important to understand that one item that uses a male or female name does NOT necessarily indicate bias. Gender bias, or any other bias, is not clearly seen when looking at only one assessment item. The assessment as a whole must be reviewed, and reviewers may find that bias is less obvious than shown in this sample, and more pervasive an issue than can be pointed out in one test question or one test. 

Consider this example from The National Center for Fair & Open Testing (FairTest). Before the revamping of one high-stakes assessment, male test-takers consistently received higher scores than female test-takers on the math portion of the exam, while female test-takers performed better on the verbal portion. The makers of the assessment revised the verbal portion to include more questions related to politics, business, and sports. Following this revision, the latter trend has reversed: Male test-takers now consistently outscore females on the verbal portion of the exam! 

Surprisingly, gender bias may be present in the types of assessment items that are used. Evidence reported by FairTest and the American Physical Society suggests that male students tend to score higher on timed, multiple-choice tests than female students. This may show that male students are more willing to risk guessing at the correct answers, while female students tend to fully work out problems and check their results. 

Language-based and Cultural Bias

Item three has an example of bias that is based on the language used. Can you determine how this question might be difficult for a non-native English speaker?

Idioms, such as the phrase beat around the bush, can be great to include in assessment items that test whether students can derive meaning of an unfamiliar word or phrase from the surrounding text. However, in this item, there are no clear context clues within the question text that reveal the meaning of the phrase. The answer may or may not be known to students based on their own experience. Idioms may be regional, or they may be unfamiliar to immigrants or ESL students. Test items that include references to cultural details that may be unfamiliar to specific student groups may be considered biased. 

Emotional Response

In some cases, a question or question topic may elicit a strong emotional reaction from test takers. Anxiety, anger, or emotional pain may cause unequal testing conditions. Unless this is the desired response or the topic is pertinent, and the response will be evoked from all test-takers, it is best to avoid sensitive topics (e.g., prison, serious health issues, natural disasters, divorce). Consider how a student may react to a mathematics question or a question on verb use that includes the context of a tornado if their family has experienced recent loss due to a tornado.

It’s important to note that a topic may be acceptable and appropriate when discussed in class, but may not be appropriate in a testing environment. Stereotyping, bias, and exclusion based on age, gender, culture, or ethnic groups should be avoided when possible on assessments, even in distractors.

Other Forms of Bias

Any other forms of bias, including racial, religious, and age-related bias, should also be avoided. Consider how an item relating the number of aces or spades in a deck of cards may be more difficult for a student from a religious background who has never been exposed to cards than for a student from a family who plays cards at home. Or how a third-grade student may or may not be familiar with the concept of a class reunion.

While an assessment item may still be answered correctly by a student who is not familiar with the context, a disparity is created when one group of students is familiar while another group is not. 

So, How Can I Avoid Bias?

Avoiding bias is not as difficult as it may seem. Consider using assessments that include many different item types. Instead of all multiple-choice questions, include short-answer and extended-response items that ask students to explain their understanding. Performance tasks can be great assessment tools that can be used to show not only understanding, but moving beyond understanding into applying concepts to new situations.  

When you write assessment questions, or decide if you want to use questions someone else has written, consider the items carefully. Make a checklist, similar to the one below, of things to consider as you put together an assessment.

1. Is there bias in the item content?
2. Is there bias in the language?
3. Could the item potentially leave a cultural group at a disadvantage?
4. Is there anything that might give offense in the item presentation?
5. Could the item elicit a strong emotional reaction?
6. Are there stereotypes (positive or negative) in the item?
7. If there is context, does it reflect a diverse culture?
8. Is the item fair overall?

Remember, the goal of an assessment is to determine if a student has mastered specific content. Be sure your questions aren’t setting your students up for failure. 

Helpful Resources

Avoiding Test Bias

Testing Bias, Cultural Bias & Language Differences in Assessments

Test Bias