Tip on developing T & F, MC/Matching, and Essay test items

Developing T & F test items

As instructors, most of us have looked at true/false items we have developed for a test/exam and realized, in hindsight, that they were not very effective.

With true/false items, students don’t spend their time physically writing their responses, therefore, a large amount of content can be assessed. This item format is especially useful in determining students’ knowledge of facts and important detail-oriented items, or in determining students’ yes/no decision making abilities. The following suggestions may help you to develop more effective true/false test items:

  • It is helpful to develop questions as you teach the material.
    • After each class select information that you think would best be tested using this question format (facts, details, yes/no decision making) and write a series of true statements relating to that information.
      - Sample true statements: a) “Writing objective statements precedes instructional development in the instructional design process.” b) Visual, spatial, and mathematical are three of the 9 intelligences comprising Gardiner’s Theory of Multiple Intelligences.
    • Convert half of the true statements you have written to false statements.
      -Sample conversion to false statement: “Instructional development precedes the writing of objective statements in the instructional design process”
  • Here are some guidelines for writing statements to be used with true/false questions:
    • Keep statements as concise as possible (avoid wordiness)
    • Use language/terminology students are familiar with
    • Keep true and false statements the same length ( avoid adding information to make correct statements more precise)
    • Avoid using general statements, absolutes (all, always, never, none, only) as knowledge of one instance in which it is untrue could influence students to incorrectly judge the statement as “false”. 
    • Avoid using the word “can” as knowledge of a single instance in which something could be done might influence students to incorrectly judge the statement as “true”. 
    • For the sake of clarity, avoid using ambiguous or vague terms (large, long time, regularly, some, usual) as meanings for this words are subjective.
    • Avoid the use of negatives such as “not” and if you do use them highlight them in some way (“NOT”)
    • Provide clear, concise directions as to how students should indicate their responses to questions (circle, underline, place a check, write out the word “true” or “false” etc.).
      -having students circle or underline to indicate their responses eliminates problems that could arise due to illegible handwriting (mistaking an “F” for a “T” or vice versa)
    • Once you have developed your questions, make sure that they are not arranged in a discernable pattern (FF TT FF TT FF;  FTT TFF FTT TFF)
    • It is best to have an equal number of true and false items.
    • NEVER deliberately write a ‘trick’ question.  Use true-false items for important items.  Before committing it to paper, imagine explaining why a question is true or false to the class, anticipating the reaction of student who got it wrong. 
    • Chances are students will get half the true-false format items correct just by guessing; therefore, it might be best to limit the number of this type of item.
      • Alternatively, some professors apply a ‘correction for guessing’ by reducing the total of correct responses by a fraction of a point for each incorrect response (e.g., -.25 for each wrong answer), but there is no loss of points for a blank response. Be advised that students usually do not welcome such strategies.
      • If this is a concern, you might transform some true-false items into multiple choice items.

Rationale                                                                                                                                                            

Test items that are well written enable students to easily understand what is expected and provide a valid indication of student learning.  Students can respond to more content-related items if they don’t have to physically write a response.  For students who find writing difficult, either physically painful or because of a learning disability, objective test items can be highly effective tools for testing.  Other students will prefer essay questions.  Objective questions take a long time to develop and a short time to grade.

 

Developing MC & matching test items

As Instructors, determining the types of items to include in a test/exam is important; as is the way those items are constructed, if we expect our evaluation of student learning outcomes to be valid. This tip addresses two important questions concerning multiple choice and matching test items: 1) When is it appropriate to use multiple choice and matching questions? 2) How do I develop more effective multiple choice test items?

Multiple choice, like true/false test items, can be used to test large amounts of information and require students to do a minimal amount of writing. Matching items are particularly useful when asking students to recall and associate information (dates with events, examples with concepts). Both types of test items can be used to determine lower level (recall) and higher level learning (understanding, application, synthesis).The following suggestions may help you to develop more effective multiple choice & matching test items:

  • Before writing test items determine whether you want to evaluate acquisition of content or address higher-order learning (understanding, application, synthesis).

    -Items that address higher-order learning might require students to predict the outcome of a situation… Example: If “A” & “B” occur, the result will be A, B, C, or D. (Application of knowledge)Provide an example for a concept or principle… Example: Which of the following examples best demonstrates aggressive behaviour A, B, C, D?

OR

  • Give examples ask students to match the theory, concept, or principle that applies to that example… (Understanding)
  • Write test item statements after each class so that they reflect your teaching emphasis.
    -Write the preferred response immediately after writing each statement so that you are clear about what you expect the preferred response to be.
  • Keep item statements clear and concise and state them in a positive way (by following the same measures indicated previously for true/false questions).
  • Provide 4-5 answer options. Avoid writing answer options with very little distinction between them (unless this is an objective of the course).
  • Avoid providing hints by including extraneous information in faulty answer options (students may realize that you are trying to baffle them)
  • Avoid answer options such as “all of the above” and “A & B” as this often indicates the preferred response.
  • Avoid “cute” or “funny” response options as this often indicates the non preferred response option.
  • Avoid providing verbal cues in the preferred answer (key words only used in the statement and correct answer option).
  • Ensure that preferred responses to each item are not following a discernable pattern.

Rationale                                                                                                                                                   

Test items that are well written enable students to easily understand what is expected and provide a valid indication of student learning.  Students can respond to more content-related items if they don’t have to physically write a response. Students who have physical or learning disabilities often have difficulty writing responses to essay type items, matching and multiple choice test items require limited writing and can be effective tools for testing specific types of learning. Like true and false items, these types of items require more effort to develop, but a shorter time to grade.

 

Developing and evaluating essay test items

Essay test items are effective in helping students to demonstrate higher order thinking in unique ways. However, grading can be time consuming and it is often difficult for instructors to evaluate these types of test items objectively and consistently. Here are some suggestions for developing and evaluating essay test items:

  • Before writing an essay item, clarify your objective in developing the item.
    • Know what abilities the appropriate response to the question should indicate (understanding, ability to compare, contrast, differentiate, synthesize, evaluate, analyze etc.)
  • Ensure that the response to the item will result in the type and quality of evidence you require.
    • Proper wording leaves no doubt as to what is required for a complete response to the item.
      Example A:“What is Democracy?” If the intent is to have students discuss specific aspects of Democracy, this item would not encourage such a response. Students could respond by saying “Democracy is a form of government.” The question is too broad and could be answered in a number of ways.
      Example B: “Explain the roles played by society and state in a Democracy.” There is little doubt about what is required for a complete response to the second item.
    • Avoid items that contain phrases such as “What do you think…” or “What is your opinion about…” These types of items are indeterminate, they have no clear right or wrong answer and are almost impossible to evaluate. 
      Example A: “How do you think a Democracy compares with a Dictatorship?” This item is asking for an opinion, and there can be no right or wrong answer when asking for an opinion. Example B provides a better alternative that will require the respondent to demonstrate command of specific knowledge.
      Example B: “Describe the similarities and differences between a Democracy and a Dictatorship.” This item is much more focused as it requires respondents to compare specific aspects of both.
  • Once you have developed a clear objective and worded the item in a way that helps respondents meet that objective, you might want to consider developing a rubric to help you evaluate responses to the item.
    • The rubric should specify the criteria students are required to meet in achieving full value for the test item or the value they will earn if missing certain criteria ( no spelling and grammatical errors, but weak example; no example provided; no example, spelling and grammatical errors; inappropriate or weak response to item indicating little knowledge or understanding; no response to item or response is completely off the target)
  • In addition to evaluating using a rubric, try evaluating each essay item for all students before moving on to the next item. This will eliminate the “halo” effect where students are given the benefit of the doubt if their response to the previous item was answered well or the evaluation is negatively biased due to a poor response to the previous item.
  • Ask students to put only their student # or some other numerical identifier on their test or exam so that you cannot identify the person you are evaluating.

Rationale                                                                                                                                                

Having clear goals will enable you to write essay items that will assist students in demonstrating the learning outcomes you want to evaluate.

Test items that are well written enable students to easily understand what is expected and provide a valid indication of student learning.

The time taken to develop rubrics will enable the instructor to grade more quickly and consistently.

Grading each item for all students before moving on to grade the next item, and using blind grading will also provide more consistent and objective evaluation.

References                                                                                                                                                                 

Boyle, E., & Rothstein, H. (2003). Essentials of college and university teaching: A practical guide. Stillwater, Okla.: New Forums Press.

Facione, P., Facione, N., & Giancarlo, C. (2000). The disposition toward critical thinking: Its character, measurement, and relationship to critical thinking skill. Informal Logic, 20(1), pp. 61-84.

Gross Davis, B. (1993). Tools for teaching. San Francisco: Jossey-Bass. 

Johnson, G. R. (1995). First steps to excellence in college teaching. Madison, WI: Atwood Publishing.

Plaomba, C. & Banta, T. (1999). Assessment essentials: Planning, implementing, and improving assessment in  higher education. San Francisco: Jossey-Bass.

Pregent, R. (2000). Charting your course: How to prepare to teach more effectively. Madison, Wisconsin:   Atwood Publishing.

Ralph, E.G. (1998). Motivating teaching in higher education: A manual for faculty development. Stillwater,  Oklahoma: New Forums Press.