Moss, Pamela A.. “Testing the Test of the Test: A Response to “Multiple Inquiry in the Validation of Writing Tests” AW
In Moss’ response to Richard Haswell’s Multiple Inquiry in the Validation of Writing Tests, she takes a closer look at how we might think about testing the validating of some assessment measures, namely those for programmatic assessment. At the onset, Moss claims—referencing Haswell—that writing assessment is a social act: “these issues highlight testing as a social practice with social origins and social consequences, not just for th person assessed, but for all stakeholders in the assessment process” (150). Given these social consequences, it matters how we validate tests because there may be unintended consequences that emerge from a test.
Defined, validity research “is designed around an explicit understanding of the purpose(s) the test will serve and of the kinds of interpretations that are intended to be drawn. Thus, validity research involves attention both to the meaning of the test scores or other test results as well as their effects” (152). Validity research, then, is guided by “plausible rival hypothesis” which refers to the potential alternative explanations or meanings from a test score based on the kind of data collected. Evaluations of validity, then can be organized around a set of questions: “what are the arguments for and against the intended aims of the test? And what does the test do in the system other than what it claims, good or bad?…It requires examine the intended interpretations and effects (use), as well as those that are unintended” (153).
In designing any assessment, it should be made clear “what the substantive goals of the program are—what counts as good writing—or what the theoretical rationale underlies the placement of students into different courses” (155). In other words, clearly state both the purpose and the intended interpretations or meaning of test scores.
Finally, Moss argues why such research on validation is important: namely, like much in writing assessment, it has to do with the construct of writing. “the importance of investigating the extent to which the interpretations ‘we’ produce become ‘part of the taken-for-granted definitions and categories by which members of communities define themselves and others’ and how this, in turn, impacts their access to material resources and locates them within social relations” (157). These understandings often operate tacitly.