Moss, Pamela. “Can There Be Validity Without Reliability?” (7 pages) AW
Moss questions whether there can be validity without reliability. Despite the commonly touted axiom that there can be no validity without reliability (not vice versa), the simple answer to her inquiry is yes, there can be validity without reliability. She evidences this claim by looking critically at how we understand reliability in relation to assessment. As she notes, at the core of reliability is consistency in examinee performance. Consitency among independent observers (interrater reliability, specifically) requires a significant level of standardization. However, such high levels of standardization is often not sought out in many assessments.
She frames her discussion in a hermeneutic approach to assessment: “hermeneutics characterizes a general approach to interpretation of meaning reflected in any human product, expression, or action, often referred to as a text or ‘text analog’” (85). The hermeneutic approach seeks to understand the whole in light of its parts. She places particular attention to critical or depth hermeneutics which refers to the “importance of considering social dynamic that may distort meaning” (86): “a hermeneutic approach to assessment would involve holistic, integrative interpretations of collected performances that seek to understand the whole light of its parts, that privilege readers who are most knowledgeable about the context in which the assessment occurs, and that ground those interpretations not only in the textual and contextual evidence available, but also in a rational debate among the community of interpretants” (86).
When looking at hermeneutics, we also likewise look toward interpretive communities (see: Stanley Fish). Decisions are often based on the consensus or compromise of community discussions; members often include people form a variety of expertise or credentials in order to “cover the knowledge and political bases that a thoughtful decision requires” (87). In this case, there is no reliability among raters; however, such discussions “construct a coherent interpretation of the collected performances” where interpretations are continually revised “until they account for all of the available evidence” (89). She continues succinctly, “inconsistency in student performance across tasks does not invalidate the assessment. Rather, it becomes an empirical puzzle to be solved by searching for a more comprehensive or elaborated interpretation that explains the inconsistency or articulates the need for additional evidence” (89).
To summarize, Moss sees inconsistency as a source or prompt for further dialogue and discussion which can be valid and desirable for some assessment decisions.