Condon, William. “Large-scale assessment, locally-developed measures, and automated scoring of essays: Fishing for red herrings?” Assessing Writing 18.1 (2013): 100-108. (8 pages)
Automated Essay Scoring (AES) is limited in how it both defines and describes its construct, i.e. writing. As Condon makes clear, “No AES System can achieve the kind of understanding necessary to evaluate writing on the semantic level—on the level of meaning, let alone the level of awareness of occasion, purpose, and audience demanded by any form of real-world writing, whether within the academy or in the workplace” (102). Not only that, but the use of such a scoring method has an impact on what students are learning when taught (or trained) in the classroom to write to a machine that judges their writing (often poorly) based on syntactical or mechanical word- or sentence-level errors.
However, while he opens by describing the issues of AES, his sees it as the product that logically emerged from timed-essay assessments embraced by many large-scale assessment companies. In other words, AES is a red herring for what’s at the core of the issue: “AES is not the culprit here, but it has been the red herring, at least for those engaged in college-level writing instruction and assessment. The real issue is the kind of writing AES can score. Rather than objecting to AES as the mechanism for scoring these tests, we need to argue against the tests themselves as an ineffective and even irrelevant measure of writing ability” (104). In fact, as Condon writes, some have noted that AES systems are reliable to assess the Six-Trait model of holistically scored writing rubrics; however, given the profound pitfalls of AES, the issue may lie on the holistically scored tests rather than the AES system itself. Namely, such writing may not, already, be able to assess for rhetorical knowledge, critical thinking, and control of the writing process.
Condon concludes, “we need not focus much attention on AES itself. Instead, we need to examine the testing practices that have led to the development and use of AES as we know it today….Once we make that transition, if AES survives, it will survive because it can measure far more than syntax and because it can rate far more than pseudo-essay. Let the high, multiple-validity tests lead, and allow AES to follow. If (or when) it can” (106-7).