Neal, Michael. Writing Assessment and the Revolution in Digital Texts and Technologies. NY: Teachers College, 2011. (168 pages)
Introduction: Writing Assessment as and with Technology
Neal approaches writing assessment from the perspective of technology in order to illuminate “how the cultural expectations of technologies shape our expectations of writing assessment” (5). Assessments, themselves, are technologies, but he specifically makes a distinction between writing assessment with technology and writing assessment as technology.
As technologies, we can look to the ways that writing assessments are threaded in other aspects of the education process. Much like how technologies are constitutive of society (or, in other words, what makes society possible), “writing assessments are constitutive of writing instruction” (8). He writes further that “we can see not only how they are designed to function but also the often unintended consequences that accompany their implementation” (8).
With technologies, such a perspective “reveals how assumptions and narratives explored previously play out in the digital age” (9). Here, Neal uses this perspective to look at the ways we use writing assessments to do the work of assessing students; he specifically points to the mechanization of assessment and the assumptions that underlie such mechanization.
Chapter 1: Underlying Assumptions of Writing Assessments as Technologies
Neal makes the argument that all writing assessments—both the material objects of the assessments and the knowledge and procedures involved in its implementation—are technologies. Of course, this means that many of the procedures and method in place—even before digital or online assessments—can also be labeled technologies. As such, Neal points out that a key component to understand how we might be able to better respond to student writing with better judgment, we can identify patterns in how we have used old writing assessment technologies.
Specifically, Neal seeks to dispel to hardwired cultural assumptions pertaining to technologies that have a particular impact on how we come to understand writing assessments: (1) Instrumentalism refers to the tendency to focus solely on the object or instrument in discussions of technology, thus having the tendency to encourage technological neutrality; (2) determinism refers to the tendency to give agency to the technologies thus ridding human agents from their culpability in participating in assessment.
Instrumentalism. Neal points to some common definitions of technology; namely, that they are “things people use or create to accomplish a specified purpose” (18). Such a definition focuses our attention to “the materiality of assessment” or, rather, the things that we use to accomplish tasks. However, Neal adds that technologies involve much more than the thing itself, but the processes, procedures, and techniques involved in using those instruments. Ignoring the contexts of us of technologies and only focusing on the objects of it “conceals [the technologies] complex ideological qualities, the processes by which the technologies function, and the wide-ranging social consequences of their production and use” (19). Neal continues, “While writing assessments are composed in part by their physical apparatus, they do not accurately represent the full extend to which writing assessments are technologies” (19). “by focusing exclusively on the instrument, we often ignore deeper underlying assumptions about writing assessment…when our focus Is on the test or the instrument, it is easy to ignore or miss the larger technological system at work within writing assessment” (19). Under the instrumentalist understanding of technology, when assessments go awry, the issue is wrongly identified as the reliability of the instrument. Instead, we should be considering “how and why decisions are made based on the results gathered by the assessments” (19). Such is the basis of validity: contexts of use.
Instrumentalism, as Neal writes, tends to see technologies as value-free or neutral tools. Looking to guns as an example, he points to the common adage, “guns don’t kill people; people kill people” (21). In this way, the instrument or gun itself is seen as neutral rather than ideological: the neutrality of technology-as-instrument ignores the wider systems involved in manufacturing, the history, function and capacity of the human-made technology, and how the gun will be used. As Neal appears to argue, we must include the usages and contexts of use as part of how we understand technologies, and, as such, the technology itself—including the instrument, but expanded to include its context of use—is always ideological. Looking to assessment, the perspective from ETS and College Board is to vouch for the accuracy of the assessments, the instruments, but, as such, does not contend with the kinds of ways the results are being used. They relieve themselves of responsibility since they do not contend with its context of use, but for Neal, this is a necessary aspect of its technology.
Neal writes, “we need to consider the entire writing assessment system, including the accuracy and appropriateness of decisions we make based on it, to find more valid ways of assessing student writing within local contexts and individual needs (23).
Determinism. Defined, technological determinism refers to the “belief that the technologies themselves shape and define the culture, largely independent of human interaction” (23). As Neal writes, such a perspective results in the invisibility of technology and, likewise, does not make people accountable for the creation and implementation of technologies. To put another way, Neal is confronting the inevitability of technology and saying that people make technologies, and, accordingly, people can enact change in how we use, design, and implement these technologies. Neal challenges the idea of technological imperative that suggests that “(1) because technologies are inevitable, they should be simply accepted, but it also means that (2) because technologies are available, they should be used or implemented” (24). A technology that’s available and widely used—or even supported by administrators—can often render it as natural and goes uncontested. Neal offers the example of the implementation of a portfolio system from higher administration: “once it was accessible and effectively marketed to the decision makers in the institution (not faculty, in this case), it became more difficult to talk and think about technology in critical ways” (25). Like his conclusion for instrumentalism, we must examine and discuss the viability of new technologies, particularly digital writing and assessment technologies.
Also in this section on determinism, Neal also begin to discuss the ways (1) we humanize technologies to have active, human agency and (2) the ways that we mechanize people to contend with technologies. “By humanizing our technologies, we create the illusion that they are more personal and relational than they are” (24), but more to Neal’s central focus, people are becoming mechanized within the system: “I am concerned that the types of writing assessments we use are mechanizing the minds of students in ways that are contrary to what we are or should be teaching them about writing” (24). I’m thinking here of Ong: technologies are interiorized and, thus, restructure consciousness. In this way, we can’t say that technologies, themselves, don’t have some kind of an impact on our practices: they invite particular modes of behavior, or at least afford for particular kinds of behavior/consciousness. But, For Neal, we must not see these behaviors as unchangeable; rather, he sees these as negotiable and capable of change.
Chapter 2: Cultural Narratives that Characterize Writing Assessment as Technologies
Technotopic Narratives of Writing Assessment as Technology. “technological advances have been inextricably related to the grand narrative of societal change” (34). However, Neal discusses the idea of technicism: “this belief espouses a type of false hope or promise that technological innovations are propelling us in mostly constructive, mutually beneficial directions. Usually presented with critical edge, technicism is often used similarly to Stanley Fish’s ‘theory hope’: something that promises much but doesn’t live up to expectation” (34). For writing assessment, Neal points to the widespread adoption of portfolios that often left it uncritically observed—it operated as a kind of savior of all problems. However, as Neal points out, only after it received widespread acclaim and acceptance did the scholarship begin to be more nuanced and cautious.
Technophobia: Pessimistic narratives of technology. “I would suggest that the neo-Luddites within our own discipline reflect a similar technological bias. Instead of being antitechnology, they are merely invested in older technologies in which they have developed expertise and that have afforded them accomplishments and gave them standing within the community” (41).
Chapter 3: Decision Making and Development of New Assessment Technologies
Neal, as he outlines at the onset, is very much concerned with the ways teachers and educators are involved in the technologies of assessment. As he has pointed out earlier, teachers should be taking a critical gaze to these new technologies and observing the consequences of the information we receive from those assessment technologies. With this goal in mind, Neal points to three stages “during which those of us with writing expertise and teaching expertise should exert influence for new assessment technologies” (44):
- Implementation: “We may not always be happy with the choices we have at our disposal, but we need to determine which writing and assessment technologies will provide the best opportunities for teaching and learning in order to measure student performance and support them to improve their writing” (46).
- Prediction: “Ultimately, we want to get out in front of new technological developments and exert some influence over them rather than merely making do with what is made available to us. Therefore, it is essential that we do a better job at predicting the trajectories of new assessment technologies” (46). Neal points to four questions that would be helpful to predict potential writing assessment technologies and uses: (1) how will the technology evolve? (2) What will the technology be used for? (3) Who will the technology’s users be? And (4) Who will decide how the technology will be used? Other claims should be considered:
- Prediction is difficult, even for experts
- New technologies are market driven
- Innovations proliferate rapidly
- Best design does not always win
- Uses of new projects are hard to foresee
- Design: “how are assessment technologies developed and by whom?” (53)/
Chapter 5: Hyperactive Hypertechs
Neal notes that students are writing in digital spaces under their own volition: “students know these spaces, have reasons to write in them, and would benefit from a more critical examination of how they interact in them” (77). However, he also notes that there appears to be a disconnection between the writing students are doing outside of school and the kind of writing that was being ask of them inside of school: “what happens to composition when we delegitimize or marginalize the writing students do outside class, even those done in digital spaces that do not look much like writing students have traditionally done in the academy” (78)? In response, Neal explores “how technology is (re)shaping communication through texts and how we can and should respond via writing assessments with technologies” (78). He points to three kinds of composing that assessment would need to begin to contend with: the hypertext, hypermedia, and hyperattention.
He points to the concept of hypertext as a starting point; as he writes, hypertext describes connectivity: “linked texts, especially those that result in non- or multilinear arrangement, provide a rhetorical nuance for writers, since they have to anticipate multiple paths via which a reader might navigate a text…hypertext invites multiple paths and options often in more obvious ways” (79). The initial use of hypertext was regarded as avant-garde and poststructuralist; however, some publications in hypertext “betrayed its potential by simply reproducing print online instead of creating sustainable, new types of texts” (79). The movement to electronic portfolios, then, marks a shift toward observing the interactive potential of hypertext. ePortfolio assessment, “as an artifact or space to collection, select, and reflect on multiple texts” (80), allowed for reflection through the representation of connections made to demonstrate learning. “Connections are the key, and hypertext provides a means—through not exclusively—to represent those relationships” (81).
The ability to create linkages between artifacts becomes a cornerstone of the ePortfolio. However, Neal points to an example of a portfolio initiative where the institution administration offered a digital database for students to host portfolios, eliminating the opportunity for students to contend with design.
Howeve,r it is not simply that we provide opportunities for students to use visual design to create connections, but systematic opportunities to articulate and make explicit what has typically been tacit composing knowledge is key for assessing student’s composing abilities with hypertext. Neal offers the genre of a rhetorical rationale to accompany hypertext creations. In assessing the rationale, Neal observes: “claims with no evidence demonstrates either a lock of self-awareness on the part of the writers or that the writer is trying to claim something that he or she has not actually accomplished. Equally problematic, however, is evidence of solid work without the student’s making any claims about it. Again, this shows either a lack of self-awareness or that the writing skills remain tacit in an environment in which part of demonstrating learning means students becoming aware as writers. Effective reflection demonstrates a balance between claims and evidence that are mutually reinforcing” (89).
Hypermedia refers to the combination of one or more media or communication modality in digital platforms. With the emergence of digital media, we also see the redefinition of old media/platforms, themselves experiencing a kind of new media. Referring to Wysocki, any medium—including older media of handwritten letters, etc—“can be classified as new media as long as the reader and writer are aware, for example, of the ways the letter allows for certain type of communication that is distinct from other media choices” (92). In other words, handwritten letters—as with many old media—has a more specific usage than has been noticed before the emergence of a new media landscape.
Neal also points out that “new media texts are necessarily mathematical as well as linguistic thus becoming programmable… a majority of students are producing numerically represented textes, which allows for this type of programming of texts” (93). In other words, while hypermedia texts are linguistic in that they involved words, they are also code, and as code, students—and others—can manipulate the linguistic code in ways not possible before.
Given these considerations, Neal points to three common but insufficient approaches to assessing hypermedia:
- Treating hypermedia the same as print texts
- Importing criteria from other disciplines. See Sorapure.
- Developing reductive grading rubrics
Unlike the other two descriptions of media or genres, hyperattenion is itself an attribute of writers and readers in the digital age. Defined by Hayles, hyperattention is “characterized by switching focus rapidly between different tasks, preferring multiple information streams, seeking a high level of stimulation, and having a low tolerance for boredom…Although we can see its effects in hypertext and hypermedia, hyperattention is a physiological characteristic of people who have been influenced by digital texts” (99; See: Ong).
Neal offers a couple points of departure in considering hyperattenion, but he points to the combined spaces of virtual and the physical: “spaces, though, are just a part of the larger picture of textual hypridization. Students are producing a variety of hybrid digital texts. One particular student-produced text is interesting to consider in relation to its hybridity and its demonstrating how students can compose in multiple communication modes” (102). In the example he provides, a student who combines multiple media to comment on how the Dodger’s training camp moved from Florida to Arizona. She was able to combine music, pop culture clips, etc to create a comment on the physical landscape.