APPLE Lecture Presents Dr. Paul Deane

APPLE Lecture Series 2022 Presents Dr. Paul Deane

Title: New Ways to Measure Literacy: Combining Scenario-Based Assessment with an Automated Writing Trait Model

Abstract:

Literacy is a complex ability, requiring people to deploy flexible combinations of reading, writing, and thinking skills and strategies. Measuring it well requires us to look at how people perform in realistic contexts -- and that is not easy to do, especially for writing and other complex performance tasks.

SCENARIO-BASED ASSESSMENT is one approach to this problem. In a scenario-based assessment (or SBA), people are asked to perform a sequence of tasks that simulate steps that an expert might take to solve a problem. The tasks are designed to measure specific supporting skills necessary to complete the performance task being simulated, while modeling the purposes and strategies that would motivate someone to deploy those skills in the first place. For instance, before writing an argument essay of their own, a writer might read, summarize, and analyze relevant sources, and evaluate the arguments they contain. An SBA based on this scenario might therefore include summary evaluation, summary writing, argument analysis, and argument writing tasks. This kind of SBA has been the focus of much research at ETS, as part of the CBAL research iniative (CBAL=Cognitively Based Assessments of, for and as Learning).

STEALTH ASSESSMENT is another approach to this problem. In a stealth assessment, people are asked to perform normal literacy activities, but aspects of that performance or of its work products are measured in the background. Stealth assessment has a natural alliance with automated measurement methods, including natural language processing, process log analysis, and other artificial intelligence, or AI methods. Both NLP and process log analysis lend themselves naturally to a TRAIT MODEL approach, in which automated methods are used to identify an array of distinct performance measures that place student work in an implicit n-dimensional space. These may include features of writing quality, such as organization, development, text cohesion, vocabulary, syntactic complexity, and adherence to conventions, or features of writing performance, such as time on task, typing speed, and use of planning, editing, or revision strategies. In principle, once student performances have been mapped into such a trait space, we can characterize learning, and other changes to student performance, as movement from one region in this space to another.

In this presentation, I will discuss research we have been doing at ETS designed to marry scenario-based assessment with trait models, and will discuss data that show how this kind of assessment can be used to provide a rich picture of student literacy skills.

Biography:

Paul D. Deane is a principal research scientist in Research & Development at the Educational Testing Service. He earned a Ph.D. in linguistics at the University of Chicago in 1987. He is author of Grammar in Mind and Brain (Mouton de Gruyter, 1994), a study of the interaction of cognitive structures in syntax and semantics, and taught linguistics at the University of Central Florida from 1986 to 1994. From 1994 to 2001, he worked in industrial natural language processing, where he focused on lexicon development, parser design, and semantic information retrieval. He joined Educational Testing Service in 2001. His current research interests include automated essay scoring, vocabulary assessment, and cognitive models of writing skill. During his career at ETS he has worked on a variety of natural language processing and assessment projects, including automated item generation, tools to support verbal test development, scoring of collocation errors, reading and vocabulary assessment, and automated essay scoring. His work currently focuses on the development and scoring of writing assessments, as well as examining use of keystroke logs to analyze in writing in high-stakes assessments and in learning environments.