With school districts placing more importance on test scores, it's important to know what all of these numbers mean. Using statistics, probability and computer coding, Kikumi Tatsuoka has come up with the Rule-Space Model which can determine individual strengths and weaknesses based on how a test taker responds to questions on the Scholastic Aptitude Test (SAT).

Tatsuoka, a full-time project director in TC's Department of Human Development, Distinguished Research Scientist and Adjunct Professor of Statistics and Education, developed the model while working at ETS. She has been working on it under various grants since 1980. The College Board is beginning to use her model to report diagnostic, or scores broken down into relatively specific achievement areas and cognitive subskills, with this fall's SAT scores.

"Professor Tatsuoka is one of the most eminent scholars nationally in the field of Educational Measurement. Her Rule-Space Model (RSM) is seen as one of the most viable alternatives to the traditional unidimensional models of Item Response Theory (IRT)," said James Corter, Chair of Human Development.

Tatsuoka noticed that the "deterministic approach" of finding a student's errors or "bugs" didn't work well. Sometimes students change their rules for choosing a certain answer for no reason. In addition, that student may not remember why he or she chose that particular answer. It's difficult to figure out the logic behind an answer choice because it's not something that they can observe while it's happening.

Her methodology for the Rule-Space Model follows pattern analysis, which helps computers to read letters like humans do, each question has "feature variables." For example, when a computer distinguishes between the letter "A" and the letter "H," it looks at what differentiates the two characters. Those differences are the letters "features," or "attributes." These features are translated into code, a series of patterns of zeros and ones. Then, they match stored patterns with the new patterns. The computer processes them and reads the letters. It is the same approach used to read DNA.

Instead of looking at DNA or the alphabet, Tatsuoka is looking at questions such as, "Why can't Johnny read?" or "Why can Jane do fraction problems?" To figure this out, she uses probability and statistics to determine the student's "knowledge state," which is a combination of the attributes that she compiles using pattern analysis.

Using this information, she can plot the numerical SAT scores and their related various attributes. While examining these numbers, she found that test preparation courses make SAT scores increase minimally, by only about 50 points. However, other attributes, such as good deductive thinking skills, can make SAT scores jump up much higher. "This method makes it clear what area we need to work on with students," she said.

With this system of scoring, each student gets a personal attribute mastery report showing how he or she did individually. The school district and state would also get the report. This individual report breaks down the total score into categories of mastery and where the student needs improvement.

"Students may have similar scores, but entirely different strengths and weaknesses," said Tatsuoka. The Rule-Space Model analyzes the score and looks at the test takers' specific skills based on the answer choice he or she made.

Two students who got 500 on the verbal portion of the SAT have very different individual reports. For example, Joe, who scored 500, isn't good at bringing material together from two passages or processing grammatically complex tests. He doesn't understand the main idea when not explicitly stated. While Nancy also scored 500, she is not good at synthesizing scattered information and applying general background knowledge. She doesn't have a good global understanding.

Tatsuoka's method tells schools and parents to see exactly what the lump score of 500 means. It tells them where the students went wrong and provides suggestions for improvement in specific areas. For example, when for Nancy's lack of global understanding, the report suggests, "working on summarizing the main idea of a text or a section of text. When you read a paragraph, ask yourself, 'What is the author driving at here? What is the most important point that the writer is trying to make in this essay?'"

With these new individual reports, SAT scores will be more than just numbers. They can provide a snapshot of a student's abilities and provide suggestions to build on them.

Published Thursday, Feb. 27, 2003