University of Oregon

Measured Progress Team

PDFPrintE-mail

Accessible Assessment

Research Site:       Measured Progress, Nimble Assessment Division, Boston, Massachusetts
Researchers:         Michael Russell, Jennifer Higgins, Tom Hoffmann
Study Title            MeTRC Accessible Assessment Research; Study 1

Study Description

The goal is to evaluate strategies for altering or supporting the presentation of text within math learning and assessment activities to increase accessibility and understanding. This research focused on the reading of expressions by examining whether a literal or interpretive description of an expression affected student understanding of the expression and/or their ability to use the expression to solve a given problem.

 Research Question. To what extent does the application of different scripting rules to generate alternate audio-based representations of mathematics expressions affect student understanding of that content or the test question designed to measure that content?

This question focuses on the presentation of mathematical expressions in an audio form. State assessment programs currently recommend two approaches for reading mathematical expressions aloud. The first approach focuses on exact presentation of content and requires the reader to state the content that is displayed in text-based form reading from left to right.  As an example, the expression “3x+y=2x(y+1)2 ” would be presented as “three x plus y equals two x, open parenthesis y plus one close parenthesis to the second.” The second approach presents a version of the content that is designed to assist the student in accessing the meaning of the content. This strategy would present the above expression as “three x plus y equals two x multiplied by the quantity of y plus one raised to the power of two.” The goal of this research question is to deepen our understanding of how different approaches to audio presentation of mathematical expressions affect student interpretation of those audio representations.

To this end, two approaches (literal and interpretive) were applied to samples of algebraic expressions. To disentangle alternate representations of different elements of expressions, the contrasts isolated the following elements of expressions:

  1. parenthetical expressions (e.g., x(y+1) )
  2. exponents (e.g., xy, or xy+4, or xy+4 )

Methods

The literal description for parentheses used the words “open parentheses” and “close parentheses” while the interpretive description used the term “the quantity of” when referring to the parenthetical expression. For items containing exponents, the literal description used the words “X to the n” to describe Xn while the interpretive description used the words “X raised to the power of n.” After responding to the parentheses items, students were asked two survey questions about their preference for how items were read aloud. The same two survey items were asked after the exponent items.

After scripting all math and survey items, the items were voice recorded using a human reader and inserted into NimbleTools, the universally designed computer based testing system used for this research. Two research forms, each consisting of the same 20 items in the same order, but with different variations of audio rules were then created.

Cognitive lab interviews were also conducted with a sample of students to help better understand student views on the audio presentation of exponent and parenthetical math items. A researcher worked with students as they viewed and completed four exponent and four parentheses items. Students were asked to explain their thinking in answering items, for example if there was any wording that was confusing to them or helped them understand the math content, and asking about their preference for audio scripting rule.

Sample

Three groups of students were recruited to participate in this study.  The first group was comprised of students with low vision who typically employ large-print versions of instructional and assessment materials.  The second group was comprised of students who have been identified with dyscalculia or a learning disability that might affect the reading of mathematical notation or representations.  The third group was comprised of students that have not been identified with a physical or learning disability. Grade 9-12 students were recruited in Florida and New Hampshire by sending emails and calling school contacts. Students from two Florida and three New Hampshire schools participated. The goal was to recruit 30 students in each group. While this goal was met in the “math needs” and “no need” groups, only one “low vision” student was successfully recruited for this portion of the research.

Students used a three-digit identification number to log in to the NimbleTools test session. Teachers administering this research were given a list of three columns of identification numbers: 1) low vision 2) math disability and 3) no identified need. Numbers in each of these columns were randomly assigned to either form 1 or form 2. By assigning id numbers according to a student’s access need, stratified random sampling of students to test forms was achieved.

In addition, students were recruited to participate in a cognitive lab that focused on students’ understanding and interpretation of the audio representation, with the goal of having 12 cognitive lab participants. Two “low vision,” eight “math needs,” and four “no need” students participated in a cognitive lab.

Findings

Descriptive data show that overall, participating students struggled with correctly answering items on the test forms. Table 1 presents the sample size, mean and standard deviation of student performance (represented as percent correct) on items by content element and scripting rule.

Table 1: Test Performance Descriptive Statistics

Content Element

Audio Scripting rule

Total (n=64)

M

SD

Exponents

Literal (X to the n)

35.6

25.1

 

Interpretive (X raised to the power of n)

31.8

23.2

Parentheses

Literal (open/close parentheses)

45.6

30.6

 

Interpretive (the quantity of)

24.3

22.1

Paired samples t-tests were employed to estimate the effect that the application of a given audio scripting rule had on difficulty for parenthetical items (literal versus interpretive) and exponent items (literal versus interpretive). There was no statistically significant difference in performance overall or for subgroups for the exponent content element. For the parentheses content element, students performed significantly better when items were presented literally versus interpretively. This finding is consistent for students with an identified math access need and for students with no identified need.

Descriptive data show that participating students preferred that exponent items be presented interpretively (X raised to the power of n), while parenthetical items be presented literally (open/closed parentheses). Table 3 presents the sample size, number of students and percentage of students reporting a preference (represented as percent correct) for each audio scripting rule.

Table 2: Survey Preference Descriptive Statistics

Content Element

Audio Scripting rule

Total (n=55)

n

%

Exponents

Literal (X to the n)

15

27.3

 

Interpretive (X raised to the power of n)

40

72.7

Parentheses

Literal (open/close parentheses)

47

80.0

 

Interpretive (the quantity of)

12

20.0

Cognitive Labs

Cognitive lab interviews were intended to help better understand student views on the audio presentation of exponent and parenthetical math items. A researcher worked with students one-on-one as the student viewed and completed four exponent and four parentheses items. Students were asked to explain their thinking in answering items. After completing all items associated with one content element, students were asked if there was any wording that was confusing to them or helped them understand the math content. Students were also asked to answer the survey items asking about their preference for audio scripting rule.

Descriptive data show that participating students preferred that exponent items be presented interpretively (X raised to the power of n), while parenthetical items be presented literally (open/close parentheses).

The cognitive labs yielded important qualitative information to this study. Students consistently explained why they preferred the interpretive representation of parentheses items:

‘It’s familiar and I like how it tells you what to do’

They also provided qualitative evidence that “the quantity of” is not a term often used in instruction:

 ‘I have no idea how to solve with quantity of; I’m totally confused’

Similarly, for the exponent items, students verbalized that they preferred the interpretive representation because of the instructive nature of the phrase “raised to the power of”:

Easier ‘because you know you need the exponent’

While other students noted that the extra words in the interpretive representation of exponent items was cumbersome:

Prefer x to the n ‘raised to the power of is too wordy.’

This research study builds on work conducted by Nimble Innovation Lab to apply principles of universal design to mathematics assessment. Findings consistently indicate that students prefer performing tests using computer-based accessibility tools as compared to having accommodations provided through a paper-based test experience and that test scores provide a more valid representation of students’ mathematical understanding when using the computer-based tools.

Results from both the experimentally designed component and from cognitive labs show that students prefer and perform better on parenthetical items when presented verbally in a literal manner, using the words “open/close parentheses” instead of using “the quantity of.” Results for the exponent items are less clear. While there were no statistically significant differences in performance, 76% of students in the experimentally designed component and 50% of students in the cognitive labs preferred the interpretive presentation (X raised to the power of n). Students reasoning for these preferences, as expressed in the cognitive labs, centered on having auditory descriptions that are instructive. As one student said ‘It helps me understand what they are trying to ask me.’

Connect with Us

FacebookTwitter
seo