When setting test rigor, Oklahoma education officials wanted to avoid a ‘thumb on the scale’

OKLAHOMA CITY — As Oklahoma prepared to set a new scoring method for statewide reading and math tests, state Superintendent Ryan Walters said he decided his administration would “not put our thumb on the scale in any way.”

He said he instructed his staff at the Oklahoma State Department of Education to give neutral oversight as it gathered committees of teachers over the summer who would recommend new performance expectations, called cut scores, which designate what skills students should demonstrate on state tests to be labeled proficient, advanced or below grade level in reading and math.

The resulting cut scores ultimately lowered the state’s expectations for on-grade-level performance, artificially inflating proficiency rates across Oklahoma. A lack of announcement from the Education Department about the change caused frustrations among educators and parents, who initially believed the rising test scores were the result of true academic improvement.

Although his stated goal was to limit the agency’s influence, the Education Department was anything but hands off during the cut score setting process, known as standard setting, according to a technical report that details what happened.

Walters’ administration and its testing vendor, Cognia, developed benchmark scores that teacher committees used as an optional reference point when setting the final cut scores. Those benchmarks suggested lowering Oklahoma’s standard for reading and math proficiency, the technical report shows.

As Oklahoma public schools approach the next round of annual state testing, Walters is calling for the cut scores to be recalibrated immediately to align with high national standards. He said another state agency involved is responsible.

“This is an issue that really needs to be fixed,” Walters told Oklahoma Voice. “(It) really needs to be fixed moving forward so we don’t find ourselves in this place again. The students of Oklahoma and parents should know in a crystal-clear way that standards and expectations on academic success in Oklahoma will always be high.”

Methods ‘influence cut scores,’ expert says

If the Education Department wished to remain neutral, it shouldn’t have created its own suggested cut scores, said Marianne Perie, a testing expert who has helped Oklahoma and other states set cut scores and design student assessments.

She said the state could have given the teacher committees a blank slate without a target range to set the new expectations.

“It looks like it was a well-done standard setting, but decisions were made,” Perie said. “And decisions were made about how to engage with the educators (in a way) that influences cut scores. Good, bad or indifferent, it influences them.”

Rather than adhere to the state’s suggestions, the teacher committees set expectations for reading performance even lower than what the Education Department proposed.

The committees’ recommended cut scores labeled 51% of third graders as proficient or advanced in reading while the state’s benchmark suggested 44%, according to the technical report. The state’s previous cut scores would have found only 28% of the same group of third graders to be proficient, the report shows.

At the end of the process, the Education Department presented the committees’ exact recommendations, without making any changes to them, to the Commission for Educational Quality and Accountability, a separate state board responsible for giving final approval. CEQA voted in July to approve the cut scores with no adjustments.

Walters said his administration had faced false accusations of exerting an improper level of influence in other elements of Oklahoma’s public education system. That’s why he said he was determined his administration would not tamper with the cut scores for state tests.

“As a state Department of Education, we really oversee a process and allow this to play out,” Walters said. “Not tell the teachers where to set cut scores, not try to push them in a direction, but utilize their expertise, their understanding.”

Response to Walters’ accusation of ‘political interference’

After the teacher committees recommended certain cut points, Walters said it was up to CEQA to “then go in and calibrate those accordingly” with national standards.

State law requires CEQA to vote on the cut scores and conduct a review of the standard setting process.

The CEQA is now preparing to carry out that review and could change the cut scores, if it deems it necessary to do so, said Megan Oftedal, who leads the small state agency that answers to the commission. Commissioners last week discussed hiring a vendor to conduct the cut score study and give them training on the standard setting process.

Walters accused the commission and Gov. Kevin Stitt, who appoints its members, of undermining the integrity of state test results by not completing the review before approving the cut scores. State law doesn’t mandate a specific time the review must take place, only that it be “ongoing.”

Walters said the state Legislature should clarify the law to ensure the review takes place in advance of a commission vote.

The CEQA’s chairperson, Oklahoma Education Secretary Nellie Tayloe Sanders, said last week the board is establishing a training regimen on standard setting “so that no matter what, this never happens again.”

A week after Walters accused the board of “political interference” in test results, Sanders said during the board’s meeting that CEQA is in a “knock-down, drag-out fight” for the future of Oklahoma’s children, and “there’s not one agency that was designed to fight for them more than this one.”

“We also know that our agency has a lot of work to do to get our structure even better so we can fight harder and have even more transparency (and) more truth because we have parents that are desperate for honest answers about whether their kids are ready for life,” Sanders said during the meeting. “And you know what? We’re going to be the agency that can answer that question for them.”

New cut scores are reversal from 2017 goals

The procedure Walters envisioned for Oklahoma’s standard setting is distinctly different from what happened the last time the state had a major cut score change.

In 2017, the Education Department had an expressed goal of raising its cut scores to align with standards found in the National Assessment of Educational Progress. NAEP tests students in all 50 states every two years and compiles the results in the Nation’s Report Card.

The Education Department set this policy goal after convening a task force and conducting town halls across the state, according to the 2017 technical report.

“Because it is critical that Oklahoma have national comparability in its assessments, the (state Education Department) is intent on closing the honesty gap — the discrepancy between proficiency as defined by a state and proficiency as defined by NAEP,” the agency wrote in a 2017 report to the U.S. Department of Education detailing its state plan for public schools.

The honesty gap state officials had sought to avoid appeared again last year when state test results suggested reading and math levels were on the rise, but NAEP found Oklahoma had made no significant progress academically.

The state’s goal of matching national standards was evident as the 2017 teacher committees started outlining their expectations of student performance. The teachers used NAEP’s proficiency score as a reference point while setting Oklahoma’s cut scores.

That process resulted in NAEP-aligned recommendations that needed only limited adjustments from the Education Department after the committees finished their work, said Perie, who worked with Oklahoma during the 2017 standard setting.

Making sweeping changes at the end of the process, like what Walters has demanded, is rare for states to do, Perie said.

“Their priority is usually to respect what the teachers did,” Perie said. “They want to respect the teachers’ expertise and the process they went through. So I have rarely seen major changes after a standard setting.”

Oklahoma Voice (oklahomavoice.com) is an affiliate of States Newsroom, the nation’s largest state-focused nonprofit news organization, supported by grants and donations. Oklahoma Voice provides nonpartisan reporting, and retains full editorial independence.