2016-07-01 / Front Page

Students correct national school data

By Wm. Duke Harrington Staff Writer

KENNEBUNK — Although still chagrined over mistakes made by a national magazine that knocked Kennebunk off a list of best Maine high schools, RSU 21 has turned the slight into a teachable moment.

For the last few weeks of the school year, students in two of Alan Carp’s Advanced Placement statistics classes at Kennebunk High School corrected the rankings issued this past spring by U.S. News and World Report.

Historically, KHS has scored in the middle of the top 10 percent on the annual listing, placing as high as fifth overall. But this year it slipped off the list completely.

According to Superintendent Katie Hawes, it was determined after much back and forth with state and federal education officials that data used by the magazine’s contracted research firm to create a “performance index” for each school was subject to bureaucratic bungling.

The research firm, North Carolina based RTI International, tried to weigh standardized testing at the high school level – in Maine’s case, SAT test scores – by factoring in the number of students from low- and moderate-income homes. However, while test data sent by Maine to the feds was accurate, it got conflated in the federal Common Core Database with census numbers. But because many students who live in Arundel attend high schools other than KHS, the figures did not match.

“The two numbers didn’t match, so they just moved kids around,” Hawes said, noting that while KHS actually has 667 students, RTI calculated its numbers based on the census figure of 849 high-school aged children.

Using the U.S. News ranking methodology and the correct enrollment numbers, Carp’s students calculated a performance index that placed KHS 10th among the 101 schools with reportable test scores from the 2013-2014 school year, used by RTI.

For schools that ranked in the Top 10 percent in a state, or had scores within a set standard deviation from the top score, U.S. News then had RTI rank the schools that made the cut by calculating a “college readiness” level using results of Advanced Placement or International Baccalaureate testing. The “or” is important because KHS is one of three high schools in Maine to offer both tests, but U.S. News only allowed RTI to use one or the other.

“They [U.S. News] have admitted their numbers are flawed and that they recognize the problem and are committed to having the correct numbers next year,” Hawes said. “But they don’t seem to understand how they could possibly use both AP and IB in terms of their algorithm. So, they’re pretty firm on not counting both tests and just picking one randomly per school.”

“They actually acknowledge in the appendix to their report that this is not fair, but then they just left it at that,” Carp said.

According to Carp, his students calculated that had RTI used the number of students taking and passing an IB test instead of AP, KHS would have landed in 15th place among all Maine high schools. If RTI had found a way to count both tests, as Carp’s students did, KHS would have ranked sixth.

“In either case, we would have been eligible for a silver medal, or at worst a bronze,” Carp said, in a June 21 report to the school board, referring to the final “awards” given out by U.S. News.

“The have some very sophisticated software, we did all of our work using a TI-84 calculator,” Carp said. “But in the end, our performance substantially increased when we used the correct reporting data. There are some questions about their methodology, but even if it is sound, the adage applies, when you put garbage in, you get garbage out. They really ought to try getting their numbers together a little bit better next year.

“I think if our AP stats class had had just a little more time, they probably cold have come up with a better methodology,” Carp said. “RTI is using a MySQL database and so using numbers that are easy to import from one spreadsheet to another and easy to get a hold of.”

“We’ve made it clear if they are to remain in the business of ranking schools, they need to have accurate data, because they are impacting our kids,” Hawes said.

In the meantime, she added, students will have copies of the corrected rankings placed in their transcripts, in hopes of mitigating whatever effects the U.S. New faulty report might have on college admissions.

“We also will feed this back to U.S. News and World Report,” Hawes said. “They’ve promised the report will be corrected for next year. What we are trying to do now is to get them to go back and correct the rankings for this year’s report,” she said.

Meanwhile, KHS Principal Susan Cressey said Newsweek, which is preparing a similar high school ranking, called her this past week to confirm its data. That courtesy call is something U.S. News used to do, she said, before the change in research firms this past year.

It was one Kennebunkport parent, Traci Gere, who helped RSU 21 tracked down the hands behind the report.

“My sense is the reason RTI got the contract is that they cut cost by taking out any human involvement in actually reality-checking the data,” Gere said at the June 21 school board meeting.

Simple logic, she said, should have told RTI their results were suspect, given past reports. For example, Brewer High School, which got tagged in the common core database for a lot fewer students than it actually has – the opposite problem that befell KHS – climbed as high in the ranking as KHS plummeted. Like KHS, the Maine School of Science and Mathematics charter school in Limestone, perennial holder of the top podium spot in the U.S. News ranking, also fell off the list this year.

Both Hawes and Gere have said that U.S. News’ chief data strategist, Robert Morse, at first “completely denied” any problem with the data.

In a May 11 letter to Hawes, Morse laid any fault with those who collect and compile the data sets it uses.

“We must rely on states to send accurate data to the federal government, and for the federal government to report those data accurately,” he wrote.

And, because the federal common core database has been off because of state reporting being conflated with census data, “A quality-checking step that compares enrollment across years would not have caught that,” Morse claimed.

The Post twice emailed Morse for a comment, but did not receive a reply.

But even after Morse finally acknowledging an issue, Gere says, RTI continued to throw up their hands on possible solutions, including bothering to verify data it imported out of federal reporting databases.

“What they’ve been saying is, ‘We have no way of doing that without putting more person power into it,’” Gere said. “That’s the fundamental issue. I think more publicity on this would help to put the pressure on them. Having more negative press certainly would help to motivate them.”

More media scrutiny might not be in RTI’s best interest, given that the majority of its work is actually in compiling statistics for the health care industry.

“If they’re a major researcher for the health care industry and they’re messing up the statistics for ranking a high school, that’s kind of concerning,” school board member Erin Nadeau said.

Staff Writer Duke Harrington can be reached at news@kennebunkpost.com.

Return to top