2016-05-13 / Front Page

Officials: Magazine used faulty statistics

By Wm. Duke Harrington Staff Writer

School officials at both the state and local level say a ranking of Maine high schools recently published by a national magazine is based on data so “fundamentally flawed” the results are practically worthless.

“I don’t know how this happened, why this happened or if this has happened before,” said Anne Gabbianelli, director of communications for the Maine Department of Education, on Monday. “We definitely want it adjusted. There’s no doubt about that.”

“We’re not happy,” RSU 21 Superintendent Katie Hawes said. “We’ve also contacted the magazine that put out the report. We’ve contacted the research agency that did the report. We’ve contacted the Maine Department of Education. We’ve contacted the federal Department of Education.

“The concern for us is the impact this is going to have on our students in terms of scholarships and college recruiting,” Hawes said. “It also can impact our towns, because people really do use school rankings when deciding where to move.”

However, as of Monday afternoon Hawes said, she had yet to receive a reply from the magazine to a complaint filed last week.

At issue is the annual ranking of high schools in each state circulated by U.S. News & World Report. Historically, Kennebunk High School (KHS) has scored in the middle of the top 10 percent, placing as high as fifth, overall. But last year the school slipped off the list completely.

Hawes says school officials presumed there was an error in the data that would be corrected this year. But when the 2016 rankings – based on data for the 2013-2014 school year – came out late last month, her phone started to ring.

“There was a group of middle school and high school people, as well as some parents, who were quite upset,” Hawes said. “We’ve been trying to get to the bottom of it ever since.”

One of those parents is Traci Gere of Kennebunkport, owner of Make It KPT, a creativity store located in Lower Village, who has a child set to enter KHS next year.

Gere holds an MBA from Boston University and, prior to opening her store, spent 25 years doing market research in the high-tech, tourism, and consumer products industries.

“I am passionate about doing good research and using good research to improve businesses and organizations,” she said on Monday. “U.S. News & World Report wants to be the Good Housekeeping seal for high schools, so they have a responsibility to the public to have good quality control mechanisms in place, to be transparent about their methods, and to own up to mistakes when they make them. They should have realized that there were problems with the Maine report last year. Now they should take this opportunity to correct the Maine report, publish a retraction, and update the rankings.”

Hawes was even more forthright.

“It’s sloppy work. It’s not accurate data,” she said, noting that the change in KHS’ fortunes coincides with a switch two years ago in research firms used by U.S. News for its high school rankings – from the American Institute for Research, a Washington, D.C. firm, to North Carolina-based RTI International.

“Up until the switch two years ago, we used to get sent information to confirm our demographic data,” Hawes said. “That doesn’t happen anymore.

“If U.S. News does not have the capacity to rate schools appropriately, they should not put their name behind it,” Hawes said. “We’re not out to take anybody down. We just want the data to be accurate.”

According to Gere, RTI used data from the National Center for Education Statistics (NCES) that was “fundamentally flawed.” The managers of the federal Common Core Database (CCD) of student information appear to have used correct student count data submitted by Maine for each school, Gere said, but then adjusted the counts in an apparent attempt to make the figures match U.S. Census data for towns served by each school district.

According to Charlotte Ellis, data reporting coordinator for the MDOE, the federal department of education requires two reports on student enrollment, an “attendance count” at the school level, and a “resident count” on the district level.

“Census doesn’t understand the differences between the attending and resident counts, and they want them to match – so they make them match, by moving student counts around,” Ellis said. “This ends up creating incorrect data sets.”

While Hawes says early talks with state and federal officials pointed the smoking gun at how the Common Core Database deals with rural school districts, more recent conversations have pulled back on that assumption.

“The source of that error is being investigated.” Gabbianelli, wrote in an April 27 email to Gere.

The update was based on a two–sentence reply to Gabbianelli’s inquiry, given by Adam Honeysett, state and local engagement director at the U.S. Department of Education.

“I have elevated this request to the appropriate individuals at NCES. They are reviewing the issue and will get back to me as soon as possible to steps for a resolution,” he wrote.

The most recent communication from Honeysett, Gabbianelli said, came on May 9.

“Just a note to let you know the review is still in process. For now, we have a note on the CCD web site,” Honeysett wrote. “It states ‘NCES is currently reviewing and updating the 2013-14 school-level enrollment data. Please use caution when using this data.’”

Data deconstructed

While the feds review their datasets, Gere says she has the exact numbers they’re looking for, and has identified the source of the problem. The student count for KHS used in the U.S. News ranking included about 140 students from Arundel who attend Thornton Academy, she said. The report claims KHS has 841 students. In fact, Hawes said, it has 682.

That becomes important because the math and reading test data used by U.S. News to rank high schools is not merely a regurgitation of the SAT tests taken by area students as part of the annual Maine High School Assessments system. Instead, in an effort to more equitably compare poor and affluent communities, RTI takes the raw test scores and weights it according to the number of low- to moderate-income students served by each school district.

“In its ranking methodology, U.S. News adjusts each school’s performance on standardized tests to reflect the percentage of low-income students it serves, as measured by the percentage of students receiving free or reduced lunch,” Gere said. “In other words, the methodology gives schools a higher performance score when their low-income students do better than expected.

“When the data is correct, this practice helps to make sure schools in all communities are measured fairly,” Gere said. “They’re actually doing something that makes sense, the problem is they’re doing it wrong.”

And, just as incorrect high student counts pushed down the free lunch percentage, deflating KHS’ performance score, other Maine schools in the ranking saw the opposite effect. Of particular note, said Gere, is Brewer High School, which the U.S. News ranking credited as having 379 students, even though it actually serves 692.

“Basically, the ranking are wrong across the state. A number of schools where affected by this, in both directions,” Gabbianelli said.

But, Gere says, the flaws don’t stop there.

“For the next step in its methodology, the U.S. News report uses a regression analysis to identify the range of performance scores for each percentage of low-income students, and to then identify the schools that did better than expected,” she explained. “However, the regression analysis used the incorrect performance scores, compounding the problem and resulting in incorrect expected performance ranges.

“As the saying goes, ‘garbage in, garbage out,’” Gere said. “Because the data inputs were wrong, the regression analysis was wrong, as is the identification of the ‘outperforming’ schools that were then selected to be ranked in the report.” Finally, a critical component of the U.S. News ranking was the number of students deemed to be college ready. That score is based on the number of students taking either Advanced Placement (AP) or International Baccalaureate (IB) testing. KHS is one of only three high schools in Maine to offer both, Hawes said, with about 65 percent of all students taking one or the other to prove their worthiness to post-secondary schools.

However, the U.S. News report is only set up to include only AP or IB counts, not both. KHS got a double ding in the datasets because only its IB numbers were used, and then that count was compared to an artificially high enrollment number.

“My suspicion of this being bad research has been confirmed,” Gere says.

Regrets the error

For now, Hawes and KHS Principal Susan Cressey say they are still hoping to convince U.S. News to revise the methodology of its rankings with correct demographic data, and then re-release its 2016 report.

For Gere, however, it all comes down to a much simpler goal.

“I just want people to know our school system is a good school system, on par with any school system in the state as far as being a high performing school,” she said. “I’d encourage parents to reach out to U.S. News and demand that they fix the data that’s out there now for this year, but, at the very least, they should make sure when looking at any kind of rakings like these, that they have confidence in how the rankings were done.”

Meanwhile, Gabbianelli says she has yet to contact U.S. News, preferring to let her federal counterparts take the lead on that.

“We are waiting on guidance from them on how to proceed,” she said. “But if they were to contact the magazine, I think that would get the issue resolved rather quickly.”

Like Hawes, the Post was unable to secure a reply from Bob Morse, the chief data strategist at U.S. News, who wrote an April 19 piece detailing why the rankings changed so dramatically, attributing the new results to a “major change” in methodology.

Ellis says she will continue to lobby with the feds on behalf of Maine schools, including KHS.

“I have sent NCES several examples of these issues,” she said of the errors in this year’s Best High Schools rankings. “While they have stated that there is nothing that they can do about this for the current data being displayed, they assure me that they will do everything possible to ensure that this issue is not repeated for the 2014-2015 data that is being processed right now. I will be following up with NCES on this issue and meeting with them directly in July.

Return to top