Two Hospital Safety Reports, Different Rankings

The Harlem Hospital Center in Manhattan received 20 points out of 100 in Consumer Report's hospital ratings. The New York City Health and Hospitals Corporation disputed the rating, claiming the report used outdated information. Flickr/David.R.Carroll

When deciding where to get a hospital treatment, many people rely on personal recommendations or hearsay about a particular hospital’s reputation.  Until recently, comparing the safety of various hospitals required wading through large amounts of data from disparate sources.

Two new reports, by Consumer Reports and the Leapfrog Group, have now organized that information and rated hospitals around the country. However, there is  still some digging to do. In their findings for New York state, the reports gave quite different ratings for individual hospitals.

Consumer Reports released its first-ever hospital safety report last week. It rates 1,159 hospitals in 44 states on a scale from 1-100, 1 being worst and 100 being the best. Only 31 out of 117 New York State hospitals that received a score — many did not have enough information to be ranked — scored a 50 or above.

The director of the Consumer Reports Health Ratings Center, John Santa M.D., explained the report as a type of advocacy. “We’re doing this in part because 12 years ago the Institute of Medicine made the same suggestions that we’re making. This kind of information needs to be publicly reported, these problems need to be solved, but the hospitals still haven’t done it,” he said.

It’s kind of like having two different book reviews. Different reviewers have different interests.

In June, Leapfrog Group, a Washington D.C.-based hospital safety advocacy group, created its own safety rankings, but unlike Consumer Reports’ numerical scores, Leapfrog used A, B and C letter grades, similar to New York City’s restaurant grading system that only has three grades.

Different Results, Different Data

When you compare the two reports, you might be baffled. While Consumer Reports gave Cayuga Medical Center in Ithaca a 61, the highest rating in New York state, Leapfrog gave the hospital its lowest safety grade, a C.

“We each looked at some different measures,” said Leah Binder, CEO of the Leapfrog Group. “Particularly, they [Consumer Reports] looked at patient satisfaction measures and rates of CT scans. We didn’t look at those things. We looked at injuries, errors and accidents only. It’s kind of like having two different book reviews. Different reviewers have different interests.”

Consumer Reports used pre-existing data from multiple sources, including some of Leapfrog’s, in six sets of criteria to create a composite ranking for each hospital.

The criteria includes rates of infection (data from the Leapfrog Group and state health departments); patient readmission (data from the Centers for Medicare and Medicaid Services); communication between staff and patients (data based on a survey by the Hospital Consumer Assessment of Healthcare Providers and Systems); CT scanning and use of electronic health records (data from the American Hospital Association and Centers for Medicare and Medicaid Services); and complications and mortality (data from the Agency for Health Quality Research).

“We intentionally weighted four variables [infection, readmission, communication and CT scans] the same, they were all 20 percent,” said Santa. “We looked at a variety of options in terms of weighting things differently and then it didn’t really change their rating much. The measures hospitals were most likely to do well in were hospital infections and CT scanning measures, so those tended to bring hospitals up. The toughest measures were communication and discharge.”

Bellevue Hospital in Manhattan received an "A" letter grade from Leapfrog, but a 40 out of 100 from Consumer Reports. Flickr/edenpictures

In addition to the different grading systems, Leapfrog’s ratings differed from Consumer Reports’ in two fundamental ways.

First, as Binder said, Leapfrog looked at somewhat narrower criteria that didn’t include patient experience, underutilization of electronic record-keeping or overuse of CT scans. Second, whereas Consumer Reports created its rankings based exclusively on external data, Leapfrog actually sent out surveys to 2,652 hospitals (1,100 of which responded) and combined those results with data from the Agency for Healthcare Research and Quality, the Centers for Disease Control and Prevention and the Centers for Medicare and Medicaid Services.

The combined data was used to create a composite score, which in turn was used to create the letter grades, but wasn’t shared with the public.

“We do have a composite numerical score, which we shared with the hospitals. It’s based on up to 26 measures, and we weighted those measures under the guidance of our blue ribbon expert panel,” said Binder. “The board of Leapfrog then assigned grades. The A hospitals are safer than the B hospitals and the B hospitals are safer than the C hospitals and so on. What we didn’t want is a meaningless horse race over which hospital has one digit higher than another. We wanted a letter grade.”

Binder said certain variables were more heavily weighted than others. For instance, high rates of  dangerous central line bloodstream infections would seriously hurt a hospital’s overall score.

Criticizing Famous Hospitals

While their findings are different, both reports gave fairly low rankings to a number of hospitals with excellent reputations. For instance, Consumer Reports gave Bellevue Hospital a 40 out of 100 for what it said were bad rates of bloodstream infections and weak communication to patients about drug information. On the other hand, Leapfrog gave Bellevue an A.

Both reports gave the well-regarded New York Presbyterian Hospital low marks: Leapfrog gave it a C and Consumer Reports gave it a 32 out of 100.

“Many of our best hospitals do have a lot on their plate. They do research, they teach, they take care of a lot of uninsured people, they’re huge employers and they want to treat their employees well, but I think that means a lot of priorities are competing,” said Santa. He added, “This is the kind of stuff where the CEO of a hospital has to say ‘we’re going to do this.'”

Both reports have received criticism from certain quarters of the medical community.

The Medical Community Responds

The New York City Health and Hospitals Corporation released a statement claiming Consumer Reports’ ratings used “flawed data” that “failed to represent an accurate picture of excellent patient care and significant improvements achieved by these hospitals.” The Health and Hospitals Corporation said that Harlem Hospital Center, ranked lowest in the state by Consumer Reports, had actually replaced its CT scanners in 2009, which if acknowledged in the report, would have bumped the hospital’s rating by 20 points.

Santa said he found that ironic.

“Do we acknowledge that some of the data is older than we’d like it to be? Yes, but that’s up to the hospitals. It frustrated me to hear hospitals critical of something they have lobbied for,” said Santa, explaining that hospitals have often delayed giving updated information to the agencies that Consumer Reports consulted.

He explained, “What they will argue is that ‘this [updating data] takes a lot of time and money to do and we need to make sure we get it right. If you want this in six months versus a year, it’s going to cost us twice as much.’ So the legislature says, ‘okay, 12 months.’ And when the data comes out they say ‘this is twice as old as it should be!'”

The American Hospital Association released a statement saying the Leapfrog report “has supported several good quality measures, but many of the measures Leapfrog uses to grade hospitals are flawed, and they do not accurately portray a picture of the safety efforts made by hospitals.”

Binder disagreed.

“I think there are hospitals in New York City that will say ‘we have a great reputation and we get accolades from other organizations, so why are these two organizations [Leapfrog and Consumer Reports] giving me a lower grade?’ For both of us, we looked at safety, we looked at errors, injuries and accidents, things you don’t want to have happen. Safety should be the first, but not the only consideration, when choosing a hospital,” said Binder.

She added, “Sometimes hospitals will say, ‘well we serve a very sick population and they need to take into account that our population is different from others.’ That may be true for some things, but it’s not true for errors and injuries. What I think is remarkable is that some of the top hospitals, like Bellevue in New York City, have the toughest populations.”

Of the 151 New York state hospitals that Leapfrog ranked, 41 are located in New York City.  Of the 117 hospitals that Consumer Reports ranked in New York state, 36 are in New York City. Leapfrog’s report is available free online but Consumer Reports’ requires a subscription.

©2022 WNET. All Rights Reserved. 825 Eighth Avenue, New York, NY 10019

WNET is a 501(c)(3) nonprofit organization. Tax ID: 26-2810489