Surgery News - May 2008 - (Page 20)

PRACTICE TRENDS SURGERY NEWS • M AY 2 0 0 8 Stats Vary for Same Patients Risk Data • from page 1 Data that are collected by a trained nurse reviewer and validated by an external reviewer. Access to almost any information on inpatient and outpatient medical records, as well as direct access to patients if questions arise. Stepwise logistic regression analysis. UHC’s membership of 101 academic medical centers and 170 of their affiliated hospitals includes about 90% of nonprofit academic medical centers. The organization uses the Centers for Medicare and Medicaid Services’ (CMS) system for classifying severity of illness, the All Patient Refined Diagnosis Related Groups, which is based on information in a patient’s medical record. Prior to this year, when CMS began specifying whether a comorbidity was present on admission, it had been impossible to distinguish a comorbidity from a complication. Additional modifiers further refine a patient’s severity of illness and risk of death. UHC also develops a mainstream population, which excludes specialty hospitals and those with fewer than 150 beds, and performs a logistic regression analysis for each individual diagnosis-related group in that population. The results of the analysis are applied to the entire patient population. “From our point of view, [UHC’s methodology] is somewhat more complex than the NSQIP methodology,” said Dr. Steinberg, an ACS Fellow. There were significant differences in risk-adjusted mortality, the incidence of comorbidities, and the rate of outcomes between NSQIP and UHC. According to NSQIP, Ohio State’s ratio of observed to expected mortality was 0.76, placing it in the top quartile. But UHC calculated a ratio of 1.45, putting it in the bottom quartile. A ratio less than 1 indicates that the hospital is performing better than expected given the complexity of its patient population and surgical case complexity. Overall, NSQIP tallied significantly few- er comorbidities per person after risk adjustment than did UHC (1.38 vs. 2.85). These included significantly discordant results between NSQIP and UHC for the rates of hypertension (47% vs. 43%, respectively) and diabetes (11% vs. 14%), as well as cardiac (10% vs. 12%) and pulmonary comorbidities (18% vs. 23%). The investigators calculated discordance as the number of cases in which a comorbidity or outcome differed between the two programs, divided by the total number of cases (120). Significant discordance also occurred between NSQIP and UHC results for all complications combined (28% vs. 11%). “Clearly, not all risk adjustment is the same. Both NSQIP and the University HealthSystem Consortium risk adjustment of data cannot be kept at our institution because they are so different,” Dr. Steinberg said. “From my point of view, NSQIP has more face validity than the UHC system, not just because we did better [on NSQIP] but because it’s something that I can understand, whereas I have great difficulty in being able to understand the UHC process.” Several audience members thought that the results illustrate the problems with using retrospective analyses of administrative data sets to evaluate outcomes, rather than prospective databases that are maintained by a trained and dedicated nurse, as is the case with NSQIP. The difference in the ratio of observed to expected mortality between these quality improvement programs could be attributable to a number of factors: Problems with documentation and coding (although this is unlikely, according to Dr. Steinberg). Differences in the participation of medical centers in each quality improvement program (although 56 centers participate in both NSQIP and UHC). Possible incorrect classification—for example, UHC defines a service line by ICD9 codes, not whether a patient was ever actually on a service. Differences in the programs’ risk-adjustment methodologies. Dr. Steinberg advised his colleagues to note each program’s strengths and weaknesses and to engage in choosing the programs used by their institutions. Database Test Finds Flaws in Mortality Rate Estimates B Y M A R K S. L E S N E Y Else vier Global Medical Ne ws volving large congenital heart surgery Deliberatea error-seeding experiments indatabase showed that even small levels of miscoding can substantially change mortality estimates. This was especially true for miscoding of procedure type and for operations with mortality below 10%. Such error-driven variations in mortality estimates are especially troubling in an era when registry databases increasingly are expected to form the foundation of risk analysis for various operations, and might even be used to evaluate the doctors and the institutions that perform these operations. In addition, errors in databases are not uncommon. One recent study of a carefully audited California database reported at least one diagnostically relevant error in 63% of patient records, according to Dr. Steve Gallivan and colleagues in the March issue of the European Journal of Cardio-Thoracic Surgery. Computer simulation techniques were used to create realistic analysis scenarios based upon data from the Toronto Cardiovascular Surgery Database for Congenital Heart Surgery, which contains information on nearly 18,000 operations. This includes outcomes for 132 operation types from which 30 marker operations were chosen, each of which had been reported at least 100 times in the database and had nonzero mortality. Four thought experiments were performed using the data on the marker operations. In the first experiment, the only errors introduced were random miscoding of outcomes with three scenarios: error rates of 1%, 3%, and 5%. Each of these scenarios showed considerable changes in mortality rates, especially when the true mortality rate was small (Eur. J. CardioThorac. Surg. 2008;33:334-40). In the second experiment, the only errors introduced comprised random omission of data at rates of 0%, 10%, or 20%, with the miscoding of outcomes fixed at 1%; these scenarios showed that random omission of data had no discernible effect on inaccuracies in mortality rate estimates. This was predicted by mathematical modeling. In the third thought experiment, errors introduced comprised random outcome miscoding at different rates for deaths and survivors; a progressive increase in estimation error was seen when mortality rates fell, “and the scale of such overestimation is alarming for mortality rates below 10%,” according to Dr. Gallivan of University College, London, and his international colleagues. The final thought experiment regarded introduced errors from miscoding of the operation type with no data omission or outcome miscoding. Three operations illustrated the potential dangers of such errors: ASD/secundum repair (recorded mortality rate 0.2%), TGA repair/arterial switch (mortality rate 9.0%), and the Norwood operation (mortality rate 36.3%). The assumption was made that each operation had an equal probability of being miscoded as one of the other two. As predicted from mathematical modeling, as the miscoding rate increased, the gross mortality rate for ASD/secundum repair became increasingly overestimated, the rate for TGA repair remained relatively the same, and the rate for Norwoods became increasingly underestimated, according to the authors. “The results reported here sound a loud note of caution and perhaps it is time for a reappraisal of the clinical database structure. There is often a somewhat misplaced belief that if one gathers a lot of data, then, if analyzed cleverly enough, they will reveal a new truth. This view is wrongheaded; the reality is that the more data items that are collected, the more errors occur,” the authors stated. “The results we describe are alarming. Even moderate levels of error can lead to substantial inaccuracy in estimates of mortality rates and in some circumstances these inaccuracies can be gross, especially at the low mortality rates that are now prevalent in cardiothoracic surgery,” they concluded. http://www.facs.org/cancer/coc/comingtogether2008.html http://www.facs.org/cancer/coc/comingtogether2008.html

Table of Contents Feed for the Digital Edition of Surgery News - May 2008

Surgery News - May 2008 Contents New Lung Approach Speeds Extubation Innovative GI Procedures May Improve Diabetes Quality Programs Differ on Risk Data Crystal Ball Medical Modeling Ventricular Valve Taking Stock

Surgery News - May 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.