Administrative coding of medical diagnoses has become mandatory in French hospitals in order to perform epidemiological studies and to calculate medical reimbursement costs. Most databases are used by hospital administrators, according to the local system for hospital funding, which is derived from the Diagnosis-Related Group (DRG) in the US 1. In the French national system, the medical diagnoses are coded by the physician treating the patient, collected by the Department of Medical Information (DMI) in the hospital, and transmitted to a national service that determines the hospital costs to be reimbursed by the health care insurance system 2. As in other countries 34, French intensive care unit (ICU) physicians have established a number of databases collating information from multiple centers in order to perform epidemiological studies and/or benchmarking 5. The medical information in these databases, which share either a financial or a scientific objective, must be reliable. Most databases use a diagnostic thesaurus 6 extracted from the International Statistical Classification of Diseases and Related Health Problems (ICD) 7. The 10th revision of this classification, the ICD-10, is used in France in the national funding database 2 and in the two main ICU databases used for clinical research 58. The same revision is used in these databases to simplify data collection and comparisons.

In France, as in most Western countries, patients' medical records are now computerized in order to improve activity assessment. As diagnosis coding is a fastidious and time-consuming process, several groups have begun to develop automatic coding systems based on data available in hospital information systems 9. However, preliminary results suggest that diagnosis coding in economic databases is inconsistent between physicians and administrative personnel 1011.

The Outcomerea database was set up in 1998 in order to perform clinical research on ICU cohorts. It contains a pre-established set of physiological data, clinical diagnoses, and therapeutic procedures collected every day during a patient's ICU stay. It receives data from 12 French ICUs 5. Each year, the participating ICUs must collect data during the complete ICU stay of at least 50 patients staying for more than two consecutive days. Good reliability of physiological data designed to calculate severity scores has been documented following biannual audits 12. The diagnoses are coded according to the guidelines published by the French Society of Intensive Care Medicine (SRLF) and the French Society of Anesthesiology and Intensive Care Medicine (SFAR) in 1999 13. Large cohorts based on coded diagnoses are regularly published and used to document epidemiological trends and the outcome of acute diseases such as sepsis 141516. However, the results of these studies are regularly challenged 17.

Our hypothesis was that the poor reproducibility of medical diagnoses observed in administrative databases is also found in research databases. The present study tested the reliability of coding of medical diagnoses, and specifically the diagnoses of septicemia and hemodynamic shock, in the Outcomerea database.

In this study investigating the reliability of diagnostic coding by physicians trained to collect data in ICU patients, we observed that coding by an external physician after examination of a patient's medical record did not modify the total number of diagnoses made for the patient. Agreement between coders was most often moderate regarding the choice of codes. This was also true for the principal diagnosis used for the DRG system as well as for the codes used to indicate septicemia and shock.

Hospital databases are used to estimate reimbursement costs of medical care, to determine human resources for clinical units, or to perform epidemiological studies. Accurate coding of diagnoses is a cornerstone of these three objectives. Quality analyses of coding have been performed mainly in the area of resource allocation. At the hospital level, these analyses have shown that coding is poorly reliable. It has been estimated that external coding in European countries and the US would modify 32% to 42% of diagnoses 10. The quality control system of Medicare showed that reliability was as poor between external coders as between physicians and hospital administrators 11. Finally, the use of trained experts to carry out coding increases the number of diagnoses but the level of agreement between experts is less than 70%. In American ICUs, the codes describe the reason for admission in less than 50% of cases, devaluing hospitals with ICUs and making the administrative database nonapplicable for quality-of-care assessment 21.

Coding reliability appears to be even worse in medical ICUs. In ICU patients, coding errors concern as many as 46% of cases, with a resultant financial loss of 18.4% 22. Coding of therapeutic procedures plays an important role in most systems derived from North-American DRGs. This accounts for the better accuracy of DRGs in elective surgical patients 23. Accordingly, in contrast to diagnostic coding, the French network CUB-Rea of 35 ICUs around Paris showed that the reliability of coding of severity scores and therapeutic items was acceptable 8. The poor reliability we found for diagnoses could be due to the frequent combination of multiple diseases and organ failure in a single patient, which plays a cumulative role in resource utilization, mortality, and secondary morbidity 24. ICD codes are often used in large epidemiological studies as a surrogate for the cause of ICU admission 141516. However, the use of such codes in classifying ICU patients has been widely debated and other tools for classifying ICU admissions have been proposed 325. Thus, coding requires complex and precise rules 13, especially in the ICU setting, to select diagnoses with objectivity. This can be obtained through an automated algorithm using an expert system 9. We have recently designed software that selects the codes from the patient's electronic record, based on linguistic treatment exploring inductive mechanisms and extracting concepts rather than words from textual medical reports 26. Testing of this software is currently under way in a pilot cohort of patients 26.

We chose to perform this study with real data from patients admitted to ICUs corresponding to French quality standards 19 and sharing a routine practice in database exploitation 5. External coding was performed by two independent experts who had been trained in coding in a similar way and had similar experience in ICU practice.

Despite these precautions, our study has several limitations due to the small sample size, the methods used, and the fact that codes were determined by physicians rather than trained administrative coders. First, external coding was performed after the ICU stay by practitioners following a post hoc chart review. It is more likely that the initial diagnosis made by the physician treating the patient was accurate and that the chart review may not have correctly captured the appropriate diagnosis and is therefore inaccurate. This could also account for the poor reliability between the two external coders. This suggests that neither a gold standard nor an expertise for diagnosis coding exists in the ICU. Second, the external coders worked in hospitals with different case mixes and could have had different areas of scientific interest. Thus, their method of coding could have been influenced by their professional expertise. We attempted to control for this factor by training them to code according to specific guidelines. However, these guidelines, even if they should be considered as the French 'gold standard', include the 662 codes considered to be the most common, and this number might be too large to use with good reliability. Third, we did not control the quality of the medical records, corresponding to 'real-life' recording in France. However, all the summaries corresponded to the quality criteria required for French hospital certification procedures 27. Again, this does not account for the poor reliability between the external coders as they worked on the same source documents. Finally, the reliability of coding septicemia and shock requires further assessment, particularly to optimally interpret both previous and future cohort studies using administrative data.

Using a quality-assured database designed for clinical research, we observed that coding of medical diagnoses was unreliable in ICU patients despite specific training of physicians. From an economic point of view, this could explain the poor results of the DRG system in ICU patients which have been previously published. This lack of reliability could limit the interpretation of epidemiological and clinical research programs based on diagnoses such as sepsis. The reliability of diagnoses should be tested in other research databases, and systems of automatic computerized data collection 9 should be analyzed. The results of our study will be used as a comparator in a forthcoming investigation of automatic coding in ICU patients.

• Coding diagnoses is necessary to categorize patients in epidemiological studies.

• Multiple symptoms or diseases are characteristic of intensive care unit (ICU) patients.

• The International Statistical Classification of Diseases and Related Health Problems provides a profusion of medical codes.

• The selection of codes by ICU physicians is unreliable. This weakens the conclusions of cohort studies using diagnosis as an inclusion criterion.

DMI = Department of Medical Information; DRG = Diagnosis-Related Group; ICD = International Statistical Classification of Diseases and Related Health Problems; ICD-10 = International Statistical Classification of Diseases and Related Health Problems: Tenth Revision; ICU = intensive care unit; SD = standard deviation; SFAR = French Society of Anesthesiology and Intensive Care Medicine; SRLF = French Society of Intensive Care Medicine.

The authors declare that they have no competing interests.

BM, DN, and J-FT participated in the conception and design of the study and in the writing of the article. AV participated in the writing of the article. All of the authors participated in the acquisition of data, analysis and interpretation of data, critical revision of the manuscript for intellectual content, and approval of version to be published. All authors read and approved the final manuscript.