|
|
Background Little is known about the magnitude of deficits in the quality of care delivered to children, since comprehensive studies have been lacking.
Methods We assessed the extent to which care processes recommended for pediatric outpatients are delivered. Quality indicators were developed with the use of the RAND–UCLA modified Delphi method. Parents of 1536 children who were randomly selected from 12 metropolitan areas provided written informed consent to obtain medical records from all providers who had seen the children during the 2-year period before the date of study recruitment. Trained nurses abstracted these medical records. Composite quality scores were calculated by dividing the number of times indicated care was documented as having been ordered or delivered by the number of times a care process was indicated.
Results On average, according to data in the medical records, children in the study received 46.5% (95% confidence interval [CI], 44.5 to 48.4) of the indicated care. They received 67.6% (95% CI, 63.9 to 71.3) of the indicated care for acute medical problems, 53.4% (95% CI, 50.0 to 56.8) of the indicated care for chronic medical conditions, and 40.7% (95% CI, 38.1 to 43.4) of the indicated preventive care. Quality varied according to the clinical area, with the rate of adherence to indicated care ranging from 92.0% (95% CI, 89.9 to 94.1) for upper respiratory tract infections to 34.5% (95% CI, 31.0 to 37.9) for preventive services for adolescents.
Conclusions Deficits in the quality of care provided to children appear to be similar in magnitude to those previously reported for adults. Strategies to reduce these apparent deficits are needed.
Research and policy related to children have focused on expanding eligibility for public insurance programs, but expanding access to a system that does not deliver necessary services will not result in optimal outcomes. Deficits in the delivery of care must be identified if appropriate strategies to close the gaps are to be developed and implemented.
In an attempt to address the limitations of previously published studies of the quality of care provided to children, we developed a comprehensive method for evaluating quality on the basis of information in medical records. We recruited a nationally representative sample of children by means of collaboration with the Community Tracking Study (CTS), conducted by the Center for Studying Health System Change.12 We sought to answer five questions. First, how good is the quality of care for children overall? Second, does quality vary according to the type of care (care for acute or chronic medical problems or preventive care)? Third, does quality vary across the continuum of care functions (screening, diagnosis, treatment, and follow-up)? Fourth, does quality vary according to the mode of care (history taking, physical examination, laboratory testing or radiography, medication, immunization, encounter, education, or counseling)? Fifth, does quality vary according to the type of clinical area?
Methods
Development of Quality Indicators
Members of the RAND staff reviewed established national guidelines and the medical literature and developed indicators of quality for the continuum of care functions — including screening, diagnosis, treatment, and follow-up — for the most common childhood health care needs.13 A nine-member expert panel assessed the validity of the proposed indicators, using the RAND–UCLA modified Delphi method.14 We solicited nominations for panelists from the American Academy of Pediatrics, the American Academy of Family Physicians, the Ambulatory Pediatric Association, and the Society for Adolescent Medicine. The panel consisted of four general pediatricians, two family practitioners, two specialists in adolescent medicine, and one specialist in pediatric infectious diseases (see Appendix 1 of the Supplementary Appendix, available with the full text of this article at www.nejm.org).
Panelists rated indicators on a 9-point scale, with a score of 1 denoting not valid and a score of 9, very valid. Indicators with a median validity score of 7 or higher were included in the study. Previous work has shown this method of selecting indicators to be reliable and to have content, construct, and predictive validity in other applications.15,16,17 The criteria for selecting the clinical areas, literature reviews, procedures followed by the panel, and final indicators have been reported elsewhere.18 (For more details, see the Technical Appendix in the Supplementary Appendix.) Table 1 provides brief descriptions and classifications for a sample of the 175 indicators that we selected for use from the original 242 (Appendix 2 in the Supplementary Appendix). The indicators were categorized according to type of care (preventive care, care for acute conditions, or care for chronic conditions), function of care (care serving as screening, diagnosis, treatment, or follow-up), mode of care (encounter, medication, immunization, physical examination, or laboratory testing or radiography), and type of clinical area (e.g., acne). We excluded indicators associated with modes of care for which the adequacy of documentation may be a concern (i.e., history taking, counseling, and education).
|
The CTS recruited households in 12 metropolitan areas (Boston; Cleveland; Greenville, SC; Indianapolis; Lansing, MI; Little Rock, AR; Miami; Newark, NJ; Orange County, CA; Phoenix, AZ; Seattle; and Syracuse, NY), using a random-digit-dial telephone survey. The communities were randomly selected to represent metropolitan areas with a population of more than 200,000. Between October 1998 and August 2000, we telephoned participating households that had a child enrolled in the CTS. We interviewed the adult in the household who was most familiar with the child's medical history to obtain demographic information and both oral and written informed consent to request copies of the child's medical records from all providers seen during the 2-year period before the date of the interview. The results are based on care delivered between October 1996 and August 2000.
Response Rates
The study was approved by the RAND Human Subjects Protection Committee. We began with an initial sample of 4096 children who had participated in the CTS, for which the response rate was 62.5%. Of these children, 398 (9.7%) were deemed ineligible, primarily because their families had moved. We interviewed the parents of 2851 of the 3698 eligible children (77.1%) and excluded 77 (2.7%) because they had not seen a health care provider during the prior 2 years. Among the 2774 children who had at least one visit to a provider, parents provided oral informed consent to obtain records for 2415 children (87.1%) and written informed consent to do so for 1813 children (65.4%). We received 2264 of the 3597 medical records (62.9%) for which we had written informed consent. We obtained at least one medical record for 1536 of the 1813 children for whom we had written informed consent (84.7%). Children for whom we obtained at least one medical record (1553 of the 3698 eligible children [42.0%]) were included in the analyses.
Abstracting of Charts
All charts were sent to RAND for abstraction. We developed computer-assisted abstraction software on a Visual Basic platform (version 6.0, Microsoft). The software allowed the abstraction to be tailored to the record being reviewed and permitted checks of the range and consistency of the data, calculations (e.g., determination of the presence of fever), and classifications (e.g., determination of the drug class) during abstraction. Seven trained registered nurses abstracted the medical records. Charts were abstracted separately for each health care provider of each child.
To assess interrater reliability, we re-abstracted charts from a randomly selected 10.4% of participants (160 participants). Average reliability, indicated by the kappa statistic, ranged from substantial to almost perfect19 at three levels: the presence or absence of a given clinical area (
=0.89; 95% confidence interval [CI], 0.86 to 0.91), the child's eligibility for the care represented by a given indicator (=0.95; 95% CI, 0.94 to 0.96), and the participant's score for that indicator (=0.83; 95% CI, 0.80 to 0.85).
Individual and Composite Scoring of Indicators
We determined whether each child was eligible for the care represented by each indicator (whether indicator eligibility was met) using data collected from the abstracted charts, such as age, diagnosis, and presenting symptoms. For children who were eligible, we determined whether the required care had been received on the basis of documentation in the chart that included orders, prescriptions, patterns of visits, visit notes, discharge abstracts, and correspondence.
Each indicator was scored at one of three levels — that of the child, the child–provider dyad, or the episode of care — depending on the care process being evaluated. The scoring level determined the number of times indicator eligibility was met (which was the denominator in the calculation of the composite score). Child-level indicators were given a score of "pass" if any of the child's health care providers delivered the indicated care (e.g., immunizations). Indicators scored at the level of the child–provider dyad (e.g., limiting of the use of nasal decongestants to 4 days) were scored separately for each provider who saw the child. Episode-level indicators generally required coordination of care provided by multiple providers (e.g., hearing evaluation in patients with persistent bilateral otitis media).
Composite scores were constructed with the use of an opportunity-score approach.20 Specifically, they were calculated by dividing the total number of times the indicated care was noted in the record as having been ordered or delivered by the total number of times indicator eligibility was met.
Statistical Analysis
Because all children who were eligible for the study had participated in the CTS, we had a rich set of variables with which to assess nonresponse. We estimated the relationship between individual characteristics of the children (age, race, income, parent-reported level of use of physicians and hospitals, insurance status, and health status) and participation in the current study, using logistic-regression analysis. Although we adjusted the regression model for all individual characteristics, only race and health status were predictive of participation. Blacks and other nonwhites were less likely to participate than whites (P<0.001), and children in excellent or very good health were less likely to participate than those in good, fair, or poor health (P=0.001) (Appendix 3 in the Supplementary Appendix). We used the results of the logistic-regression analysis to create weights to adjust for nonresponse and to make the respondents representative of the study population.
All means and standard errors incorporate adjustments for sampled population and nonresponse, as well as for the clustering of eligibility events for each patient (with the use of generalized estimating equations). The survey procedures in SAS software, version 9.2, were used to perform these analyses. P values of less than 0.05 were considered to indicate statistical significance. We also conducted a number of sensitivity analyses to assess threats to the validity of our findings by recalculating composite scores for different subgroups of indicators and using t-tests to determine whether the differences between the original and recalculated results were significant.
Results
Characteristics of Participants
Detailed results of the analysis comparing the 3698 children who had participated in the CTS and who were eligible for this study with various nonrespondent subgroups are given in Appendix 3 in the Supplementary Appendix. Study participants were more likely than the average child in the United States to be white and to have private insurance (Table 2) but were less likely to live in households with annual incomes of $50,000 or more.21 We received medical records from an average of 2 providers per child (range, 1 to 10). On average, children were eligible 8 times (range, 1 to 44) for care represented by quality indicators.
|
Table 3, Table 4, and Table 5 show the number of indicators included in each composite score, the number of children eligible for the care represented by one or more indicators within each category, the total number of times indicator eligibility was met, and the weighted mean percentage of indicated care received (adherence rate and 95% confidence interval). On average, according to the data documented in the charts, children received 46.5% (95% CI, 44.5 to 48.4) of the indicated care (Table 3). They received 67.6% (95% CI, 63.9 to 71.3) of the indicated care for acute medical problems, 53.4% (95% CI, 50.0 to 56.8) of the indicated care for chronic medical conditions, and 40.7% (95% CI, 38.1 to 43.4) of the indicated preventive care. Adherence rates for the continuum of care functions ranged from 37.8% (95% CI, 34.6 to 41.0) of the indicated screening processes to 65.9% (95% CI, 62.4 to 69.4) of indicated treatment processes (Table 3).
|
|
|
Problems with Quality of Care
The rates of documented adherence were lower for indicators characterized by underuse of services (42.4% [95% CI, 40.2 to 44.6]) than for those characterized by overuse (73.1% [95% CI, 64.6 to 81.6]) or misuse (90.2% [95% CI, 87.8 to 92.5]).
Variation in Clinical Areas
In the group of 11 clinical areas for which data for at least 50 children were included in the composite-score calculation, the rates of documented adherence ranged from 92.0% (95% CI, 89.9 to 94.1), for indicated care for upper respiratory tract infection, to 34.5% (95% CI, 31.0 to 37.9), for indicated preventive care for adolescents (Table 5).
Sensitivity Analyses
In analyses involving all 242 original indicators (including those requiring documentation of medical histories and of counseling or education), the overall adherence rate was 42.2% (95% CI, 40.4 to 43.9). The adherence rate for the 110 indicators with a median validity score of 8 or 9 (the maximum possible score) was 42.1% (95% CI, 39.9 to 44.4), and the rate for the 61 indicators with a median validity score of 9 was 41.5% (95% CI, 39.2 to 43.8). The inclusion of only the 99 indicators based on expert consensus pediatric guidelines reduced the overall adherence rate to 39.5% (95% CI, 37.0 to 42.1). Among the 895 children for whom we had all medical records, the overall adherence rate was 46.8% (95% CI, 44.2 to 49.4), and it was 46.3% (95% CI, 42.6 to 49.9) among the 457 children for whom just one record was missing.
Discussion
On the basis of medical record documentation, deficits in the delivery of indicated care to children (for which the overall adherence rate was 46.5%) are similar in magnitude to those previously reported for adults (for which the overall adherence rate was 54.9%).1 These deficits may result in avoidable adverse health outcomes. For example, only 44.0% of children with asthma who were noted to be using 
2-agonists at least three times per day had a prescription for an antiinflammatory medication recorded in the chart. Similarly, studies of children with persistent asthma have shown that only 39 to 51% were treated with antiinflammatory medications.22,23,24 Children with persistent asthma who are treated with inhaled antiinflammatory drugs, as compared with those who are not, have fewer asthma-related symptoms and improved pulmonary function,25 are hospitalized less frequently,26 and have lower asthma-related mortality.27
Immunizations are effective in protecting children against a variety of serious childhood diseases. Only 49.8% of children in our study who reached 2 years of age during the study period were fully immunized, according to their records. The rate of immunization during this period ranged from 47 to 54%, according to the Health Plan Employer Data and Information Set (HEDIS), which is based on a combination of data from chart review and medical claims.28
According to chart data, urine cultures were obtained for 16.2% of children 3 to 36 months of age who presented with fever of unknown origin and who were thought to be at high risk for sepsis. The reported prevalence of urinary tract infection is high (4 to 5%) among children 2 months to 2 years of age who have fever without an identified source of infection on the basis of the history and physical examination.29,30 Early diagnosis of urinary tract infection might lead to earlier identification of high-grade vesicoureteral reflux, allowing for the prevention of recurrent infections, worsening renal damage, and chronic renal failure.31,32
Only 41.5% of eligible adolescent girls in the current study had charts showing evidence of laboratory orders for tests for Chlamydia trachomatis or of the results of such testing, as compared with 37.0% of adolescent girls enrolled in Medicaid and 24.0% of those with commercial health insurance, according to data for 2000 from HEDIS.28 Screening for chlamydia is important, because 75% of such infections are asymptomatic,33 and it is reported that 40% of untreated women and adolescents will have pelvic inflammatory disease. Of that 40% of women, 20% will have infertility due to tubal factors and 9% will have life-threatening complications during pregnancy.34 Broad-based screening, early detection, and treatment have decreased the incidence of pelvic inflammatory disease associated with chlamydia in adolescent girls by 60%, lowering rates of hospitalization and complications.11,35
Our present study has a number of limitations. Nonresponse bias is a concern because the sample we analyzed included only 42.0% of the children who were eligible, though the direction of that bias is unclear. Our study participants included children who had seen a provider at least once in 2 years and who were more likely than the average child in the United States to have private insurance. We would expect these children to have a higher quality of care than the average child. We did not study children living in rural areas and those without telephones; we would expect their quality of care to be lower. Children in excellent health were less likely to participate; we would expect their quality of care to be higher. The adjustments for nonresponse and sampled population were used to account for as much of this type of bias as possible.
We did not have all medical records for all the children, which raises the question of bias due to missing data. We examined whether the rates of performance varied on the basis of whether we had all the charts, were missing one chart, or were missing two or more charts; there were no significant differences among these groups. We abstracted all the information available in each chart, which gave us some information on the care delivered by providers for whom we were missing charts. In many cases, the chart that has the information necessary to determine whether a child is eligible for a care process is also the chart that contains information on whether the care was delivered or ordered, so a missing chart is likely to have caused us to miss information on both the indicator eligibility and the scoring.
We relied on medical records to determine both indicator eligibility and score. Concordance between the content of medical records and direct observations, audiotapes, or videotapes of the encounters described in the records varies according to the type of care.36,37,38,39 We restricted the results reported here to the subgroup of indicators (175 of the original 242) for which the documentation was generally good. However, some care that was delivered may not have been documented, and some care that was documented may not have been delivered.
The data on which our results are based are 7 to 11 years old, which raises the question of whether patterns of practice are different today. Most quality measurements, reporting of quality assessments, improvement efforts, and incentive payments have been focused on care for adults. In the National Healthcare Quality Report by the Agency for Healthcare Research and Quality, the median level of improvement has been about 3.1% per year — mostly in hospital-based care of adults for heart attack, heart failure, and pneumonia.40 Thus, it appears that the quality of health care for adults is improving only slowly, despite considerable attention. There has been no equivalent commitment to improve health care for children, and it therefore seems unlikely that quality has changed markedly over time.
Apparent deficits in the quality of care for children are similar in magnitude to those previously reported for adults.1 Although the data in this study are based on recorded care delivered from 1996 to 2000, it seems unlikely that quality has improved substantially since that period.40 Expansion of access to care through insurance coverage, which is the focus of national health care policy related to children, will not, by itself, eliminate the deficits in the quality of care.
Supported by grants from the Robert Wood Johnson Foundation for data collection and analysis, the Centers for Medicare and Medicaid Services for the development of the indicators, and the California HealthCare Foundation for the development and testing of medical-record–abstraction software.
No potential conflict of interest relevant to this article was reported.
We thank Maureen Michael, James Knickman, and Robert Hughes at the Robert Wood Johnson Foundation for their support; Paul Ginsburg at the Center for Studying Health System Change for his support of this collaboration; Richard Strauss at Mathematica Policy Research for developing systems for passing the initial sample from the CTS household survey to RAND for this study; RAND's Survey Research Group (Josephine Levy and Laural Hill) and the telephone interviewers for recruiting participants; Peggy Wallace, Karen Ricci, and Belle Griffin for their assistance in the design of the data-collection tool, for hiring and training the nurse abstractors, and for overseeing the data-collection process; Liisa Hiatt for serving as the project manager; and Vector Research for developing the data-collection software.
Source Information
From the Department of Pediatrics, University of Washington, and Children's Hospital and Regional Medical Center — both in Seattle (R.M.-S.); RAND, Santa Monica, CA (A.H.D., C.M.S., J.K., D.J.K., J.L.A., M.A.S., E.A.M.); and the Departments of Pediatrics and Health Services, University of California at Los Angeles, Los Angeles (M.A.S.).
Address reprint requests to Dr. Mangione-Smith at the Department of Pediatrics, University of Washington, Child Health Institute, 6200 NE 74th St., Suite 210, Seattle, WA 98115-8160, or at ritams{at}u.washington.edu.
References
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
Related Letters:
Children and the Quality of Ambulatory Care
Meropol S. B.
Extract |
Full Text |
PDF
N Engl J Med 2008;
358:532-533, Jan 31, 2008.
Correspondence
This article has been cited by other articles:
HOME | SUBSCRIBE | SEARCH | CURRENT ISSUE | PAST ISSUES | COLLECTIONS | PRIVACY | HELP | beta.nejm.org Comments and questions? Please contact us. The New England Journal of Medicine is owned, published, and copyrighted © 2008 Massachusetts Medical Society. All rights reserved. |
Previous
