|
|
| |||||||||||||||||||||||||||||||||
Background The National Surgical Adjuvant Breast and Bowel Project (NSABP) Protocol B-06, a clinical trial sponsored by the National Cancer Institute (NCI), has provided evidence of the value of lumpectomy and breast irradiation for treating women with breast cancer in an early stage. Publicity generated by the discovery that the study included fraudulent data on patients enrolled by St. Luc Hospital in Montreal aroused concern about the overall accuracy of the data and conclusions. To address this concern, the NCI conducted an audit of other participating institutions.
Methods In 1994, data on 1554 of the 1809 randomized patients (85.9 percent) enrolled by centers other than St. Luc Hospital were audited at 37 clinical sites in North America. The audit included data on eligibility, survival, disease-free survival, the length of time to a recurrence of cancer in the ipsilateral breast, and documentation of signed informed consent.
Results End points were assessed for all 1554 patients, and eligibility was assessed for 1507 patients; 47 patients were excluded because their forms were not complete or not returned. A total of 1429 patients had their eligibility status verified. Of a total of 7770 data points examined with respect to the number of positive nodes at base line, treatment characteristics, first events (excluding death), recurrence of cancer in the ipsilateral breast, and survival, 7577 (97.5 percent) were verified, 123 (1.6 percent) could not be verified, and 70 (0.9 percent) were discrepant with the NSABP file. Of the 1554 patients, 1340 (86.2 percent) had all audited items (including eligibility) verified, 69 (4.4 percent) had at least one discrepant item, and 113 (7.3 percent) had at least one unverified item (as a result of missing or incomplete data); 32 (2.1 percent) were not assessed for eligibility but had no other discrepant or unverifiable items. Written informed consent was documented for 1098 patients before surgery and 210 after surgery; no date appeared on the signed form for 137. The informed-consent status was not verified for 71 patients and could not be determined for 38. The rates of verification of end-point data and documentation of written informed consent were similar among the total-mastectomy group, the lumpectomy group, and the group treated by lumpectomy and breast irradiation.
Conclusions The audit confirms the adequacy of the data on which the reanalysis of Protocol B-06 and the results after 12 years of follow-up are based.
4 cm in size (tumor, node, metastasis classifications, T1 or T2, N0 or N1, M0), accrued patients between 1976 and 1984. The initial results were reported in 19851 and updated in 1989.2 The most recent analysis appears elsewhere in this issue of the Journal.3 In 1991, the NSABP detected scientific misconduct by an investigator who had been a major contributor to Protocol B-06. After an investigation, the Office of Research Integrity of the Public Health Service determined that Dr. Roger Poisson, of St. Luc Hospital in Montreal, had fraudulently enrolled 99 patients in NSABP protocols,4 including 6 in Protocol B-06 for whom eligibility data were falsified or fabricated.
Two therapeutic options — mastectomy or conservative surgery plus breast irradiation — are currently available to women with breast cancer in an early stage. Several randomized trials5,6 of these two approaches have yielded similar outcomes. Protocol B-06 was the largest of these studies, and any change in the results of this study could affect prevailing treatment recommendations.7 In early 1994, publicity surrounding the inclusion of fraudulent data in the study aroused public concern about the overall quality of the data and the validity of the conclusions.
To assess the accuracy of the data base as a whole and to address the public's concern, the National Cancer Institute (NCI) audited the Protocol B-06 data. Although the auditors looked for evidence of fraud, the primary focus was on verifying the accuracy of the data on which the study's original analysis and conclusions were based.1,2 Attempts were made to distinguish between deliberate falsification and error. The purpose of the audit was to assess independently sufficient eligibility and outcome data to permit a reanalysis based only on audited data.
Details of the design of Protocol B-06, including eligibility criteria, treatment plans, and statistical methods, have been published previously.1,2,3 Major study end points included disease-free survival, survival free of disease at distant sites, overall survival, and the length of time to the recurrence of tumor in the ipsilateral breast.3
Methods
Audit Methods
In Protocol B-06, 2163 patients underwent randomization at 89 institutions. The exclusion of the 354 patients enrolled by St. Luc Hospital decreased the number of patients available for the audit to 1809. The NCI audit targeted all hospitals or clinics that had enrolled at least 10 eligible patients, except St. Luc Hospital in Montreal, and Royal Melbourne Hospital in Melbourne, Australia (because of geographic distance). Ultimately, two additional institutions were excluded: Letterman Army Hospital in San Francisco, which had closed, and Michigan State University, a consortium of eight institutions in Michigan, each of which had accrued three or fewer patients. This plan targeted 1554 of 1809 patient records (85.9 percent) (Table 1). Site visits were conducted between March 28, 1994, and July 20, 1994, at 37 of the 89 institutions that had enrolled patients in the study.
|
After gaining experience in the initial site visits, the NCI revised the audit forms. The outcome-assessment form was clarified, and the eligibility-assessment form was shortened from a 25-point checklist to a list of 11 criteria deemed critical to the outcome of the study. The eligibility criteria (Table 2) focused on key physical and pathological characteristics of the tumor and lymph nodes and the patients' status with respect to prior therapy for breast cancer and concomitant malignant conditions.
|
The auditors did not attempt to clarify consent issues beyond noting the nature of the consent (written vs. oral) and the date of consent relative to the definitive surgery. Patients were not declared ineligible or unable to be evaluated because of the lack of documentation of informed consent. Institutions were not asked to provide additional information, as they were for other missing or ambiguous items.
The NSABP provided specific information on individual patients (updated through September 1993) from their data base; such information was preprinted on the audit forms. Auditors used any primary-source documentation available at the site to verify data from the NSABP files, including records from hospitals, outpatient clinics, physicians' offices, and separate primary-research records. The NSABP data forms were not considered adequate source documentation and were not used, except to obtain the date of the most recent follow-up visit. Because many of the study participants were no longer being seen, followed, or treated at the institution that had enrolled them in the study, these NSABP forms were often the only documentation of the date of the most recent contact. The major audited items used in the reanalysis are described in Table 3.
|
All audited data forms were submitted to the EMMES Corporation (Rockville, Md.) for processing, review, and analysis. When data items were identified as discrepant, unverified, or missing, the EMMES Corporation contacted the principal investigators and the chairpersons of the institutional review boards at the participating institutions to request additional relevant documentation. These requests did not provide the data-base values for the items in question. Additional information on survival was obtained by a search of the National Death Index. The NSABP provided copies of relevant documents from its files, such as copies of pathology reports documenting a date or site of first recurrence.
A data-resolution panel composed of three of the authors (Jeffrey S. Abrams, Michaele C. Christian, and Richard S. Kaplan) reviewed the data to resolve all remaining questions of eligibility and outcome. Members of the NSABP staff were invited to these meetings and attended two of four meetings.
An advisory committee of experts from outside the NCI was convened to review the audit findings and the end-point reanalysis based on audited data and to assess the overall process.8
Statistical Analysis
With respect to the determination of the number of patients to be sampled, the major finding of Protocol B-06 to be verified with the use of audited data was the similar rates of disease-free survival and overall survival in the group treated by mastectomy and the group treated by lumpectomy plus irradiation. On the basis of the reanalysis of the original data9 by the EMMES Corporation, the hazard ratio (relative risk) for survival among eligible patients (excluding patients enrolled by St. Luc Hospital) was 0.88 (95 percent confidence interval, 0.72 to 1.07). The hazard ratio for disease-free survival was 0.91 (95 percent confidence interval, 0.77 to 1.07). These hazard ratios reflect near-equivalence, with a slight advantage for the group treated by lumpectomy plus irradiation.
On the basis of analytic calculations and computer simulations, if a random sample of approximately 80 percent of the eligible patients (excluding patients enrolled by St. Luc Hospital) was analyzed (and the original data were verified by the audits) then, with 95 percent power, the upper limits of the 95 percent confidence intervals for the hazard ratios would be less than 1.20. A value below 1.20 would preclude a hazard-rate disadvantage for survival or disease-free survival of more than 20 percent for lumpectomy plus irradiation as compared with mastectomy. To account for the fact that there would be changes to the data base as a result of the audits (on the basis of a preliminary analysis of the first nine audited institutions), we decided to audit institutions that accounted for approximately 87 percent of the eligible patients randomized in the trial.
A revised data base was constructed through the auditing process. When a difference in a data item was found between the original source and the original NSABP file, the revised audit data base was modified to reflect the original source. In addition, data on some patients were censored because of possible withdrawal of informed consent. The revised audit data base, with a listing of changes from the original NSABP files, was provided to the NSABP to update their analysis files. Other plans, involving sampling of institutions, were considered but eventually rejected because they would have required audits at substantially more than 37 institutions.
Results
Of the 1554 audited patients, 1340 (86.2 percent) had all audited items (including eligibility) verified, 69 (4.4 percent) had at least one discrepant item, and 113 (7.3 percent) had at least one unverified item (with no discrepant items); the remaining 32 patients (2.1 percent) were not assessed for eligibility and had no discrepant or unverifiable items. The rates of verification were similar among the three treatment groups: 84.9 percent for the total-mastectomy group, 85.9 percent for the lumpectomy group, and 87.9 percent for the group treated by lumpectomy and irradiation. With respect to the five other variables listed in Table 3 — the number of positive nodes, treatment characteristics, first event excluding death, recurrence of tumor in the ipsilateral breast, and survival — 1432 patients (92.1 percent) had all items verified, 68 (4.4 percent) had at least one discrepant item, and 54 (3.5 percent) had at least one unverified item (and no discrepant items). No evidence of systematic attempts to manipulate the data was uncovered by the audit.
Eligibility
Eligibility was assessed in 1507 patients (Table 1), regardless of whether they had been declared ineligible by the NSABP (there were 62 such patients). The eligibility of 47 patients at six sites could not be assessed for a variety of reasons, including failure to provide eligibility-assessment forms to the auditors at one site, which was audited with different forms at different times (21 patients); inaccessibility of data at two sites, where the charts of 16 patients who withdrew consent were sealed; inability to locate the charts of 6 patients at two sites; and inability to locate the audit forms for 4 patients at one site. Because these omissions were discovered late in the auditing process and were not the result of institutional deficiencies, the institutions were not asked to supply additional documentation. These 47 patients were assessed for end-point items but were excluded from the analysis of eligibility.
Eligibility status was completely verified for 1429 patients, or 94.8 percent. The auditors were unable to verify one or more of the eligibility items for 77 patients (5.1 percent), and 1 patient (0.1 percent) classified as eligible by the NSABP was deemed ineligible by the auditors (discrepant result). Verification rates were similar among the three treatment groups: 94.6 percent for the total-mastectomy group, 94.2 percent for the lumpectomy group, and 95.7 percent for the group treated by lumpectomy and irradiation.
Patients' Characteristics, Treatment Assignments, and Outcome
The results of the audit of data on base-line characteristics, treatment, and outcomes for the 1554 patients are presented in Table 4. Overall, 7577 of 7770 data points (97.5 percent) covering the number of positive nodes at base line, treatment characteristics, first events (excluding death), recurrence of tumor in the ipsilateral breast, and survival were verified. A total of 123 data points (1.6 percent) could not be verified. In the case of 70 data points (0.9 percent) from 25 institutions, the audited charts and the NSABP files were discrepant. Verification rates were similar among the treatment groups: 97.8 percent for the total-mastectomy group, 97.5 percent for the lumpectomy group, and 97.3 percent for the group treated by lumpectomy and irradiation.
|
|
The distribution of age, tumor size, nodal status, and tumor-margin status was similar among the 1340 patients who had all data verified, the 69 patients who had at least one discrepant data point, and the 145 for whom no discrepant data points were identified, but for whom some data could not be verified. The 1554 audited patients were slightly older than the 255 patients who were not audited (mean age, 51.8 vs. 50.3 years). Differences in the other variables were minor.
The presence of residual tumor at the margins of the surgical specimen (positive margins), or its absence, after lumpectomy often could not be confirmed because the original pathology reports from the period (1976 to 1984) did not describe the margins precisely. To address this issue, the NSABP had assigned a pathologist at each site to review the specimens and to complete a special form. Although the status of margins was one of the items originally targeted for auditing, there was no reasonable means of verifying this end point without the use of the NSABP forms.
The presence of a signed, informed-consent form or documentation of oral consent was also assessed (Table 6). A total of 1445 patients were verified to have given written informed consent, 1098 (70.7 percent) before surgery, and 210 (13.5 percent) after surgery; the date on which the form was signed was not recorded for 137 patients (8.8 percent). The rates of verification of written informed consent were similar among the three treatment groups. The informed-consent status could not be documented for 71 patients (4.6 percent), and data on informed consent provided by the auditors were inconsistent or insufficient to make an assessment for 38 patients (2.4 percent). Unverifiable and indeterminate results were found in 22 institutions. A chart review showed that 53 patients gave oral consent before surgery.
|
The panel chose to use the earliest date on which consent may have been withdrawn in order to acknowledge the apparent wishes of the patients with regard to participation. A change in the revised NCI data base resulting from the use of an earlier date (e.g., censoring a patient's data in 1982 instead of 1984) was not considered a discrepancy since it was based on a difference in judgment regarding the appropriate handling of these data, not on a factual disagreement.
Discussion
Discrepancies between the NSABP data and the audit results were uncommon, and most of the items considered in the audit were verified. The rates of verified items, discrepant items, and unverifiable items were similar among the treatment groups, as would be expected in a randomized study. We did not address the relative performance of individual institutions that participated in Protocol B-06, because that was not considered in the design and methods of the audit and because some centers enrolled too few patients (range, 11 to 171) to allow meaningful comparisons.
Because the mandate and primary focus of the audit was to provide an accurate data set on which to base an independent reanalysis of the results of Protocol B-06, eligibility and end-point data were emphasized and data on informed consent were handled differently. Institutions were not asked to provide additional documentation of signed consent when the auditors could not find signed consent forms during the site visit. In addition, except for cases in which consent was withdrawn, the data-resolution panel did not review informed-consent issues.
The qualitative and quantitative aspects of data monitoring differ from study to study, and there are few published reports of on-site audits of clinical-research trials. Large-scale retrospective reviews of the case records of an entire study population are uncommon. Such audits have been done by the pharmaceutical industry in support of applications for drug approval; however, the results of such audits are usually not published. Most audits involve a more contemporaneous review of the data than ours.
The Cancer and Leukemia Group B reviewed its audits conducted in four periods between 1982 and 1992 and involving nearly 1500 patients at more than 230 institutions; the group noted substantial improvements over time with regard to informed-consent deficiencies involving the content, signing, date, or witnessing of these documents (18.5 percent in cycle 1 vs. 3.9 percent in cycle 4). Overall, the eligibility of 94.5 percent of the patients was verified, the tumor response was verified in 96.4 percent of cases, and the rate of what they considered major deviations from the dosing protocol remained steady at 11 percent.10 Preliminary results of the review of audits conducted over a seven-year period by the Southwest Oncology Group11 of data on 1751 of their 28,613 patients showed that eligibility was confirmed in 94.8 percent of the patients, treatment compliance was confirmed in 78.5 percent, and response assessment was confirmed in 95 percent.
In 1982 the NCI established a formal quality-assurance program in which its cooperative groups were given the responsibility for auditing participating institutions at least once every three years, according to specific guidelines. The NCI reviewed the results of audits conducted between 1982 and 1984 that involved 812 institutions and 5988 patients enrolled by 17 cooperative groups. The ineligibility rate was 7 percent, serious deviations from the treatment protocol occurred in 12 percent of patients, the response was inadequately documented or incorrect in 6 percent, and the data could not be verified because of a lack of documentation or because of discrepancies in 5 percent.12
Protocol B-06 had complicated randomization and consent procedures, involved multiple surgical procedures, and required long-term follow-up even of participants who were no longer being seen at the institutions that had enrolled them in the study. Nonetheless, the verification rates are consistent with those obtained through audits of other trials. The NCI audit confirms the adequacy of the data on which the reanalysis of Protocol B-06 and the results after 12 years of follow-up are based.3
We are indebted to all the investigators and staff members for their cooperation, assistance, and conscientious efforts to locate the necessary records, charts, and follow-up documents.
Source Information
From the Cancer Therapy Evaluation Program, Division of Cancer Treatment, National Cancer Institute, 6130 Executive Blvd., Rm. 715, Rockville, MD 20892, where reprint requests should be addressed to Dr. Christian.
References
| |||||||||||||||||||||||||||||||||
This article has been cited by other articles:
HOME | SUBSCRIBE | SEARCH | CURRENT ISSUE | PAST ISSUES | COLLECTIONS | PRIVACY | HELP | beta.nejm.org Comments and questions? Please contact us. The New England Journal of Medicine is owned, published, and copyrighted © 2008 Massachusetts Medical Society. All rights reserved. |
Previous
