fraud

Statistics and Ethics in ResearchDavid L. DeMets, Ph.D.Department of Biostatistics and Medical Informatics : Statistics and Ethics in Research David L. DeMets, Ph.D. Department of Biostatistics and Medical Informatics


Data Analysis and Ethics : Data Analysis and Ethics “Fishing, Trimming, and Cooking” Statistician Vulnerable to Pressures


Fishing : Fishing Find the Right Definition or Cutpoint


Dichotomizing Outcome : Dichotomizing Outcome Common practice to define a response (S,F) from a non-dichotomous variable By changing our definition, we can alter results Thus, definitions stated in advance Example Heart Rate Trt A Trt B Subject Pre Post D Pre Post D 1 72 72 0 72 70 2 2 74 73 1 71 68 3 -- 25 73 73 0 79 79 0 Mean 74.0 73.2 .8 74.4 74.0 .4


Proper Analysis vs Fishing : Proper Analysis vs Fishing Data set constructed such that there is not difference between to treatment arms Comparison of means or medians produce no significant results, so no abstract or publication Investigator begins to explore other analyses (with your help)


“Cutpoint” Analysis : “Cutpoint” Analysis Break continuous data into categories of success or failure Define various cut points for the categorization Examine results for various cut points


3 Possible Analyses (1) : 3 Possible Analyses (1) Change D Trt A Trt B P-Value F = < 7 23 25 .49 S = > 7 2 0 Analysis 1


3 Possible Analyses (2) : 3 Possible Analyses (2) Change D Trt A Trt B P-Value F = < 5 19 25 .02 S = > 5 6 0 Analysis 2


3 Possible Analyses (3) : 3 Possible Analyses (3) Change D Trt A Trt B P-Value F = < 3 17 18 .99 S = > 3 8 7 Analysis 3


Choosing the Right Analysis : Choosing the Right Analysis Which of the three analyses is the correct one? Which should be reported?


Analysis Principles to Consider : Analysis Principles to Consider Intention-to-treat Analysis Subgroup Cautions Multiple Outcomes Cautions Surrogate Outcomes Cautions


Cooked or Fabricated Data : Cooked or Fabricated Data Tough to detect Even tougher if detected


Investigating A Fraud CaseA Personal Experience : Investigating A Fraud Case A Personal Experience Hopefully, no one will experience a similar case May, however, have experiences that involve some level of inappropriate scientific conduct Such circumstances challenge statistician’s professional standards


Research in Minimizing Impact of Heart Attacks : Research in Minimizing Impact of Heart Attacks Heart attacks cause serious morbidity and mortality Need to seeks ways to reduce the burden Prevention Treatment Treatments considered to reduce impact by reducing damage to heart muscle


Variation in Animal Models : Variation in Animal Models Typically, drugs are tested in an appropriate animal model before testing in humans Several laboratories tested drugs (eg verapamil, ibuprofen) believed to limit the size of the infarct Results varied from laboratory to laboratory, even with the same drug


AMPIM Study : AMPIM Study Dog trial evaluating drugs to protect heart muscle during ischemia Four laboratory centers One statistical center Data analysis compared centers Discovered treatment by center interaction


AMPIM StudyBethesda Four : AMPIM Study Bethesda Four Richard Schwartz, PhD Project Officer Patrick Sullivan Contract Officer Kent Bailey, PhD Biostatistician David DeMets PhD Biostatistician


AMPIM Investigator MeetingSeptember 1981 : AMPIM Investigator Meeting September 1981 Discussed center differences No rational explanation forthcoming Learned about May fabrication of EKG strip on AMPIM dog


Slide20 : Figure 1B. Ventricle weight versus dog weight, sites C and D.


AMPIM Analysis Challenged : AMPIM Analysis Challenged Division director challenged statistician analyses and probing Senior leadership at NIH became uncomfortable; director vacancy Analysis team “threatened” NIH Division of Research grants formed external review committee Analysis team became support staff


AMPIM Study : AMPIM Study Ross Committee Report - February 1982 (Science) Formed by host institution to investigate allegations Concluded that problem limited to single fabricated EKG Morgan Committee - January 1982 Begins a 6-month investigation


AMPIM Review : AMPIM Review AMPIM hallmark was lack of variability Seven previously published papers conducted experiments with similar components Morgan committee member noticed remarkable reproducibility across studies (papers)


Panel/Center Review Meeting : Panel/Center Review Meeting Analyses presented to University senior leadership Lab director challenged statistical analyses of additional papers as being possibly fraudulent Analyses Implicated years of research & lab prestige Quick Response needed


AMPIM Review : AMPIM Review Violation of Central Limit Theorem 95% Confidence Intervals Mean Difference


AMPIM Analyses : AMPIM Analyses Evidence of fraudulent data largely based on statistical arguments Up till then, no hard evidence Criminal case being considered by US Dept of Justice Thoughts of testifying …….. Then 3 smoking pistols emerge


AMPIM Study3 Smoking Pistols : AMPIM Study 3 Smoking Pistols The Bagged Dog More Dogs Than Logs No “Hot” Dogs


The Bagged Dog : The Bagged Dog To conduct experiment, dog had to be sacrificed with heart extracted Research fellows noticed colleague working on AMPIM left lab early and put an experimental AMPIM dog into the freezer in a bag Fellows went into freezer and removed heart from experimental dog Challenged fellow at lab meeting about measurements obtained from this dog


More Dogs than Logs : More Dogs than Logs Many papers written from a series of experiments, each using dogs in either experimental or control arms All of the experiments funded by NIH research grants/contracts Contract officer noticed that the number of dogs in the papers exceeded the number of dogs paid for in the contract Later, discovered more dogs in the papers than in the dog lab logs


No “Hot” Dogs : No “Hot” Dogs To conduct the measurements on the dog hearts, a radioactive substance is introduced Radioactive substance allows identifying the amount of blood flow to various regions of the heart Sacrificed dogs with radioactive substance must be disposed of using special procedures Ashes of these dogs retrieved and no radioactivity could be found Suggested experiments never done on several dogs included in the AMPIM paper


3 Smoking Pistols : 3 Smoking Pistols This evidence gave investigating team hard evidence for fraud Removed the need to base arguments on probabilities of unusual results Allowed investigative team to rapidly conclude their report


Impact of Darsee Case : Impact of Darsee Case Affected the careers of research fellows, junior faculty member, senior faculty member Brought new standards for conduct of laboratory research Led to investigation of Darsee’s research at previous training institutions Darsee barred from NIH funding Bethesda Four survived


Summary: AMPIM Study : Summary: AMPIM Study One center quite different Plots of data revealed implausible relationships Further investigation leads to discovery of possible fraud Data lacked appropriate variability


Montreal Breast Cancer Fraud Case : Montreal Breast Cancer Fraud Case References Fisher (Nov, 1995) Breast Cancer, NEJM Christian (Nov, 1995) Breast Cancer, NEJM Multicenter Breast cancer trial evaluating radical vs modified mastectomy plus adjuvant chemo therapy Statistical Center and Study Chair at the University of Pittsburg Montreal Investigator modified dates of xrays to meet eligibility criteria


Montreal Fraud Case : Montreal Fraud Case Records of 6 patients affected Real patients, real disease, real treatment & real outcomes Statistician at Coordinating Center discovered discrepancies in dates on the data forms in 1991 Reported the problem to Study Chair and the funding agency, the National Cancer Institute


Montreal Fraud Case : Montreal Fraud Case The analysis indicated that the results of the published paper did not change as a result of including or excluding affected patients Modified mastectomy as effective as radical mastectomy No updated analysis presented until 1996 No communication of the fraud case was presented to public in 1991 A couple of years later, story broke in Chicago newspaper about the previous discovery of fraud in a key breast cancer study


Montreal Fraud Case : Montreal Fraud Case Following the news story The biostatistician was fired The Study chair was fired Public advocacy groups challenged all the results from this breast cancer research team Congress got involved in hearings Congress & Public expected research such as this trial to have no errors (same as their bank)


Montreal Fraud Case : Montreal Fraud Case Congressional pressure eventually forced NCI Director to resign In 1995, NCI conducted a thorough investigation, reported by Christian et al No real changes in the statistical analysis, or the conclusions about the role of modified mastectomy Analysis excluded 6 patients with modified records & exclude all patients from Montreal site


Montreal Fraud Case : Montreal Fraud Case Law suits ensued by Study Chair against the University of Pittsburg Eventually Study chair re-instated Statistician (whistle-blower) re-instated Neither’s career was the same Montreal investigator disbarred from the cancer research group


Slide45 : With whom should he/she seek advice? How sure is “sure enough” Are the “whistleblower” risks worth the chance? If Statistician Discovers Possible Cooking or Fraud