Pick'em Leagues: THE BIG SHOW CASUAL BEST OF THE REST Single Event PvP: FANTASY POOLS Betting Leagues: THE BIG SHOW BEST OF THE REST
  MMA Playground 4.0 is live!     Returning members: check out the patch notes     New members: visit our getting started guide, read the FAQ's and start playing!

The science of doping

Print  
Posted By Message

Rush

Laying down the beats

Rush Avatar
4



 
 
 


 
 
Posts:6,291
Career:759-539
Joined:Jan 2007
Chips:
835
Nature 454, 692-693 (7 August 2008) | doi:10.1038/454692a; Published online 6 August 2008


The science of doping
See associated Correspondence: Ljungqvist et al. , Nature 455, 1176 (October 2008)Faber, Nature 455, 1176 (October 2008)

Donald A. Berry1

Donald A. Berry is head of the Division of Quantitative Sciences, chair of the Department of Biostatistics and Frank T. McGraw Memorial Chair of Cancer Research, MD Anderson Cancer Center, University of Texas, 1400 Pressler Street, Houston, Texas 77030-1402, USA.
Email: dberry@mdanderson.org

Top of pageAbstractThe processes used to charge athletes with cheating are often based on flawed statistics and flawed logic, says Donald A. Berry.

Recently, the international Court of Arbitration for Sport upheld doping charges against cyclist Floyd Landis, stripping him of his title as winner of the 2006 Tour de France and suspending him from competition for two years. The court agreed with the majority opinion of a divided three-member American Arbitration Association (AAA) panel and essentially placed a stamp of approval on a laboratory test indicating that Landis had taken synthetic testosterone. Although Landis asserts his innocence, his options for recourse have all but dried up.

Already, in the run-up to this year's Olympic Games, vast amounts of time, money and media coverage have been spent on sports doping. Several doping experts have contended that tests aren't sensitive enough and let dozens of cheaters slip through the cracks. And some athletes are facing sanctions. Upon testing positive for clenbuterol, US swimmer Jessica Hardy was held back from the Olympic team and faces a two-year ban from the sport. She is attesting her innocence. China has already banned several athletes, some of them for life, on doping charges. Indeed, many world-class athletes will find their life's accomplishments and ambitions, their integrity and their reputations hinging on urine or blood tests. But when an athlete tests positive, is he or she guilty of doping? Because of what I believe to be inherent flaws in the testing practices of doping laboratories, the answer, quite possibly, is no.

In my opinion, close scrutiny of quantitative evidence used in Landis's case show it to be non-informative. This says nothing about Landis's guilt or innocence. It rather reveals that the evidence and inferential procedures used to judge guilt in such cases don't address the question correctly. The situation in drug-testing labs worldwide must be remedied. Cheaters evade detection, innocents are falsely accused and sport is ultimately suffering.

Prosecutor's fallacy
One factor at play in many cases that involve statistical reasoning, is what's known as the prosecutor's fallacy1. At its simplest level, it concludes guilt on the basis of an observation that would be extremely rare if the person were innocent. Consider a blood test that perfectly matches a suspect to the perpetrator of a crime. Say, for example, the matching profile occurs in just 1 out of every 1,000 people. A naive prosecutor might try to convince a jury that the odds of guilt are 999:1, that is, the probability of guilt is 0.999. The correct way to determine odds comes from Bayes rule2, 3, 4 and is equal to 999 times P/(1-P) where P is the 'prior probability' of guilt. Prior probability can be difficult to assess, but could range from very small to very large based on corroborating evidence implicating the suspect. The prosecutor's claim that the odds are 999:1 implies a prior probability of guilt equal to 0.5 (in which case P and 1-P cancel). Such a high value of P is possible, but it would require substantial evidence. Suppose there is no evidence against the suspect other than the blood test: he was implicated only because he was from the city where the crime occurred. If the city's population is one million then P is 1/1,000,000 and the odds of his guilt are 1001:1 against, which corresponds to a probability of guilt of less than 0.001.

The prosecutor's fallacy is at play in doping cases. For example, Landis's positive test result seemed to be a rare event, but just how rare? In doping cases the odds are dictated by the relative likelihood of a positive test assuming the subject was doping ('sensitivity') against a positive result assuming no doping (which is one minus 'specificity'). Sensitivity and specificity are crucial measures that must be estimated with reasonable accuracy before any conclusion of doping can be made, in my opinion.

The studies necessary to obtain good estimates are not easy to do. They require known samples, both positive and negative for doping, tested by blinded technicians who use the same procedures under the same conditions present in actual sporting events. In my view, such studies have not been adequately done, leaving the criterion for calling a test positive unvalidated.

Laboratory practices
Urine samples from cyclists competing in the 2006 Tour de France were analysed at the French national anti-doping laboratory (LNDD) in Châtenay-Malabry. This is one of 34 laboratories accredited by the World Anti-Doping Agency to receive and analyse test samples from athletes. The LNDD flagged Landis's urine sample following race stage 17, which he won, because it showed a high ratio of testosterone to epitestosterone.

Based on the initial screening test, the LNDD conducted gas chromatography with mass spectrometry, and isotope ratio mass spectrometry on androgen metabolites in Landis's sample. Such laboratory tests involve a series of highly sophisticated processes that are used to identify the likelihood of abnormal levels of plant-based androgen metabolites (from dietary or pharmaceutical sources) in a urine sample. The goal is to differentiate from endogenous androgen metabolites normally found in urine.

Mass spectrometry requires careful sample handling, advanced technician training and precise instrument calibration. The process is unlikely to be error-free. Each of the various steps in handling, labelling and storing an athlete's sample represents opportunity for error.

In arbitration hearings, the AAA threw out the result of the LNDD's initial screening test because of improper procedures. In my opinion, this should have invalidated the more involved follow-up testing regardless of whether or not sensitivity and specificity had been determined. Nevertheless, the AAA ruled the spectrometry results sufficient to uphold charges of doping.

During arbitration and in response to appeals from Landis, the LNDD provided the results of its androgen metabolite tests for 139 'negative' cases, 27 'positive' cases, and Landis's stage 17 results (see Fig. 1). These data were given to me by a member of Landis's defence team. The criteria used to discriminate a positive from a negative result are set by the World Anti-Doping Agency and are applied to these results in Fig. 1b and d. But we have no way of knowing which cases are truly positive and which are negative. It is proper to establish threshold values such as these, but only to define a hypothesis; a positive test criterion requires further investigation on known samples.

The method used to establish the criterion for discriminating one group from another has not been published, and tests have not been performed to establish sensitivity and specificity. Without further validation in independent experiments, testing is subject to extreme biases. The LNDD lab disagrees with my interpretation. But if conventional doping testing were to be submitted to a regulatory agency such as the US Food and Drug Administration5 to qualify as a diagnostic test for a disease, it would be rejected.

The problem with multiples
Landis seemed to have an unusual test result. Because he was among the leaders he provided 8 pairs of urine samples (of the total of approximately 126 sample-pairs in the 2006 Tour de France). So there were 8 opportunities for a true positive — and 8 opportunities for a false positive. If he never doped and assuming a specificity of 95%, the probability of all 8 samples being labelled 'negative' is the eighth power of 0.95, or 0.66. Therefore, Landis's false-positive rate for the race as a whole would be about 34%. Even a very high specificity of 99% would mean a false-positive rate of about 8%. The single-test specificity would have to be increased to much greater than 99% to have an acceptable false-positive rate. But we don't know the single-test specificity because the appropriate studies have not been performed or published.

More important than the number of samples from one individual is the total number of samples tested. With 126 samples, assuming 99% specificity, the false-positive rate is 72%. So, an apparently unusual test result may not be unusual at all when viewed from the perspective of multiple tests. This is well understood by statisticians, who routinely adjust for multiple testing. I believe that test results much more unusual than the 99th percentile among non-dopers should be required before they can be labelled 'positive'.

Other doping tests are subject to the same weak science as testosterone, including tests for naturally occurring substances, and some that claim to detect the presence of a foreign substance. Detecting a banned foreign substance in an athlete's blood or urine would seem to be clear evidence of guilt. But as with testing for synthetic testosterone, such tests may actually be measuring metabolites of the drug that are naturally occurring at variable levels.

Whether a substance can be measured directly or not, sports doping laboratories must prospectively define and publicize a standard testing procedure, including unambiguous criteria for concluding positivity, and they must validate that procedure in blinded experiments. Moreover, these experiments should address factors such as substance used (banned and not), dose of the substance, methods of delivery, timing of use relative to testing, and heterogeneity of metabolism among individuals.

To various degrees, these same deficiencies exist elsewhere — including in some forensic laboratories. All scientists share responsibility for this. We should get serious about interdisciplinary collaborations, and we should find out how other scientists approach similar problems. Meanwhile, we are duty-bound to tell other scientists when they are on the wrong path.

References
Buchanan, M. The prosecutor's fallacy. The New York Times (16 May 2007).
Berry, D. A. Stat. Sci. 6, 175–205 (1991). | Article |
Berry, D. A. Statistics: A Bayesian Perspective (Duxbury Press, California, 1996).
Berry, D. A. & Chastain, L. A. Chance 17, 5–8 (2004).
http://www.fda.gov/cdrh/osb/guidance/1620.pdf



Fig legend
Plots show the distribution of 167 samples of the metabolites etiocholanone and 5 -androstanediol (a, b), and androsterone and 5 -androstanediol (c, d). Panels b and d show samples the French national anti-doping laboratory (LNDD) designate to be 'positive' (red crosses) or 'negative' (green dots); the values from Landis's second sample from stage 17 is shown as a blue dot. Axes display delta notation, expressing isotopic composition of a sample relative to a reference compound.

Image Attachment(s):

Photo Attachment 1

Last edited 2/26/09 2:01PM server time by Rush
Edit note/reason: n/a

Post #1   2/26/09 2:00:48PM   

Rush

Laying down the beats

Rush Avatar
4



 
 
 


 
 
Posts:6,291
Career:759-539
Joined:Jan 2007
Chips:
835
Doping: probability that testing doesn't tell us anything new
Geoffrey Baird1

Department of Laboratory Medicine, Division of Clinical Chemistry, University of Washington, Seattle, Washington 98195, USA
Email: gbaird@u.washington.edu

Sir
In his Commentary 'The science of doping' (Nature 454, 692–693; 2008), Donald Berry discusses Bayes' rule, noting that consideration of P, the prior probability of guilt, is essential in interpreting a positive doping result. He fails, however, to mention what the actual value of P might be in Floyd Landis's case, which I think misses an opportunity to address an important problem.

Athlete acquaintances and the news media have led me to believe that P can be very high, and in fact approach unity, in some sports. If this is true, then anti-doping measures should cease — and not because of the statistical arguments that Berry raises, rather because the testing isn't telling us anything we don't already know.

If P is close to 1, then negative tests are most likely to be false negatives. Those who test positive might only be those who are least adept at hiding their drug use.







Doping: similar problems arise in medical clinics
Eric L. Altschuler1

Department of Physical Medicine and Rehabilitation, University of Medicine & Dentistry of New Jersey, Newark, New Jersey 07103, USA
Email: altschel@umdnj.edu

Sir
In his Commentary (Nature 454, 692–693; 2008), Donald Berry warns about the dangers of poor statistical understanding and misinterpretation of drug-testing results in Olympic athletes. Unfortunately, this same problem arises on a daily basis around the world in medical clinics, often with even greater consequences.

Berry illuminates the failure to use proper Bayesian reasoning in interpreting doping tests and also the problem of not having sufficient control-population norms for the tests to determine correctly whether an athlete is taking a banned substance or not. Clinicians typically have less understanding of Bayesian statistics than drug-testing officials and even fewer resources to interpret or norm such tests.

Take urine testing of patients on opiate therapy to make sure that they test positive for opiates (to show the patient is taking the medicine rather than, say, selling it) and that they are not using illegal drugs. Either a negative test for opiates or a positive test for an illegal substance can typically be sufficient to preclude a patient from receiving another prescription for opiates or to put the clinician in the position of having to explain the test result before prescribing the medicine.

Such tests need to be reported with the appropriate Bayesian interpretation. Also, as Berry advocates for Olympic athletes, patients should have the right (and access) to a statistical 'consultation' if they feel the test is in error.







Doping: ignorance of basic statistics is all too common
Matthew Fero1

Clinical Research Division, Fred Hutchinson Cancer Research Center, Seattle, Washington 98109, USA
Email: mfero@fhcrc.org

Sir
Donald Berry's Commentary 'The science of doping' (Nature 454, 692–693; 2008) is like a breath of fresh air in the murky world of drug testing. Unfortunately, a lack of competence in basic statistics is all too common in biology and the clinical sciences. As Berry points out, there is often a lack of accounting for pre-test probabilities in the application of tests with known sensitivities and specificities, as well as for issues arising from multiple testing.

Even those who grasp the principles of Bayes' rule frequently make the mistake of not empirically confirming the utility of confirmatory assays. Take steroid testing, as illustrated in Berry's Figure 1 for Floyd Landis's case in 2006. Given the high sensitivity and specificity of the assay, androsterone plus 5a-androstanediol is assumed to form the basis of a conclusive set of tests for confirming positive screening results with etiocholanone plus 5b-androstanediol. In fact, the confirmatory tests can provide little additional information unless they have been shown to be independent predictors of drug positivity.







Doping: a paradigm shift has taken place in testing
Pierre-Edouard Sottas1, Christophe Saudan1 & Martial Saugy1

Swiss Laboratory for Doping Analyses, Chemin des Croisettes 22, 1066 Epalinges, Switzerland
Email: pierre-edouard.sottas@chuv.ch

Sir
In his Commentary 'The science of doping' (Nature 454, 692–693; 2008), Donald Berry claims that anti-doping tests are based on flawed statistics. Your Editorial 'A level playing field?' (Nature 454, 667; 2008) goes even further in concluding that the anti-doping authorities act unscientifically. These claims neglect an abundant body of literature and ignore the paradigm shift that has taken place in anti-doping science.

Anti-doping is a forensic science, not a medical one. In medical diagnostics, biostatisticians have all the leeway to set sensitivity and specificity to an appropriate level. Such freedom is restricted in forensics: the risk of a false positive must be minimized at every step of the development, validation and application of a test. This fact alone explains why anti-doping tests do not necessarily rely on statistical reasoning, and certainly not solely on a specificity threshold, something Berry seemingly takes for granted. For the detection of exogenous testosterone in particular, anti-doping laboratories establish intervals for a reference population throughout validation processes that also include quality controls for batch acceptance. To date, no false positive has been reported among all the negative controls.

The nature of scientific evidence is also different. In forensics, the traditional assumptions of 'absolute certainty' and 'discernible uniqueness' are being progressively abandoned in favour of an empirical and probabilistic approach (see M. J. Saks and J. J. Koehler Science 309, 892–895; 2005). In the fight against doping, this is embodied by the 'athlete's biological passport', an electronic document that stores an individual's information pertaining to indirect markers of doping.

In multiplying the probabilities to estimate the specificity for the Landis case, Berry makes a basic statistical error. Indeed, successive tests are not independent in a longitudinal follow-up (P. E. Sottas et al. Forensic Sci. Int. 174, 166–172; 2008).

A more thorough literature search would have prevented Berry from attempting to reinvent the wheel and from concluding that anti-doping scientists are "on the wrong path", which is presumptuous and disrespectful. The role of anti-doping science (not "doping science") is to protect clean athletes. Your Editorial may have just the opposite effect.

Last edited 2/26/09 2:03PM server time by Rush
Edit note/reason: n/a

Post #2   2/26/09 2:01:49PM   

Loot613

MMA Regular

Loot613 Avatar
1
 
 
 


 
 
Posts:114
Career:1,249-863
Joined:Apr 2007
Chips:
12
Holy moley MMA fans cannot read that jargon nor can we read that much....


Post #3   2/26/09 2:58:52PM   

roadking95th

In Cael We Trust!

roadking95th Avatar
7
 
 
 
 


 
 
Posts:1,030
Career:1,351-714
Joined:Jan 2007
Chips:
180
Rush, this is very interesting. Could you give us a brief layman's version?

Post #4   2/26/09 4:10:01PM   

chitownsfinest15

MMA Regular

chitownsfinest15 Avatar
 
 
 


 
 
Posts:85
Career:413-266
Joined:Dec 2007
Camp: KMX 7249
Chips:
6
he is definately right....

Post #5   2/26/09 6:00:27PM   

mictlantechutli

MMA Regular

mictlantechutli Avatar
1
 
 
 


 
 
Posts:87
Career:660-408
Joined:Apr 2007
Chips:
12
"Whether a substance can be measured directly or not, sports doping laboratories must prospectively define and publicize a standard testing procedure, including unambiguous criteria for concluding positivity, and they must validate that procedure in blinded experiments. Moreover, these experiments should address factors such as substance used (banned and not), dose of the substance, methods of delivery, timing of use relative to testing, and heterogeneity of metabolism among individuals."

Here's one of multiple problems with his argument. If you openly define and publicize a standard testing procedure, you're essentially giving the drug manufacturers all the information they need to beat the tests! Providing the details of the testing is like giving Steroid Manufacturer "A" the keys to the castle. Berry is just expecting us to count on the moral authority of drug manufacturers??
On another note, he's got 4 sources, 3 of which are his own articles. Hardly what I would call an exhaustive review of relevant literature.

Last edited 2/26/09 6:18PM server time by mictlantechutli
Edit note/reason: n/a

Post #6   2/26/09 6:09:03PM   

jakeiceman

Banned

jakeiceman Avatar
 
 
 


 
 
Posts:805
Career:185-141
Joined:Apr 2007
Chips:
140

Posted by roadking95th

Rush, this is very interesting. Could you give us a brief layman's version?



I agree it was a good read, definately refreshing (atleast it was not vaselinegate). Not gonna lie, I had to read some parts a bit slower

Post #7   2/26/09 6:22:50PM   

roadking95th

In Cael We Trust!

roadking95th Avatar
7
 
 
 
 


 
 
Posts:1,030
Career:1,351-714
Joined:Jan 2007
Chips:
180

Posted by jakeiceman


Posted by roadking95th

Rush, this is very interesting. Could you give us a brief layman's version?



I agree it was a good read, definately refreshing (atleast it was not vaselinegate). Not gonna lie, I had to read some parts a bit slower



I'm not going to lie either! Was going a little cross eyed there.

I remember Rush had one of his quizzes. I believe the subject is what he is going to school for. Anyway, I looked up some of the stuff to get answers...AND I STILL GOT IT WRONG

Post #8   2/26/09 8:55:11PM   

Rush

Laying down the beats

Rush Avatar
4



 
 
 


 
 
Posts:6,291
Career:759-539
Joined:Jan 2007
Chips:
835
I'll do my best to post a summary, but I am not interested in posting on forums right now, so I can't make promises. This was just a copy of an article I read yesterday that I thought might be interesting.


edit:
One thing I do recall is the author's criticism of committees setting a threshold for a particular substance or hormone. He suggested that studies need to be done and in some cases looked at (the ones that are ignored by doping committees) and address the statistical significance of whether the test was a false positve or false negative. What he was trying to say was (or rather ask the question) if an athlete is accused of doping based on a test result and that result (statistically) had a 60% chance of being a false positive, does that mean the athlete is guilty? Since they don't use these types of statistical analyses (according to the author) in current doping tests, there are a number of people wrongly accused and punished that are innocent.


Also, be sure to read the resposes in the second post as they offer a more balanced opinion on the matter.

I also debated posting another one I read about multi-drug resistent staph (and other bacteria) in hopes that maybe some dojos and fighters kept their shit a little cleaner.

Last edited 2/27/09 2:15PM server time by Rush
Edit note/reason: n/a
3 total post edits

Post #9   2/27/09 2:08:07PM