2012 lecture 2 test performance charactaristics

Views:
 
Category: Entertainment
     
 

Presentation Description

No description available.

Comments

Presentation Transcript

Test Performance Characteristics:

1 1 Test Performance Characteristics What is normal? Is the result different from the last time this patient was tested Diagnostic utility of tests

Is the Result OK, NORMAL??:

2 Is the Result OK, NORMAL?? How to define healthy (normal) from unhealthy (diseased, not normal, at risk, etc.) First need to define the values against which to compare the patient’s result.

PowerPoint Presentation:

3 Repeated measure of a single sample will give a “normal” distribution of values. Measurement of a homogenous population will often give a “normal’ distribution of results. Choose mean + 2 SD as “normal” – 2.5% on each side “not normal”

Problems with Normal:

4 Problems with Normal Normal has both statistical and health meanings. Not all analytes are normally distributed or can be transformed to normal. Abnormal result ≠ pathology and normal result ≠ absence of pathology Upper and lower 2.5% Normal ≠ Ideal ≠ Health If you look hard enough, almost all “normal, healthy” people will have one or more tests that are “abnormal” Screening and panel testing

Distribution of Total Cholesterol values in Canadians aged 20 -70 :

5 5 Distribution of Total Cholesterol values in Canadians aged 20 -70 Concentrations > 5.2 mmol/L associated with significant increased risk for Cardiovascular disease

Alternative to Normal:

6 Alternative to Normal REFERENCE INTERVAL Removes the word “normal” Allows comparison of the result to a value, or range of values that can be derived in different ways. Statistical Risk-based Based on therapeutic effect

Is it different?:

7 Is it different? What amount of change between two results is required to be sure that there has been a ‘real’ change? If the analytical and biological variability for an analyte is known, then the total variability can be calculated Changes greater than this can be assumed to be real ? Clinical significance See case history 1.2

PowerPoint Presentation:

8 The critical difference between two results (at a statistical significance of p <.05), taking into account only analytical variability is 2.8 times the analytical variability. Analytical variability can be easily determined in each laboratory by repeated measurement of the same sample, or by using published literature describing analytical performance of tests. Mr. Smith has his blood taken for determination of total protein concentration. The result is 70 g/L. One hour later his blood is taken again and the result this time is 72 g/L. Are these two results different? Figure 1.5 indicates that the analytical variability is 1g/L These two results are unlikely to be statistically different.

PowerPoint Presentation:

9 Biological variability can also be determined Requires much more work Or taken from the published literature See figure 1.5

PowerPoint Presentation:

10 So – we need to take into account both analytical variability and biological variability Total variability = 2.8 x SD ana 2 + SD biol 2 Mr. Jones has plasma sodium measured. The result is 141 mmol/L (the reference interval for this test is 135-145 mmol/L) Two days later he has this test done again. This time the result is 146 mmol/L. He is quite concerned that the result now appears to have changed from “normal” to “abnormal” 2.8 x 1.1 2 + 2 2 = 6.4 Mr. Jones can be reassured that this change is unlikely to be clinically significant and is most likely the analytical and biological variability of this analyte.

Diagnostic Characteristics and Utility :

11 11 Diagnostic Characteristics and Utility What are the characteristics of tests that can help us to decide whether or not the test is useful? In clinical practice? In research?

The Diagnostic Process:

12 12 The Diagnostic Process Obtain evidence about patient to form initial impression Create a list of possible diagnoses - “Pre-test probability” Decide what further information is required to rule in or rule out the presumptive diagnosis. Perform “test” to increase or decrease the probability of the various options – “Post-test Probability”.

PowerPoint Presentation:

13 13 (Prevalence)

Some important terms:

14 Some important terms Prevalence Criterion Standard or “Gold Standard” Sensitivity Specificity Positive Predictive Value Negative Predictive Value

Prevalence Criterion Standard:

15 Prevalence Criterion Standard The total number or proportion of individuals who have the outcome or condition of interest At any given point in time. In the population of interest. The test or criterion that is regarded as definitive. The ultimate arbiter of “truth” The “ Gold Standard ” “Outcome”

Sensitivity Specificity:

16 16 Sensitivity Specificity The proportion of individuals with the outcome of interest who have a positive test. A highly sensitive test will have few false negatives Tests with high sensitivity can be used to “rule out” a diagnosis. The proportion of individuals without the outcome who have a negative test. A highly specific test will have few false positives Tests with high specificity can be used to “rule in” a diagnosis.

Predictive Values:

17 Predictive Values Positive Predictive Value (PPV) The proportion of all positive results that are “true positives” Given a positive test, how likely is it that the patient has the outcome of interest? Negative Predictive Value (NPV) The proportion of all negative results that are “true negatives” Given a negative test, how likely is it that the patient is free of the outcome of interest?

PowerPoint Presentation:

18 Healthy volunteers Sick patients Value of Test parameter Frequency of Response Decision threshold between positive and negative

PowerPoint Presentation:

19 19

How Do We :

20 How Do We Convert these definitions into usable concepts ? Assess the diagnostic utility of a test? Manipulate these parameters to optimize the performance of a test?

It all Starts with a 2 x 2 Table:

21 21 It all Starts with a 2 x 2 Table Outcome = Criterion Standard or Gold Standard

PowerPoint Presentation:

22 22 Sensitivity = The proportion of individuals with the outcome of interest who have a positive test Specificity = The proportion of individuals without the outcome who have a negative test.

Diagnosis of Biochem 3H03itis can a blood test diagnose academic success or failure?:

23 Diagnosis of Biochem 3H03itis can a blood test diagnose academic success or failure? Gold standard (pos/neg outcome) – fail/pass, based on final mark 150 students in the class 15 are found to have 3H03itis by looking at final mark (Prevalence = 10 %) Potential blood test is available to Dx this disorder Sens 80%, Spec 85% OUTCOME " Biochem 3H03itis " Pos Neg Blood Test Pos 12 20 Neg 3 115 12/15 =80% 115/135 =85%

Characteristics of Sensitivity and Specificity :

24 24 Characteristics of Sensitivity and Specificity Provide information about how the test behaves in populations. Difficult to apply to individual patients Not affected by prevalence. Can be affected by spectrum bias Choice of decision point will affect the values (threshold effect)

PowerPoint Presentation:

25 25 % Spectrum Bias Very healthy volunteers Very sick patients Test parameter Test threshold

PowerPoint Presentation:

26 26 Threshold effect

Design the test to maximize the sensitivity reduce test threshold – minimise false negatives will result in increased false positives:

27 OUTCOME Pos Neg Blood Test Pos 15 40 Neg 0 95 15 135 100 70.37 Design the test to maximize the sensitivity reduce test threshold – minimise false negatives will result in increased false positives

Design test to maximize specificity Increase test threshold –minimise false positives Will increase the false negative rate:

28 Design test to maximize specificity Increase test threshold –minimise false positives Will increase the false negative rate OUTCOME Pos Neg Blood Test Pos 8 3 Neg 7 132 15 135 53.33 97.78

Predictive Values :

29 29 Predictive Values Positive Predictive Value – given a positive result, how likely is it that the individual has the outcome. Negative Predictive Value – given a negative result, how likely is it that the individual does not have the outcome. Are useful in applying test information to individual patients. PPV and NPV are highly dependent on prevalence.

PowerPoint Presentation:

30 30 Sensitivity = TP/TP + FN = a/a + c Specificity = TN/ TN + FP = d/ b + d PPV = TP/TP + FP =a/ a + b NPV = TN/TN + FN = d/ d + c Outcome Pos Neg Test Pos True Positive (a) False Positive (b) a + b Neg False Negative (c) True Negative (d) c + d a + c b + d

PowerPoint Presentation:

31 If you have positive test – how likely is it the you will end up with 3H03itis in May? If you have a negative test – how likely is it that you will pass the course? OUTCOME " Biochem 3H03itis " Pos Neg Blood Test Pos 12 20 12/32 =37.5% Neg 3 115 115/118 =97.5% 12/15 =80% 115/135 =85%

PPV and NPV are highly dependent on prevalence. Prevalence = 10%:

32 PPV and NPV are highly dependent on prevalence. Prevalence = 10% OUTCOME " Biochem 3H03itis " Pos Neg Blood Test Pos 12 20 12/32 =37.5% Neg 3 115 115/118 =97.5% 12/15 =80% 115/135 =85%

PowerPoint Presentation:

33 OUTCOME " Biochem 3H03itis " Pos Neg Blood Test Pos 40 15 40/55 = 72.7% Neg 10 85 85/95 = 89.4% 40/50 =80% 85/100 =85% What if the prevalence was 33% rather than 10%? Sensitivity and Specificity do not change PPV and NPV change quite a bit.

Manipulating Test Characteristics can Change the Utility of a Test :

34 Manipulating Test Characteristics can Change the Utility of a Test When screening, it is important that everyone with the condition be identified. Missing an outcome (False Negative) is not good. Set the threshold for decision very low – high sensitivity. Downside is a high False positive rate – low specificity.

Design the test to maximize the sensitivity reduce test threshold – minimise false negatives will result in increased false positives:

35 OUTCOME Pos Neg Blood Test Pos 15 40 55 27.27 Neg 0 95 95 100 15 135 100 70.37 Design the test to maximize the sensitivity reduce test threshold – minimise false negatives will result in increased false positives

PowerPoint Presentation:

36

PowerPoint Presentation:

Is Screening for Breast Cancer Using Mammograms Effective? The Globe and Mail Friday, Dec. 02, 2011 Outcome: Preventable Breast Cancer Death

PowerPoint Presentation:

38 In some cases it is important that the false positive rate be very low. If the therapy is risky and expensive, then only those who will surely benefit should be offered the treatment. Set the threshold for decision very high – high specificity Downside will be a high false negative rate – low sensitivity

Design test to maximize specificity Increase test threshold –minimise false positives Will increase the false negative rate:

39 Design test to maximize specificity Increase test threshold –minimise false positives Will increase the false negative rate OUTCOME Pos Neg Blood Test Pos 8 3 11 72.73 Neg 7 132 139 94.96 15 135 53.33 97.78

PowerPoint Presentation:

40

How can we summarise the data regarding performance at various thresholds? :

41 How can we summarise the data regarding performance at various thresholds? Receiver Operator Characteristic (ROC) Curves are a convenient way to summarize the sensitivity and specificity of a test at various decision thresholds Allows one to optimize sens, spec, or to take the best combination of both. Calculate sens and spec for each decision point Plot sensitivity vs 100-specificty

PowerPoint Presentation:

42 Plot sens vs 100-spec for each decision point. Decision point nearest Upper left hand corner is The best combination of Sens and Spec Closer the line gets to the Left hand corner, the better the test. The Area Under the Curve (AUC) represents the overall efficiency of the test

PowerPoint Presentation:

43 14 Present 13 absent 13 absent 13 absent 12 absent 12 Present 11 absent 10 Present 10 absent 9 absent 9 absent 9 absent 8 absent 7 absent 6 absent 6 absent 5 absent 4 absent 4 absent 3 absent 2 absent 2 absent Test Value Outome 25 Present 20 Present 19 absent 18 Present 18 absent 17 Present 16 Present 15 absent 15 Present 15 absent 14 absent Test Value Outome For each test value determine # of TP, TN, FP, FN Calculate Sens and Spec Plot An Example How well does our test for Biochem 3H03 perform?

PowerPoint Presentation:

44 Test Value   (abnormals above cut-off)   Sensitivity Specificity TP TN FP FN - 100.0% 0.0% 9 0 24 0 2 100.0% 8.3% 9 2 22 0 3 100.0% 12.5% 9 3 21 0 4 100.0% 20.8% 9 5 19 0 5 100.0% 25.0% 9 6 18 0 6 100.0% 33.3% 9 8 16 0 7 100.0% 37.5% 9 9 15 0 8 100.0% 41.7% 9 10 14 0 9 100.0% 54.2% 9 13 11 0 10 88.9% 58.3% 8 14 10 1 11 88.9% 62.5% 8 15 9 1 12 77.8% 66.7% 7 16 8 2 13 77.8% 79.2% 7 19 5 2 14 66.7% 83.3% 6 20 4 3 15 55.6% 91.7% 5 22 2 4 16 44.4% 91.7% 4 22 2 5 17 33.3% 91.7% 3 22 2 6 18 22.2% 95.8% 2 23 1 7 19 22.2% 100.0% 2 24 0 7 20 11.1% 100.0% 1 24 0 8 25 0.0% 100.0% 0 24 0 9

PowerPoint Presentation:

45

PowerPoint Presentation:

46 46 (Prevalence)

The Ideal Test :

47 47 The Ideal Test Ideal Marker Absolutely sensitive - few or zero false negatives Absolutely specific – few or zero false positives Easily measurable Quantity reflective of severity of disease Early detection following onset of disease Not affected by other biological disturbances There are few, if any, perfect biological markers for disease.

authorStream Live Help