False Positive Released

NICOR NACSA False Discovery Rate Analysis  17th August 2015

Due to the large number of tests being conducted  in the NACSA audit we can expect false positives to occur in the hospital and consultant level analyses, even if all have acceptable performance. The proportion of those units or surgeons found to be outliers that are false positives gives the chance of a positive finding being a false positive, the “False Discovery Rate”, i.e. the chance that a unit determined to be outlying is in fact performing within the ‘control limits’.

Estimate of False Discovery Rate

We used the upper bound estimate for False Discovery Rate

suggested by Professor Sir David Spiegelhalter (“How confident can we be that ‘outlying’ units are ‘truly outlying’?”, communication July 2015) for the False Discovery Rate where  is the number of units eligible for reporting,  is the level of significance and  is the number of outliers found.

Numbers of units

There are 39 hospitals eligible for reporting (and 282 eligible consultants).

Alarms

Using the level of significance  =0.001 (99.8% limits), we found 2 hospitals and 2 consultants to be Alarms.

For hospitals the expected number of chance Alarm findings, assuming all hospitals to have acceptable performance, is 0.039 each year (1 every 26 years). The Alarm False Discovery Rate is estimated to be 0.02, i.e. we can expect at least 98% of the 2 hospitals found to be Alarm to be true outliers.

For consultants the expected number of chance Alarm findings, assuming all hospitals to have acceptable performance, is 0.282 each year (1 every 3.5 years). The Alarm False Discovery Rate is estimated to be 0.14, i.e. we can expect at least 86% of the 2 consultants found to be Alarm to be true outliers.

Alerts

Using the level of significance  =0.025 (95.0% limits), we found 5 hospitals (2 Alarms) and 14 consultants (2 Alarms) as Alert or Alarm.

For hospitals the expected number of chance Alert or Alarm findings, assuming all hospitals to have acceptable performance, is 0.975 each year. The Alert or Alarm False Discovery Rate is estimated to be 0.20, i.e. we can expect at least 80% of the 5 hospitals found to be Alert or Alarm to be true outliers.

For consultants the expected number of chance Alert or Alarm findings, assuming all hospitals to have acceptable performance, is 7.05 each year. The Alert or Alarm False Discovery Rate is estimated to be 0.50, i.e. we can expect at least 50% of the 14 consultants found to be Alert or Alarm to be true outliers.

Significantly Higher than Expected

Using the level of significance  =0.001 (99.8% limits), we found 3 hospitals and 0 consultants as Significantly Higher than Expected.

For hospitals the expected number of chance Significantly Higher than Expected findings, assuming all hospitals to have acceptable performance, is 0.039 each year (1 every 26 years). The Significantly Higher than Expected False Discovery Rate is estimated to be 0.01, i.e. we can expect at least 99% of the 3 hospitals found to be Significantly Higher than Expected to be true outliers.

Higher than Expected

Using the level of significance  =0.025 (95.0% limits), we found 5 hospitals (3 Significantly Higher than Expected) and 7 consultants (0 Significantly Higher than Expected) as Higher than Expected.

For hospitals the expected number of chance Higher than Expected findings, assuming all hospitals to have acceptable performance, is 7.05 each year. The Higher than Expected False Discovery Rate is estimated to be 0.20, i.e. we can expect at least 80% of the 5 hospitals found to be Higher than Expected to be true outliers.

For consultants the expected number of chance Higher than Expected findings, assuming all hospitals to have acceptable performance, is 7.05 each year. The Higher than Expected False Discovery Rate is estimated to be 1.00, i.e. we cannot expect any of the 7 consultants found to be Higher than Expected to be true outliers.