Unit and Outcome Data

The process has been outlined at the recent Annual Business Meeting and was unanimously supported.  The process will be starting this year and it is anticipated that the final endpoint will be reached in 2024, with a wider range of outcome measures being analysed. This will entail gathering data from more clinical areas than we currently do. Where possible, the NICOR dataset will be expanded to collect this data and in other areas a regular survey will need to be completed by unit leads. 

Quality Assurance Survey

Fundamentally the process for patient data for each operation will remain the same. The main change is that every three months there will be an online survey sent to each unit of high-level themes, which will take around ten minutes to complete. Rather than waiting three years before analysing performance, this new process asks each unit to review its outcomes every three months, so that if there is negative variance, supportive measures can be utilised. 

There will be some overlap with the GIRFT initiative (with whom we have also been collaborating), but we envisage the Society being a central repository for the unit level data and thereby providing a national picture for members and the general public. The aim of this programme is to look at the quality of care throughout the whole patient pathway from pre-admission decisions to post-operative care and discharge. The programme will also look at departmental governance processes and internal audit mechanisms for addressing variation in outcomes. In addition to mortality, units will be expected to regularly review reopening rates, the incidence of new renal failure, permanent stroke and deep sternal wound infections.  

Initially, the unit level data collected will be relatively broad and by the means described above. In addition to the standard surgical mortality and morbidity measures, this programme will look at other areas, such as multidisciplinary team meetings, quality indicators related to infrastructural support, including accreditation of both perfusion and echocardiography services, as well as indicators of working practices (e.g., consultant of the week). 

In order to support units and members, the SCTS has run a series of webinars, an example of which can be seen below:

As the programme matures, it is anticipated that the data will become more granular and that this detail will inform individual units of their outcomes at a much earlier stage than the current reporting process. The immediate plan is to start trialling data collection from April 2021 and work towards the first data for reporting to be collected in July 2021. 

This quality assurance programme will only be considering the data at unit level. There is no intention to look at an individual doctor’s/healthcare professional’s performance. Any data that it is put onto the Society website or in the public domain will be at a unit level only. The Society is currently looking at ways in which data can be portrayed to provide meaningful insights but equally do not easily lend themselves to producing league tables.  

Once we have a cohort of preliminary data, this will be discussed and presented to the adult cardiac surgical subcommittee, the board of representative’s meetings and the wider membership through the annual business meeting. Feedback from these discussions will help to further refine the programme. 

Simon Kendall, SCTS President
Uday Trivedi, SCTS Adult Cardiac Surgery Audit Lead
Andrew Goodwin, NICOR Adult Cardiac Surgery Audit Lead
Doug West, SCTS Audit Sub-committee Co-chair
Narain Moorjani, SCTS Honorary Secretary

A joint report from the Association for Cardiothoracic Anaesthesia and Critical Care, the British Cardiovascular Intervention Society, the British Cardiovascular Society, the British Heart Valve Society and the Society for Cardiothoracic Surgery

Read the full Report

This paper proposes the changes to the structure of the audit that allow it to more effectively assess modern surgical practice, and to more effectively support quality assurance and quality improvement in the future.

Read the full paper in the link below:

Proposal to Improve the Audit - 2021 to 2024 and beyond

For the latest and previous Thoracic Blue Books visit the Reports page.

Cardiac surgery is an integral part of the TAVI Heart team. On site cardiac surgery is an absolute requirement to support a TAVI service and is recommended by ESC / EACTS and North American guidelines. Cardiac surgery service provides:

• Direct MDT input for all potential TAVI patients
• Emergency surgical bailout for complications including left ventricular perforation by guide wire and annular rupture, or valve embolization which are immediately life threatening and can only be successfully treated with immediate surgery
• Emergency surgery for peripheral vascular access complications.
• Alternate TAVI approaches (e.g. transapical, subclavian or direct aortic) which are led by cardiac surgeons.

Details of all TAVI procedures and their outcomes are submitted to NICOR. Criteria for defining outlier performance is currently being agreed, but it is expected that the BCS Outlier policy will be used to implement Society advice. At present 30-day mortality and major complications including rate of vascular complication or stroke are used to measure safety. However other outcomes such as change in symptoms and quality of life may be used in the future.

Visit the BCIS website to find links to the latest data.

2018 - 2019 TAVI Data

EuroSCORE
The simple additive EuroSCORE model is now well established and has been validated in many patient populations across the world. It is easy to use, even at the bedside. It is very valuable in quality control in cardiac surgery and gives quite a useful estimate of risk in individual patients. However, particularly in very high-risk patients, the simple additive model may sometimes underestimate the risk when certain combinations of risk factors co-exist.

Logistic EuroSCORE

Using the same risk factors as the additive model, the logistic regression version of the score (the “logistic EuroSCORE”) can be calculated.  For a given patient, the “logistic EuroSCORE”  is the predicted mortality according to the logistic regression equation.

Roques F, Michel P, Goldstone AR, Nashef SA. The logistic EuroSCORE. Eur Heart J. 2003 May;24(9):882-3

For the analysis of surgeon and unit specific reslts, NICOR uses a calibrated logistic EuroSCORE.

EuroSCORE II
The previous additive EuroSCORE models are now out of date. A new model has been prepared from fresh data and was launched at the 2011 EACTS meeting in Lisbon. The model is called EuroSCORE II. The full logistic version of EuroSCORE (EuroSCORE II) produces more accurate risk prediction for a particular high-risk patient. Its main disadvantage is that the risk has to be calculated in quite a complex way - not by mental arithmetic or "on the back of an envelope".

STS Calculator
The STS Short-Term Risk Calculator allows you to calculate a patient’s risk of mortality and morbidities for the most commonly performed cardiac surgeries. The Risk Calculator incorporates STS risk models that are designed to serve as statistical tools to account for the impact of patient risk factors on operative mortality and morbidity.

In the new Euroscore II model the previous definitions of operative urgency were expanded to 4 classes: 

1. Elective : routine admission for operation 

2. Urgent: patients who have not been electively admitted for operation but who require intervention or surgery on the current admission for medical reasons. These patients cannot be sent home without a definitive procedure. 

3. Emergency: operation before the beginning of the next working day after decision to operate. 

4. Salvage: patients requiring cardiopulmonary resuscitation (external cardiac massage) en route to the operating theatre or prior to induction of anaesthesia. This does not include cardiopulmonary resuscitation following induction of anaesthesia. 

Currently Elective and Urgent are included in surgeon-specific outcomes reporting by NICOR and Emergency and Salvage are not (a number of other operative categories are also excluded by NICOR e.g. LVAD, pericardiectomy). The problem however is the definitions used here can be interpreted differently depending on the clinical scenario and the involvement of any different surgeons at different time-points. There is ambiguity in the definition of emergency as it does not state who makes the decision to operate and whether that decision is valid across all the surgeons in a unit. A patient could have been in hospital a number of days before a decision to operate is made. One surgeon may choose not to operate and a day later another surgeon may decide to operate, would that be an emergency case? 

The second issue is what constitutes completion of induction of anaesthesia? Arrival in the anaesthetic room or once patient intubated or when the patient is ready to enter the operating room? There are numerous ways to interpret the Euroscore definitions and variability in operative urgency classification may lead to invalid comparison between units and surgeons. 

To gain a better picture of how individuals and units classify operative urgency a Survey Monkey poll was carried out by the Society. This consisted of ten clinical scenarios and the respondent was asked to classify the operative urgency of each case. The results of this survey are available here

These findings were presented at the Board of Representative meeting on 27th September 2019 and discussed at a meeting of the Adult Cardiac Sub-committee on 8th October 2019. From the findings and discussions the following recommendations are made. 

1. Cases that are admitted electively and go to surgery without significant clinical change should be classed as Elective

2. Patients who are kept in for surgery following non-elective admission and not operated on within 24 hours (irrespective of weekday or weekend) should be classified as Urgent cases. 

3. Case operated on within 24 hours of admission for life-threatening conditions (e.g. dissection; VSD) or where clinical deterioration of an inpatient warrants surgical intervention within 24 hours (such as ongoing ischaemia), should be classed as Emergency 

4. Patients in whom there is rapid clinical deterioration (irrespective of route of admission), who need to have surgery expedited as a life saving measure or need external massage/inotropes started/in shock prior to commencement of surgery (knife to skin) should be classed as Salvage. In some patients it may be more appropriate for them to have non-surgical therapeutic interventions to either stabilise prior to deferred cardiac surgery or as a definitive treatment. 

For patients who come under the definitions described in points 3 & 4, it is recommended that where possible a mini-MDT is convened and the decision documented in the notes.

The following guidance is to assist units in deciding how to allocate cases for the dual consultant operating where the outcome will be measured at a unit rather than surgeon specific level. The guiding principle for this is to provide best practice to ensure high quality care and outcomes for patients and counter any individual risk averse behaviour. This guidance is for elective and urgent cases where there would be time for discussion and scheduling to accommodate two consultant surgeons. In emergency situations we do not recommend delay in treatment and the current system does not include emergency/salvage cases in surgeon specific outcomes.

Guidelines

1. The decision to allocate any patient for dual consultant operating has to be made prior to the date of surgery. This should be clearly documented at an MDT meeting and recorded in the notes.
2. The unit audit lead is expected to keep a log of all cases which will have dual consultant operating. This will need to be presented at local clinical governance meetings on a regular basis.
3. Both consultant surgeons are expected to be present for the duration of the case and scrubbed for the majority of the time at the operating table.
4. There should be documented planning of the operative strategy that has been agreed by both surgeons.
5. Cases of mentoring will not count towards dual consultant operating and will still be allocated to a single surgeon.

Additional Information

It is recognised that individual units vary significantly at a national level in terms of infra-structure, size and caseload. Due to this no specific guidance can be given to units about which cases should be included. It is expected that the units will be looking at high risk or complex cases in the main. A case could be made for dual consultant operating when innovative procedures are being introduced in order to maintain patient safety. However, this would be for truly innovative surgery and not a procedure which is new to the unit. In the latter case the appropriate training, governance and proctoring are expected, and the cases will be attributed to a single surgeon.

This initiative is to ensure that high risk cases which would benefit from surgery are not denied surgery. NICOR will monitor the percentage of cases being allocated under this system and further feedback will be provided. Units may notice an increase in the risk score of the patients they operate on. The SCTS recognises that over a number of years, this may lead to a rise in raw mortality.

NICOR SOP – NCAP Outlier Policy

Guidance on how to respond to a negative NCAP individual operator ‘alert’ or ‘alarm’ for:

• Medical Directors
• Chief Executives,
• Local NCAP audit leads

The recommendations below are to offer guidance to a hospital following notification from NICOR of a negative individual operator alert or alarm from the NCAP.

Introduction

Delivering the highest quality of care possible and ensuring patients are protected from avoidable harm is the highest priority. However, it is also fundamental that before reacting to potentially negative outcome notifications, that the quality of data is confirmed as being correct. Individuals with a suspected or confirmed alert need appropriate support and also to be managed in a constructive and sensitive way. Medical directors need to ensure that a transparent and objective action plan is developed that should focus on addressing concerns so that an individual operator can be expected to return to independent practice, accepting that on rare occasions this may not always be possible.

Identification of an individual consultant as a potential outlier is an extremely stressful period for the consultant involved. This event can have a major impact on the individual’s wellbeing as well as the direct impact professionally. Therefore, it is imperative that individuals are appropriately supported and that a clear set of actions are followed.

If analysis of NCAP data suggests that an individual operator is a potential outlier that satisfies the definition of ‘alert’ or ‘alarm’ status, then NICOR will contact the Medical Director/ Chief Executive of the relevant trust.

Definitions

Suspected outlier– initial analysis of NCAP audit data has potentially identified an individual consultant as a negative outlier but the quality of the audit data has not been reviewed and confirmed as correct.

Confirmed outlier– following review and revision of submitted audit data (if indicated), re-analysis demonstrates that the individual is a negative outlier

Handling of suspected Outliers at ‘alert’ or ‘alarm’ Level

Any identification of ‘outlier’ status indicates a statistically significant value and does not necessarily mean outlying performance by a consultant. Outlier status is a statistical warning and may be due to chance alone.

Risk of False Positive Outlier status and False Discovery rate: It is important that Medical Directors and Chief Executives appreciate that due to the large number of analysis that underpin the individual consultant NCAP analyses, false positive alerts will be expected to occur, even when an individual may have acceptable performance. This is especially relevant for ‘alert’ status and a statistical outlier may not equate with poor performance.

Judgements on performance can only be made after a full examination of all the issues involved in the delivery of care, and this may be multi-factorial and complex. Medical directors are advised not to solely react to the statistical data without appreciating the limitations of the statistical methodology and having considered all potential explanations for the outlier status.

A key priority for a hospital/ department with a clinician suspected of being an outlier is to undertake a detailed and comprehensive review of the clinical audit data.

The medical director must ensure that adequate resources are allocated to the investigation and it is undertaken rapidly in order for the hospital to work towards the timelines laid out in the NCAP Outlier SOP.

The Medical Director/ Chief Executive is responsible for acknowledging receipt of the letter from NICOR.

Local National Clinical Audit Lead for the domain within the hospital is responsible for:

• Providing written confirmation of the accuracy of data underpinning the analysis following initial notification
• Amending the live data if the data underpinning the outlier analysis are inaccurate
• Discussing with the NCAP Domain Clinical Lead, NCAP Chief Operating Officer if a potential outlier is detected and there is a ‘case to answer’
• Providing NCAP with the reasons why the original data were inaccurate and outlining changes that have been made to prevent a recurrence
• Discussing the findings with the local clinical and governance leads

If the individual consultant with a confirmed alert or alarm has a position of responsibility relating to the management of the service (e.g. audit lead, clinical director, medical director) then their responsibilities with regard to the management of the notification, should be delegated to another appropriate individual. The Chief Executive should be responsible for this action.

Handling of Confirmed Outliers at ‘Alert’ Level

General principles

A confirmed alert from NICOR (after an initial review of data quality) should trigger a review of the whole of a consultant’s practice including relationships with colleagues and other aspects of team functioning.

A review should cover all sites at which the doctor practises, including any private practice.

The hospital/ trust must undertake a review and further analysis of local data, which may include a review of data prior to the time period included in the alert and also a review of more contemporary data that may not have been included in the analysis time period.

Hospitals must provide the support and facility for further analysis of locally held data, requesting support from NICOR and the specialist societies where appropriate.

Engagement and support of individual consultant

Medical directors (and the relevant Clinical Director):

• must support colleagues who are identified as a potential outlier
• should ensure good communication and engagement with the individual consultant
• should ensure that individual consultants are informed of any investigations and kept updated about progress of any investigations/ review of practice
• should allow individual clinicians to comment on any findings
• must support an individual colleague who requests assistance
• Responsibility of the individual consultant with potential outlier status

Individual doctors must engage with outlier status management and investigation

Non-engagement should result in prompt referral to the Medical Director

Suggested (but not mandatory) areas for review include but are not limited to:

• Detailed review of audit data quality/ accuracy and completion
• Review of individual case notes of deceased patients
• Cross-checking of catheter laboratory or theatre records
• Review of MDT records
• Review of case load and case mix
• Consider subspecialist practice not adjusted for by the risk prediction model
• Consideration of institutional factors (e.g. referral practices, provision of intensive care, post-operative services)
• Undertake a time trend analysis of mortality to ascertain at what stage survival rates started to decline

Team dynamics: investigation may include the assessment of an individual’s role within the wider clinical team

Consider external review from relevant Royal College if appropriate

Actions to be considered with regard to a clinician’s current practice:

• mentoring of individual consultant (internal or external colleague or group of colleagues) 
• temporary modification in practice
• restriction of cases to elective case for defined period
• review of waiting list and removal/ redistribution of higher risk cases to colleagues
• partial or complete supervision of practice
• having new high risk case referrals reviewed by colleague/s for second opinion
• joint consultant operating for all or high risk cases
• operative planning of individual cases (or cases or just high risk cases) with nominated colleague/ mentor
• case-mix adjustment
• further training, if further training felt to be helpful:

Consider a broad range of additional training opportunities

• sabbatical/ visiting other centres
• peer review and support
• recognised training courses
• utilising simulator training

Involvement of external bodies

If internal investigation of outcomes raises concerns that cannot be dealt with internally then advice should be sought from the relevant professional society or GMC at an early stage.

For cardiac surgery this is the Society of Cardiothoracic Surgeons in collaboration with the Royal College of Surgeons: https://www.rcseng.ac.uk/standards-and-research/support-for-surgeons-and-services/irm/

For cardiology the British Cardiovascular Society in Collaboration with the British Cardiovascular Intervention Society and the Royal College of Physicians: https://www.rcplondon.ac.uk/invited-reviews

Outcome of review

The outcome of the review process and any actions should be agreed and signed off by the clinical director and documented in the individual consultant’s appraisal folder.

This documentation should also be presented as part of their evidence for revalidation.

The Responsible Officer must be fully informed of any concerns, reviews and remedial action

Actions from Confirmed Outliers at ‘Alarm’ Level

NICOR will inform the Chief Executive and the Medical Director of the individual consultant outlier alarm status. The individual clinician and clinical director will also be contacted by NICOR.

Chief Executive will be required to:

• Acknowledging receipt of the letter in all cases
• Confirm that a local investigation will be undertaken and that the GMC Employment Liaison Advisor has been informed
• inform relevant bodies about the NCAP concerns
  • Clinical Commissioning Groups
  • NHS Improvement
  • relevant professional society and Royal College
  • the GMC Employment Liaison Advisor (ELA)
  • Care Quality Commission.
• Consider initiating an external review of the consultant’s practice through the relevant Royal College and professional society (For cardiac surgery contact the Society for Cardiothoracic Surgery in collaboration with the Royal College of Surgeons: https://www.rcseng.ac.uk/standards-and-research/support-for-surgeons-and-services/irm/ For cardiology contact the British Cardiovascular Society in Collaboration with the Royal College of Physicians: https://www.rcplondon.ac.uk/invited-reviews)

Medical Director:

• Needs to be responsible for the quality of audit data if alarm status is confirmed after initial review and investigation of local data
• Ensure individual consultant is adequately supported
• Undertake a risk assessment of individual clinicians continued practice
• Make a decision around continued practice, supervised practice or temporary or permanent exclusion
• Ensure current practice status of individual consultant is communicated to the clinical team and referring colleagues
• Ensure any problems and issues to be addressed are clearly defined and documented
• Document the remedial action plan (in collaboration with external review)
• Develop and document the mechanism of how patients or relatives can discuss concerns they may raise
• Develop and document the mechanism of how the organisation will respond to media enquiries
• Consider seeking advice from HQIP and appropriate professional organisations
• By the time outlier alarm results are published the organisation should have a good understanding of why the survival rates are not as expected
• Set an appropriate specified time period for when a full investigation and documented action plan will have been completed
• Consider all of the options for investigation and remedial actions described above under Handling of Confirmed Outliers at ‘alert’ Level

Where a problem has been identified and appropriately addressed

Medical Director:

• Robust action plan of remedial actions must be documented and made available (including to patients and other external stakeholders if required)
• Ensure that both external review and validation of findings and any decisions related to practice has been undertaken
• Ensure that there is a comprehensive communication plan to stakeholders when an individual outlier status has been addressed/ resolved/ explained

Due to the large number of tests being conducted in the NACSA audit we can expect false positives to occur in the hospital and consultant level analyses, even if all have acceptable performance. The proportion of those units or surgeons found to be outliers that are false positives gives the chance of a positive finding being a false positive, the “False Discovery Rate”, i.e. the chance that a unit determined to be outlying is in fact performing within the ‘control limits’.

Estimate of False Discovery Rate

We used the upper bound estimate for False Discovery Rate suggested by Professor Sir David Spiegelhalter (“How confident can we be that ‘outlying’ units are ‘truly outlying’?”, communication July 2015) for the False Discovery Rate where  is the number of units eligible for reporting,  is the level of significance and  is the number of outliers found.

Numbers of units

There are 39 hospitals eligible for reporting (and 282 eligible consultants).

Alarms

Using the level of significance =0.001 (99.8% limits), we found 2 hospitals and 2 consultants to be Alarms.

For hospitals the expected number of chance Alarm findings, assuming all hospitals to have acceptable performance, is 0.039 each year (1 every 26 years). The Alarm False Discovery Rate is estimated to be 0.02, i.e. we can expect at least 98% of the 2 hospitals found to be Alarm to be true outliers.

For consultants the expected number of chance Alarm findings, assuming all hospitals to have acceptable performance, is 0.282 each year (1 every 3.5 years). The Alarm False Discovery Rate is estimated to be 0.14, i.e. we can expect at least 86% of the 2 consultants found to be Alarm to be true outliers.

Alerts

Using the level of significance =0.025 (95.0% limits), we found 5 hospitals (2 Alarms) and 14 consultants (2 Alarms) as Alert or Alarm.

For hospitals the expected number of chance Alert or Alarm findings, assuming all hospitals to have acceptable performance, is 0.975 each year. The Alert or Alarm False Discovery Rate is estimated to be 0.20, i.e. we can expect at least 80% of the 5 hospitals found to be Alert or Alarm to be true outliers.

For consultants the expected number of chance Alert or Alarm findings, assuming all hospitals to have acceptable performance, is 7.05 each year. The Alert or Alarm False Discovery Rate is estimated to be 0.50, i.e. we can expect at least 50% of the 14 consultants found to be Alert or Alarm to be true outliers.

Significantly Higher than Expected

Using the level of significance =0.001 (99.8% limits), we found 3 hospitals and 0 consultants as Significantly Higher than Expected.

For hospitals the expected number of chance ‘Significantly Higher than Expected’ findings, assuming all hospitals to have acceptable performance, is 0.039 each year (1 every 26 years). The Significantly Higher than Expected False Discovery Rate is estimated to be 0.01, i.e. we can expect at least 99% of the 3 hospitals found to be Significantly Higher than Expected to be true outliers.

Higher than Expected

Using the level of significance =0.025 (95.0% limits), we found 5 hospitals (3 Significantly Higher than Expected) and 7 consultants (0 Significantly Higher than Expected) as Higher than Expected.

For hospitals the expected number of chance Higher than Expected findings, assuming all hospitals to have acceptable performance, is 7.05 each year. The Higher than Expected False Discovery Rate is estimated to be 0.20, i.e. we can expect at least 80% of the 5 hospitals found to be Higher than Expected to be true outliers.

For consultants the expected number of chance Higher than Expected findings, assuming all hospitals to have acceptable performance, is 7.05 each year. The Higher than Expected False Discovery Rate is estimated to be 1.00, i.e. we cannot expect any of the 7 consultants found to be Higher than Expected to be true outliers.

Please refer to the link below to view the 'Outlier identification and monitoring of institutional or clinician performance: an overview of statistical methods and application to national audit data'

https://rdcu.be/c29Lo