Purpose 

The purpose of an independent audit is two-fold. First, an audit assures stakeholders (Database participants, patients, researchers, consumers, regulators, payers, and industry) that the clinical data in the five STS National Database components—Adult Cardiac Surgery, Congenital Heart Surgery, General Thoracic Surgery, Intermacs, and Pedimacs—are consistent, accurate, and complete. Second, the audit process is designed as an educational opportunity to complement the participant’s internal quality controls established to maximize the integrity of the data collected. 

Policy

1. Participation agreements in the Adult Cardiac Surgery Database (section 1.2.1 or 1.3.1), Congenital Heart Surgery Database (section 1.2.1), General Thoracic Surgery Database (section 1.2.1), Intermacs Database (section 1.2.4) and Pedimacs Database (section 1.2.4), require participants to submit accurate and complete data and provide cooperation and assistance with any independent audit.  

“Participant hereby warrants that all data submitted for inclusion in the Database will be accurate and complete, and acknowledges that such data may be subject to independent audit in accordance with STS Policies. Participant will use its best efforts to address any data or related deficiencies identified by the independent data warehouse service provider, and agrees to cooperate with and assist STS and its designees in connection with the performance of any independent audit.”

2. STS responsibilities: 

(a) Contract with an external independent organization (audit company) to provide secure, HIPAA-compliant auditing services
(b) Notify sites selected for STS audit participation
(c) Provide education to the audit company regarding data definitions 
(d) Explain the audit process and expectations to sites
(e) Send summary results to sites
(f) Provide oversight in review and interpretation of the audit results, including identification of data quality outliers and provision of remediation assistance, as deemed appropriate by the associated workforce or task force.
(g) Provide access to Database-specific educational resources to audited sites, especially those with results that do not meet the expectation of STS (please see section 7). These resources include: Optional attendance to Advances in Quality & Outcomes: A Data Managers Meeting (AQO), Database-specific training manuals, educational webinars, STS National Database Data Manager Collaborative Program, National Database News, and the FAQ inbox. 

3. Participant responsibilities: 

(a) Cooperate with external audit organization to schedule the audit 
(b) Provide records (at no charge) or access to records in a timely fashion 
(c) Provide records in a HIPAA-compliant format
(d) Participate in post-audit review and educational opportunities

4. Each of the Database task forces is responsible for identifying audit variables.

5. Audit site selection:

(a) Each year approximately 10% of sites in each Database will be audited
(b) To be included in the audit pool, a site must a. be an active Database participant for all 12 months of the audited time period and b. capture the specified minimum number of procedures/admissions during the audit time period.

DatabaseMinimum Number of Procedures/Admissions
Adult Cardiac Surgery Database20 isolated CABG procedures and 10 Valve or Valve + CABG procedures
Congenital Heart Surgery Database30 index operations
General Thoracic Surgery Database20 total lung cancer or esophageal cancer cases
Intermacs Database10 patients implanted
Pedimacs Database 3 patients implanted

(c) All US and Canadian sites are included in the audit pool except:
     1.    Sites that have been identified as not meeting expectations during a recent audit (Please see section 7).
       2.    Sites that have successfully completed an STS audit within the past 3 years.
(d) STS reserves the right to utilize various methodologies, including random and targeted selection.  
      1.    Targeted audits may be scheduled if requested by a participant or deemed appropriate by STS.  

6. Audit summary results: 
A final Summary Report will be emailed by STS to each site. The rate of agreement and data completeness will be considered when evaluating each site and identifying sites that do not meet the STS expectations (please see section 7).  
    
7. Data quality
(a) Definitions:   
     1.    Data Accuracy (Data Agreement) – rate of agreement between data submitted by the participant and data in the medical record as adjudicated by the auditors 
      2.    Data Completeness/Missing Data – data fields not completed or, for certain critical variables such as those related to Operative Mortality, entered as Unknown
      3.    Data Completeness/Missing Records – discrepancy between the case log and cases entered into the Database
(b) Data accuracy will be calculated three ways:
     1.    Variable accuracy (agreement) will be measured for each audited variable. The denominator will equal the total number of records audited. The numerator will be the number of records where that variable was answered completely and correctly. This will be reported back as a percentage to sites. 
     2.    Section accuracy (agreement) will be measured for each section (i.e., demographics, risk factors, complications, discharge, mortality, etc.). The denominator will equal the total number of variables within the corresponding section where the variable was available or should have been available to abstract, times the total number of records abstracted. The numerator will equal the total number of variables within the specific section that were abstracted completely and correctly times the total number of records. 
                a.    For example, if the demographics section is audited on 20 records and there are three variables within the demographic section and an answer was expected for each variable on all records, then the denominator will be 60. If the site had one abstraction error within the demographic section for each record for a total of 20 total errors, then the result would be 20/60 or 66.6% accuracy for the demographic section.
     3.    Overall site accuracy (agreement) will be measured for each site as an overall agreement rate. The denominator will equal the total number of records audited times the total number of variables that were available or should have been available for abstraction of each record. The numerator will equal the total number of variables that were abstracted completely and correctly times the total number of records audited at the site.
     4.    Each agreement rate (variable, section, and overall) will also be calculated using the entire cohort to achieve the overall STS Database-specific rates.  
(c) For Congenital, the rate of agreement and rate of completeness are calculated using the following methods. 
      1.    Rate of Agreement = (Match + Acceptable Variance) / (Match + Acceptable Variance + Unacceptable Variance)
       2.    Rate of Completeness = (Match + Acceptable Variance + Unacceptable Variance) / (Match + Acceptable Variance + Unacceptable Variance + Missing Data) 
                           i.    Acceptable variance is defined as differing nomenclature among providers. See Appendix A for the formal definition of Acceptable Variance: Primary Diagnosis and Primary Procedure. 
(d) For all sections (excluding the Post Operative Events or Mortality variables listed in Appendix B, C and D) and overall site accuracy (agreement) rates, the following grading system will be provided to sites.
1.    98.0% or greater is defined as a site that exceeds expectations. 
2.    90.0% to 97.9% is defined as a site that meets expectations. 
3.    89.9% or less is defined as a site that does not meet expectations.
(e) For the Post Operative Events section, the following grading system will be provided to sites. See Appendix B for a breakdown of the complications section per Database. 
1.    98.0% or greater is defined as a site that meets expectations.
2.    90.0 to 97.9% is defined as a site that does not meet expectations.
3.    A site achieving 89.9% or less for accuracy on the Post Operative Events section will require a re-audit.
(f) For the Mortality section, the following grading system will be provided to sites. See Appendix C for a breakdown of the mortality section per Database. 
1.    98.0% or greater is defined as a site that meets expectations.
2.    90.0 to 97.9% is defined as a site that does not meet expectations.
3.    A site achieving 89.9% or less for accuracy on the Mortality section will require a re-audit.
(g) For the Mortality section, verification of post-procedure status is required to be submitted to the audit company. This includes any tracking logs, EHR, or other documentation to support coding of those variables listed in Appendix D. 
1.    100.0% accuracy is defined as meeting expectations.
2.    99.9% or less for accuracy is defined as not meeting expectations and will require re-audit.
(h) Sites identified as not meeting expectations for overall site accuracy (89.9%) or sites having 89.9% or less for accuracy on the Post Operative Events section and Mortality section or 99.9% or less on the Mortality verification of post-procedure status variables will be required to undergo a follow-up audit within one to two years.
(i) Sites identified as continuously not meeting expectations (with a lower-than 90.0% agreement rate for the Post Operative Events and Mortality sections, or lower-than 100.0% for the Mortality verification of post-procedure status variables) following the second audit jeopardize their participation in the STS National Database. A participant will not receive a star rating (except in the instance of CHSD and GTSD participants, who do not receive star ratings), will not be included in the aggregate data, and will not receive a risk-adjusted report of data. The site will also be denied the option to publicly reporting their data. 

Appendix A

Definition of Acceptable Variance: Primary Diagnosis and Primary Procedure

1. Surgeon auditor deems the participant’s choice of primary diagnosis or procedure to be materially interchangeable with that of the auditor (primum ASD vs. partial AVSD).
2. Surgeon auditor’s choice for primary diagnosis or procedure is on the list of all diagnoses or procedures submitted by the participant.
3. Participant chooses to submit an “atomic” vs. “molecular” procedural description.

(a) Algorithmically generated “molecular” procedure is on the “Congenital Combo Code” list.
(b) “Atomic vs. “molecular” procedural description does not impact the STAT score of the Index Procedure.

Appendix B

See below the audited variables that are included in the complications section (Adult Cardiac), post operative events section (General Thoracic and Congenital Heart) and adverse events section (Intermacs and Pedimacs).

Adult Cardiac Surgery Database
Short NameSequence Number
SurSInf6690
DeepSternInf6700
COpReBld6755
COpReVlv6765
CReintMI6771
CAortReint6774
COpReOth6778
COpReNon6780
CNStrokP6810
CPVntLng6835
CRenFail6870
COtAFib6945
MtOpD7124
Readmit7140

 

General Thoracic Surgery Database
Short NameSequence Number
POEvents3660
PostOpProc3670
Pneumonia3720
ARDS3740
RespFail3760
Bronchopleural3770
PE3780
Vent3810
Trach3820
AtrialArryth3830
MI3870
PostopProcAL3920
DelayCondEmp 3910
LaryngealNerve 4080

 

Congenital Heart Surgery Database
Variable NameShort NameSequence Number
Major Postoperative Event: Unplanned cardiac reoperationPostopEventMulti_2222 (harvest code)
Major Postoperative Event: Unplanned interventional cardiovascular catheterization procedurePostopEventMulti_2424 (harvest code)
Major Postoperative Event: Non-cardiac reoperationPostopEventMulti_2626 (harvest code)
Major Postoperative Event: Renal failurePostopEventsMulti_570570 (harvest codes)
Major Postoperative Event: Postoperative/Postprocedural mechanical circulatory supportPostopEventsMulti_4040 (harvest code)
Major Postoperative Event: Arrhythmia necessitating pacemaker, Permanent pacemakerPostopEventsMulti_7474 (harvest code)
Major Postoperative Event: Paralyzed DiaphragmPostopEventsMulti_300300 (harvest code)
Major Postoperative Event: Unexpected Cardiac ArrestPostopEventsMulti_3030 (harvest code)
Postoperative Events - Neurological Deficit Present At DischargePOENeuroDefPAtDis4802

For the CHSD, only major post-operative complications are considered at present in the agreement and completeness calculations. However, other complications may be included in these calculations in the future at the discretion of the Database with advance notice to participants.

Intermacs Database
WBDE FormShort Name
Rehospitalization FormPRIM_ADMIS_RSN
Adverse Event FormAE_INFECTION_ONSET_DT
Adverse Event FormINFECT_LOC_PUMP_DRIVELINE
Adverse Event FormBLEEDING_SOURCE_LOW_GASTRO
Adverse Event FormNEURO_CAT
Adverse Event FormHEPATIC
Adverse Event FormRENAL_DYS
Adverse Event FormRESP_FAIL
Adverse Event FormAE_DEV_MALF_EVNT
Adverse Event FormHEMOLYSIS
Adverse Event FormRHF

 

Pedimacs Database
WBDE FormShort Name
Rehospitalization FormPRIM_ADMIS_RSN
Adverse Event FormAE_INFECTION_ONSET_DT
Adverse Event FormINFECT_LOC_PUMP_DRIVELINE
Adverse Event FormBLEEDING_SOURCE_LOW_GASTRO
Adverse Event FormNEURO_CAT
Adverse Event FormHEPATIC
Adverse Event FormRENAL_DYS
Adverse Event FormRESP_FAIL
Adverse Event FormAE_DEV_MALF_EVNT

 

Appendix C

See below the audited variables that are included in the mortality section (Adult Cardiac, General Thoracic, and Congenital Heart) and the follow-up section (Intermacs and Pedimacs).

Adult Cardiac Surgery Database

Adult Cardiac Surgery Database

Short NameSequence Number

Mt30Stat

7001

DischDt

7006

DischMortStat

7007

DCASA

7060

DCADP

7070

DCBeta

7105

DCLipLowStat

7115

MtDate

7121

MtOpD

7124

General Thoracic Surgery Database

General Thoracic Surgery Database

Short NameSequence Number

MtDCStat

4220

Mt30Stat

4310

Congenital Heart Surgery Database

Congenital Heart Surgery Database

Variable NameShort NameSequence Number
Date of AdmissionAdmitDt570
Date of SurgerySurgDt580
Operation TypeOpType1755
Date of Hospital DischargeHospDischDt4875
Mortality Status at Hospital DischargeMtHospDisStat4880
Date of Database DischargeDBDischDt4935
Mortality Status at Database DischargeMtDBDisStat4940
Mortality 30-Day StatusMt30Stat4945
Mortality DateMtDate5005

 

Intermacs Database

Intermacs Database

WBDE FormVariable

Death Form

Death Date

Death Form

Completed

One-Month Follow-Up Form

Completed

Three-Month Follow-Up FormQoL Questionnaire Completed
Three-Month Follow-Up Form6-minute walk
Three-Month Follow-Up FormNYHA Class

 

Pedimacs Database

Pedimacs Database

WBDE FormVariable

Death Form

Death Date

Death Form

Completed

One-Month Follow-Up Form

Completed

Three-Month Follow-Up Form

QoL Questionnaire Completed

Three-Month Follow-Up Form

Hemolysis Zone Record

Three-Month Follow-Up Form

Right Heart Failure Zone Record

Three-Month Follow-Up Form

6-minute walk

Three-Month Follow-Up Form

NYHA Class

 

Appendix D

See below the post-procedure verification variables for each Database.

Adult Cardiac Surgery Database

Adult Cardiac Surgery Database

Short NameSequence Number

Mt30Stat

7001

General Thoracic Surgery Database

General Thoracic Surgery Database

Short NameSequence Number

Mt30Stat

4310

Congenital Heart Surgery Database

Congenital Heart Surgery Database

Variable NameShort NameSequence Number
Mortality Status at Database DischargeMtDBDisStat4940
Mortality 30-Day StatusMt30Stat4945
Intermacs Database

Intermacs Database

Short NameNotes

Three-Month Follow-Up Form 

Check that it was completed

Death Form

Check that it was completed

Pedimacs Database

Pedimacs Database

Short NameNotes

Three-Month Follow-Up Form 

Check that it was completed

Death Form

Check that it was completed