Purpose 

The purpose of an independent audit is two-fold. First, an audit assures stakeholders (Database participants, patients, researchers, consumers, regulators, payers, and industry) that the clinical data in the five STS National Database components—Adult Cardiac Surgery, Congenital Heart Surgery, General Thoracic Surgery, Intermacs, and Pedimacs—are consistent, accurate, and complete. Second, the audit process is designed as an educational opportunity to complement the participant’s internal quality controls established to maximize the integrity of the data collected. 

Policy

1. Participation agreements in the Adult Cardiac Surgery Database (section 1.2.1 or 1.3.1), Congenital Heart Surgery Database (section 1.2.1), General Thoracic Surgery Database (section 1.2.1), Intermacs Database (section 1.2.4) and Pedimacs Database (section 1.2.4), require participants to submit accurate and complete data and provide cooperation and assistance with any independent audit.  

“Participant hereby warrants that all data submitted for inclusion in the Database will be accurate and complete, and acknowledges that such data may be subject to independent audit in accordance with STS Policies. Participant will use its best efforts to address any data or related deficiencies identified by the independent data warehouse service provider, and agrees to cooperate with and assist STS and its designees in connection with the performance of any independent audit.”

2. STS responsibilities: 

(a) Contract with an external independent organization (audit company) to provide secure, HIPAA-compliant auditing services
(b) Notify sites selected for STS audit participation
(c) Provide education to the audit company regarding data definitions 
(d) Explain the audit process and expectations to sites
(e) Send summary results to sites
(f) Provide oversight in review and interpretation of the audit results, including identification of data quality outliers and provision of remediation assistance, as deemed appropriate by the associated workforce or task force.
(g) Provide access to Database-specific educational resources to audited sites, especially those with results that do not meet the expectation of STS (please see section 7). These resources include: Optional attendance to Advances in Quality & Outcomes: A Data Managers Meeting (AQO), Database-specific training manuals, educational webinars, STS National Database Data Manager Collaborative Program, National Database News, and the FAQ inbox. 

3. Participant responsibilities: 

(a) Cooperate with external audit organization to schedule the audit 
(b) Provide records (at no charge) or access to records in a timely fashion 
(c) Provide records in a HIPAA-compliant format
(d) Participate in post-audit review and educational opportunities

4. Each of the Database task forces is responsible for identifying audit variables.

5. Audit site selection:

(a) Each year approximately 10% of sites in each Database will be audited
(b) To be included in the audit pool, a site must a. be an active Database participant for all 12 months of the audited time period and b. capture the specified minimum number of procedures/admissions during the audit time period.

Database Minimum Number of Procedures/Admissions
Adult Cardiac Surgery Database 20 isolated CABG procedures and 10 Valve or Valve + CABG procedures
Congenital Heart Surgery Database 30 index operations
General Thoracic Surgery Database 20 total lung cancer or esophageal cancer cases
Intermacs Database 10 patients implanted
Pedimacs Database  3 patients implanted

(c) All US and Canadian sites are included in the audit pool except:
     1.    Sites that have been identified as not meeting expectations during a recent audit (Please see section 7).
       2.    Sites that have successfully completed an STS audit within the past 3 years.
(d) STS reserves the right to utilize various methodologies, including random and targeted selection.  
      1.    Targeted audits may be scheduled if requested by a participant or deemed appropriate by STS.  

6. Audit summary results: 
A final Summary Report will be emailed by STS to each site. The rate of agreement and data completeness will be considered when evaluating each site and identifying sites that do not meet the STS expectations (please see section 7).  
    
7. Data quality
(a) Definitions:   
     1.    Data Accuracy (Data Agreement) – rate of agreement between data submitted by the participant and data in the medical record as adjudicated by the auditors 
      2.    Data Completeness/Missing Data – data fields not completed or, for certain critical variables such as those related to Operative Mortality, entered as Unknown
      3.    Data Completeness/Missing Records – discrepancy between the case log and cases entered into the Database
(b) Data accuracy will be calculated three ways:
     1.    Variable accuracy (agreement) will be measured for each audited variable. The denominator will equal the total number of records audited. The numerator will be the number of records where that variable was answered completely and correctly. This will be reported back as a percentage to sites. 
     2.    Section accuracy (agreement) will be measured for each section (i.e., demographics, risk factors, complications, discharge, mortality, etc.). The denominator will equal the total number of variables within the corresponding section where the variable was available or should have been available to abstract, times the total number of records abstracted. The numerator will equal the total number of variables within the specific section that were abstracted completely and correctly times the total number of records. 
                a.    For example, if the demographics section is audited on 20 records and there are three variables within the demographic section and an answer was expected for each variable on all records, then the denominator will be 60. If the site had one abstraction error within the demographic section for each record for a total of 20 total errors, then the result would be 20/60 or 66.6% accuracy for the demographic section.
     3.    Overall site accuracy (agreement) will be measured for each site as an overall agreement rate. The denominator will equal the total number of records audited times the total number of variables that were available or should have been available for abstraction of each record. The numerator will equal the total number of variables that were abstracted completely and correctly times the total number of records audited at the site.
     4.    Each agreement rate (variable, section, and overall) will also be calculated using the entire cohort to achieve the overall STS Database-specific rates.  
(c) For Congenital, the rate of agreement and rate of completeness are calculated using the following methods. 
      1.    Rate of Agreement = (Match + Acceptable Variance) / (Match + Acceptable Variance + Unacceptable Variance)
       2.    Rate of Completeness = (Match + Acceptable Variance + Unacceptable Variance) / (Match + Acceptable Variance + Unacceptable Variance + Missing Data) 
                           i.    Acceptable variance is defined as differing nomenclature among providers. See Appendix A for the formal definition of Acceptable Variance: Primary Diagnosis and Primary Procedure. 
(d) For all sections (excluding the Complications or Mortality variables listed in Appendix B, C and D) and overall site accuracy (agreement) rates, the following grading system will be provided to sites.
1.    98.0% or greater is defined as a site that exceeds expectations. 
2.    90.0% to 97.9% is defined as a site that meets expectations. 
3.    89.9% or less is defined as a site that does not meet expectations.
(e) For the Complications section, the following grading system will be provided to sites. See Appendix B for a breakdown of the complications section per Database. 
1.    98.0% or greater is defined as a site that meets expectations.
2.    90.0 to 97.9% is defined as a site that does not meet expectations.
3.    A site achieving 89.9% or less for accuracy on the Complications section will require a re-audit.
(f) For the Mortality section, the following grading system will be provided to sites. See Appendix C for a breakdown of the mortality section per Database. 
1.    98.0% or greater is defined as a site that meets expectations.
2.    90.0 to 97.9% is defined as a site that does not meet expectations.
3.    A site achieving 89.9% or less for accuracy on the Mortality section will require a re-audit.
(g) For the Mortality section, verification of post-procedure status is required to be submitted to the audit company. This includes any tracking logs, EHR, or other documentation to support coding of those variables listed in Appendix D. 
1.    100.0% accuracy is defined as meeting expectations.
2.    99.9% or less for accuracy is defined as not meeting expectations and will require re-audit.
(h) Sites identified as not meeting expectations for overall site accuracy (89.9%) or sites having 89.9% or less for accuracy on the Complications section and Mortality section or 99.9% or less on the Mortality verification of post-procedure status variables will be required to undergo a follow-up audit within one to two years.
(i) Sites identified as continuously not meeting expectations (with a lower-than 90.0% agreement rate for the Complications and Mortality sections, or lower-than 100.0% for the Mortality verification of post-procedure status variables) following the second audit jeopardize their participation in the STS National Database. A participant will not receive a star rating (except in the instance of CHSD participants, who do not receive star ratings), will not be included in the aggregate data, and will not receive a risk-adjusted report of data. The site will also be denied the option to publicly reporting their data. 

Appendix A

Definition of Acceptable Variance: Primary Diagnosis and Primary Procedure

1. Surgeon auditor deems the participant’s choice of primary diagnosis or procedure to be materially interchangeable with that of the auditor (primum ASD vs. partial AVSD).
2. Surgeon auditor’s choice for primary diagnosis or procedure is on the list of all diagnoses or procedures submitted by the participant.
3. Participant chooses to submit an “atomic” vs. “molecular” procedural description.

(a) Algorithmically generated “molecular” procedure is on the “Congenital Combo Code” list.
(b) “Atomic vs. “molecular” procedural description does not impact the STAT score of the Index Procedure.

Appendix B

See below the audited variables that are included in the complications section (Adult Cardiac, General Thoracic, and Congenital Heart) and adverse events section (Intermacs and Pedimacs).

Adult Cardiac Surgery Database
Short Name Sequence Number
SurSInf 6690
DeepSternInf 6700
COpReBld 6755
COpReVlv 6765
CReintMI 6771
CAortReint 6774
COpReOth 6778
COpReNon 6780
CNStrokP 6810
CPVntLng 6835
CRenFail 6870
COtAFib 6945
MtOpD 7124
Readmit 7140

 

General Thoracic Surgery Database
Short Name Sequence Number
POEvents 3660
PostOpProc 3670
Pneumonia 3720
ARDS 3740
RespFail 3760
Bronchopleural 3770
PE 3780
Vent 3810
Trach 3820
AtrialArryth 3830
MI 3870
PostopProcAL 3920
DelayCondEmp  3910
LaryngealNerve  4080

 

Congenital Heart Surgery Database
Variable Name Sequence Number Short Name
Unplanned cardiac reoperation during the postoperative or postprocedural time period, exclusive of reoperation for bleeding (22) and unplanned interventional cardiovascular catheterization procedure during the postoperative or postprocedure period (24) and unplanned non-cardiac reoperation during the postoperative time period (26) and bleeding, requiring reoperation (240) 4203 MajorCompUnplannedProcedure
22_24_26_240
Acute renal failure requirement temporary dialysis with the need for dialysis not present at hospital discharge (223) and acute renal failure requiring temporary hemofiltration with the need for dialysis not present at hospital discharge (224) and acute renal failure requiring dialysis at the time of hospital discharge (230) 4204 MajorCompDialysis
223_224_230
Postoperative/postprocedure mechanical circulatory support (IABP, VAD, ECMO, or CPS) (40) 4205 MajorCompMechCircSupport40



 
Arrhythmia-necessitating pacemaker, permanent pacemaker (74) 4206 MajorCompPermPacer74
Unexpected cardiac arrest (30) 4207 MajorCompCardArrest30
Neurological deficit persisting at discharge (320) 4208 MajorCompNeuro320
Paralyzed diaphragm (possible phrenic nerve injury) (300) 4209 MajorCompParDiaphragm300

For the CHSD, only major post-operative complications are considered at present in the agreement and completeness calculations. However, other complications may be included in these calculations in the future at the discretion of the Database with advance notice to participants.

Intermacs Database
WBDE Form Short Name
Rehospitalization Form PRIM_ADMIS_RSN
Adverse Event Form AE_INFECTION_ONSET_DT
Adverse Event Form INFECT_LOC_PUMP_DRIVELINE
Adverse Event Form BLEEDING_SOURCE_LOW_GASTRO
Adverse Event Form NEURO_CAT
Adverse Event Form HEPATIC
Adverse Event Form RENAL_DYS
Adverse Event Form RESP_FAIL
Adverse Event Form AE_DEV_MALF_EVNT
Adverse Event Form HEMOLYSIS
Adverse Event Form RHF

 

Pedimacs Database
WBDE Form Short Name
Rehospitalization Form PRIM_ADMIS_RSN
Adverse Event Form AE_INFECTION_ONSET_DT
Adverse Event Form INFECT_LOC_PUMP_DRIVELINE
Adverse Event Form BLEEDING_SOURCE_LOW_GASTRO
Adverse Event Form NEURO_CAT
Adverse Event Form HEPATIC
Adverse Event Form RENAL_DYS
Adverse Event Form RESP_FAIL
Adverse Event Form AE_DEV_MALF_EVNT

 

Appendix C

See below the audited variables that are included in the mortality section (Adult Cardiac, General Thoracic, and Congenital Heart) and the follow-up section (Intermacs and Pedimacs).

Adult Cardiac Surgery Database

Adult Cardiac Surgery Database

Short Name

Sequence Number

Mt30Stat

7001

DischDt

7006

DischMortStat

7007

DCASA

7060

DCADP

7070

DCBeta

7105

DCLipLowStat

7115

MtDate

7121

MtOpD

7124

General Thoracic Surgery Database

General Thoracic Surgery Database

Short Name

Sequence Number

MtDCStat

4220

Mt30Stat

4310

Congenital Heart Surgery Database

Congenital Heart Surgery Database

Short Name

Sequence Number

Gender

340

MtDate

490

AdminDt

780

SurgDt

790

AgeDays

820

OpType

1056

HospDischDt

4220

MtHospDisStat

4230

DBDischDt

4250

Mt30Stat

4300

MtDBDisStat

4260

 

Intermacs Database

Intermacs Database

WBDE Form

Variable

Death Form

Death Date

Death Form

Completed

One-Month Follow-Up Form

Completed

Three-Month Follow-Up Form

QoL Questionnaire Completed

Three-Month Follow-Up Form

6-minute walk

Three-Month Follow-Up Form

NYHA Class

 

Pedimacs Database

Pedimacs Database

WBDE Form

Variable

Death Form

Death Date

Death Form

Completed

One-Month Follow-Up Form

Completed

Three-Month Follow-Up Form

QoL Questionnaire Completed

Three-Month Follow-Up Form

Hemolysis Zone Record

Three-Month Follow-Up Form

Right Heart Failure Zone Record

Three-Month Follow-Up Form

6-minute walk

Three-Month Follow-Up Form

NYHA Class

 

Appendix D

See below the post-procedure verification variables for each Database.

Adult Cardiac Surgery Database

Adult Cardiac Surgery Database

Short Name

Sequence Number

Mt30Stat

7001

General Thoracic Surgery Database

General Thoracic Surgery Database

Short Name

Sequence Number

Mt30Stat

4310

Congenital Heart Surgery Database

Congenital Heart Surgery Database

Short Name

Sequence Number

Mt30Stat

4300

MtDBDisStat

4260

Intermacs Database

Intermacs Database

Short Name

Notes

Three-Month Follow-Up Form 

Check that it was completed

Death Form

Check that it was completed

Pedimacs Database

Pedimacs Database

Short Name

Notes

Three-Month Follow-Up Form 

Check that it was completed

Death Form

Check that it was completed