STS National Database Audit Policy

Purpose 

The purpose of an independent audit is two-fold. First, an audit assures stakeholders (Database participants, patients, researchers, consumers, regulators, payers, and industry) that the clinical data in the five STS National Database components—Adult Cardiac Surgery, Congenital Heart Surgery, General Thoracic Surgery, Intermacs, and Pedimacs—are consistent, accurate, and complete. Second, the audit process is designed as an educational opportunity to complement the participant’s internal quality controls established to maximize the integrity of the data collected. 

Policy

1. Participation agreements in the Adult Cardiac Surgery Database (section 1.3.1), Congenital Heart Surgery Database (section 1.2.1), General Thoracic Surgery Database (section 1.2.1), Intermacs Database (section 1.2.4) and Pedimacs Database (section 1.2.4), require participants to submit accurate and complete data and provide cooperation and assistance with any independent audit.  

“Participant hereby warrants that all data submitted for inclusion in the Database will be accurate and complete, and acknowledges that such data may be subject to independent audit in accordance with STS Policies. Participant will use its best efforts to address any data or related deficiencies identified by the independent data warehouse service provider, and agrees to cooperate with and assist STS and its designees in connection with the performance of any independent audit.”

2. STS responsibilities: 

(a) Contract with an external independent organization (audit company) to provide secure, HIPAA-compliant auditing services
(b) Notify sites selected for STS audit participation
(c) Provide education to the audit company regarding data definitions 
(d) Explain the audit process and expectations to sites
(e) Send summary results to sites
(f) Provide oversight in review and interpretation of the audit results, including identification of data quality outliers and provision of remediation assistance, as deemed appropriate by the associated workforce or task force.
(g) Provide access to Database-specific educational resources to audited sites, especially those with results that do not meet the expectation of STS (please see section 7). These resources include: Optional attendance to Advances in Quality & Outcomes: A Data Managers Meeting (AQO), Database-specific training manuals, Database educational webinars, STS National Database Mentorship Program, STS National Database News, and the FAQ inbox. 

3. Participant responsibilities: 

(a) Cooperate with external audit organization to schedule the audit 
(b) Provide records (at no charge) or access to records in a timely fashion 
(c) Provide records in a HIPAA-compliant format
(d) Participate in post-audit review and educational opportunities

4. Each of the Database task forces is responsible for identifying audit variables.

5. Audit site selection:

(a) Each year approximately 10% of sites in each Database will be audited
(b) To be included in the audit pool, a site must a. be an active Database participant for all 12 months of the audited time period and b. capture the specified minimum number of procedures/admissions during the audit time period.

Database Minimum Number of Procedures/Admissions
Adult Cardiac Surgery Database 20 isolated CABG procedures and 10 Valve or Valve + CABG procedures
Congenital Heart Surgery Database 30 index operations
General Thoracic Surgery Database 20 cases total of lobectomy or esophagectomy for primary cancer
Intermacs Database 10 patients implanted
Pedimacs Database  3 patients implanted

(c). All US and Canadian sites are included in the audit pool except:

(i) Sites that have been identified as not meeting expectations during a recent audit (Please see section 7.)
(ii) Sites that have successfully completed an STS audit within the past 3 years.

(d) STS reserves the right to utilize various methodologies, including random and targeted selection.  

(i) Targeted audits may be scheduled if requested by a participant or deemed appropriate by STS.

6. Audit summary results: 

A final Summary Report will be emailed by STS to each site. The rate of agreement and data completeness will be considered when evaluating each site and identifying sites that do not meet the STS expectations (please see section 7).  

7. Data quality: 

(a) Definitions

(i) Data Accuracy (Data Agreement) – rate of agreement between data submitted by the participant and data in the medical record as adjudicated by the auditors 
(ii) Data Completeness/Missing Data – data fields not completed or, for certain critical variables such as those related to Operative Mortality, entered as Unknown
(iii) Data Completeness/Missing Records – discrepancy between the case log and cases entered into the Database

(b) Data accuracy will be calculated three ways:

(i) Variable accuracy (agreement) will be measured for each audited variable. The denominator will equal the total number of records audited. The numerator will be the number of records where that variable was answered completely and correctly. This will be reported back as a percentage to sites. 

(ii) Section accuracy (agreement) will be measured for each section (i.e., demographics, risk factors, complications, discharge, mortality, etc.). The denominator will equal the total number of variables within the corresponding section where the variable was available or should have been available to abstract, times the total number of records abstracted. The numerator will equal the total number of variables within the specific section that were abstracted completely and correctly times the total number of records. 

For example, if the demographics section is audited on 20 records and there are three variables within the demographic section and an answer was expected for each variable on all records, then the denominator will be 60. If the site had one abstraction error within the demographic section for each record for a total of 20 total errors, then the result would be 20/60 or 66.6% accuracy for the demographic section.

(iii) Overall site accuracy (agreement) will be measured for each site as an overall agreement rate. The denominator will equal the total number of records audited times the total number of variables that were available or should have been available for abstraction of each record. The numerator will equal the total number of variables that were abstracted completely and correctly times the total number of records audited at the site.

(iv) Each agreement rate (variable, section, and overall) will also be calculated using the entire cohort to achieve the overall STS Database-specific rates.

(c) For Congenital, the rate of agreement and rate of completeness are calculated using the following methods.   

(i) (Rate of Agreement = (Match + Acceptable Variance) / (Match + Acceptable Variance + Unacceptable Variance)
(ii) Rate of Completeness = (Match + Acceptable Variance + Unacceptable Variance) / (Match + Acceptable Variance + Unacceptable Variance + Missing Data)

Acceptable variance is defined as differing nomenclature among providers. See Appendix A for the formal definition of Acceptable Variance: Primary Diagnosis and Primary Procedure. 

(d) For all sections (excluding the Complications or Mortality sections) and overall site accuracy (agreement) rates, the following grading system will be provided to the sites

(i) 98% or greater is defined as a site that exceeds expectations. 
(ii) 90% to 98% is defined as a site that meets expectations. 
(iii) Less than 90% is defined as a site that does not meet expectations.

(e) For the Complications or Mortality sections, the following grading system will be provided to sites. See Appendix B for a breakdown of the complications section per Database. 

(i) 98% or greater is defined as a site that meets expectations.
(ii) 90 to 98% is defined as a site that does not meet expectations.
(iii) A site achieving less than 90% accuracy on the Complications or Mortality section will require a re-audit.

(f) Sites identified as not meeting expectations for overall site accuracy (agreement) or sites having less than 90% accuracy on the Complications or Mortality sections will be required to undergo a follow-up audit within 2 years.

(g) Sites identified as continuously not meeting expectations or with a lower-than 90% agreement rate for the Complications or Mortality sections following the second audit jeopardize their participation in the STS National Database. A participant will not receive a star rating (except in the instance of CHSD participants, who do not receive star ratings), will not be included in the aggregate data, and will not receive a risk-adjusted report of data. The site will also be denied the option to publicly reporting their data. 

Appendix A

Definition of Acceptable Variance: Primary Diagnosis and Primary Procedure

1. Surgeon auditor deems the participant’s choice of primary diagnosis or procedure to be materially interchangeable with that of the auditor (primum ASD vs. partial AVSD).
2. Surgeon auditor’s choice for primary diagnosis or procedure is on the list of all diagnoses or procedures submitted by the participant.
3. Participant chooses to submit an “atomic” vs. “molecular” procedural description.

(i) Algorithmically generated “molecular” procedure is on the “Congenital Combo Code” list.
(b) “Atomic vs. “molecular” procedural description does not impact the STAT score of the Index Procedure.

Appendix B

See below the audited variables that are included in the complications section for each registry.

Adult Cardiac Surgery Database
Short Name Sequence Number
SurSInf 6690
DeepSternInf 6700
COpReBld 6755
COpReVlv 6765
COpReGft/CReintMI 6771
CAortReint 6774
COpReOth 6778
COpReNon 6780
CNStrokP 6810
Prolonged Vent 6835
CRenFail 6870
COtAFib 6930
MtOpD 7124
Readmit 7140

 

General Thoracic Surgery Database
Short Name Sequence Number
PostOpInvProc 3330
Pneumonia 3460
Vent 3520
AtrialArryth 3560
ARDS 3470
RespFail 3480
PostOp MI 3580
DelayCondEmp Delayed conduit emptying requiring intervention 3650
LaryngealNerve Recurrent Laryngeal Nerve Paresis 3780

 

Congenital Heart Surgery Database
Variable Name Sequence Number Short Name
Unplanned cardiac reoperation during the postoperative or postprocedural time period, exclusive of reoperation for bleeding (22) and unplanned interventional cardiovascular catheterization procedure during the postoperative or postprocedure period (24) and unplanned non-cardiac reoperation during the postoperative time period (26) and bleeding, requiring reoperation (240) 4203 MajorCompUnplannedProcedure
22_24_26_240
Acute renal failure requirement temporary dialysis with the need for dialysis not present at hospital discharge (223) and acute renal failure requiring temporary hemofiltration with the need for dialysis not present at hospital discharge (224) and acute renal failure requiring dialysis at the time of hospital discharge (230) 4204 MajorCompDialysis
223_224_230
Postoperative/postprocedure mechanical circulatory support (IABP, VAD, ECMO, or CPS) (40) 4205 MajorCompMechCircSupport40



 
Arrhythmia-necessitating pacemaker, permanent pacemaker (74) 4206 MajorCompPermPacer74
Unexpected cardiac arrest (30) 4207 MajorCompCardArrest30
Neurological deficit persisting at discharge (320) 4208 MajorCompNeuro320
Paralyzed diaphragm (possible phrenic nerve injury) (300) 4209 MajorCompParDiaphragm300

For the CHSD, only major post-operative complications are considered at present in the agreement and completeness calculations. However, other complications may be included in these calculations in the future at the discretion of the Database with advance notice to participants.

Intermacs Database
Variable Name WBDE Form
Major Infection adverse events reported and verified Driveline Major Infection Form (verify date and location)
Post-implant Major Bleeding adverse events reported correctly (maybe episodes) Lower GI Bleeding Major Bleeding Form
CVA, TIA reported post implant? Neuro AE Form
Hepatic Dysfunction Post Implant Additional Events Form
RHF Post Implant AT 1 Follow-Up Form
Renal Dysfunction Post Implant Additional Events Form
Homolysis Post Implant AT 1 Follow-Up Form
Re-hospitalization reason completed Re-hospitalization Form x1
Device Malfunction Device malfunction Form x1
Post Implant Respiratory Failure AE Additional Events Form

 

Pedimacs Database
Variable Name WBDE Form
Major Infection adverse events reported and verified Driveline Major Infection Form (verify data and location)
Post implant Major Bleeding adverse events reported correctly (maybe episodes) Lower GI Bleeding Major Bleeding Form
CVA, TIA reported post implant? Neuro AE Form
Hepatic Dysfunction Post Implant Additional Events Form
RHF Post Implant AT 1 Follow-Up Form
Renal Dysfunction Post Implant Additional Events Form
Hemolysis Post Implant AT 1 Follow-Up Form
Re-hospitalization reason completed Re-hospitalization Form x1