Silent QC: How Vendors Remove Surveys Without Disclosure — And What It Means for Your Data

Your supplier reports an 82% completion rate. You assume this means 82 out of every 100 invited respondents successfully completed the survey. In practice, it may mean something quite different: 100 respondents completed the survey, and the vendor removed 18 — for reasons that may or may not be disclosed. The difference matters enormously, and most contracts don't require vendors to tell you which scenario occurred.

What "Silent QC" Means

Silent QC refers to the practice of research vendors removing or excluding raw survey responses before data delivery, without providing buyers with any disclosure regarding the volume removed, the removal criteria applied, or who made the removal decision.

Quality control removal is a legitimate and necessary part of research practice. Removing speeders, straight-liners, duplicate device submissions, and clear bot patterns is essential to data integrity. The problem is not that removal happens — it is that the process is routinely undisclosed.

When Removal Is Acceptable vs. When Disclosure Is Required

How Undisclosed Removal Distorts Research Conclusions

The following case is reconstructed from real project data (identifying details removed):

A technology company commissioned research across five Southeast Asian markets, targeting 50 CIO/IT Director-level respondents per market (250 total). The supplier reported a "78% completion rate" on delivery. In the Malaysia subsample, however, responses to a question about "budget decision authority" showed an unusual skew toward higher budget authority compared to all other markets — a pattern inconsistent with market intelligence.

On further investigation, the supplier acknowledged that 16 Malaysia responses had been removed for "insufficient language quality" and replaced from a backup sample. These removed responses were disproportionately concentrated among respondents with lower stated budget authority — possibly because respondents with less formal business training expressed themselves in shorter English phrases.

The consequence: the Malaysia subsample systematically overstated the proportion of respondents with high budget decision authority, leading the buyer to materially misjudge the complexity of the Malaysia procurement decision landscape.

How to Build Contractual Protections

Require QC Log Delivery

Require by contract that the supplier delivers a complete QC removal log alongside final data. The log should document: raw collection volume, removals by category and rule, final delivered sample size, and written explanation for any selective removals exceeding 1% of any subsegment.

Set Disclosure Thresholds

Add a clause requiring that when any single QC rule removes more than 8% of a market or segment target, the supplier must notify the buyer in writing within 24 hours of removal execution, including the removal criteria and the affected respondent characteristic distribution.

Raw Data Retention Rights

Require the contract to guarantee buyer access to the complete raw dataset — including removed surveys with deletion flags — for at least 12 months after project completion. This enables independent quality verification and provides a baseline for future projects in similar markets.

How to Detect Undisclosed Removal Without Contractual Access

Request timestamp data. If timestamp distributions show unusual clustering in specific time periods, combined with anomalous distributions on key variables, this may indicate supplementary sampling occurred after initial fieldwork.

Check the completion rate arithmetic. If the supplier reports "78% completion rate, 250 final completes," the implied invitation volume is approximately 321. Request this number from the supplier and evaluate its plausibility.

Examine subsample consistency. If distributions on key variables differ significantly across markets without a plausible market-specific explanation, request per-market QC detail from the supplier.