A software company wants to know why customers leave. It emails a survey and offers a gift card for taking part. Responses arrive – but most come from customers who still use the product. The people whose departures prompted the research barely appear in the results.
- Do survey incentives improve the data or just participation?
- Set the reward terms before inviting respondents
- Check the response before rejecting the participant
- Decide separately whether the reward is owed
- Automate reward delivery without paying twice
- Limit the personal information passed between systems
- Measure the cost of responses you can actually use
- Test the incentive offer before expanding recruitment
- Check the retained sample before releasing cancellation findings
- Frequently Asked Questions
Incentive-driven response bias can arise when a reward changes who responds, leaving the collected sample poorly matched to the research question. Audit who received invitations, who answered and whose responses survived review. For a cancellation survey, track former customers separately from active users before trusting the completion count.
That is not a survey logistics problem. It is a data-quality defect, and it sits in the sample long before anyone runs a query. Incentive-driven response bias changes who lands in the dataset, and a healthy completion count will happily hide the gap. Before you trust the numbers, audit the composition behind them.
- Audit recruitment before drawing conclusions: compare invitations, replies and retained responses for the customer groups the research concerns.
- Set reward eligibility, payment conditions and delivery timing before sending invitations.
- Assess reward entitlement separately from whether an answer can contribute to a particular analysis.
- Keep the same reward reference through retries so a timeout does not create another order.
- Measure cost per unique retained response, including review work and rewards already owed.
In the hypothetical above, the analyst needs to check recruitment before drawing conclusions about cancellations. How many former customers received the invitation? How many replied? Did the survey let them explain their reasons, or ask questions that assumed they were still using the software?
Survey incentives deserve the same scrutiny as the questionnaire and invitation list. The offer may persuade someone to spend time answering. To judge whether it helped, the research team needs to examine whose answers it collected and whether those answers address the question that started the work.
Do survey incentives improve the data or just participation?
An incentive can encourage someone to respond. It says nothing about who took part or how they answered. More completed questionnaires may leave important customer groups underrepresented. Equally, offering no reward can favour people who already care enough about the subject to volunteer their time.
Survey researchers may seek to reduce costs by choosing not to offer an incentive. However, there is evidence that surveys with guaranteed incentives tend to improve response rates when compared to surveys with no incentive or a lottery/prize drawing (see for example, Halpern et al. 2011 and Dykema et al. 2024 ). Further, studies have shown that the provision of an incentive may increase response rates from hard-to-reach groups with little or no impact on data quality.
Michael Soszynski, https://doi.org/10.29115/SP-2024-0013 in Michael Soszynski, https://doi.org/10.29115/SP-2024-0013A 2024 study published in Survey Practice examined web surveys of utility customers in the Mid-Atlantic United States. Researchers compared responses collected with and without a $5 incentive. They found no convincing evidence that it affected the data-quality measures they examined.
Read incentive findings in the context of the people invited and the questions asked. In this study, the researchers lacked population benchmarks to establish whether respondents were demographically representative. Similar response-quality measures therefore did not establish that either group reflected the full customer population.
For a customer-research team, the practical question is who remains missing. Suppose a software company wants feedback from both active customers and those who recently cancelled. If nearly everyone responding still uses the product, a large completion count will not explain why former customers left.
Track those groups separately during recruitment. The reward may help bring people into the survey; the analyst still has to establish whether the resulting sample supports the business question.
Set the reward terms before inviting respondents
Decide who qualifies for the reward, what they will receive and when it will arrive before sending invitations. Participants should be able to understand the offer without completing the survey first.
A guaranteed gift card and entry into a prize draw make different promises. If everyone who meets the stated conditions receives a reward, say so. If only selected winners receive something, make that distinction clear in the invitation. Someone deciding whether to spend fifteen minutes answering questions needs to know which offer applies.
Timing matters too. A prepaid incentive reaches someone before they respond. A reward promised after completion gives the research team a different administrative task: keeping track of what it owes each eligible participant. Choose the arrangement that fits the study and budget, then carry the same terms through the invitation, questionnaire and confirmation message.
Avoid choosing an amount simply because another survey used it. Consider the time required and the people you need to reach. A short questionnaire about a recent purchase asks less of someone than a detailed survey that requires them to look up business records. Test whether the proposed reward is useful and accessible to the intended audience.
If payment depends on a review, explain that condition upfront. Lehigh University’s guidance for online research recommends disclosing review conditions and recording the reasons for decisions not to compensate participants. Its guidance also makes clear that checks must suit the study and follow the institution’s research requirements.
For a business survey, write the payment rules before responses arrive. Discovering an unexpected answer halfway through fieldwork is a reason to investigate the questionnaire or review the record. It is not a sound basis for inventing a new condition on a reward already promised.
Check the response before rejecting the participant
A fast response deserves a look at the questionnaire the person actually received. Someone routed past several pages has less to answer. Before excluding the record, check the questions shown and whether the answers make sense together.
The same care applies to apparent duplicates. Two employees answering from one office may share an IP address. If the study allows both to participate, that shared connection is a poor reason to reject either. Compare the invitation records and eligibility answers before deciding that one person submitted twice.
Lehigh’s guidance on online research recommends using several indicators to flag responses for further review. It also calls for documenting decisions not to compensate participants. A useful review note explains what the reviewer found and which disclosed condition the response failed to meet. “Flagged by system” leaves the next person with the same unanswered question.
Sometimes the questionnaire needs correcting. If several otherwise credible respondents stumble over one instruction, inspect that question before excluding everyone who missed it.
Decide separately whether the reward is owed
An analyst may be unable to use an answer even though the participant earned the reward.
Take an optional question about spending. A customer who leaves it blank cannot contribute to the spending calculation, but may have completed everything required by the invitation. Record the missing value in the analysis and assess payment against the terms offered.
Treat criticism the same way. A low satisfaction score with no written explanation may disappoint the research team, but it does not establish poor-quality participation. If an explanation was optional, the reviewer should not make it compulsory after seeing the answer.
Automate reward delivery without paying twice
A reward request times out. The survey platform has no confirmation, so it tries again. If the first request succeeded before the connection dropped, that retry could purchase a second reward for the same participant.
Give each approved reward a stable reference and keep it through retries. Tango’s API documentation specifies an externalRefID for this purpose. Within the same account, retrying with that reference returns the original order rather than creating another. A different reward transaction needs a new reference.
This protects against duplicate orders. The research team still needs its own checks for someone entering the survey more than once.
Keep a small operational record for each reward:
- The internal participation reference and payment decision.
- The reward reference and provider’s order ID.
- The latest delivery status and any unresolved error.
Assign someone to review exceptions. An accepted order does not establish that the participant received the email, and an email-delivery problem should not automatically trigger another purchase. Check the existing order first.
Keep administrator instructions separate from participant messages. If staff need help finding the sender dashboard, the independent Tango portal login guide explains that access route; it is not Tango’s official website. Participants receiving a Reward Link follow their redemption instructions instead.
Limit the personal information passed between systems
The person sending rewards needs delivery details. The analyst usually needs the answers and enough background information to interpret them. Design those exports separately rather than circulating one spreadsheet containing everything.
Use an internal reference to reconcile the records, and restrict access to the file that connects that reference to a person. This still leaves the records linkable, so describe the arrangement as confidential or pseudonymised where appropriate. Separating names from answers does not, by itself, make a survey anonymous.
Before launch, submit a test response and follow it through approval, ordering and delivery. Then simulate a retry. Confirm that it retrieves the existing reward and that the team can identify who should investigate a failed delivery.
Measure the cost of responses you can actually use
Divide the cost of the survey by the number of unique responses retained under your predefined analysis rules. Include recruitment, incentives and the staff time spent reviewing submissions or resolving reward problems. This gives the team a practical cost measure, provided everyone uses the same definition.
Consider a hypothetical project that costs $3,000 and collects 600 submissions. The headline cost is $5 per submission. If 450 unique responses meet the analysis rules, the cost per retained response is about $6.67.
Neither figure tells you whether the sample answers the research question. If the project concerns customer cancellations but few former customers respond, inexpensive recruitment may still leave you short of useful evidence.
A compact dashboard can keep those distinctions visible:
| Measure | Calculation or comparison | What to investigate |
|---|---|---|
| Retained-response share | Unique retained responses ÷ submissions received | Why records were excluded |
| Cost per retained response | Attributable project cost ÷ unique retained responses | Whether recruitment or review costs changed |
| Sample composition | Respondents compared with recruitment targets | Which groups remain underrepresented |
| Reward-delivery failures | Failed deliveries ÷ delivery attempts in the same period | Incorrect contact details or delivery problems |
Count rewards already owed when assessing project cost, even if they have not yet been sent. Otherwise, a payment backlog could make one recruitment period appear cheaper than another. If no responses qualify for analysis, report that outcome alongside the money spent; a cost-per-response calculation would be undefined.
Test the incentive offer before expanding recruitment
A pilot should establish whether the offer reaches the intended participants and whether the team can fulfil it. Agree on the review rules and measures before invitations go out.
Where practical, randomly assign eligible invitees to different offers while keeping the questionnaire, invitation wording and recruitment period consistent. If random assignment is not feasible, record the differences between groups. Comparing one offer sent to recent customers with another sent to long-inactive accounts would make the results difficult to interpret.
Examine who responded as well as how many. In the utility-customer study published in Survey Practice, the researchers considered sample composition and interest in the topic alongside quality measures. They found some evidence of differences in composition and topic interest, although evidence of differences diminished after adjustment for multiple comparisons. That broader view helps identify changes a completion counter would miss.
Review participant complaints and a selection of rejected records before scaling up. Check that promised rewards arrived and that retries did not create duplicate orders. An offer worth expanding should bring the team closer to its intended sample, with payment terms it can reliably honour.
Check the retained sample before releasing cancellation findings
For your cancellation survey, compare former customers in the invitation list with those in the retained dataset before releasing findings. Keep that comparison separate from the total completion count. The Survey Practice study found similar data-quality measures between groups without establishing that either sample represented the population.
Your reward records also need a check before recruitment expands. A timeout followed by a retry can leave the team unsure whether it owes a payment or has already ordered one. Match the approved reward reference to the provider’s order ID and delivery status. Tango documents reuse of externalRefID within the same account so retries retrieve the original order. Keep unresolved deliveries visible while assessing the cost of the retained sample.
Frequently Asked Questions
What is an example of response bias?
An example of response bias occurs when a software company offers a gift card for a cancellation survey, but active satisfied customers supply the majority of responses while former customers barely respond. The resulting response bias creates a misleading sample that obscures true churn reasons.
What is the response bias explanation?
The response bias explanation is that financial incentives systematically change who chooses to participate. When a reward attracts specific demographic groups into the survey while others decline, the resulting response bias produces a distorted sample that diverges from the broader target population.
What does response bias mean?
Response bias means a systematic distortion where respondents who complete a survey differ fundamentally from non-respondents. In incentive-driven research, response bias means the collected sample reflects participants motivated by the reward rather than the target population being studied.
