The Export Reflex
Here's how it usually goes. A product lead sends a Slack message at 4pm: "Do enterprise customers complain about onboarding more than SMB?" The researcher opens the study, clicks Export, and twenty minutes later is filtering a 2,400-row spreadsheet by a segment column someone filled in three weeks ago. They count rows that contain "onboarding" and paste a bar chart into the thread. The product lead reacts with a thumbs-up and moves on.
We call this the Export Reflex: when a question comes in, you turn rich interview data into flat rows, because a spreadsheet feels like the only place you can do analysis that someone can check. It feels rigorous. It isn't. By the time the data is in a CSV, the things that made it qualitative are gone: who said what, what they were responding to, what they said ten minutes later, and whether "onboarding" meant the setup wizard or the three-month rollout to their team.
This post covers why the reflex costs you, and then gives you ten questions to ask your data directly, with the exact wording that tends to produce answers you can defend.
Why the CSV Quietly Lies
There are three ways it goes wrong.
First, rows break up conversations. A transcript is a sequence. Something a participant says at minute 34 often changes the meaning of something they said at minute 6. Once you export, each row is cut loose from what came before and after it. Keyword filters then reward whoever repeated a term most often, not whoever explained it best.
Second, segment columns get frozen. The segment field in an export reflects how someone tagged the participant at recruitment. Half your "SMB" participants may describe enterprise-style procurement in the interview itself. The spreadsheet can't know that, so every comparison you build on that column starts out wrong.
Third, counting replaces reasoning. "Onboarding mentioned in 41% of enterprise rows vs 28% of SMB rows" sounds like a finding. It's actually a measure of how often a word appears, mixed up with interview length and how hard the moderator probed. The decision it drives (put onboarding investment into enterprise) may be right, but nothing in the analysis shows that.
The alternative is not "ask an AI and trust the answer". It's asking questions of the full, structured corpus in a way that makes every answer point back to the evidence. The ten questions below are written that way.
Segment Comparison
1. "Where do these two segments disagree, and where do they only seem to disagree?"
Don't ask "what do enterprise users say about X". Ask for the contrast directly, and ask for false contrasts to be pointed out:
"Compare how enterprise and SMB participants describe onboarding. List the points where they really differ, the points where they say the same thing in different words, and the points where one segment simply wasn't asked."
The third category is the one that matters. Many apparent segment differences are really differences in the discussion guide: the moderator probed enterprise participants on admin setup and never raised it with SMB. As we argued in our piece on the Persona Collapse Problem, averaging hides real segments. Comparing badly invents segments that aren't there.
2. "Which participants behave like a different segment from the one they were tagged as?"
"For each participant, compare their recruitment segment with how they describe their team size, buying process and usage. Flag anyone whose interview contradicts their tag, with the supporting quote."
In B2B studies this usually reclassifies 10 to 20 percent of the sample. Do it before any segment comparison, not after.
Contradiction Finding
3. "Where does a participant contradict themselves?"
"Find places where a single participant makes a claim and later says something inconsistent with it. Give both quotes with timestamps and the question that prompted each."
Inconsistency within one participant is often where the real story is: "the setup was easy" at minute 5 and "it took us a month to get anyone using it" at minute 40. Both are true. The first is about the product, the second is about the organisation. A CSV filter on "easy" only finds the first.
4. "What evidence goes against our leading theme?"
"Our working theme is that pricing drives churn. List every passage that weakens, complicates or contradicts it, ranked by how directly it does so."
This is negative case analysis in practice, and it's the question teams skip most often, because nobody exports a CSV to look for reasons they might be wrong. Ask it every time a theme is about to go into a deck.
Onboarding Pain
5. "At what point in the first 30 days does each participant first describe friction?"
"For each participant, identify the earliest moment they describe being stuck, confused or slowed down during onboarding. Note what they were trying to do, how they got past it, and whether they mention it again later."
Grouping by moment rather than by theme turns a vague "onboarding is hard" into "seven of twelve stalled at inviting their first teammate". That's something an engineer can act on.
6. "What workaround did they invent?"
"List every workaround participants describe during setup: spreadsheets, scripts, asking a colleague, skipping a step. Quote each one."
Workarounds tell you more about onboarding pain than complaints do. People complain about what irritates them. They build workarounds for what actually blocks them. Workaround descriptions rarely contain the word "onboarding", so keyword exports miss almost all of them.
Churn Reasons
7. "What was the stated reason, and what happened before it?"
"For each churned participant, separate the reason they give for leaving from the events they describe in the months before. Where these differ, show both."
Churn interviews nearly always open with a tidy reason: price, a missing feature, a competitor. The causal story is usually further back: a champion who left, a failed rollout, a quarter in which nobody logged in. Ask for both layers separately, or the tidy reason wins by default.
8. "Which churn reasons were set off by a single event?"
"Group churn reasons by whether the participant describes a gradual decline or one specific trigger event. For trigger events, name the event and quote the description."
Gradual decline and trigger events call for opposite responses (fixing engagement loops vs fixing incident handling). Lumping them together under "churn reasons" is one of the most expensive simplifications in retention research.
Quote Retrieval
9. "Give me the three quotes that best represent this theme, and one that doesn't fit."
"For the theme 'admins feel blamed for low adoption', return three quotes that represent the typical expression of it, from three different participants, plus one quote that is an outlier. Include participant ID, segment and timestamp."
The "different participants" constraint matters. Without it you get the three best lines from your most articulate participant, which is how the Verbatim Overweighting Effect takes over a readout. The outlier quote stops the slide from looking unanimous.
10. "Show me the full exchange around this quote."
"For this quote, return the moderator question that prompted it, the participant's complete answer, and the next two turns."
This is the question that matters most for credibility. A quote with its prompt shows whether the moderator led the participant. A quote without it is an assertion. If a decision is going to rest on a line, a stakeholder should be able to see the conversation around it. That's the whole argument of the Insight Attribution Gap.
What Makes a Question Answerable
All ten questions share four features, and you can apply them to any question you write:
- They ask for a contrast or a structure, not a summary. "What do people say about X" gets you mush. "Where do A and B differ, and where do they only appear to" gets you a finding.
- They demand citations. Participant ID, timestamp, prompting question. An answer you can't trace back to the transcript isn't evidence.
- They name the absences. "Who wasn't asked" and "what contradicts this" stop the system from filling gaps with plausible text.
- They spread retrieval across participants. Engineers building retrieval systems run into the same issue: what gets pulled in, and in what order, shapes the answer more than the question does. The retrieval ordering problem is a useful technical parallel for why "from different participants" belongs in your prompts.
When You Should Still Export
Exports are fine for archiving, for handing a codebook to a quant team, or for counting a deliberately coded variable across hundreds of sessions. The problem is using the export as your analysis environment. If the question includes "why", "how" or "where they differ", the answer is in the conversations, not in the rows.
How This Works in Qualz.ai
Qualz.ai keeps every interview as a structured, queryable conversation, not a set of rows. You can ask any of the ten questions above across a whole study or several studies. Answers come back with participant IDs, timestamps and the moderator turn that prompted each quote, so the evidence travels with the claim into your readout. That's the standard we set out in the Evidence Density Test. Segment tags can be checked against what participants actually said, and contradictions show up without anyone having to go looking for them.
If your team answers stakeholder questions by exporting spreadsheets, book a Qualz.ai information session and bring a real study. We'll run these questions on it live.
Practical Takeaways
- Before answering any segment question, run question 2 and reclassify participants whose interviews don't match their tags.
- Phrase comparisons as three-way splits: real differences, differences in wording only, and topics one segment was never asked about.
- Run a contradicting-evidence query (question 4) on every theme before it goes into a deck.
- Map onboarding pain by the moment it happened and by workaround, not by keyword.
- Separate the reasons churned participants give from what happened beforehand, and label each case as a gradual decline or a trigger event.
- Retrieve representative quotes from different participants and always include one outlier.
- Never present a quote without the moderator question that prompted it.
