The Reaction You Cannot Trust
A participant opens your new feature for the first time. They lean in. They explore. They forgive a confusing label because the whole thing feels fresh and interesting. They rate the experience highly and tell you, sincerely, that they would use it every day. Your usability session looks like a triumph.
Then you ship it, and the daily-use numbers never materialize. The feature that tested beautifully turns out to be quietly abandoned within a week. Nothing in your research predicted this, because your research captured the one moment when the product was most flattering to itself: the first encounter.
This is the novelty confound. In a first-session usability test, the participant is reacting to two variables that you cannot separate -- the design itself, and the simple fact that it is new. Novelty inflates attention, tolerance for friction, and self-reported satisfaction. It is a temporary state, and it decays fast. If your evaluative research lives in first sessions, you are systematically measuring the wrong thing.
Why Novelty Is Not Noise -- It Is Bias
It would be convenient if novelty were random error that averaged out across participants. It is not. Novelty biases in a consistent direction: it makes almost every new design look better than it will perform in sustained use. That directional consistency is what makes it dangerous. Random noise widens your error bars; systematic bias moves your conclusions.
Consider what novelty specifically suppresses. In a first session, a user has no muscle memory to violate, so awkward interactions feel like learning rather than friction. They have no accumulated frustration, so a minor annoyance is absorbed rather than compounded. And they are often aware they are being observed, which layers the observer effect on top of the novelty -- two inflating forces pointing the same way.
The friction that actually drives abandonment -- the tenth-time-you-do-this irritation, the shortcut you wish existed, the step that feels fine once and infuriating daily -- is invisible in a first session by construction. You cannot observe durable friction in a single encounter, because durable friction is defined by repetition.
The Task-Completion Mirage
Teams often defend first-session testing by pointing to hard behavioral metrics: task completion rates, time on task, error counts. Surely those are objective? They are objective, but they are still measuring the novel state. A user who completes a task on first try while fully engaged tells you the task is completable under ideal attention. It tells you nothing about completion under the distracted, habituated conditions of real use.
This is the same trap we described in the surrogate endpoint problem, where task completion rates mislead about real-world adoption. The metric is real; its relationship to the outcome you care about is not what you assume. First-session completion is a surrogate for sustained usability, and novelty makes it a badly calibrated one.
What the First Session Actually Measures
The first session is not worthless. It is a precise instrument -- for learnability. It tells you how a fresh user forms a mental model, where their first assumptions break, whether the initial value is legible. Those are real, useful questions. The error is treating learnability data as if it were usability data. They are different constructs, and conflating them is a version of the generative versus evaluative confusion, where mixing discovery and validation in one study produces neither.
What the first session cannot measure is the thing product teams most want to know: will people keep using this? That question requires watching the novelty burn off.
Designing Novelty Out of Your Findings
You defeat the novelty confound by extending your observation window past the point where novelty decays -- usually the second or third real-world exposure. Several methods do this.
Multi-session protocols. Bring participants back after they have lived with the product for a few days. The delta between session one and session three is where durable friction reveals itself. This is closely related to what we covered in the second interview effect and how follow-up data differs.
Diary studies. When you cannot re-run moderated sessions, longitudinal self-report captures the decay curve. As we argued in diary studies reveal what interviews miss, the friction that surfaces on day four is the friction that predicts churn.
Explicit novelty accounting. At minimum, annotate first-session findings with a novelty flag. Separate "reactions likely inflated by newness" from "structural observations," the way an analyst separates strong from weak evidence.
There is a systems analogy worth borrowing here. Enterprise AI teams learned not to trust launch-day performance because systems degrade silently over time -- which is why they invest in observability and monitoring of AI systems in production rather than one-time acceptance tests. Usability has the same shape: the launch reading is the least representative one. And just as rigorous teams practice eval-driven development, testing AI systems continuously rather than once, usability evaluation should be continuous across sessions, not frozen at first contact.
The Discipline of Waiting
The hardest part of defeating the novelty confound is organizational, not methodological. Stakeholders love first-session enthusiasm. It confirms the roadmap, energizes the team, and arrives quickly. Telling them that the glowing first-session data is the least trustworthy data you have is unwelcome news. But it is the truth, and the teams that internalize it stop shipping features that test well and die quietly.
Qualz.ai is built for research that spans time, not just moments -- helping teams run and analyze multi-session and longitudinal studies so the decay of novelty becomes visible in the data rather than in the abandonment metrics three weeks after launch. The first session tells you what a product feels like. Only the sessions after it tell you what it is.
