Hardwired to Underestimate: The Cognitive Science Behind Chronic Project Estimation Failures — and a Proven Framework to Fix Them

Every experienced project manager has lived through a version of the same story. A confident team submits an estimate. Stakeholders approve it. Six months later, the project is over budget, behind schedule, and leadership is asking uncomfortable questions. The postmortem diagnosis is almost always the same: the original estimates were too optimistic.

The conventional response is to demand more rigorous planning next time — better templates, more detailed work breakdown structures, tighter scope definitions. And yet the cycle repeats. Quarter after quarter, across industries and organization sizes, project estimates continue to miss the mark in the same direction: they are almost always too low and too short.

The problem is not a process failure. It is a cognitive one. And until organizations acknowledge that distinction, no amount of new tooling will close the gap.

Why Your Brain Is a Terrible Estimator

In the 1970s, behavioral economists Daniel Kahneman and Amos Tversky identified what they called the planning fallacy — the near-universal human tendency to underestimate the time, cost, and risk associated with future tasks, even when the estimator has direct experience with similar failures in the past. This is not a trait of inexperienced or careless planners. Nobel laureate Kahneman himself admitted to falling victim to it during the writing of his own book on decision-making.

Layered on top of the planning fallacy is optimism bias — the cognitive distortion that leads individuals to believe their project will be the exception to the historical pattern. Studies published in the Harvard Business Review have consistently found that large infrastructure and technology projects overrun their original cost estimates by an average of 27 percent, with schedule overruns routinely exceeding 20 percent. In IT-specific projects, the figures are even more sobering.

Then there is anchoring, the psychological phenomenon where an initial number — even an arbitrary one — disproportionately influences all subsequent estimates. Once a project sponsor floats a desired timeline in an early meeting, that figure becomes a cognitive anchor that subtly warps every estimate the team produces thereafter.

The compounding effect of these biases creates what researchers call the optimistic planning spiral: each layer of the organization produces estimates that feel reasonable in isolation but are structurally biased toward underestimation at every level.

Why More Experience Does Not Automatically Help

One of the most counterintuitive findings in estimation research is that seniority does not reliably improve accuracy. In fact, expert estimators frequently perform worse than statistical models because their experience gives them greater confidence without proportionally improving their calibration.

This phenomenon — sometimes called the inside view problem — occurs when planners focus on the specific details of the task at hand rather than consulting the broader historical record of similar projects. A senior software architect estimating a new platform migration draws primarily on their mental model of this migration, not on aggregate data from the last fifty comparable migrations across the industry. Their expertise makes their inside view more vivid and more convincing, which makes it harder, not easier, to override.

McKinsey & Company's research on large-scale capital projects has documented this pattern extensively. Their analysis found that projects with experienced, highly credentialed leadership teams were not statistically less likely to overrun budgets than those with less decorated teams. The differentiator was not experience — it was the method used to generate estimates.

How Elite Organizations Restructure Their Forecasting

The U.S. military, which manages some of the most complex and high-stakes projects on earth, has long grappled with estimation failure. Defense acquisition programs have historically overrun original cost estimates by margins that would be catastrophic in the private sector. In response, the Department of Defense has institutionalized independent cost estimation — a practice in which a team entirely separate from the program office produces a parallel cost forecast using only historical data, with no access to the program team's internal estimates. The divergence between the two figures is then treated as a signal, not a negotiation.

The consulting industry has adopted analogous practices. At firms operating at the frontier of large transformation engagements, there is a growing norm of separating the team that builds the project plan from the team that validates the estimate. The validating team is explicitly instructed to ignore the internal plan and instead anchor their forecast to comparable historical projects.

This approach has a name in the academic literature: reference-class forecasting.

Reference-Class Forecasting: A Practical Framework

Developed by Kahneman and later operationalized by Oxford professor Bent Flyvbjerg — whose research on megaproject failures is widely cited in both academic and policy circles — reference-class forecasting is the most empirically validated method available for counteracting optimism bias in project estimation.

The method follows a structured sequence that any project management team can implement:

Step 1: Identify the Reference Class Select a group of past projects that are genuinely comparable to the current one in type, scale, and organizational context. Resist the temptation to define the class so narrowly that it excludes uncomfortable data points. A software implementation project is not unique simply because it uses a particular vendor.

Step 2: Establish the Distribution of Outcomes For the selected reference class, compile actual outcomes — final costs, actual durations, realized benefits — and build a distribution. Where does the median project land relative to its original estimate? What does the 80th percentile look like? This distribution becomes the empirical baseline.

Step 3: Position the Current Project Within the Distribution Using only the outside-view data, determine where the current project is most likely to fall. Unless there is specific, documented evidence that this project is structurally different from the historical norm, assume it will land near the median of the reference class — not at the optimistic end.

Step 4: Adjust for Known Risk Factors Only after anchoring to the reference class should the team introduce project-specific adjustments. Novel technology, organizational change management complexity, and regulatory exposure are legitimate reasons to shift the estimate toward the higher end of the distribution. Confidence in the team is not.

Step 5: Build in a Documented Optimism Uplift Flyvbjerg's research recommends applying an explicit optimism uplift factor — a percentage added to the base estimate to account for the statistically predictable tendency to underestimate. For IT projects, his data suggests uplifts in the range of 15 to 30 percent are frequently warranted.

The Tool Trap

It is worth addressing a common organizational response to estimation failure directly: investing in new project management software. Gantt chart tools, AI-assisted scheduling platforms, and integrated portfolio management systems are genuinely valuable for execution visibility. But they do not correct cognitive bias. A more sophisticated tool applied to a biased estimate produces a more sophisticated version of the same wrong number.

The organizations that have made measurable progress on estimation accuracy share a common characteristic: they changed their process, not just their software. They introduced structured pre-mortems — asking teams to imagine the project has failed and work backward to identify why — before estimates were finalized. They created psychological safety for team members to surface pessimistic scenarios without social penalty. And they held estimation reviews accountable to historical data rather than stakeholder preference.

The Leadership Imperative

For senior leaders, the estimation problem carries a specific responsibility. Organizational culture frequently punishes pessimistic estimates and rewards the team willing to commit to the aggressive timeline. When leaders respond to a conservative forecast by asking whether the team can "find a way" to hit a shorter date, they are not motivating performance — they are manufacturing bias.

Smart project leadership means creating the conditions in which accurate estimates are valued more than convenient ones. It means asking not "Can we do it in six months?" but rather "What does the historical record tell us about projects like this one?"

The science is unambiguous: the brain will not fix this problem on its own. The organizations that close the estimation gap are the ones that build systems specifically designed to override the brain's natural inclinations — and that have leaders willing to demand that discipline even when the numbers are uncomfortable.