Andrew Brown, a Principal Consultant at Trissential’s parent company, Expleo, discusses the top five ways that human factors hold back software development.

Information technology and software development have delivered enormous benefits to humankind. They have improved living standards for most of us and made many of our lives more enjoyable and rewarding. However, far too many software development projects have either failed outright or not delivered the expected benefits.

Contrary to popular belief, the main reasons for failed projects are not technical but instead are human factors. We must recognize the importance of human factors and address the problems they cause. If we don’t, our industry will continue to experience failed projects.

1. Poorly estimated projects in the planning stage

The main reason projects fail, are delivered late, or are over budget is that they are poorly estimated. Our poor estimations are driven by four human biases; anchoring, overconfidence effect, optimism bias, and the show-off hypothesis.

  • Anchoring is where our estimate is anchored to an initial value. If we fail to adjust far enough away from that value, it causes low estimation as follows. The business asks for a cost and time estimate. But whilst doing this it also asks if the software can be delivered by, say, the end of the year. Naturally, this date is far too early. When we produce an estimate, we adjust it upwards. However, we do not adjust far enough.
  • The overconfidence effect is where we have excessive confidence in our own judgment that is not warranted by objective accuracy. This causes us to be overly confident in our own skills, ability to influence others, and ability to deliver. Hence leads us to believe that we can deliver far earlier and at a lower cost than is realistic.
  • Optimism bias is where we have unwarranted optimism. When estimating a project, the bias makes us believe that we’ll be the lucky few not hit by an adverse event. Even despite ample evidence that most projects encounter several such events.
  • The show-off hypothesis is where we engage in unnecessarily risky activities. In our evolutionary past, we engaged in such activities to demonstrate the quality of our genes to potential mates. On a modern-day project, this leads us to take on more risk than is prudent.

2. Irrational risk-taking

Once a project has been underestimated, sooner or later it will become overdue or over budget. When this occurs, the project tends to take crazy risks. Such as risks where the expected costs exceed the benefits, or there is a high chance of catastrophe. Additionally, overdue projects may behave irrationally in other ways, such as discarding partially developed code or features that have value.

So, we take crazy risks on projects. When that project becomes overdue, our situational perception changes from one of the gains to one of the losses. This in turn causes us to switch our risk appetite from risk-averse to risk-seeking, encouraging us to take crazy risks.

3. Organizations have too much technical debt

Most organizations complain that they have too much technical debt. However, as they complain, they are probably running a project which will add to that level of technical debt.

Technical debt is not a technical problem, although it sounds as if it should be. Technical debt is a problem of trade-off, between present and future, between costs and benefits. It is a classic vice versus virtue dilemma. Technical debt is a consequence of how we make decisions, which is by using a process that involves our emotions; the affect heuristic.

Organizations have excessive levels of technical debt. When making the trade-off between technical debt and new features, technical debt does not weigh heavily, using the affect heuristic. Consequently, organizations choose new features over technical debt and then complain about the technical debt they suffer.

4. Adding quality processes does not improve quality

Many organizations are unhappy with the quality of their finished software product. Therefore, they add in additional quality processes to try and catch more bugs. However, they are often disappointed to find that quality does not improve as much as they hoped.

This is because of risk compensation, sometimes known as the Peltzman Effect. This effect causes us to take more risks when we perceive safety has improved. It is why we drive faster when wearing a seatbelt. On a software project, it causes us to become a little less careful. Especially when we know the project has put in additional quality and test processes, to catch any mistakes we make.

5. We continue to fund zombie projects

Irrational escalation, sometimes known as sunk cost effect, is a bias that causes us to continue investing in a decision. This is due to the past investment that we have ‘sunk’ into it. Even though evidence shows that continued investment will never be repaid.

This bias causes organizations to continue to invest in zombie development projects. Instead, they would be better off killing and investing resources elsewhere.

Conclusion

All these problems, plus others, are important issues that hold back software development for most organizations. Although these problems can be driven by technical factors, human factors play a far more important role.

We have had these problems within software development for many decades. We will continue to have these problems until we begin to recognize and address the human factors underlying these problems.

Other industries, such as aviation, medicine, and government policy, are already addressing similar problems caused by human factors. These problems are difficult, but by no means insurmountable. The rewards to organizations that successfully address these human factors and overcome these problems will be huge.