The Primary Causes of Project Failures in Software Development and How to Prevent Them

Blurred path
A project with unclear requirements is like a driving with poor visibility of your destination.

The CIO magazine reported in a study that more than 50% of IT projects fail.

A Project Management Institute (PMI) survey in 2017 of companies with failed projects, said that the #1 cause of failure is “change in organization’s priorities”.  If corporate objectives have changed and a project is no longer serving the priorities of a company then it makes sense to end the project.  It is debatable whether we can call that a failure or not.  Accepted.

Primary causes of project failures
Primary causes of project failures.  Source: PMI

Now, let’s look at the second most common cause of failure.  In the same PMI survey, 39% of the respondents said that their projects failed due to “Inaccurate Requirements Gathering.”   This is a preventable situation, yet time and time again, this same scenario happens in many organizations.   A common problem with an uncommon solution.

How many software development projects have we seen that start with unspecific requirements yet promise a specific budget and specific target finish date.  What was the basis for the estimates?  Guesstimate?  

A successful project should have all of the above realized within expectations – we should deliver the requirements, within budget, and on time.   It always involves balancing these three:

  • requirements – what do we plan to achieve?
  • budget – how much will it cost?
  • schedule – when can it be done?

Among these three, the business requirement is the major factor that influences both budget and schedule and yet it is this factor (among the three) that gets the least attention.  If we do not get the requirement accurately, we will always miss our budget and our schedule.

Even if we have the best programming talent in our team but if our requirements are not clearly stated before we start coding, our team will be in a state of continuous improvisation – where we refine requirements while the project is already underway.   Then what?  We will be surprised with “new” things that we sometimes call “changes” that we only realize in the middle of construction or worse during testing.   These surprises may require small or big changes.  It may require rework or scrapping the design and rebuilding things from scratch.  Regardless of the size, these revisions will always move schedule and/or increase expenses.   We will have a hard time hitting our deadline and controlling our expenses.  Maybe we can complete the project after long delays and an exponential increase in cost.  Still, it is a project that missed its targets.

How do we ensure that requirements are accurate?  Or perhaps, at least clear enough to prevent any chance of being misinterpreted?

How To Improve Accuracy of Business Requirements

  • Have the project sponsor define the project charter in very specific parameters.  Example: “Reduce our turn-around time for loan applications to one hour or less for all applications with complete requirements.  This should be live by June 1, 2019.”.  Explicitly specify that any feature that does not serve this mission is out of scope.  This last statement may sound imposing but it will help your organization maintain focus on the mission.  It will reduce scope creep where well meaning users insert their nice-to-have features and declare them as “very important”.
  •  Communicate all business requirements in quantifiable terms.  Specifications that are quantifiable are verifiable and will not invite any dispute when validation time comes.  Any data presented using the characters 0 ~ 9 or logically verifiable will not be misinterpreted. Examples:  “Every transaction will be charged a PHP 150 processing fee.  This fee should be maintainable by the designated administrator as needed.” ,  “All system response time should not go beyond 10 seconds.”.  These are all quantifiable requirements and since they are known  before programming starts, it definitely impacts the way we approach construction.   You are your client will also have a clear way of measuring what is acceptable or not instead of arguing about it come user acceptance test period.  It will save the entire project team valuable time and it will save your company from unexpected expenses.
  • Create an inventory of all business requirements in a searchable repository ensuring every requirement has a designated unique number.   This is so it is easy for all stakeholders to be specific when referring to a business requirement.  This can be done using a feature tracking tool like Trello, Redmine, JIRA or other suitable tools.  This same inventory database of features will be used to update pass/fail when the time comes to validate the system.
Total Project Cost is Lower With Precise Requirements
Total Project Cost is Lower With Precise Requirements

Situational Examples on How To Arrive At Quantifiable Requirements

Let’s say you are gathering requirements and  your client tells you:  “Make sure the system has a fast response time.”  What are you going to say?  “Of course, we have experienced engineers who know how to optimize applications.  Nothing to worry.”  Your client nods, confident in your promise. 

Are we done?  Not so fast.  Does the client mean fast in relation to their current system that gives results after an hour?  Or do they mean fast like Google search that gives sub-second response time for search results?  There is a million dollar cost difference between the two.  We need to ask the client for some specific questions.  Example questions:

1)  How many users do you expect to be using the system at any given time?  Example of a precise answer: 150 concurrent users maximum.

2)  How many seconds is the response time for you to consider it fast?  Example of a quantifiable answer: Under 1 second.

3)  Do all your users have access to internet with at least 10mbps speed?

4)  Are you willing to spend 1 Million pesos a month to achieve that speed requirement?  This invites a specific Yes/No answer.   If the answer is ‘Yes’, we proceed.  If the answer is ‘No’, we go back to #2 and compromise on the speed vs cost.  For example, we can say — “We can give you 10 second response time for a system server that is not going to be more than PHP 100k per month operating expense”.   

The answers to these sample questions are quantifiable.  We should always drill down to precise and measurable parameters that nobody will misinterpret when it comes to testing time.  Words can be misinterpreted even if we all speak the same language.  Quantifiable requirements leave little for misinterpretation.

Impact of Clear Requirements During Client Testing Phase

Very often, inaccurate business requirements show their ugly head after construction and during testing.  When the time comes that the business owners validate the work, any misunderstanding with the requirement will start to manifest around this period, leading to rework, more testing and many more cycles of testing and rework, leading to project delay and budget overruns.  If we do not spend enough time to come up with clearly defined business requirements before we start construction, it will surely translate to much higher expenses later on in the project.

Let’s revisit our example on response time above.  If we did not drill down to numbers, the word “fast” will fail in the eyes of the load tester if they test it against 1000 concurrent users.  “Fast” will fail in the eyes of an end-user tester that expects it to give a response in under 1 second.   You will not have your sign-off document because the specific parameter for passing the criteria is not written down.

If we have accurate requirements well documented – they’re like the height of the bar set for a high jumper.  The jumper, the judge, the audience are all aware of what needs to be passed.  You either clear it or you don’t.

The height of the bar we need to clear is precise.  There is no gray area between failure and winning.  You either clear it or you don’t.

All the project team members and the client representatives who will test the system will know the specific parameters for passing or failing the test of a feature.  Knowing the specific pass/fail criteria reduces any misunderstanding and rework to a minimum.

The common way software development is done is to perform the quality assurance activity after the product is ready for testing.  That is a natural progression but the problem with involving QA only later in the project phases means they are only there to do quality control (not quality assurance) — which is a reactive activity that measures unclear requirements.  The effect will be a test phase full of misunderstanding between the project team and the client simply because the parameters for passing or failing it are not clearly articulated during the planning phase.

It will be advantageous to the project to clearly define fail or pass parameters before construction even starts so that the project team know what they are supposed to build and the testers will know what they are suppose to test.

Yes, gathering precise requirements is hard work but there are many benefits to this early hard work —  we come up with more accurate business requirements which translates to more accurate cost estimate and a more precise completion date.  Let’s move the hard work very early in the project planning phase, during requirements gathering, before construction, so that we have a much clearer view of the risks we face.  Knowing the risks early allows the project team to make adjustments early in the project phase.  

Our project will have a much higher chance of completing a software development projects on time and within budget if you took the time to ensure the business requirements are quantifiable.

Author: Galileo Valiente

The author has 25+ years of experience working with custom enterprise application development in multinational corporations.   With hands-on experience as a software developer and a certified project manager.