Measuring Project Success with Business Intelligence Dashboard

This system is quite easy to use and understand as it is based on a simple 0 to 100 scale. Depending on the rating, the overall project’s success falls into three groups, each marked with a different colour (red, yellow or green). Later on in the article, we’ll dive into more details to show you how we built the in-house BI system.

How to get an extensive picture of the project’s overall performance

When it comes to measuring a project’s success, all criteria may be grouped into five well-known categories:

• Scope – to characterize the specific project goals, functions, tasks, deadlines, requirements
• Budget – to track the project's costs usage
• Time – to define if the project is on schedule
• Quality – to assess the overall status of the project's quality: process, testing procedures, issue identification, and resolution
• Resources – to evaluate project resource management.

It may be common practice to measure project success considering time and budget as the key parameters. But is it really the right method? Doubtfully. The fact that a project is finished on time and budget doesn’t mean it is a success. Obviously, such an approach doesn’t provide the full answer and does not address other factors contributing to the project’s success or failure. So, to provide a more extensive and up-to-date overall picture of the projects, we decided to build a system that could combine a comprehensive list of criteria in all five areas and present the results on the system’s dashboard.

So, how to measure the key 5 success indicators?

After deciding to use a combination of criteria, it was time to start building a complex math model capable of calculating and analyzing individual factors that affect the overall project health. To design the unique metric system, powered by Data Science, we implemented the Bayesian Network (BN), a directed probabilistic graphical model that deals with uncertainty and complexity. One of the most distinct advantages of this network tool is its ability to perform various types of analysis as well as to model complex relations between various faсtors. It can also perform some sophisticated interrogations of the system and include data from various information sources.

In the picture below, you can see a ready-to-use model based on data extracted and incorporated in the BN from the following sources:

• An issue tracking system used at ELEKS
• A team collaboration software at ELEKS
• MS Dynamics CRM
• Our proprietary HRM (a system for tracking human resource management processes)
• ELEKS’ code repository
• The internal financial management system.

A BN is structured into a directed acyclic graph, which consists of the set of variables (denoted by nodes) representing the probabilistic dependencies among those variables. The conditional dependencies in the graph are estimated by using the known statistical and computational methods. Constructing such a network requires expert knowledge of the underlying domain. So, at first, we built a directed acyclic graph and then included the conditional probability distributions for each node in the graph, taking into account the stakeholders’ experience and expectations.

How the model actually works

To explain how the model works, let’s model the following case. First, observations about the project are received, and then the state of the particular node changes and the reasoning process triggers the calculation of the probability values for all the dependable nodes. Here are some factors that cause reasoning:

• Customer satisfaction index
• Delivering a release on a specific date
• Changes in a cycle time value
• Changes in the number of issues that are in the ‘in-progress’ status
• Changes in the number of reopened bugs
• Changes in the ratio of closed and opened bugs
• Financial data in comparison to the company’s KPIs
• Project stage
• Project type
• Environment and hardware availability, etc.

The graphical model below demonstrates that the project’s success equals 87.36%, taking into account only expert knowledge.

But, as soon as the information is updated, it causes the value of the project’s success to change. Below, you can find various factors that might affect the overall project’s performance:

1. The cooperation model used for this project is the fixed bid model.
2. The project has reached a stable stage.
3. The cycle time value is decreasing in comparison with the previous check.
4. The number of the opened and reopened bugs is lower in comparison with the previous check.
5. The project’s financial values fit the company’s KPI values.
6. All the environment tools are set and in-use.
7. There are no blocking requests concerning software and hardware.
8. There are no open vacancies.
9. The latest customer satisfaction rate is high.

Everything looks quite good, doesn’t it? But, here are some more factors, worth paying attention to:

1. There is an increasing number of tasks in the ‘in-progress’ status.
2. The predicted release date is currently out of the schedule.

Being out of schedule and having low performance is quite risky for project delivery, as it might radically change the level of the project’s success (especially when we are dealing with the fixed bid cooperation model).

All these new observations lead to the recalculation of the dependable states of nodes within the model, resulting in a new project success value - 12.71%. Fig. 4 shows how the model values have changed comparing to the Fig. 3.

How we proceeded to adapt our model for measuring project success internally

Well, our next step was to validate and implement the model within all the company’s projects. To do this, we questioned the company’s PMs about the status of their projects. Then, we compared their assessments with information that was obtained from the BN model and discovered that the results were pretty close to the project managers’ expectations. This process of validating the model unveiled the area for improvement. It concerned CPD (Conditional Probability Distribution) correction. Changing the CPD values helped us adjust the influence of one node to another, and as a result, we made them more or less sensitive regarding the changing observations.

And only after having conducted a thorough analysis, we proceeded in collecting data for all the company’s projects and then calculated their success. Now we have the system that is currently available for ELEKS’ senior management and project managers, offering them a new, better and advanced way to keep a finger on the pulse of each of their projects.

Figure 5 is the screenshot from the system’s BI dashboard, showing the state of the sample projects. Our aim was to simulate the real-life case: