print preview

«We use the AHP method to determine the overall benefit of the candidates»

Recently the candidates submitted their second proposal. From the perspective of the project teams, this marks the start of another important phase: the analysis of the proposal, the determination of the total benefit and the total cost, and the final reporting. Xavier Comby, responsible for the engineering part of the NFA-project, explains how the determination of the total benefit is carried out by using the so-called «AHP method».

Aeronautical Systems, Air2030 Programme

Xavier Comby stands in front of a flipchart and explains an equation
Xavier Comby, sub-project manager technology «New Fighter Aircraft»

AHP stands for «Analytic Hierarchy Process» and is a method of decision support used to determine the total benefit of each candidate. The name of this method refers to three of its most important features. The method is:

  1. Analytical, because the total benefit is determined based on mathematical calculations. 
  2. Hierarchical, because the so-called evaluation points are organised hierarchically in the same way as a decision tree, i.e. from the main evaluation criteria to the individual evaluation points.
  3. Process-oriented and systematic respectively, and therefore transparent and comprehensible.

The AHP method was developed by the mathematician Thomas Saaty at the end of the 1970s and has since been the subject of numerous research papers. When defining the evaluation process in the NFA and Bodluv GR projects, the latest research studies concerning AHP were thoroughly analysed and validated by the project team, for example with the help of specially developed simulations with the Matlab software.

When you compare the traditional benefit analysis with the AHP method, you will find that the latter is mathematically somewhat more demanding. It is essentially based on so-called matrix multiplications and the determination of eigenvectors. One of the advantages of the method is that not only is an evaluation of the various evaluation points carried out, but also the consistency of an evaluation is calculated and thus becomes visible. Furthermore, the AHP method makes visible how strong the consensus of an evaluation team and thus of a decision making process is.

The AHP method is particularly suitable for the evaluation of more complex systems such as weapon systems. In the NFA and Bodluv GR project, an established software is used for this purpose, which was procured on the basis of market clarifications and discussions with other, uninvolved procurement authorities and was then thoroughly tested and validated before its application.

How does the AHP method work?

In principle, the method works in such a way that, as part of a clearly defined process, the participants in a team of experts compare certain evaluation points in pairs. This pair-wise comparison makes it easier for the evaluators to assess the performance characteristics systematically.

In principle, the four main evaluation criteria have been defined in the Air2030 projects with the corresponding weightings at project start: These are effectiveness with 55%, product support with 25% and the opportunities for cooperation and direct industrial participation with 10% each. In the NFA project these criteria were divided into about 80 evaluation points. The AHP method is used to determine the benefit of the candidates for each of these evaluation points.

The basis for comparing the candidates are the so-called technical reports. On the one hand, these summarise the findings from the analysis of the proposals or the answers to the questionnaire which the candidates had to answer with the proposal. The findings from the testing activities are also incorporated into these technical reports, mainly by validating the candidates' answers. Each technical report refers to one evaluation point per candidate and is not comparative. With around 80 evaluation points for the NFA project and four candidates, 320 technical reports must therefore be produced. The comparison of the candidates is based on objective measures per evaluation point, which were established at an early stage of the project.

«Each technical report refers to one evaluation

   point per candidate and is not comparative»

                                            Xavier Comby

As a possible example of an evaluation point the «ease of maintenance» can be used. This evaluation point would be assigned to the main evaluation criterion «product support». Measures that could be used in this example would be, for example, the average time taken to repair a defective aircraft and release it for flight operations. The data source for these parameters is the information provided by the manufacturers, which was validated during the ground tests in Switzerland.

These data and their analysis are documented and summarised as results for each candidate individually in a Technical Report «Ease of Maintenance» by the corresponding expert team.

Based on these technical reports, the candidates are then compared with each other using the AHP method. To stay with our example: The candidate for whom such a repair can be carried out most efficiently receives more points than the other candidates.

AHP sessions

The comparison of the candidates takes place in so-called AHP sessions, a process of which is formally defined. This can be roughly summarised as follows using our example:

  1. The persons in charge of the Expert Team on «Ease of Maintenance» present the findings on this evaluation point to the evaluation team on the basis of their technical report.
  2. The presentation is followed by a question and answer session.
  3. Each individual member of the evaluation team will then compare the candidates independently. The consistency of these individual evaluations is checked by the AHP tool.
  4. Then, the aggregation of the individual evaluations into an overall evaluation follows. The AHP-Tool again determines how consistent the overall evaluation is and how the consensus within the evaluation team is.
  5. Significant inconsistencies in the evaluation team are detected by the tool and must be clarified before an evaluation is finally accepted.

Comparison of total benefits with total costs

The result of the comparison of total benefit and total cost is presented in a diagram in which the vertical y-axis represents the total benefit. The horizontal x-axis is the cost axis. Here, both the procurement costs of the systems and their operating costs over a 30-year are taken into account for the evaluation of the total costs. Seen in this light, the total benefit and the total costs each flow 50% into the diagram or into the comparison of total benefit to total costs.

Benefit analysis with AHP

The AHP method is initially used in the NFA and Bodluv GR projects. The experiences of other procurement authorities and our own experience to date show that the method is robust. It is particularly advantageous in comparison with the traditional benefit analysis when more complex systems are evaluated. The experience gained in the Air2030 programme will be documented and passed on for further projects. For a successful application of the method, it must not be forgotten that a project must create the conditions early in the process to be able to use AHP later in the process. The requirements are directly linked to the questions in the questionnaire of the request for proposal and these in turn are linked to the individual main evaluation criteria or evaluation points. This means that the project team must be clear early on what it wants to evaluate a few years after the project has started.

Brief profile

Xavier Comby (46) is a graduate ETHZ physicist and holds a Master in Aviation from the ISAE (Institut Supérieur de l'Aéronautique et de l'Espace) in Toulouse/France. He joined armasuisse in 2014 after holding positions in aviation companies in Switzerland and abroad both as a systems engineer and in management roles. In the NFA project, he is responsible for the engineering part of the project. At the level of the Air2030 programme, he manages the system-of-systems engineering and creates technical specifications for cross-programme aspects.