print preview

«At the end of the evaluation process, a comprehensive comparison between the candidates is possible»

The evaluation of the new fighter aircraft is complex. Which methodology is used, which rules apply in regards to the cooperation within the project and with the candidates, which tools are used, which interfaces have to be considered or for which eventualities one must be prepared for? In order to answer these and other questions and to give interested parties an insight into the course of an evaluation, the project team created an explanation video using the example of the new fighter aircraft. Darko Savic, Program Manager «New Fighter Aircraft» explains in an interview what is of particular importance during the evaluation process.

Aeronautical Systems, Air2030 Programme

Darko Savic is standing on the stairs. At the back you can see the staircase of building B of the VZ G1.
Darko Savic, Program Manager «New Fighter Aircraft»

Evaluation process by the example of the procurement of the new fighter aircraft

The evaluation of military systems is complex. Which methodology and tools are used, which interfaces have to be considered or for which eventualities one must be prepared for? The explanation video (in German) provides an insight into the course of an evaluation using the example of the new fighter aircraft.

Darko Savic, how do you communicate with the candidates?

In the forefront of the evaluation, we carefully thought through how to execute the cooperation with the candidates in a goal-oriented process. After all, the ultimate goal is that we can communicate to them exactly what we want from them. To achieve this, the way we communicate must be clear and disciplined; administrative wastage needs to be minimized wherever possible. In addition, we pay particular attention to ensure that we treat all candidates equally and that identical discussion points and procedures are used in all meetings.
More specifically, we have exchanged virtually no e-mails with the candidates. Furthermore, only the project office communicates with the candidates, which relieves the individual project managers. Instead of e-mails, we work together with the candidates on a SharePoint. The collaboration is based on clear rules, which we communicated to them together with the request for proposal. For example, the candidates may ask us questions about the request for proposal based on a defined process. Answers that concern all candidates are shared with the other candidates.
Thanks to the clear communication process, we are also able to limit the number of meetings with candidates to the minimum, which reduces the workload for all parties. Experience shows that unclear guidelines and processes will inevitably lead to an increase in the number of meetings.

As we saw in the video, the candidates are compared at the end of the evaluation process, why?

The evaluation methodology is designed in such a way that a comparison of the candidates is only possible once we have solid and definitive data.
The video shows that there are two proposal rounds. Only with the information from the second proposal, our analysis for each candidate can be finalized. Based on this analysis, we compare the candidates with each other on the benefit side. This is followed by the comparison of total benefits and total costs in the evaluation report. Only then, at the end of the evaluation process, a comprehensive comparison between the candidates is possible.

How are the candidates compared with each other?

We summarize the findings from the proposals and from the various testing activities in collaboration with the Armed Forces Staff, the Air Force, the Armed Forces Logistics Organisation and the Armed Forces Command Support Organisation in technical reports for each candidate and for each subject area separately. These technical reports form the basis for comparing the candidates.
For simplification and as a hypothetical example, one can imagine that a total of 1,000 points can be distributed to the candidates on the benefit side. Of these 1,000 points, a certain number of points are assigned to a subject area, for example the ease of maintenance. When comparing the candidates, it is now a matter of distributing these points among the candidates. The best candidate in this area from the Swiss point of view wins the most points, the second best candidate the second most points, and so on. There are various subject areas to which we allocate points to the candidates. In turn, different specialists are dedicated to each subject area.

What are your present experiences from the NFA evaluation process - what would you like to share with other project managers?

We invested a lot of time in the development of our evaluation methodology in the forefront of the evaluation. To do so, we not only took into account the findings and lessons learned from previous evaluations. We were also interested in how other procurement agencies work when evaluating complex systems. Against the background of these findings, we have further developed our evaluation methodology. Experience to date shows that the methodology is robust and consistent.
It is imperative to have a clear picture of how to manage the project in the run-up of the project. This saves a lot of trouble and time later on.

Who is part in the project team?

No matter how sophisticated the evaluation methodology is, the success of the project depends on the people involved. I can rely on an experienced team. Even if the list is not exhaustive, it seems important to me to show which functions interact with each other in a project of this complexity.
It starts with the Armed Forces planners and the partners from the Armed Forces who define the requirements for the system and assign and support armasuisse with the execution of the project. Employees from different units and professions are involved in the execution of the project. Engineers who analyse the systems from their technical perspective and who are later responsible for integrating them into the existing system landscape. Engineers from the fields of life cycle management and product support, whose methods must keep an eye on the support of the system over the next 30 to 40 years. Test pilots and flight test engineers who test the aircraft thoroughly. Quality managers, who take care of the acceptance and later help to approve the systems for safe operation. Commercial managers and business economists who, in cooperation with lawyers, ensure that commercial processes are adhered to, and contracts are concluded. Communications experts who relieve the project of communication tasks. Specialists who carry out noise measurements during the flight tests. Legal experts who assess the weapon system from the perspective of international law and approve them for use. Frequency management specialists who ensure that the systems do not have a negative impact on the Swiss frequency landscape. The project office, which supports the project management and the entire project team in project management tasks, thus ensuring the smooth operation of the business. The list could be further extended and is intended to show as an example that many players are involved and thus make an important contribution to the success of the project.

Why does the evaluation take so long?

The video shows which steps are necessary to perform a comprehensive evaluation of a high technology system. In the case of the New Fighter Aircraft, it takes almost three years from the submission of the first request for proposal to the preparation of the evaluation report. A phase in which, in addition to the two proposal rounds, a test phase is also integrated. With this phase, we tested the aircraft in the operational environment in Switzerland, which also enabled us to validate the information provided by the candidates. Obviously, during this process one also gains good knowledge of the respective aircraft, which is an advantage in the negotiations from the Swiss point of view.
Finally, this is also a strategic and cost-intensive procurement, which must be handled with the necessary seriousness. After all, we want to use the system operationally for 30 to 40 years and be sure that we have considered everything. On the other hand, the candidates have to analyze our request for proposal, which is about 600 pages long, create answers, and comply with internal business approval processes and so on. That takes time.

The video talks about a specially developed, self-contained data centre. Why is this required and how does it work?

We primarily focus on the protection of information, which is of essential importance to us for three reasons:
First, we need classified information from the candidates in order to be able to carry out a solid evaluation. We are thus committed to protecting the sensitive military information accordingly. This is of particular importance because the fighter aircraft of the respective candidates are important elements in the defence structure of the candidate's countries. The disclosure of sensitive information could therefore cause significant damage. It is therefore a matter of trust between the DDPS - even Switzerland - and the candidate's countries. Any indiscretion could harm Switzerland's foreign policy interests or international relations.
Second, from an industrial and commercial point of view, it is also important to prevent the disclosure of business and manufacturing secrets of the companies involved.
Third, the project could suffer damage due to an unintended disclosure of information. This could, for example, disturb the well-organized evaluation process and undermine confidence in the project organization.
To ensure the information protection, we use a self-contained data centre without network access. In addition, we work according to the principle «knowledge only when necessary». The data centre allows a clear allocation of access rights. That way, we ensure that the respective specialists only see the information in the proposal that is necessary for their task. In addition, the technical specialists, who assess the candidates from their technical perspective, have no insight into the cost structure.

Brief profile

Darko Savic (42) is a graduate mechanical engineer. He has completed a master's degree in project management and is IPMA Level B certified. After working as mechanic, design engineer, systems engineer and project manager in Swiss aviation companies he joined armasuisse in 2009. From 2009 - 2014 he worked in the project of the partial replacement of the F-5 Tiger in the function of project manager product support. In addition, he managed projects in the field of the F-5 Tiger as well as in the field of pilot and parachute equipment. From 2014 to 2017, he led the F/A-18 Service Life Extension Program in the function of program manager, and since 2018, he has been in charge of the «New Fighter Aircraft» program.