3.3 Delphi Technique
Consensus reaching methods generally used in health care are Delphi panel, nominal group or consensus conference. They are useful to organize “qualitative judgments and, which is concerned to understand the meanings that people use when making decisions about health care.” (Black, 200688, page 132). They are not as such qualitative methods because they may use quantitative data collection tools (questionnaires, scales), and quantitative element in the analysis (statistics).
All the consensus methods cited here are characterized by the provision of information prior to the discussion, privacy (participants express their opinion in private), opportunity for participants to change their view and explicit and transparent derivation of the group decision, based on (statistical) analysis88.
3.3.1 Description of the method
The Delphi method (named so because of the Delphi Oracle) was initiated by the RAND corporation, a nonprofit institution that helps improve policy and decision making through research and analysis[a]. The original definition given in the 50s was that it “entails a group of experts who anonymously reply to questionnaires and subsequently receive feedback in the form of a statistical representation of the "group response," after which the process repeats itself. The goal is to reduce the range of responses and arrive at something closer to expert consensus.”89 Today, the method has evolved and Delphi surveys could aim at different goals or have several designs[b]. It could be define more as “a method for structuring a group communication process” and not as a method to produce consensus90. The method could also be defined as a systematic collection and aggregation tool of informed judgment from a group of experts on specific questions and issues” (Hasson, 201191, p. 1696).
Delphi surveys are used in several domains (politics, psychology, agriculture, etc.) and could vary in different ways. Several types of Delphi often used in health research (non exhaustive) are presented in Table 10.
Table 10 – Types of Delphi designs
Design Type | Aim | Target panellists | Administration | Number of rounds | Round 1 design |
Classical | To elicit opinion and gain consensus | Experts selected based on aims of research | Traditionally postal | Employs three or more rounds[3] | Open qualitative first round, to allow panelists to record responses |
Modified | Aim varies according to project design, from predicting future events to achieving consensus | Experts selected based on aims of research | Varies, postal, online, etc. | May employ fewer than 3 rounds | Panelists provided with pre-selected items, drawn from various sources, within which they are asked to consider their responses |
Decision | To structure decision-making and create the future in reality rather than predicting it | Decision makers, selected according to hierarchical position and level of expertise | Varies | Varies | Can adopt similar process to classical Delphi |
Policy | To generate opposing views on policy and potential resolutions | Policy makers selected to obtain divergent opinions | Can adopt a number of formats including bringing participants together in a group meeting | Varies | Can adopt similar process to classical Delphi or 1- preformulating the obvious issues by the research team; |
Real time/consensus conference | To elicit opinion and gain consensus on real time | Experts selected based on aims of research | Use of computer technology that panelists use in the same room to achieve consensus in real time rather than post or via Internet94 | Varies | Can adopt similar process |
Adapted from Hasson, 201191, p. 1697 and Keeney, 201195
[b] See the special issue 78 of the review ‘Technological Forecasting & Social change” (2011) available at http://www.journals.elsevier.com/technological-forecasting-and-social-c….
[3] Note that the number of rounds should ideally be based on the saturation of the responses and is difficult to fix in advance
3.3.2 Specific questions suitable for the method
The following questions could be answered by using a consensus reaching method such as the Delphi panel:
- To help the decision making process.
- When personal contact is not necessary96.
- To choose the most appropriate method or tool (e.g. data collection technique, scales, questionnaires, etc.).
- To identify the best choice of treatment (when no other evidence is available or to complete it).
- To identify the form of a programme.
- To clarify professional roles97.
- To develop clinical guidelines98.
3.3.3 Strengths and weaknesses of the method
3.3.3.1 Strengths
- Lower production cost99.
- Relatively rapid results99.
- Participant can express their opinion anonymously96, without external (perceived) pressure while the process allows to catch the view of the entire group96.
- Avoid domination by individuals or professional interests97;
3.3.3.2 Weaknesses
- Success depends on the qualities of the participants.
- Reliability increases with the number of participants (and the number of rounds). In addition, it is difficult to keep everybody in successive rounds96.
- Coordination is difficult96.
- The existence of a consensus does not necessary mean that it reflects an appropriate or “correct” answer97.
3.3.4 How to plan the research design?
A Delphi survey takes several weeks, even if the number of participants is small.
It has to be planned in the beginning of the project or, if the necessity to conduct such a study appears late in the course of the project, it is important to realize that the whole process takes several weeks, depending on the number of rounds needed. The next figure illustrates the whole process and the time needed.
Figure 5 – The Delphi process
Adapted from Slocum et al.93
3.3.5 Modalities of data collection
Delphi could be administrated ‘paper-and-pencil’ by mail or e-mail.
Online Delphi’s are more and more carried out. Software is available to support the data collection and the analysis (Delphi_Survey_Web (DSW)100, Mesydel©101)
The number of rounds is not necessarily defined a priori (often because of budgetary, time or human resources limitations): data collection must stop when the saturation or the consensus is reached.
3.3.6 Data collection tools
The Delphi method uses iterative (e-)mailed questionnaires in successive rounds. Because there is no interaction between the respondent and the researcher, the formulation of the questions has to be clear, and definitions should be given where necessary.
The questionnaire of the first round encompasses open-ended questions, to identify items to include in the second round.
Next rounds could be exclusively qualitative or composed of closed questions with scales (from totally agree to totally disagree, i.e. from 1 to 9), or combining both qualitative and quantitative questions. They present a synthesis of the results issued from the previous round.
In the case of closed questions, agreement is usually summarized by using the median and consensus assessed by presenting interquartile ranges for continuous numerical scales97. Graphical presentations of the results are welcomed.
In KCE reports the questionnaires used in each round are presented in appendices.
3.3.7 Sampling
Participants have to be carefully chosen because of their expertise, experience or knowledge in the field of the research question. In addition, the variety of positions in the field or opinions regarding the subject, should be covered. In that way, lay people could be added to increase the variety of viewpoints102.
They could be identified through publically available bibliographic information102. Snowballing recruitment could be useful to secure easy agreement to panelist invitation and strengthen panelist retention102.
There is no practical limit to the number of participants in a Delphi survey89.
3.3.8 Human resources necessary
The administrator of the survey develops the questionnaires, identifies, mobilizes and recruits participants, analyses findings and reports them. He/she is responsible for keeping a low attrition rate and insure the coherence between the different steps of the method.
Administrative support could be needed to (e-)mail the questionnaires and manage reminders and answers.
3.3.9 Practical aspects
- It is important to clearly explain the goal of the questionnaire and the way it will be analysed. The redaction of the invitation/introduction letter is thus crucial. “Stressing the practical policy application of the Delphi yield to experts panelists to aid their retention” (Rowe, 2011102, p. 1489).
- The research team should have managers skills to follow up the returned questionnaires and mailing.
- The utilization of online tools could be very useful as well for the research team (rapid results) as for the participants.
- While anonymity in the process of the Delphi is required, “using social rewards for recognition in participation, such as subsequently publishing panel membership listings” (Rowe, 2001102, p. 1489) could improve panelists recruitment and retention.
3.3.10 Analysis
Each step of the Delphi requires a specific analysis.
In a classical Delphi, open-ended questions from round 1 should be content analysed ‘in order to group statements generated by the experts panel into similar areas’95.
Round that uses closed questions should be statistically analysed. Summary statistics are used to decide whether or not consensus is reached. The level of the consensus has to be defined in advance (i.e. 70% of agreement).
There is no agreement on the threshold indicating a consensus, nor how to choose this threshold95. Each researcher has to reflect on it, case by case.
The proposals that have reached consensus should be eliminated from the next round.
3.3.11 Reporting of findings
Intermediary results are reported directly in the successive questionnaires.
All the consensus and dissensus items are listed and discussed at the end of the process.
3.3.12 Quality criteria
It seems that no consensus exists with regards to the standard of methodological rigor to apply. And that “no definitive evidence exists which demonstrates the reliability or validity of the technique” (Keeney, 201195, p. 104). This is partly due to the variety of the Delphi surveys and the constant evolutions in this field91.
We have not identified any checklists to assess the quality of a Delphi survey.
However, the following aspects of the survey could be assessed (adapted from Jillson103 and Hasson91):
- Applicability of the method to the specific research problem
- The quality of the composition of the Delphi panel. Participants have to be carefully chosen in function of their expertise and position in the group.
- Design and administration of the questionnaire
- Feedback
A Delphi survey should be reviewed in terms of reliability, validity and trustworthiness to judge its worth91.
3.1.13 Examples of KCE reports using the method
- Impact of academic detailing on primary care physicians104
- Burnout among general practitioners: prevention and management72
- Methods for including public preference values in reimbursement decision making processes for health interventions. Exploration of the feasibility of different models in Belgium (ongoing project, publication foreseen end 2012)
3.1.14 Basis references
For practical tips see the report of the King Baudouin Foundation available in French, Dutch and English93