6.1. Usefulness of quality criteria to evaluate qualitative research

“Whatever the method, it needs to be well-defined, well-argued, and well-executed” (Snijders, 2007)

The increasing demand for qualitative research within health and health services research has emerged alongside an increasing demand for the demonstration of methodological rigor and justification of research findings (Reynolds, 2011) . Not only is qualitative research challenged by the current evidence-based practice (EPB) movement in healthcare, also the emergence of meta-analyses (e.g. meta-synthesis) of qualitative research findings urges for quality criteria. Although in quantitative health sciences research, there exist widely-recognized guidelines, no comparable standardized guidelines exist for qualitative research. This can be explained by a lack of consensus related to how to best evaluate “rigor” in qualitative research ^{(Nelson, 2008)}. Every qualitative paradigm has its own implications regarding the definition of good quality research. First, we  introduce the reader briefly in the debate about quality criteria, second, we present the framework of Walsh and Downe (Walsh, 2006) as the most complete and comprehensible list of quality criteria to appraise qualitative research studies, and the framework of Côté and Turgeon as a shorter and practical alternative. For other checklists we refer to Appendix 1.

Among qualitative researchers there is a debate going on between those demanding for explicit criteria, for example in order to serve systematic reviewing and evidence-based practice, and those who argue that such criteria are neither necessary nor desirable^{(Hammersley, 2007)}. The quest for quality criteria assumes that qualitative research is a unified field, but this image does not fit reality. In fact, apart from a variety of other positions (e.g. symbolic interactionism, hermeneutics, phenomenology, ethnography) three main paradigms can be discerned in relation to this discussion:

• The interpretativist paradigm assumes that social realities are multiple, fluid and constructed. This framework values research that illuminates subjective meanings and multiple ways of seeing a phenomenon. These researchers question the need for and the utility of quality criteria for qualitative research or apply specific criteria for qualitative research, such as clear delineation of the research process, evidence of immersion and self-reflection, demonstration of the researcher’s way of knowing (e.g. tacit knowledge)^{(Cohen, 2008)}.
• The positivist approach stands at the other end of the continuum and assumes that there is a single objective reality that is knowable. Positivists apply traditional quantitative criteria, such as validity and reliability to qualitative work.
• The realist perspective is positioned in between. It maintains a belief in an objective reality, but knowledge of reality is always imperfect^{(Cohen, 2008)}. Realists use techniques such as triangulation, member validation of findings, peer review of findings, deviant or negative case analysis and multiple coders of data, to promote to verify findings. The realist perspective adopts a philosophy of science that is in line with positivism, but at the same time embracing the complexity of social life and recognizing the importance of social meanings. “By maintaining a belief in an objective reality and positing truth as an ideal qualitative researchers should strive for, realists have succeeded at positioning the qualitative research enterprise as one that can produce research which is valid, reliable, and generalizable, and therefore, of value and import equal to quantitative biomedical research” ^{(Cohen, 2008,} p. 336).

The position one takes in the debate about quality criteria is heavily influenced by the paradigm one feels most attracted to, or identifies with.

6.2. General quality criteria

Most of the quality criteria are applicable to all research, both quantitative and qualitative. For example in 2008, Cohen and Crabtree ^{(Cohen, 2008)} reviewed and synthesized published criteria for good qualitative research. They identified the following general evaluative criteria: 1) ethical research, 2) importance of the research, 3) clarity and coherence of the research report, 4) use of appropriate and rigorous methods, 5) importance of reflexivity or attending to researcher bias, 6) importance of establishing validity or credibility, 7) Importance of verification or reliability. Researcher bias, validity, and reliability are most heavily influenced by quantitative approaches. Table 6 bridges quantitative and qualitative research by illustrating the parallels between criteria for conventional quantitative inquiries and qualitative research.

Table 6 – Lincoln and Guba’s translation of terms

Source: Adapted from Finley,2006

In what follows we pay attention to some keywords appearing in Table 6.

Reflexivity

“Reflexivity is an awareness of the self in the situation of action and of the role of the self in constructing that situation.” (Bloor and Wood, 2006, p. 145)

Because in qualitative research, the researcher could not be ‘blinded’, he/she has to take into account subjectivity in an explicit way. To demonstrate this reflexive awareness during the research process, the following ‘good practices’ can be used (Green, 2009, p. 195):

• Methodological openness: report steps taken in data production and analysis, the decisions made, and the alternatives not pursued.
• Theoretical openess: theoretical starting points and assumptions should be adressed.
• Awareness of the social setting of the research itself: be aware of the interactivity between the researcher and the researched.
• Awareness of the wider social context, including historical and policy contexts and social values.

Triangulation

“Qualitative research is inherently multimethod in focus (Flick, 2002, p.226-227). However, the use of multiple methods, or triangulation, reflects an attempt to secure an in-depth understanding of the phenomenon in question. Objective reality can never be captured. We know a thing only through its representations. Triangulation is not a tool or a strategy of validation, but an alternative to validation (Flick, 2002, p. 227). The combination of multiple methodological practices, empirical materials, perspectives, and observers in a single study is best understood, then, as a strategy that adds rigor, breadth, complexity, richness, and depth to any inquiry (See Flick, 2002, p. 229)” (Denzin and Lincoln, 2008, p. 7).

Triangulation is the use of several scientific methods, both qualitative and quantitative, to answer the same research question^{(Bloor, 2006)} . Often triangulation is understood as producing the same results by means of several methods, sources or analysts. However, different methods or types of inquiry are sensitive to different nuances, so that they may lead to somewhat different results. In fact, triangulation is more about finding inconsistencies to gain deeper insight into the relationship between the inquiry approach and the subject under study. Thus, finding inconsistencies do not weaken the credibility of the results, but rather strengthen it (Patton, 1999).

Five kinds of triangulation can contribute to the quality and consistency of qualitative data analysis:

1. Methods triangulation: Information obtained through several methods is compared. These methods can be qualitative, or quantitative or both. Often qualitative and quantitative data can be fruitfully combined as they mostly elucidate complementary aspects of the same phenomenon^{(Patton, 1999)} .
2. Triangulation of sources: Information derived at different times and by different means is compared, e.g. comparing observational data with interview data, but also comparing what people say in public with what they say in private (Patton, 1999) .
3. Analyst triangulation: Several observers, interviewers, researchers or analysts are used. By this way the potential bias that comes from a single person doing all the data collection and/or data analysis is reduced. In addition to several researchers or data analysts, analytical triangulation may also be to have those who were studied review the findings^{(Patton, 1999)} .
4. Theory/perspective triangulation:  It involves the use of different theoretical perspectives to look at the same data. Also, for example, data can be examined from the perspective of various stakeholder positions (Patton, 1999) .
5. Member validation: It is a popular kind of triangulation that consists of “checking the accuracy of early findings with research respondents” (Bloor and Wood, 2006, p. 170).

These kinds of triangulation protect the researcher against the accusation that findings are an artifact of a single method, or source or investigator’s biases (Patton, 1999).

Transferability

Earlier in this report we argued that qualitative research is context sensitive and it is not aimed at making generalizations to the wider population. This may appear to contradict with the notion of transferability which is just about the extent to which findings of one study can be applied to other situations (external validity) ^{(Merriam, 1998)}.

Transferability refers to the responsibility of the researcher to provide sufficient contextual information about the fieldwork to enable the reader to determine how far he can be confident in transferring the findings to other situations^{(Firestone, 1993)}. However, the situation might be complicated by the possibility that factors considered by the researcher to be unimportant, and consequently unaddressed in the research report, may be critical in the eyes of a reader^{(Firestone, 1993)}.

6.3. Checklists

We have found four papers (Reynolds, 2011; ^{Walsh, 2006;Cohen, 2008;} Côté and Turgeon, 2005) reviewing the literature on quality criteria or guidelines for qualitative research. One of them (Walsh, 2006⁾ provides us with a synthesis of eight existing checklists and summary frameworks (see Table 7). This checklist is quite detailed and is designed in function of meta-synthesis, which is a kind of systematic review of qualitative research papers.

The list of criteria was built in order to rigorously appraise studies first before submitting them to the meta-synthesis technique. Agreement on criteria to judge rigor was necessary in order to decide which studies to include in the meta-synthesis. Walsh and Downe^{(Walsh, 2006)} tabulated the characteristics mentioned in each of the papers in their review. Then they mapped together the characteristics given in all the included papers, sorting them by the number of checklists in which they appeared. In the next step both authors independently attempted a synthesis before coming together to discuss. Redundant criteria were excluded if both authors agreed that the exclusion would not change the final judgment on the meaningfulness and applicability of a piece of qualitative research. Finally the table below was constructed, structured into three columns, namely stages, essential criteria and specific prompts. Although some criteria may seem self-evident, others are less obviously fundamental (Walsh, 2006). This list of criteria is very detailed. In some studies, especially those with short time frame, a shorter and more pragmatic hands-on list could be practical. Therefore we also added the grid of Côté and Turgeon [c] (Table 8) which is shorter, adapted to the specific context of heath care and easier to use for researchers who are less familiar with qualitative research. Other checklists are described in Appendix 1.

The use of a checklist may improve qualitative research, however they should be used critically: not every criterion is appropriate to every research context (Barbour, 2001). For example the list of Coté and Turgeon mentions interpretation of results in an innovative way as a quality criterion (point 10, Table 8), while this is not necessarily the case. Most important is a systematic approach during research process. For example the credibility of data analysis could encompass the use of software (Table 7), triangulation and/or member checking (point 7, Table 8), whereas a systematic approach with a detailed description of each step in the research process could have been sufficient.

Table 7 – Summary criteria for appraising qualitative research studies

Source: Walsh and Downe, 2006

Table 8 – Grid for the critical appraisal of qualitative research articles in medicine and medical education

Source: Côté and Turgeon,2005

[a]           Informants may be asked to read transcripts of dialogues in which they have participated to check whether their words match with what they actually intended ^{(Shenton 2004)}, or they may be asked to check the accuracy of early findings (Bloor 2006) ³⁵.

[b]           Thick description refers to rich qualitative data allowing not only the description of social behaviour, but also to connect it to the broader context in which it occurred (Mortelmans 2009).

[c]           A French-speaking version is also available (Côte 2002)

6.4. Conclusion

To conclude this chapter on quality criteria we wish to warn against a rigid use of checklists and quality criteria in qualitative research and to argue instead for flexible use. Moreover this also applies to quantitative research.

Barbour criticizes the widespread use and description of assumed quality indicators like theoretical sampling, grounded theory, multiple coding, and triangulation in scientific articles, as an unequivocal guarantee of robustness. These dimensions of qualitative research should be embedded within a broader understanding of the qualitative research design and not “stuck on as a badge of merit” (Barbour, 2001, p. 1115).

We agree with Walsh and Downe (Walsh, 2006) that a checklist is indicative of good quality research, but not a guarantee.

Key messages

• Although in quantitative health sciences research, there exist widely-recognised guidelines, no comparable standardised guidelines exist for qualitative research.
• Among qualitative researchers there is a debate going on between those demanding for explicit criteria, for example in order to serve systematic reviewing and Evidence-Based Practice, and those who argue that such criteria are neither necessary nor desirable.
• The framework of Walsh and Downe as an comprehensible example of quality criteria checklist to appraise qualitative research studies. The grid of Côté and Turgeon is more simple and could be recommended as tool for evaluation in KCE reports.