6.3. Checklists

Laurence.Kohn Tue, 11/16/2021 - 17:41

We have found four papers (Reynolds, 2011; ^{Walsh, 2006;^{Cohen, 2008;}}Côté and Turgeon, 2005) reviewing the literature on quality criteria or guidelines for qualitative research. One of them (Walsh, 2006⁾ provides us with a synthesis of eight existing checklists and summary frameworks (see Table 7). This checklist is quite detailed and is designed in function of meta-synthesis, which is a kind of systematic review of qualitative research papers.

The list of criteria was built in order to rigorously appraise studies first before submitting them to the meta-synthesis technique. Agreement on criteria to judge rigor was necessary in order to decide which studies to include in the meta-synthesis. Walsh and Downe^{(Walsh, 2006)} tabulated the characteristics mentioned in each of the papers in their review. Then they mapped together the characteristics given in all the included papers, sorting them by the number of checklists in which they appeared. In the next step both authors independently attempted a synthesis before coming together to discuss. Redundant criteria were excluded if both authors agreed that the exclusion would not change the final judgment on the meaningfulness and applicability of a piece of qualitative research. Finally the table below was constructed, structured into three columns, namely stages, essential criteria and specific prompts. Although some criteria may seem self-evident, others are less obviously fundamental (Walsh, 2006). This list of criteria is very detailed. In some studies, especially those with short time frame, a shorter and more pragmatic hands-on list could be practical. Therefore we also added the grid of Côté and Turgeon [c] (Table 8) which is shorter, adapted to the specific context of heath care and easier to use for researchers who are less familiar with qualitative research. Other checklists are described in Appendix 1.

The use of a checklist may improve qualitative research, however they should be used critically: not every criterion is appropriate to every research context (Barbour, 2001). For example the list of Coté and Turgeon mentions interpretation of results in an innovative way as a quality criterion (point 10, Table 8), while this is not necessarily the case. Most important is a systematic approach during research process. For example the credibility of data analysis could encompass the use of software (Table 7), triangulation and/or member checking (point 7, Table 8), whereas a systematic approach with a detailed description of each step in the research process could have been sufficient.

Table 7 – Summary criteria for appraising qualitative research studies

Stages	Essential criteria	Specific prompts
Scope and purpose	Clear statement of, and rationale for, research question / aims / purposes	Clarity of focus demonstrated Explicit purpose given, such as descriptive/explanatory intent, theory building, hypothesis testing Link between research and existing knowledge demonstrated
	Study thoroughly contextualized by existing literature	Evidence of systematic approach to literature review, location of literature to contextualise the findings, or both
Design	Method/design apparent, and consistent with research intent	Rationale given for use of qualitative design Discussion of epistemological/ontological grounding Rationale explored for specific qualitative method (e.g. ethnography, grounded theory, phenomenology) Discussion of why particular method chosen is most appropriate/sensitive/relevant for research question/aims Setting appropriate
	Data collection strategy apparent and appropriate	Were data collection methods appropriate for type of data required and for specific qualitative method? Were they likely to capture the complexity/diversity of expereince and illuminate context in sufficient detail? Was triangulation of data sources used if appropriate?
Sampling strategy	Sample and sampling method appropriate	Selection criteria detailed, and description of how sampling was undertaken Justification for sampling strategy given Thickness of description likely to be achieved from sampling Any disparity between planned and actual sample explained
Analysis	Analytic approach appropriate	Approach made explicit (e.g. thematic distillation, constant comparative method, grounded theory) Was it appropriate for the qualitative method chosen? Was data managed by software package of by hand and why? Discussion of how coding systems/conceptual frameworks evolved How was context of data retained during analysis Evidence that the subjective meanings of participants were portrayed Evidence of more than one researcher involved in stages if appropriate to epistemological/theoretical stance Did research participants have any involvement in analysis (e.g. member checking) Evidence provided that data reached saturation or discussion/rationale if it did not Evidence that deviant data was sought, or discussion/rationale if it was not
Interpretation	Context described and taken account of in interpretation	Description of social/physical and interpersonal contexts of data collection Evidence that researcher spent time ‘dwelling with the data’, interrogating it for competing/alternative explanations of phenomena
	Clear audit trail given	Sufficient discussion of research processes such that others can follow ‘decision trail’
	Data used to support interpretation	Extensive use of field notes entries/verbatim interview quotes in discussion of findings Clear exposition of how interpretation led to conclusions
Reflexivity	Researcher reflexivity demonstrated	Discussion of relationship between researcher and participants during fieldwork Demonstration of researcher’s influence on stages of research process Evidence of self-awareness/insight Documentation of effects of the research on researcher Evidence of how problems/complications met were dealt with
Ethical dimensions	Demonstration of sensitivity to ethical concerns	Ethical committee approval granted Clear commitment to integrity, honesty, transparency, equality and mutual respect in relationships with participants Evidence of fair dealing with all research participants Recording of dilemmas met and how resolved in relation to ethical issues Documentation of how autonomy, consent, confidentiality, anonymity were managed
Relevance and transferability	Relevance and transferability evident	Sufficient evidence for typicality specificity to be assessed Analysis interwoven with existing theories and other relevant explanatory literature drawn from similar settings and studies Discussion of how explanatory propositions/emergent theory may fit other contexts Limitations/weaknesses of study clearly outlined Clearly resonates with other knowledge and experience Results/conclusions obviously supported by evidence Interpretation plausible and ‘makes sense’ Provides new insights and increases understanding Significance for current policy and practice outlined Assessment of value/empowerment for participants Outlines further directions for investigation Comment on whether aims/purposes of research were achieved

Source: Walsh and Downe, 2006

Table 8 – Grid for the critical appraisal of qualitative research articles in medicine and medical education

	Yes	+/-	No
Introduction
1. The issue is described clearly and corresponds to the current state of knowledge.
2. The research question and objectives are clearly stated and are relevant to qualitative research (e.g. the process of clinical or pedagogical decision-making).
Methods
3. The context of the study and the researchers’ roles are clearly described (e.g. setting in which the study takes place, bias).
4. The method is appropriate for the research question (e.g. phenomenology, grounded theory, ethnography).
5. The selection of participants is appropriate to the research question and to the method selected (e.g. key participants, deviant cases).
6. The process for collecting data is clear and relevant (e.g. interview, focus group, data saturation).
7. Data analysis is credible (e.g. triangulation, member checking).
Results
8. The main results are presented clearly.
9. The quotations make it easier to understand the results.
Discussion
10. The results are interpreted in credible and innovative ways.
11. The limitations of the study are presented (e.g. transferability).
Conclusion
12. The conclusion presents a synthesis of the study and proposes avenues for further research.

Source: Côté and Turgeon,2005

[a] Informants may be asked to read transcripts of dialogues in which they have participated to check whether their words match with what they actually intended ^{(Shenton 2004)}, or they may be asked to check the accuracy of early findings (Bloor 2006) ³⁵.

[b] Thick description refers to rich qualitative data allowing not only the description of social behaviour, but also to connect it to the broader context in which it occurred (Mortelmans 2009).

[c] A French-speaking version is also available (Côte 2002)