At the outset, the nature of RM was unclear. A similar problem faced the researchers evaluating total purchasing. Sites were charged with finding ways of involving doctors in management, but how this would be achieved and, if achieved, how successful it would be in improving patient care, were open questions. The researchers selected major specialties within each site, and conducted interviews with relevant staff, observed meetings and analysed documentation (see Box 1). Over time, the data were used to develop a framework that captured the essential features of RM, and the framework was then used to evaluate each site’s progress with implementation (see Box 2). A more recent example of this approach is provided by a study of the impact of business process re-engineering (BPR) in a teaching hospital.10 The study had to follow and adapt to the dynamics of the BPR programme, as it changed over the period of the case study in response to experience and external influences.
The process of selecting sites for study is central to the case study approach. Researchers have developed a number of selection strategies, the objectives of which, as in any good research study, are to ensure that misinterpretation of results is avoided as far as possible. Because resources for case study work are usually limited, it is necessary to design a "purposive" sample that is typical of the phenomenon being investigated, where a specific theory can be tested, or where cases will confirm or refute a hypothesis.19 Researchers can benefit from expert advice from those with knowledge of the subject being investigated, and can usefully build the possibility of testing findings at further sites into the initial research design. Replication of results across sites helps to ensure that findings are not due to idiosyncratic features of particular sites.
The next step is to select data collection methods, ensuring that the process is driven by criteria of validity and reliability.20 A distinctive, though not unique, feature of case study research is the use of multiple methods and sources of evidence to establish construct validity.21 The total purchasing study provides an example (see Box 3). The use of particular methods is discussed in other chapters of the book. Here it is noted that case studies often use "triangulation"22 (see Chapter 9) to maximise confidence in the validity of findings. In triangulation, all data items are corroborated from at least one other source and normally also via another method of data collection. Any one method can, arguably, produce results of weaker validity than a combination. Using different methods and sources helps to address this problem, and can strengthen researchers’ beliefs in the validity of their observations. This said, reservations have been expressed about the use of triangulation in qualitative research,21 given that different methods and sources of data will tend to provide different sorts of insights rather than contribute to a single, accumulating picture. Some studies have, by contrast, used comparison of case study data with that from a larger sample to explore how far findings might be strengthened and generalised. The evaluation of total purchasing16 employed this strategy, using survey as well as case study site data. The key point here is that the case study approach, properly used, can be viewed as a strategy for combining data and methods systematically in order to validate findings.
Another example of the central concern of case studies with establishing validity is provided in a study of purchaser-provider contracting for community health services.23 The purpose of the study was to gather information about the processes involved in contracting in the NHS internal market. A small number of sites were chosen, the selection of sites being based in part on different organisational structures and approaches to purchasing to allow for broad comparison of experiences. Observations of meetings were recorded, verbatim notes were made of discussions in situ, and documents were obtained about all stages of the contracting process. Interviews were held with patients and users of health services, and an interview survey of GPs was conducted. The data obtained using each method were analysed separately and then compared with one another across different methods and study sites.
Whatever the details, the collection of data should be directed towards the development of an analytical framework that will facilitate interpretation of findings. Again, there are several ways in which this might be done. Sometimes data are collected in order to test specific hypotheses. In the evaluation of resource management, in contrast, there was no obvious pre-existing theory that could be applied: the development of a framework during the study was crucial to help organise data and evaluate findings. The framework was not imposed on the data, but derived from it in an iterative process over the course of the evaluation; each was used to refine the other over time. Possibilities for testing emerging findings can be built into case study design - for example, by feedback in report or workshop form to participants or knowledgeable informants, or by using further small-scale case studies.
The investigator is finally left with the difficult task of making a judgement about the findings of a study and determining its wider implications. The purpose of the steps followed in designing and building the case study is to maximise confidence in the findings, but interpretation inevitably involves value judgements and the danger of bias. The extent to which research findings can be assembled into a single coherent account of events varies; individual cases may exhibit common characteristics or fare very differently. In some circumstances widely differing opinions of participants are themselves very important, and should be reflected in any report. The case study approach enables the researcher to gauge confidence in both the internal and external validity of the findings, and make comments with the appropriate assurance or with reservations.
The complexity of the issues that health professionals have to address and increasing recognition by policy makers, academics and practitioners of the value of case studies in evaluating health service interventions, suggest that the use of such studies is likely to increase in the future. Their most important use may be by regulators, possibly including bodies in the NHS such as the Commission for Health Improvement,1 so they will become part of the machinery whereby doctors and other clinicians are held to account for their work. In policy research, qualitative methods can be used within case study design to address many practical and policy questions that impinge on the lives of clinicians, particularly where those questions are concerned with how or why events or initiatives take a particular course.
Yin R. Case study research: design and methods. Second edition. Newbury Park, CA: Sage, 1994.
1 Secretary of State for Health. The new NHS: modern, dependable. Cm 3807. London: Stationery Office, 1997.
2 Klein R. ed. Implementing the White Paper: pitfalls and opportunities. London: King’s Fund, 1998.
3 Wistow G, Knapp M, Hardy B et al. Social care markets: progress and prospects. Buckingham: Open University Press, 1996.
4 National Audit Office. NHS executive: the purchase of the Read codes and the management of the Centre for Coding and Classification. HC 607, Session 1997-98. London: Stationery Office, 1998.
5 National Audit Office. Cost over-runs, funding problems and delays on Guy’s Hospital phase III development. HC 761, Session 1997-98. London: Stationery Office, 1998.
6 Audit Commission. Higher purchase: commissioning specialised services in the NHS. London: Stationery Office, 1997.
7 Gray A. Budgeting, auditing and evaluation: functions and integration in seven governments. New Brunswick, NJ: Transaction, 1993.
8 Pollitt C, Girre X, Lonsdale J, Mul R, Summa H, Waerness M. Performance or compliance? Performance audit and public management in five countries. Oxford: Oxford University Press, 1999.
9 Pollitt C, Harrison S, Hunter D, Marnoch G. The reluctant managers: clinicians and budgets in the NHS. Financial Accountability and Management 1988;4:213-33.
10 Packwood T, Pollitt C, Roberts S. Good medicine: a case study of business process re-engineering in a hospital. Policy and Politics, 1998;26:401-15.
11 Laughlin R. Empirical research in accounting: alternative approaches and a case for ‘middle range’ thinking. Accounting, Auditing and Accountability Journal 1997:10; 622-48.
12 Laughlin R, Broadbent J, Willig-Atherton H. Recent financial and administrative changes in GP practices: initial experiences and effects. Accounting, Auditing and Accountability Journal 1994;7:96-124.
13 Berwick D, Roessner A, Godfrey J. Curing health care. San Francisco, CA: Jossey Bass:191.
14 St Leger A, Schneider H, Walsworth-Bell J. Evaluating health services effectiveness. Milton Keynes: Open University Press, 1992.
15 Kogan M, Redfern S. Making use of clinical audit. Buckingham: Open University Press, 1995.
16 Mays N, Goodwin N, Killoran A, Malbon G. Total purchasing: a step towards primary care groups. London: King’s Fund, 1998.
17 Pollitt C, Harrison S, Hunter D, Marnoch G. No hiding place: on the discomforts of researching the contemporary policy process. Journal of Social Policy 1990;19:169-90.
18 Packwood T, Keen J, Buxton M. Hospitals in transition: the resource management experiment. Milton Keynes: Open University Press, 1991.
19 Patton M. Qualitative evaluation and research methods. Second edition. Newbury Park, CA: Sage, 1990.
20 Yin R. Case study research: design and methods. Second edition. Newbury Park, CA: Sage, 1994.
21 Silverman D. Interpreting qualitative research. London: Sage, 1993.
22 Jick T. Mixing qualitative and quantitative methods: triangulation in action. Administrative Sciences Quarterly 1979;24:602-11.
23 Flynn R, Williams G, Pickard S. Markets and networks: contracting in community health services. Buckingham: Open University Press, 1996.
Box 1 - Evaluation of the resource management initiative18
• Six hospitals, a mix of teaching and non-teaching
• Focus on major specialties: general surgery and general medicine
• Mix of qualitative and quantitative methods
• Methods and data sources independent of each other
• Qualitative methods included interviews, non-participant observation of meetings and analysis of documents
Box 2 - Analytical framework: five inter-related elements of resource management18
• Commitment to resource management by the relevant personnel at each level in the organisation
• Devolution of authority for the management of resources
• Collaboration within and between disciplines in securing the objectives of resource management
• Management infrastructure, particularly in terms of organisation structure and provision of information
• A clear focus for the resource management strategy
Box 3 - Methods used in the national evaluation of total purchasing pilots16
• Face-to-face interviews with key players
• Diary cards to samples of key players
• Postal questionnaires
• Telephone interviews
• Documentary analysis
• Quantitative data analysis (including NHS routine data)
7 Qualitative Methods in Health-Related Action Research
The barriers to the uptake of the findings of traditional quantitative biomedical research in clinical practice are increasingly being recognised.1,2 Certain forms of qualitative research may make it easier for research to influence day-to-day practice. The style of research known as action research is particularly suited to identifying problems in clinical practice and helping to develop potential solutions in order to improve practice.3 For this reason, action research is increasingly being employed in health-related settings. Although not synonymous with qualitative research, action research usually draws on the types of qualitative methods described in Chapters 2, 3 and 4 of this book. Action research projects are frequently written up as case studies, but this approach to research is distinct from the types of case study discussed in Chapter 6.
What is action research?
Like qualitative research in general, action research is not easily defined. It is a style of research rather than a specific method. First used in 1946 by Kurt Lewin, a social scientist concerned with inter-group relations and minority problems in the USA, it is now identified with research in which the researchers work explicitly with and for people rather than undertake research on them.4 Action research appears to be gaining credibility in a variety of practice-based disciplines, particularly within the health care professions. Its strength lies in its focus on generating solutions to practical problems and its ability to empower practitioners - getting them to engage with research and subsequent "development" or implementation activities. Practitioners can be involved when they choose to research their own practice5 or when an outside researcher is engaged to help them to identify problems, seek and implement potential solutions, and systematically monitor and reflect on the process and outcomes of change.6,7
The term "action research" tends to be used loosely. However, most definitions incorporate three important elements, namely its participatory character, its democratic impulse, and its simultaneous contribution both to social science and social change.8
Participation in action research
Participation is fundamental to action research; it is an approach which demands that participants perceive the need to change and are willing to play an active role both in the research and change process. Whilst all research requires willing subjects, the level of commitment required of those involved in an action research study goes beyond simply agreeing to answer questions or to be observed. The clear-cut demarcation between "researcher" and "researched" found in other types of research may not be apparent in action research. The research design or strategy is negotiated with participants in a continuous process in an action research study, making obtaining informed consent at the outset problematic. Action researchers, therefore, need to agree an ethical code of practice with the participants.9 Participation in the research and in the process of change can be threatening, and conflicts may arise in the course of the research, as a number of studies have shown.10,11 For example, in one action research study6 the process of asking for suggestions for improvements and honestly feeding them back to participants had a profound effect on the dynamics of a multi-disciplinary team. Not all participants in the study felt comfortable questioning their own practice and examining their own roles and responsibilities. Indeed, the nurse in charge found it particularly threatening and this resulted in her seeking employment elsewhere.9 Where an outside researcher is working with practitioners it is important to obtain their trust and agree rules on the control of data and their use, and on how potential conflict will be resolved within the project. The way in which such rules are agreed demonstrates a second important feature of action research, namely its democratic impulse.
Democracy in action research
Action research is concerned with intervening to change and improve practice whether in health, education or any other area of life.12 It engages participants in the struggle for more rational, just, democratic and fulfilling forms of "service". As such, it can be seen as a form of "critical" social science.13 This philosophical underpinning is a key difference between action research and more traditional case study approaches. "Democracy" in action research usually requires participants to be seen as equals of the researcher. The researcher works as a facilitator of change, consulting with participants not only on the action process, but also on how it will be evaluated. One benefit of designing a study in conjunction with practitioners is that it can make the research process and outcomes more meaningful to practitioners, by rooting them in the reality of day-to-day practice.
Throughout the study, findings are fed back to participants for validation and to inform decisions about the next stage of the study. This formative style of research is thus responsive to events as they naturally occur in the field and frequently entails collaborative spirals of planning, acting, observing, reflecting and re-planning (see Figure 1).14 These action-reflection spirals are characteristic of any action research study. McNiff 14 visually depicts action research as a three dimensional tree of spirals, allowing for smaller "spin-off spirals" to branch out from larger spirals of activity. This illustrates how practitioners engaging in action research can address many different problems at one time without losing sight of the main issue. It also captures the frequently seemingly chaotic nature of change in organisations and practice settings.
However, as has already been noted, the open feedback of findings to participants can be, and often is, very threatening. Democratic practice is not always a feature of health care settings. Care needs to be taken in undertaking democratic action research in such settings. An action researcher needs to be able to work across traditional boundaries (for example, between professionals, health and social care, and between hospital and community care) and juggle different, sometimes competing agendas. Excellent interpersonal skills, in addition to research ability, are clearly of paramount importance in action research.
Contribution to both social science and social change
The focus on practice and on change, and the greater level of involvement by participants in the research process, opens action research to the challenge of being "unscientific". This is compounded by the fact that action research in health care settings has tended to utilise a range of qualitative methods rather than the quantitative methods associated with clinical, biological and epidemiological research. Yet, there is increasing concern about the theory-practice gap in clinical practice in which practitioners have to rely on their intuition and experience since traditional scientific knowledge, for example, the results of randomised controlled trials, frequently does not appear to fit the uniqueness of their situation. (Clinicians will recognise this as the exercise of professional judgement.) Action research is seen as one way of dealing with this limitation of evidence-based practice by developing a different kind of knowledge more appropriate to day-to-day clinical settings.4,8
The level of interest in practitioner-led research is increasing in the UK, in part as a response to recent proposals to "modernise" the National Health Service through the development of new forms of clinical governance.15 Clinical governance and other national initiatives in the UK (for example, the NHS Research and Development Strategy, the National Centre for Clinical Audit, the NHS Centre for Reviews and Dissemination, the Cochrane Collaboration, and Centres for Evidence Based Practice) emphasise that research and development should be the business of every clinician. It is argued16 that practitioner-led research approaches, such as single case experimental designs,17 reflective case studies18 and reflexive action research,19 are ideal research methods for clinicians concerned with improving the quality of patient care. In addition, these approaches are likely to generate findings that are more meaningful and useful to practitioners, thus reducing the theory-practice gap.
In considering the contribution of action research to knowledge, it is important to note that generalisations made from action research studies differ from those made on the basis of more conventional forms of research. To some extent, reports of action research studies rely on the reader to underwrite the account of the research by drawing on their own knowledge of human situations. It is, therefore, important, when reporting action research, to describe the work in its rich contextual detail. The researcher strives to include the participants’ perspective on the data by feeding back findings to participants and incorporating their responses as new data in the final report. In addition, the onus is on the researcher to make his/her own values and beliefs explicit in the account of the research so that any biases are evident. This can be facilitated by writing self-reflective field notes when undertaking the research. According to Carr and Kemmis,8 a successful report can be characterised by "the shock of recognition" - the quality of the account enables readers to assess its relevance to themselves and their own practice situations. The systematic feeding back of findings throughout an action research study, makes it possible to check the accuracy of the account with participants. However, interpreting the relevance of the findings to any other practice situation ultimately rests with the reader (see Chapter 9 for more on issues of validity and relevance in qualitative research).
The strength of action research lies in its ability to influence practice positively in the course of the study, whilst, at the same time, systematically gathering data to share with a wider audience. The involvement of practitioners in this process ensures not only more likelihood of discovering successful solutions to everyday problems, but also of obtaining a different type of data that is, arguably, more relevant and meaningful to practitioners.
However, change is problematic and, whilst action research lends itself well to the discovery of solutions, its success should not be judged solely in terms of the size of change achieved or the immediate implementation of solutions. Instead, success can often be viewed in relation to what has been learnt from the experience of undertaking the work. For instance, a study that set out to explore the care of older people in A and E departments20 did not result in much change in the course of the study. However, the lessons learnt from the research were reviewed in the context of national policy and research and carefully fed back to those working in the organisation and, as a result, changes have already been made within the organisation to act on the study’s recommendations. For instance, some positive changes were achieved in the course of the study (for example, the introduction of specialist discharge posts in A and E), but the study also shed light on continuing gaps in care and issues that needed to be improved in future developments. Participants identified that the role of the action researcher had enabled greater understanding and communication between two services (the A and E Department and the Department of Medicine for Elderly People), and that this had left both better equipped for future joint working. In other words, the solutions emerged from the process of undertaking the research.
Different types of action research
Under the same broad heading, there are many different types of action research. Hart and Bond3 suggest that there are some key characteristics, which not only distinguish action research from other methodologies, but which also determine the range of approaches to action research. They present a typology of action research identifying four basic types: experimental, organisational, professionalising and empowering (see Box 1). They suggest that each type embodies a different theoretical perspective on society.
Whilst this typology is useful in understanding the wide range of action research, its multi-dimensional nature means that it is not particularly easy to classify individual studies. For instance, whilst a study might be classified as "empowering" because of its "bottom-up approach" in relation to the fourth distinguishing criteria of "change intervention" (see Box 2), the other distinguishing criteria may be used to classify the same study as a different action research type (experimental, organisational or professionalising). This situation is most likely to occur if the researcher and practitioners hold differing views on the nature of society. This makes classification of single studies into any one type of action research problematic. Instead, it may be more fruitful to use this typology as a framework for critiquing individual studies and, in particular for thinking about how concepts are operationalised, the features of particular settings, and the contribution of the people within those settings to solutions.21 It is worth noting that, over time, health-related action research appears to have moved away from "experimental" to more "empowering" models of research.9 However, empowering models have to be used with care.22 Health service organisations are frequently hierarchical rather than democratic. As a result, changes based on notions of practitioner empowerment can frequently be frustrated. Somekh23 reiterates this point, arguing that different occupational cultures can affect action research methodology. For this reason, she suggests that action research should be grounded in the values and discourse of the individual or group rather than rigidly adhering to a particular methodological perspective.
Action research in health care
At a time when there is increasing concern that research evidence is not sufficiently influencing practice development,24 it is not surprising that action research is gaining credibility in health care settings.25 For example, the Royal College of Physicians in England has become involved in an action research study, commissioned by the NHS Executive, which seeks to improve the practice of clinical audit. A central focus of this study is to explore the roles of clinicians, clinical audit staff and managers in implementing clinical audit and in determining how organisational problems can be overcome.26 The interest in action research within health care settings can also be demonstrated by the recent commissioning of a methodological systematic review of the action research literature, as part of the NHS Research and Development Programme. The intention behind this review is to provide guidance for funding agencies, policy makers and researchers on the criteria to use to judge the appropriateness of action research proposals and reports.
Since action research seeks to develop a process of change in practice contexts and emphasises the role of the practitioner as researcher, it allows research to become part of the clinician’s everyday work. In many ways, its use is ideally suited to health services research and its popularity is likely to be sustained.
Ong27 advocates the value of action research within health care settings on the basis of recent changes in the requirements of health care management and policy. She highlights the need for new, systematic approaches to encouraging user participation in health services. She suggests that "Rapid Appraisal" is an ideal method for engaging users in the development of health care policy and practice.28 Rapid Appraisal is a type of action research, hitherto predominantly used in developing countries, which focuses on participatory methods to foster change, using ideas derived from the field of community development. Her book gives excellent detail not only on the philosophical and theoretical underpinnings of Rapid Appraisal, but also on how such a study might be conducted.28
Action research has also been used in hospital rather than wider community settings to facilitate closer partnerships between staff and users. In a study that focused on the introduction of lay participation in care within a general medical ward of a London teaching hospital, the action researcher worked for one year in a multi-disciplinary team (see Box 2).6 In the course of the study, it emerged that in order to foster closer partnerships with users and carers, professionals needed to change their practice to work more collaboratively with one another. As a result, three main action-reflection spirals emerged in the project: reorganising the work of the ward, multi-disciplinary communication, and lay participation in care. Each action-reflection spiral generated related activities, otherwise known as spin-off spirals. For instance, stemming from the main lay participation in care action-reflection spiral, a spin-off spiral focused on the medical staff teaching patients more about their treatments.
A range of research methods was used, including depth interviews (see Chapter 2), questionnaires, documentary analysis and participant observation (see Chapter 4). Throughout the study, preliminary findings were fed back to participants through weekly team meetings to help guide the project. Whilst positive change was demonstrated over time, the analysis generated two main data sets on the health professionals’ perceptions of lay participation in care and the difficulties encountered in changing practice.29,30
The value of using qualitative methods and an action research approach can best be demonstrated in relation to the data on the health professionals’ perceptions of lay participation in care. Qualitative methods were used alongside quantitative methods such as attitudinal scales and self-administered questionnaires as part of a process of triangulation (see Chapter 9). As suggested in Chapter 1, qualitative methods can be useful in reinterpreting the findings from more quantitative methods. In this study, health professionals expressed extremely positive views about user and carer involvement when completing an attitude scale.31 However, examination of their attitudes in interviews suggested that they had some serious doubts and concerns. Subsequent observation of their practice revealed that these doubts and concerns were inhibiting the implementation of lay participation. Previous research on health professionals’ attitudes towards user and carer involvement had tended to rely solely on structured instruments and had found that health professionals hold generally positive attitudes towards it.31-34 By contrast, using mixed methods, it was possible to explore the relationship between attitudes and practices and to explain what happened when lay participation was introduced into a practice setting. Findings suggested that, whilst current policy documents advocate lay participation in care, some health professionals were merely paying lip service to the concept and were also inadequately prepared to deliver it in practice. In addition, findings indicated that health professionals needed to learn to collaborate more closely with each other, by developing a common understanding and approach to patient care, in order to offer closer partnerships with users and carers.
Whilst one cannot generalise from a single case study, this research led to serious questioning of the value of previous quantitative research, which had suggested that health professionals hold positive attitudes towards lay participation in care. By using action research and working closely with practitioners to explore issues in a practical context, more insight was gained into how the rhetoric of policy might be better translated into reality.
Action research does not focus exclusively on user and carer involvement, though clearly its participatory principles make it an obvious choice to explore these issues. It can be used more widely, for example, to foster better practice across inter-professional boundaries and between different health care settings.20,35 Action research can also be used by clinicians to research their own practice.16 It is an eclectic approach to research, which draws on a variety of data collection methods. However, its focus on the process as well as the outcomes of change helps to explain the frequent use of qualitative methods by action researchers.
Hart E, Bond M. Action research for health and social care. A guide to practice. Buckingham: Open University Press, 1995.
Susman GI, Evered RD. An assessment of the scientific merits of action research. Administrative Science Quarterly 1978;23:582-603.
1 Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practise and teach EBM. Edinburgh: Churchill Livingstone, 1997.
2 Hicks C, Hennessy D. Mixed messages in nursing research: their contribution to the persisting hiatus between evidence and practice. Journal of Advanced Nursing 1997;25:595-601.
3 Hart E, Bond M. Action research for health and social care: a guide to practice. Buckingham: Open University Press, 1995.
4 Reason P, Rowan J. Human inquiry: a sourcebook of new paradigm research. Chichester: John Wiley and Sons, 1981.
5 Childs V, Franklin F and Kemp P. Action research in social services and health care settings. Cambridge: Anglia Polytechnic University, 1997.
6 Meyer JE. Lay participation in care in a hospital setting: an action research study. London: University of London, unpublished PhD thesis, 1995.
7 Titchen, A. Changing nursing practice through action research. Oxford: National Institute for Nursing, 1993.
8 Carr W, Kemmis S. Becoming critical: education, knowledge and action research. London: Falmer Press, 1986.
9 Meyer JE. New paradigm research in practice: the trials and tribulations of action research. Journal of Advanced Nursing, 1993;18:1066-72.
10 Webb C. Action research: philosophy, method and personal experiences. Journal of Advanced Nursing 1989;14:403-10.
11 Titchen A, Binnie A. Changing power relationships between nurses: a case study of early changes towards patient-centred nursing. Journal of Clinical Nursing 1993;2:219-29.
12 Elliott J. Action research for educational change: developing teachers and teaching. Milton Keynes: Open University Press, 1991.
13 Habermas J. Knowledge and human interests. London: Heinemann, 1972.
14 McNiff J. Action research: principles and practice. London: Macmillan Education Ltd, 1988.
15 Secretary of State for Health. The new NHS: modern, dependable. Cm 3807. London: The Stationery Office, 1997.
16 Rolfe G. Expanding nursing knowledge: understanding and researching your own practice. Oxford: Butterworth Heineman, 1998.
17 Carey LM, Matyas TA, Oke LE. Sensory loss in stroke patients: effective training of tactile and proprioceptive discrimination. Archives of Physical Medicine and Rehabilitation 1993;74:602-11.
18 Stark S. A nurse tutor’s experience of personal and professional growth through action research. Journal of Advanced Nursing 1994;19(3):579-84.
19 Titchen A, Binnie A. What am I meant to be doing? Putting practice into theory and back again in new nursing roles. Journal of Advanced Nursing 1993;18:1054-65.
20 Meyer J, Bridges J. An action research study into the organisation of care of older people in the accident and emergency department. London: City University, 1998.
21 Lyon J. Applying Hart and Bond’s typology; implementing clinical supervision in an acute setting. Nurse Researcher 1999;6:39-53.
22 Meyer JE. Action research in health-care practice: nature, present concerns and future possibilities. NT Research 1997;2:175-84.
23 Somekh B. Inhabiting each other’s castles: towards knowledge and mutual growth though collaboration. Educational Action Research Journal 1994;2(3):357-81.
24 Walshe K, Ham C, Appleby J. Given in evidence. Health Service Journal 1995;105:28-9.
25 East L, Robinson J. Change in process: bringing about change in health care through action research. Journal of Clinical Nursing 1994;3:57-61.
26 Berger A. Why doesn’t audit work? Br Med J 1998;316:875-6.
27 Ong BN. The practice of health services research. London: Chapman & Hall, 1993:65-82.
28 Ong BN. Rapid appraisal and health policy. London: Chapman & Hall, 1996.
29 Meyer JE. Lay participation in care: a challenge for multi-disciplinary teamwork. Journal of Interprofessional Care 1993;7:57-66.
30 Meyer JE. Lay participation in care: threat to the status quo. In: Wilson-Barnett J, Macleod Clark J. eds. Research in health promotion and nursing. London: Macmillan, 1993:86-100.
31 Brooking J. Patient and family participation in nursing care: the development of a nursing process measuring scale. London: University of London, unpublished PhD thesis, 1986.
32 Pankratz L, Pankratz D. Nursing autonomy and patients’ rights: development of a nursing attitude scale. Journal of Health and Social Behavior 1974;15:211-16.
33 Citron MJ. Attitudes of nurses regarding the patients’ role in the decision-making process and their implications for nursing education. Dissertation Abstracts International 1978;38:584.
34 Linn LS, Lewis CE. Attitudes towards self-care amongst practising physicians. Medical Care 1979;17:183-90.
35 Street A, Robinson A. Advanced clinical roles: investigating dilemmas and changing practice through action research. Journal of Clinical Nursing 1995;4:343-57.
Box 1 - Action research typology (reproduced with permission from Hart and Bond, 1995)3
Action research type: Distinguishing criteria
Consensus model of society
Rational social management
Conflict model of society
1 Educative base
Enhancing social science/administrative control and social change towards consensus
Enhancing managerial control and organisational change towards consensus
Enhancing professional control and individual’s ability to control work situation
Enhancing user-control and shifting balance of power; structural change towards pluralism
Inferring relationshipbetween behaviour and output; identifying causal factors in group dynamics
Overcoming resistance to change/restructuring balance of power between managers and workers
Empowering professional groups; advocacy on behalf of patients/clients
Empowering oppressed groups
Social scientific bias/researcher focused
Managerial bias/client focused
2 Individuals in groups
Closed group, controlled, selection made by researcher for purposes of measurement/inferring relationshipbetween cause and effect
Work groups and/or mixed groups of managers and workers
Professional(s) and/or (interdisciplinary) professional group/negotiated team boundaries
Fluid groupings, self selecting or natural boundary or open/closed by negotiation
3 Problem focus
Problem emerges from the interaction of social science theory and social problems
Problem defined by most powerful group; some negotiation with workers
Problem defined by professional group; some negotiation with users
Emerging and negotiated definition of problem by less powerful group(s)
Problem relevant for social science/management interests
Problem relevant for management/social science interests
Problem emerges from professional practice/experience
Problem emerges from members’ practice/experience
Success defined in terms of social sciences
Success defined by sponsors
Contested, professionally determined definitions of success
Competing definitions of success accepted and expected
4 Change intervention
Social science, experimental intervention to test theory and/or generate theory
Top-down, directed change towards predetermined aims
Professionally led, predefined, process-led
Bottom-up, undetermined, process-led
Problem to be solved in terms of research aims
Problem to be solved in terms of management aims
Problem to be resolved in the interests of research-based practice and professionalisation
Problem to be explored as part of process of change, developing an understanding of meaning of issues in terms of problem and solution
5 Improvement and involvement
Towards controlled outcome and consensual definition of improvement
Towards tangible outcome and consensual definition of improvement
Towards improvement in practice defined by professionals and on behalf of users
Towards negotiated outcomes and pluralist definitions of improvement: account taken of vested interests
6 Cyclic processes
Research components dominant
Action and research components in tension; action dominated
Research and action components in tension; research dominated
Action components dominant
Identifies causal processes that can be generalised
Identifies causal processes that are specific to problem context and/or can be generalised
Identifies causal processes that are specific to problem and/or can be generalised
Changes course of events; recognition of multiple influences upon change
Time limited, task focused
Discrete cycle, rationalist, sequential
Spiral of cycles, opportunistic, dynamic
Open-ended, process driven
7 Research relationship, degree of collaboration
Practitioner or researcher/collaborators
Practitioner researcher/coresearchers/co-change agents
Outside researcher as expert/research funding
Client pays an outsideconsultant - "they who pay the piper call the tune"
Outside resources and/or internally generated
Outside resources and/or internally generated
Figure 1 - Action-reflection spirals
Box 2 - Lay participation in care in a hospital setting: an action research study
Careful negotiation to recruit willing volunteers to examine practice and initiate lay participation in care
"Bottom-up" approach to change via weekly team meetings
Researcher as facilitator and multi-disciplinary team member
Goal of empowering practitioners and lay people in this setting
Working collaboratively with multi-disciplinary team
Participants given "ownership" of the data to determine how it might be shared with wider audience
Contribution to social science and social change
Findings constantly fed back to practitioners, leading to changes (such as, improvements in inter-professional working)
Dissemination of findings of local and national relevance
Case study of multi-disciplinary team on one general medical ward in London teaching hospital using:
• qualitative methods to highlight key themes emerging in the project
• quantitative methods for comparison of sub groups
Main action-reflection spirals
Reorganising the work of the ward
• Changes in patient care planning
• New reporting system, including bed-side handover with patient
• Introduction of modified form of primary nursing system
• Weekly team meetings instituted
• Introduction of a handout for new staff and team communication sheet
• Closer liaison with community nurses before discharge
Lay participation in care
• Development of resources for patient health education
• Introduction of medicine reminder card system
• Patient information leaflet inviting patients to participate in care
Insights into health professionals’ perceptions of lay participation in care
Some positive changes achieved (e.g. improved attitudes to lay participation in care, patient education, improved ward organisation)
Identified barriers to changing health care practice
8 Analysing Qualitative Data
CATHERINE POPE, SUE ZIEBLAND, NICHOLAS MAYS
The nature of qualitative data
There is a widely held perception that qualitative research is small scale. As it tends to involve smaller numbers of subjects or settings than quantitative research it is assumed, incorrectly, that it generates fewer data than quantitative research. In fact, qualitative research can produce vast amounts of data.
As Chapters 2 , 3 and 4 have suggested, a range of different types of data may be collected during a qualitative study. These may include jotted notes, full field notes, interview and focus group transcripts and documentary material, as well as the researcher’s own records of ongoing analytical ideas, research questions and the field diary, which provides a chronology of the events witnessed, and the progress of the research. These data are not necessarily small scale: transcribing a typical single qualitative interview generates a considerable amount of raw data - anything between 20 and 40 single-spaced pages of text.
Verbatim notes or audio/video tapes of face-to-face interviews or focus groups are transcribed to provide a record of what was said. The preparation of transcribed material will depend on the level of analysis being undertaken, but even if only sections of the data are intended for analysis, the preservation of the original recording tapes or documents is recommended. Transcribing is time consuming. Each hour of material can take six or seven hours to transcribe depending on the quality of the tape and the depth of information required. Conversational analysis of audio-taped material requires even more detailed annotation of a wide range of features of the talk studied, such as the exact length of pauses and the different types of emphasis in the spoken word. There are conventions for annotating transcripts for this purpose.1 Even when the research is not concerned with analysing talk in this depth it is still important that the data provide an accurate record of what was said and done. The contribution of sighs, laughs and lengthy pauses should not be underestimated when analysing talk, and, as a minimum, these should be noted in the transcription.
Field notes of observational research contain detailed, highly descriptive accounts of several hours spent watching and listening, and often taking part in, events, interactions and conversations. This unprocessed experience needs to be transformed into notes, and from there into data that can be analysed. The jotted notes made in the field during an observational study need to be written up in full.
Whether using interviews or observation, the maintenance of meticulous records is vital - these are the raw data of the research. National qualitative data archives in Britain2 have made secondary analysis of qualitative data possible, and mean that it is even more important that full records of qualitative studies are kept to allow the possibility of further analysis in the future.
The relationship between analysis and the data collected
Transcripts of interviews and field notes of observations provide a descriptive record, but they cannot provide explanations. The researcher has to make sense of the data by sifting and interpreting them. In much qualitative research the analytical process begins during the data collection phase as the data already gathered are analysed and fed into, or shape, the ongoing data collection. This is referred to as sequential analysis3 or interim analysis4 (see Figure 1). It allows the researcher to check and interpret the data she/he is collecting continually and to develop tentative conclusions based on the data already collected, or hypotheses for subsequent investigation in further data collection. Compared with quantitative methods, this has the advantage of allowing the researcher to go back and refine questions and to pursue emerging avenues of inquiry in further depth. Crucially, it also enables the researcher to look for deviant or negative cases; that is, examples of talk or events that run counter to the emerging propositions or hypotheses, in order to refine them. This type of continuous analysis is almost inevitable in qualitative research; because the researcher is "in the field" collecting the data, it is impossible not to start thinking about what is being heard and seen.
Although some of the initial analysis can be done whilst the data are being collected, as indicated above, there is still much to do once the researcher has left the field. Textual data, whether in the form of observational field notes or interview transcripts, are explored using some variant of content analysis. The most straightforward type of content analysis is quantitative. This uses an unambiguous, predefined coding system and produces counts or frequencies that may be tabulated and analysed using standard statistical techniques. This approach is often used in media and mass communications studies. In general, qualitative research does not seek to quantify data, although simple counts can be useful in qualitative studies. One useful example of this approach is Silverman’s research on communication in clinics.5 This quantified features such as consultation length and the patient’s use of questions and combined this information with the qualitative analysis to confirm a series of propositions about the differences between private and NHS clinics. This type of analysis that counts items in the data is distinct from qualitative analyses in which the data are preserved in their textual form and indexed in order to generate and/or develop analytical categories and theoretical explanations.
Qualitative research uses analytic categories to describe and explain social phenomena. These categories may be derived inductively, that is obtained gradually from the data, or used deductively, either at the beginning or part way through the analysis as a way of approaching the data. Though less commonly associated with qualitative research, more deductive forms of analysis are increasingly being used in applied qualitative research; one example of this is the SCPR’s framework approach6 (see Applied qualitative research).
Glaser and Strauss7 coined the term grounded theory to describe the inductive process of coding incidents in the data and identifying analytical categories as they "emerge from" the data (developing hypotheses from the "ground" or research field upwards rather defining them a priori). This process involves identifying a theme and attempting to verify, confirm and qualify it by searching through the data. Once all data that match that theme have been located, the researcher repeats the process to identify further themes or categories.
The first stage in this process involves annotating or marking up themes in the field notes or interview transcripts. This is sometimes referred to as "coding", although it does not involve assigning numerical codes in the quantitative sense (where exclusive variables are defined and given preset codes or values). To avoid confusion the term "indexing" may be preferred.
Indexing qualitative data is a lengthy and sometimes tedious process. It requires reading and re-reading the material collected to identify themes and categories - these may centre on particular phrases, incidents or types of behaviour. Sometimes interesting or unfamiliar terms used by the group studied can form the basis of analytical categories. Becker and Geer’s classic study of medical school training uncovered the specialised use of the term "crock" to denote patients who were seen as less worthwhile to treat by medical staff and students.8
All the data relevant to each category are identified and examined using a process called constant comparison, in which each item is checked or compared with the rest of the data to establish analytical categories. Again, this requires a coherent and systematic approach. The process of indexing focus group or interview material may include searching for particular types of narrative - such as jokes or anecdotes, or types of interaction such as questions, challenges, censorship or changes of mind. The key point to note about this indexing process is that it is inclusive; categories are added to reflect as many of the nuances in the data as possible, rather than reducing them to a few numerical codes. It is also to be expected that sections of the data - such as discrete incidents - will include multiple themes and are thus coded using several categories. It is, therefore, important to have some system of cross-indexing that allows the analysis of data items which fit into more than one category. A number of computer software packages have been developed to facilitate this aspect of the analytical process (see Software packages designed to handle qualitative data).
The process of indexing the data creates a large number of what Perry calls "fuzzy categories"9 or units. At this stage, there is likely to be considerable overlap and repetition between the categories. Informed by the analytical and theoretical ideas developed during the research, these categories are further refined and reduced in number by grouping them together. It is then possible to select key themes or categories for further investigation. In the study mentioned earlier, Becker and Geer pursued the use of the term "crock" by medical students to see what types of patients it described and when and how it was used. This meant collating all the instances when "crock" occurred in the data. Using these data, Becker and Geer were able to explain how medical students and staff categorised patients according to their utility for teaching/learning purposes. Once this was established, it became clear why "crocks" (typically the elderly patient, or the homeless alcoholic) who offered little or no possibility for learning about new or challenging disorders, were treated with disdain.
Grouping categories together typically entails a process of cutting and pasting, that is selecting sections of data on like or related themes and putting them together. The mechanics of how to do this vary. In the past, multiple copies of notes or transcripts were used so that sections could be, literally, cut out and pasted next to each other or sorted into different piles. Cardex systems have also been used - writing out relevant chunks of data onto index cards that could then be grouped in a card filing system.10 It is also possible to create matrices or spreadsheets to facilitate this process of identifying themes. Whilst considered somewhat old-fashioned, this repeated physical contact and handling of the data has much to recommend it; the process of re-reading the data and sorting it into categories means that the researcher develops an intimate knowledge of the data, even if the process is laborious.
Word processors can be enormously helpful in searching large amounts of text for specific terms. While it is unlikely to be the sole focus of a qualitative research project, the simple frequency with which particular words or phrases appear in a piece of text can be illuminating. Word processing functions can offer considerable advantages to researchers who traditionally would have used annotations in the margins of field notes or interview transcripts, coloured pens, scissors and glue, card systems and paper files. By typing index terms directly into the computer file containing the textual data the "search" function can be used to gather chunks of text, which can then be copied and pasted. The split screen functions make this a particularly appealing method for sorting and copying data into separate analytic files.
Software packages designed to handle qualitative data
There are now several software packages that have been specifically designed for qualitative data analysis. Among the most widely used are QSR NUD*IST11 and ATLAS/Ti.12 The evolution of analytical software has been welcomed as an important development with the potential to improve the rigour of analysis.13
The current software offers functions that enable more complex organisation and retrieval of data than is possible within word processing packages. Software packages that have been designed to assist in the analysis of unstructured textual data all have code and retrieval functions and several other uses that will be recognisable to researchers. These include the ability to conduct selective retrievals and examine reports separately by any other indexing term (for example, the respondent’s use of a particular term or a shared characteristic such as gender); to use algorithms to identify co-occurring codes in a range of logically overlapping or nesting possibilities; to attach annotations to sections of the text as "memos"; to add new codes; and to join together existing codes.
Most of the packages also provide counts of code frequencies and indicators of how many of the documents (for example interview transcripts) contain specific codes or indexing terms. While such counts may, on occasion, be illuminating to the researcher, it is important to treat them with caution. The reasons for this may be illustrated by contrasting the objectives of qualitative research with that of surveys and trials. In a study where everyone within a given population has had an equal chance of being selected to participate (this assumption is the cornerstone of sampling theory) and all respondents have been asked the same questions in the same manner, it is usually helpful to report responses as frequencies and percentages (relative frequencies). Surveys are designed to recruit sufficient numbers to represent the whole population. Trials aim to randomise enough subjects so that significant differences between treatment and control groups can be identified. By contrast, the qualitative methods of interviewing and observation are intended to identify subjective meanings and generate theory, which means that data collection will often continue until a saturation point has been reached (and no new categories are being contributed by the data), rather than until the sample is large enough to be considered statistically representative. In a qualitative study where the sample has not been (and often cannot be) selected to be numerically representative of the population, and where the interview technique is flexible and responsive, it can be misleading to report relative frequencies. This particularly applies if the questions have not been asked of all respondents, or have not been phrased in the same way or delivered at the same stage in the interview.
The ability to index, gather and sort are all important functions for organising and accessing the data, but these are only the initial stages in qualitative analysis. It has been suggested that computer-assisted analysis can help the researcher to build theoretical links, search for exceptions and examine "crucial cases" where counter evidence might be anticipated. A systematic search for "disconfirming evidence" can be assisted by using Boolean operators (such as or, and, not) to examine the data. An examination of the context of the fragments may be achieved either through considering which other index terms are attached to the data or by displaying the immediate context of the extract by including the lines of text that surround it. This function should particularly appeal to researchers who are concerned about the "decontextualisation" that can result from fragmenting the data into coded chunks. The Hypersoft package14 uses what the developer calls "hyperlinks" to capture the conceptual links that are observed between sections of the data, thereby protecting the narrative structure from being fragmented.
There are many potential benefits of using a software package to help with the more laborious side of textual analysis, but some caution is advisable. Those who are used to the sampling methods used in surveys and trials are sometimes concerned that qualitative samples are small and unrepresentative. The prospect of computer-assisted analysis may persuade researchers (or those who fund them) that they can manage much larger amounts of data and increase the apparent "power" of their study. Qualitative studies, which are not designed to be representative in terms of how far they may be generalised statistically, may gain little from an expanded sample size except a more cumbersome dataset. The nature and size of the sample should be directed by the research question and analytic requirements, not by the available software. In some circumstances, a single case study design may be the most successful way of generating theory. Lee and Fielding15 warn against the assumption that using a computer package will make analysis less time consuming, although it is hoped that it may make the process more demonstrably systematic.
Taking the analysis forward - the role of the researcher
The essential tasks of studying the text, recognising and refining the concepts and coding the data are inescapably the work of the researcher. For these reasons, it is important to dispel the notion that software packages are designed to deliver qualitative analysis of textual data. A computer package may be a useful aid when gathering together chunks of data, establishing links between the fragments, organising and reorganising the display and helping to find exceptions, but no package is capable of perceiving a link or defining an appropriate structure for the analysis. To take the analysis beyond the most basic descriptive and counting exercise requires the researcher’s analytical skills in moving towards hypotheses or propositions about the data.
One way of performing this next stage is called analytic induction. Linked to grounded theory this involves an iterative testing and re-testing of theoretical ideas using the data. Bloor16 describes in some detail how he used this procedure to reconstruct the decision-making rules used by ear, nose and throat surgeons (see Box 1). In essence, the researcher examines a set of cases, develops hypotheses or constructs and examines further cases to test these propositions - not unlike the statistical tests of association used in quantitative research.
In qualitative research, indexing the data and developing analytical categories tend to be carried out by a single researcher. However, some qualitative researchers have given attention to the notion that qualitative analyses may carry greater weight when they can be shown to be consistent between researchers (particularly when they have been undertaken to inform policy makers). This is close to the concept of inter-rater reliability, which is familiar in quantitative research. For example, Perry’s study of patients with multiple sclerosis,9 Daly et al.’s study of cardiac diagnosis,17 and Waitzkin18 used more than one analyst in order to improve their analyses. However, the appropriateness of the concept of inter-rater reliability in qualitative research is contested. Some qualitative researchers claim that a qualitative account cannot be held straightforwardly to represent the social world (just as all research findings reflect the identity of the researcher and the multiple nature of so called "reality"), thus different researchers are bound to offer different accounts. Another, less radical, assertion is that each researcher has unique insights into the data, which cannot be straightforwardly checked by others.19
In a recent contribution to the methodological debate, Armstrong et al.20 attempted to answer a simpler empirical question: do qualitative researchers show consistency in their accounts of the same raw data? To test this, they asked six experienced qualitative researchers independently to analyse a single focus group transcript and to identify and rank the major themes emerging in the discussion. Another social scientist, who had not read the transcript of the focus group, then read the six reports in order to determine the main themes and to judge the extent to which the six researchers agreed. There was quite close agreement about the identity of the basic themes, but the six researchers "packaged" or linked and contextualised the themes differently. Armstrong et al. concluded that such reliability testing was limited by the inherent nature of the process of qualitative data analysis. On the other hand, the interpretations of the six researchers had much in common despite the fact that they were from both Britain and the United States and from more than one discipline (anthropology, psychology and sociology). By deliberately selecting a diverse range of analysts (albeit all experienced), Armstrong et al. constructed a tough test of inter-rater agreement and one which would be unusual in a typical research study. It would be interesting to see the same exercise repeated with quantitative data and analysis and analysts from three different social science disciplines!
Despite the potential limitations of the term "reliability" in the context of qualitative research highlighted by Armstrong et al., there may be merit in involving more than one analyst in situations where researcher bias is especially likely to be perceived to be a problem; for example, where social scientists are investigating the work of clinicians. In a study of the contribution of the use of echocardiography to the social process of diagnosing patients with suspected cardiac abnormalities, Daly et al. developed a modified form of qualitative analysis involving the sociologist researchers and the cardiologists who had managed the patients. The raw data consisted of transcripts of the consultations between the patients and the cardiologists, cardiologists’ responses to a structured questionnaire and transcripts of open-ended research interviews with the cardiologists and with the patients. First, the transcripts and questionnaire data were analysed by the researchers in order to make sense of the process of diagnosis, including the purpose of the test. From this analysis, the researchers identified the main aspects of the consultations that appeared to be related to the use of echocardiography. Next, these aspects or features of the clinical process were turned into criteria in relation to which other analysts could generate their own assessments of the meaning of the raw data. The cardiologists involved then independently assessed each case using the raw data in order to produce an account of how and why a test was or was not ordered and with what consequences. The assessments of the cardiologists and sociologists were compared statistically and the level of agreement was shown to be good. Finally, in cases where there was disagreement between the original researchers’ analysis and that of the cardiologist, a further researcher repeated the analysis. Remaining discrepancies were resolved by consensus after discussion between the researchers and the cardiologists.
Although there was an element of circularity in part of this lengthy process (in that the formal criteria used by the cardiologists were derived from the initial researchers’ analysis) and it involved the derivation of quantitative gradings and statistical analysis of inter-rater agreement, which are unusual in a qualitative study, it meant that clinical critics could not argue that the findings were simply based on the subjective judgements of an individual researcher.
Applied qualitative research
Similar considerations arise in other areas where qualitative methods are deployed. One approach to qualitative analysis known as the framework approach has been developed in Britain specifically for applied or policy relevant qualitative research in which the objectives of the investigation are typically set in advance and shaped by the information requirements of the funding body (for example, a health authority) rather than emerging from a reflexive research process. The timescales of applied research also tend to be shorter than more "basic" social research and there tends to be a need to link the qualitative analysis to findings from quantitative investigation. For these reasons, although the framework approach is heavily based in the original accounts and observations of the people studied (that is, "grounded" and inductive), it starts deductively from the aims and objectives already set for the study. It is systematic and designed so that the analytic process and interpretations can be viewed and assessed by people other than the primary analyst.
The topic guide used to collect data under the framework approach (for example, to guide depth interviews) tends to be more structured from the outset than would be the norm for much other qualitative research. The transcription process is followed by five stages of analysis, which are similar to the steps in more conventional qualitative analysis discussed in this chapter. However, they tend to be more explicit and more strongly informed by a priori reasoning6 (see Box 2). Framework analysis is most commonly used with individual interview or focus group data. It is easy to see, even with a summary of the five stages, how laborious thorough qualitative data analysis can be.
This chapter has shown that analysing qualitative data is not a simple or quick task. Done properly, it is systematic and rigorous, and therefore labour intensive for the researcher(s) involved and time consuming. Fielding contends that "good qualitative analysis is able to document its claim to reflect some of the truth of a phenomenon by reference to systematically gathered data", in contrast, "poor qualitative analysis is anecdotal, unreflective, descriptive without being focused on a coherent line of inquiry".21 At its heart, good qualitative analysis relies on the skill, vision and integrity of the researcher doing that analysis, and as Dingwall et al. have pointed out, this may require highly trained and, crucially, experienced researchers.22
Bryman A, Burgess R. eds. Analysing qualitative data. London: Routledge, 1993.
Miles M, Huberman A. Qualitative data analysis. London: Sage, 1984.
1 Heritage J. Garfinkel and ethnomethodology. Cambridge: Polity, 1984.
2 E.S.R.C. QUALIDATA: Qualitative Data Archival Resource Centre, established 1994, University of Essex.
3 Becker HS. Sociological work. London: Allen Lane, 1971.
4 Miles M, Huberman A. Qualitative data analysis. London: Sage, 1984.
5 Silverman D. Going private: ceremonial forms in a private oncology clinic. Sociology 1984;18:191-202.
6 Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In Bryman A, Burgess R. eds. Analysing qualitative data. London: Routledge, 1993:173-94.
7 Glaser BG, Strauss AL. The discovery of grounded theory. Chicago, IL: Aldine, 1967.
8 Becker HS, Geer B. Participant observation: the analysis of qualitative field data. In: Burgess RG. Field research: a sourcebook and field manual. London: Allen and Unwin, 1982.
9 Perry S. Living with multiple sclerosis. Aldershot: Avebury, 1994.
10 Scambler G, Hopkins A. Accommodating epilepsy in families. In: Anderson R, Bury M. eds. Living with chronic illness: the experience of patients. London: Unwin Hyman, 1988.
11 Richards T, Richards L. QSR NUD*IST V3.0. London: Sage, 1994.
12 Muhr T. ATLAS/Ti for Windows, 1996.
13 Kelle U. ed. Computer-aided qualitative data analysis: theory, methods and practice. London: Sage, 1995.
14 Dey I. Qualitative data analysis: A user friendly guide for Social Scientists. London: Routledge, 1993.
15 Lee R, Fielding N. User’s experiences of qualitative data analysis software. In: Kelle U. ed. Computer aided qualitative data analysis: theory, methods and practice. London: Sage, 1995.
16 Bloor M. On the analysis of observational data: a discussion of the worth and uses of inductive techniques and respondent validation. Sociology 1978;12:545-52.
17 Daly J, McDonald I, Willis E. Why don’t you ask them? A qualitative research framework for investigating the diagnosis of cardiac normality. In: Daly J, McDonald I, Willis E, eds. Researching health care: designs, dilemmas, disciplines. London: Routledge, 1992:189-206.
18 Waitzkin H. The politics of medical encounters. New Haven: Yale University Press, 1991.
19 Morse JM. Designing funded qualitative research. In: Denzin NK, Lincoln YS, eds. Handbook of qualitative research. London: Sage, 1994:220-35.
20 Armstrong D, Gosling A, Weinman J, Marteau T. The place of inter-rater reliability in qualitative research: an empirical study. Sociology 1997;31:597-606.
21 Fielding N. Ethnography. In Fielding N. ed. Researching social life. London: Sage, 1993: 155-71(168-9).
22 Dingwall R, Murphy E, Watson P, Greatbatch D, Parker S. Catching goldfish: quality in qualitative research. Journal of Health Services Research and Policy 1998;3:167-72.
Figure 1 - Models of the research process
Box 1 - Analysis
Stages in the analysis of field notes in a qualitative study of ear, nose and throat surgeons’ disposal decisions for children referred for possible tonsillectomy and adenoidectomy (T&A)16
1. Provisional classification - For each surgeon all cases categorised according to the disposal category used (for example, T&A or tonsillectomy alone)
2. Identification of provisional case features - Common features of cases in each disposal category identified (for example, most T&A cases found to have three main clinical signs present)
3. Scrutiny of deviant cases - Include in (2) or modify (1) to accommodate deviant cases (for example, T&A performed when only two of three signs present)
4. Identification of shared case features - Features common to other disposal categories (history of several episodes of tonsillitis, for example)
5. Derivation of surgeons’ decision rules - From the common case features (for example, case history more important than physical examination)
6. Derivation of surgeons’ search procedures (for each decision rule) - The particular clinical signs looked for by each surgeon
Repeat (2) to (6) for each disposal category
Box 2 - The five stages of data analysis using the framework approach
• Familiarisation - immersion in the raw data (or typically a pragmatic selection from the data) by listening to tapes, reading transcripts, studying notes and so on, in order to list key ideas and recurrent themes
• Identifying a thematic framework - identifying all the key issues, concepts and themes by which the data can be examined and referenced. This is carried out by drawing on a priori issues and questions derived from the aims and objectives of the study as well as issues raised by the respondents themselves and views or experiences that recur in the data. The end product of this stage is a detailed index of the data, which labels the data into manageable chunks for subsequent retrieval and exploration
• Indexing - applying the thematic framework or index systematically to all the data in textual form by annotating the transcripts with numerical codes from the index, usually supported by short text descriptors to elaborate the index heading. Single passages of text can often encompass a large number of different themes each of which has to be recorded, usually in the margin of the transcript
• Charting - rearranging the data according to the appropriate part of the thematic framework to which they relate and forming charts. For example, there is likely to be a chart for each key subject area or theme with entries for several respondents. Unlike simple cut and paste methods that group verbatim text, the charts contain distilled summaries of views and experiences. Thus the charting process involves a considerable amount of abstraction and synthesis
• Mapping and interpretation - using the charts to define concepts, map the range and nature of phenomena, create typologies and find associations between themes with a view to providing explanations for the findings. The process of mapping and interpretation is influenced by the original research objectives as well as by the themes that have emerged from the data themselves.
9 Quality in Qualitative Health Research
NICHOLAS MAYS, CATHERINE POPE
The nature of qualitative data
This book has outlined the main methods used in qualitative research and described some of the ways in which these methods have been applied to answer questions about health and health care. As noted in Chapter 1, qualitative methods have long been used in the social sciences, but their use in health and health care settings is comparatively recent. In the last decade, qualitative methods have appeared in areas such as health services research and health technology assessment, and there has been a corresponding rise in the reporting of qualitative research studies in medical and related journals.1 Interest in these methods and their wider exposure in the field of health research has led to necessary scrutiny of qualitative research. Researchers from other traditions are increasingly concerned to understand qualitative methods and, most importantly, to examine the claims researchers make about the findings obtained from these methods.
Qualitative research in health and health services has had to overcome prejudice and a number of misunderstandings. For example, some people believe that qualitative research is "easy" - a soft option that requires no skills or training. In fact, the opposite is more likely to be the case. The data generated by qualitative studies are cumbersome and difficult to analyse2 and their analysis requires a high degree of interpretative skill. Qualitative research also suffers from the "stigma of the small n"3 because it tends to deal with a small number of settings or respondents and does not seek to be statistically representative. However, in expert hands this feature is irrelevant to the strength of the approach.
Nonetheless, the status of all forms of research depends on assessing the quality of the methods used. In the field of qualitative quality in qualitative health research, concern about assessing quality has manifested itself recently in the proliferation of guidelines for doing and judging qualitative work.2,4-6 Those using and funding research have played an important role in the development of these guidelines as they become increasingly familiar with qualitative methods, but require some means of assessing their quality and of distinguishing "good" and "poor" quality research. To this end, the NHS Research and Development Programme recently funded a review of qualitative research methods relevant to health technology assessment.7 However, while the sponsors of this review may have hoped for a small set of simple quality guidelines to emerge, any thoughtful analysis of the issue is inevitably far more complex.
The issue of "quality" in qualitative research is part of a much larger and contested debate about the nature of the knowledge produced by qualitative research, whether its quality can legitimately be judged and, if so, how. The chapters of this book have touched on a number of issues related to the quality of qualitative research. In outlining some of the most frequently used qualitative methods and demonstrating their contemporary application in health research, each chapter has referred to the strengths and limitations of particular methods. In doing this, each author clearly had some notion of quality in mind.
This chapter attempts to bring together some of these quality issues, although it cannot do full justice to the wider epistemological debate. It outlines two views of how qualitative methods might be judged. It goes on to argue that qualitative research can be assessed with reference to the same broad criteria as quantitative research, albeit differently used, and that two main criteria of quality in qualitative research stand out - validity and relevance.8 The chapter concludes with a list of questions that might be used to begin to assess the quality of a piece of qualitative research. The list is designed simply to indicate some of the questions worth considering when evaluating qualitative research, and not as a definitive inventory.
Can we use the same quality criteria for qualitative and quantitative research? Two opposing answers
There has been considerable debate among qualitative researchers over whether qualitative and quantitative methods can and should be assessed according to the same quality criteria. The debate is complex because there is an underlying lack of consensus about precisely what qualitative research is and the variety of approaches included under this heading. Other than the total rejection of any quality criteria, it is possible to identify two broad, opposing positions.8 First, there are those who have argued that qualitative research in all its guises represents a distinctive paradigm that generates a different type of knowledge from quantitative research. Therefore, different quality criteria should apply. Second, there are those who have argued that there is no separate philosophy of knowledge underpinning qualitative research and so the same criteria should be applied to qualitative and quantitative research. Within each position, it is possible to see a range of views.
Separate and different: the anti-realist position Advocates of this position argue that since qualitative research represents a distinct paradigm that generates a distinct form of knowledge, it is inappropriate to apply criteria derived from an alternative paradigm. This means that qualitative research cannot and should not be judged by conventional measures of validity (the test of whether the research is true to some underlying reality), generalisability (the degree to which the specifics of the research can be applied more widely to other settings and populations) and reliability (the extent to which the same findings are produced by repeating the research procedures). For those who adopt this anti-realist position, it would also be inappropriate to use mixed or multiple methods in the same study.At the core of this position is a rejection of what Lincoln and Guba9 call "na•ve realism" - a belief that there is a single, unequivocal social reality or truth that is entirely independent of the researcher and of the research process. Instead, they suggest "‘truth’ is defined as the best informed . . . and most sophisticated . . . construction on which there is consensus (although there may be several constructions extant which simultaneously meet that criterion) . . . the inquirer and the inquired are interlocked in such a way that the findings of an investigation are the literal creation of the inquiry process."9There are still more extreme relativists who hold that there is no basis even for the consensus referred to by Guba and Lincoln and that all research perspectives are unique and each is equally valid in its own terms. The absence of any external standards would clearly make it impossible for one researcher to judge another’s research. Yet, as Murphy et al. note, in health services research such a relativist position precludes qualitative research from deriving any unequivocal insights relevant to action and would, therefore, command little support among applied health researchers.7Those relativists who maintain that separate criteria are required to evaluate qualitative research have put forward a range of different assessment schemes. In part, this is because the choice and relative importance of different criteria of quality depend on the topic and the purpose of the research. If the key question for qualitative researchers is: "Why do people do what they do?" then for Popay et al. research quality relates to the sampling strategy, adequacy of theory, collection and analysis of data, the extent to which the context has been understood, and whether the knowledge generated incorporates an understanding of the nature of subjective meanings in their social contexts.10 While there may be some broad similarities between quality standards in quantitative and qualitative research, the fundamental differences in the knowledge each approach generates require that quality is assessed differently in the two traditions.11Hammersley has attempted to pull together the different quality criteria and concerns of the relativists (or anti-realists), as follows:8
• The degree to which substantive and formal theory is produced and the degree of development of such theory
• The novelty of the claims made from the theory
• The consistency of the theoretical claims with the empirical data collected
• The credibility of the account to those studied and to readers
• The extent to which the description of the culture of the setting provides a basis for competent performance in the culture studied
• The extent to which the findings are transferable to other settings
• The reflexivity of the account - that is, the degree to which the effects of the research strategies on the findings are assessedand/or the amount of information about the research process that is provided to readers. These criteria are open to challenge. For example, it is arguable whether all research should be concerned to develop theory. At the same time, many of the criteria listed are not exclusive to qualitative research (for example, the extent to which findings are transferable), suggesting that there is a case for assessing both qualitative and quantitative research against the same standards, even if that assessment has to be tailored to the type of research used.
Using criteria from quantitative research: subtle realism
Authors such as Hammersley12 and Kirk and Miller13 agree that all research involves subjective perceptions and observations, and that different methods will produce different pictures of the social phenomena being studied. However, unlike the anti-realists, they argue that this does not mean that we cannot believe in the existence of phenomena independent of our claims about them; that is, there is some underlying reality that may be studied. The role of qualitative and quantitative research is to attempt to represent that reality rather than to imagine that "the truth" can be attained. Hammersley refers to this as subtle realism.
The logic of this position is that there are ways to assess the different perspectives offered by different research processes against each other and against criteria of quality common to both qualitative and quantitative research, particularly those of validity and relevance.8 However, the means of assessment may be modified to take account of the distinctive goals of qualitative research. For example, qualitative research frequently does not seek to generalise to a wider population for predictive purposes, but seeks to understand specific behaviour in a naturally occurring context. Similarly, reliability, as conventionally defined, may be of little relevance if unique situations cannot be reconstructed or if the setting studied is undergoing considerable social change.14
A comprehensive review of the literature on qualitative research in health technology assessment7 concluded by supporting Hammersley’s case8 for assessing such research according to its validity, defined as the extent to which the account accurately represented the social phenomena to which it referred, and its relevance, defined in terms of the capacity of the research to help some group of practitioners solve the problems they faced. Each broad criterion will be discussed in turn.
Assessing the validity of qualitative research
There are no mechanical or "easy" solutions to limit the likelihood that there will be errors in qualitative research. However, there are various ways of improving validity, each of which requires the exercise of judgement on the part of researcher and reader.
Triangulation involves the comparison of the results from either two or more different methods of data collection (for example interviews and observation) or, more simply, from two or more data sources (for example, interviews with members of different interest groups). The researcher looks for patterns of convergence to develop or corroborate an overall interpretation. Triangulation is generally accepted as a means of ensuring the comprehensiveness of a set of findings. It is more controversial as a genuine test of the truthfulness or validity of a study. The latter test relies on the assumption that any weaknesses in one method will be compensated by strengths in another. Occasionally, qualitative methods will reveal inadequacies in quantitative measures or show that quantitative results are at odds with observed behaviour. For example, Stone and Campbell’s depth interviews in Nepal (mentioned in Chapter 1) revealed very different attitudes towards abortion and family planning from those recorded in formal fertility surveys.15 Similarly, Meyer’s multi-method approach highlighted the gap between the findings derived from attitudinal scales and everyday talk about, and practice in relation to, lay participation in care on the ward she studied16 (see Chapter 7). However, this use of triangulation is contested. Silverman argues that data from different sources can only be used to identify the context-specific nature of differing accounts and behaviour.17 He points out that discrepancies between different data sources (such as from doctors and their patients) present a problem of adjudication between rival accounts. Thus, triangulation may be better seen as a way of making a study more comprehensive, or of encouraging a more reflexive analysis of the data (see Reflexivity) than as a pure test of validity.
Respondent validation, or member checking as it is sometimes called, includes a range of techniques in which the investigator’s account is compared with the accounts of those who have been investigated in order to establish the level of correspondence between the two sets. The reactions of those studied to the analyses are then incorporated into the study findings. Lincoln and Guba9 regard respondent validation as the strongest available check on the credibility of a research project. However, there are limitations to these techniques as validation tests. For example, the account produced by the researcher is designed for a wide audience and will, inevitably, be different from the account of an individual informant simply because of their different roles in the research process. As a result, it is better to think of respondent validation as part of a process of error reduction, which also generates further original data, which, in turn, require interpretation.18
Clear exposition of methods of data collection and analysis
Since the methods used in research unavoidably influence the objects of enquiry (and qualitative researchers are particularly aware of this), it is important to provide a clear account of the process of data collection and analysis. This is so that readers can judge the evidence upon which conclusions are drawn, taking account of the way that the evidence was gathered. For example, in an observational study, it would be particularly pertinent to document the period of time over which observations were made and the depth or quality of the researcher’s access to the research setting.
A common failing of qualitative research reports is an inadequate account of the process of data analysis. This is compounded by the inductive nature of much qualitative work in which prior conceptualisation is largely inappropriate since concepts and categories are developed through the process of undertaking the research. As a result, the processes of data collection and analysis are frequently interwoven. Nonetheless, by the end of the study, it should be possible to provide a clear account of how early, simpler systems of classification evolved into more sophisticated coding structures and thence into clearly defined concepts and explanations for the data collected. In some situations, it may be appropriate to assess the inter-rater reliability of coding by asking another researcher independently to code some of the raw data using coding criteria previously agreed. Where this is not feasible or appropriate (see Chapter 8 for more on this), it may be preferable to show that a range of potential explanations has been explored to make sense of the data collected. Finally, it is important to include in the written account sufficient data to allow the reader to judge whether the interpretation offered is adequately supported by the data. This is one of the reasons why qualitative research reports are generally longer than those of quantitative studies since it can be difficult to summarise the data that support a concept or explanation.
Reflexivity means sensitivity to the ways in which the researcher and the research process have shaped the data collected, including the role of prior assumptions and experience, which can influence even the most avowedly inductive enquiries. Researchers can keep a personal research diary alongside the data collection and analysis in which to record their reactions to events occurring during the period of the research. They can and should make their personal and intellectual biases plain at the outset of any research reports to enhance the credibility of their findings. The effects of personal characteristics such as age, gender, social class and professional status (for example that of doctor, nurse, physiotherapist, sociologist, etc.) on the data collected and the "distance" between the researcher and those researched also need to be discussed.
Attention to negative cases
As well as exploring alternative explanations for the data collected, a long-established tactic for reducing error is to search for, and discuss, elements in the data that contradict, or appear to contradict, the emerging explanation of the phenomena under study. Deviant case analysis helps refine the analysis until it can explain all or the vast majority of the cases under scrutiny. It is similar to Popper’s quest for evidence that disproves established theories in the natural sciences and can help counteract some of the preconceptions that all researchers bring to their research. In this way, it can contribute to increasing the sophistication and credibility of research reports.19 Another version of deviant or negative case analysis is to attempt to incorporate seemingly different findings from different studies into a more refined, overarching analysis.
The final technique for reducing bias in qualitative research is to ensure that the research design explicitly incorporates a wide range of different perspectives so that the viewpoint of one group is never presented as if it represents the sole truth about any situation. Dingwall20 coined the term "fair dealing" to describe this process of attempting to be non-partisan; for him, fair dealing marks the difference between social science and "muck-raking journalism". However, this concern to deal even-handedly with all those studied is not shared by all researchers. Indeed, there is a long tradition in sociology, dating from the 1930s Chicago School, of adopting the perspective of the "underdog" against the dominant views of powerful elites.21 This position has been severely mauled in recent times: Strong scathingly described it as being more concerned with being "right on" than with being right.22
Hammersley argued that good quality qualitative research had to be relevant in some way to a public concern, though this did not necessarily mean that the research should slavishly adhere to the immediate concerns or problems defined by policy makers, professionals or managers.8 Research could be relevant when it either added to knowledge or increased the confidence with which existing knowledge was regarded. Another important dimension of relevance is the extent to which findings can be generalised beyond the setting in which they were generated. Quantitative researchers frequently criticise qualitative studies for their lack of representativeness. However, it is possible to use forms of probability sampling such as stratified sampling techniques in qualitative research in order to ensure that the range of settings chosen is representative of the population about which the researcher wishes to generalise. Another tactic is to ensure that the research report has sufficient descriptive detail for the reader to be able to judge whether or not the findings apply in other similar settings.
Finally, it has to be recognised that generalisation from qualitative research does not rely exclusively on notions of statistical logic. The extent to which inferences can be drawn from one setting to another depends as much on the adequacy of the explanatory theory on which they are based as on statistical representativeness.19 Thus the test is whether categories of cases or settings that are theoretically similar behave in the same way rather than cases or settings that are substantively similar. One way of looking at this is to explore the extent to which the sample of cases studied included the full range of potentially relevant cases. This is theoretical sampling in which an initial sample is drawn to include as many as possible of the factors that might affect variability of behaviour, but is then extended, as required, in the light of early findings and emergent theory.2 Under conceptual or theoretical sampling, statistical "minority" groups are frequently over-represented in order to test whether the emerging explanations are equally robust when applied to widely differing populations. The full sample, therefore, attempts to include the full range of settings relevant to the conceptualisation of the subject.
Is there any place for quality guidelines? The hotly contested debate about whether quality criteria should be applied to qualitative research, together with the differences of view between "experts" about which criteria are appropriate and how they should be assessed, should warn against unthinking reliance on any one set of guidelines. A number of practical checklists have been published recently in the UK to help with judging the quality of qualitative work.2,4,5 Indeed, a crude checklist was included in the previous edition of this book.6 The checklists cover a wide range of issues that may potentially be relevant to the rigour of qualitative studies. However, following Hammersley8 and the recent overview by Murphy et al.,7 two central criteria of rigour stand out - validity and relevance. Neither criterion is straightforward to assess. Each requires judgements to be made. Thus, instead of setting out yet another set of guidelines, the final section of this chapter consists of some possible questions to ask of any piece of qualitative research with an emphasis on relevance and validity. The questions could also be used by researchers at different times during the life of a particular research project in order to improve its quality.Some questions that might be asked of a qualitative study
Was this piece of work worth doing at all? Has it contributed usefully to knowledge?
• Clarity of research question
If not at the outset of the study, by the end of the research process was the research question clear? Was the researcher able to setaside her/his research preconceptions?
• Appropriateness of the design to the question
Would a different method have been more appropriate? Did the study require the use of qualitative methods? For example, if a casual hypothesis was being tested, was a qualitative approach really appropriate?
Is the context/setting adequately described so that the reader can relate the findings to other settings?
Did the sample include the full range of possible cases/settings so that conceptual rather than statistical generalisations could be made (that is, more than convenience sampling)? If appropriate, were efforts made to obtain data that might contradict or modify the analysis by extending the sample (for example, to a different type of area)?
• Data collection and analysis
Were the data collection and analysis procedures systematic? Was an "audit trail" provided such that someone else could repeat each stage, including the analysis?How well did the analysis succeed in incorporating all the observations? Was there unexplained variation? To what extent did the analysis develop concepts and categories capable of explaining key processes or respondents’ accounts or observations? Was it possible to follow the iteration between data and the explanations for the data (theory)? Did the researcher seek disconfirming cases?
• Reflexivity of the account
Did the researcher self-consciously assess the likely impact of the methods used on the data obtained? Were sufficient data included in the report of the study to provide sufficient evidence for readers to assess whether analytical criteria had been met?
Although the issue of quality in qualitative health and health services research has received considerable attention, a recent paper was able to argue, legitimately, that "quality in qualitative research is a mystery to many health services researchers".23 This chapter has tried to show how qualitative researchers endeavour to address the issue of quality in their research. It has outlined the broad debates about the nature of the knowledge produced by qualitative research and indicated some of the questions it is worth asking of a particular study or research report.
As in quantitative research, the basic strategy to ensure rigour, and thus quality, in qualitative research, is systematic, self-conscious research design, data collection, interpretation and communication. Qualitative research, as this book has shown, has much to offer. Its methods can, and do, enrich our knowledge of health and health care. It is not, however, an easy option or the route to a quick answer. As Dingwall et al. conclude, "qualitative research requires real skill, a combination of thought and practice and not a little patience."23
Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology assessment: a review of the literature. Health Technology Assessment 1998:2(16).
Dingwall R, Murphy E, Watson P, Greatbatch D, Parker S. Catching goldfish: quality in qualitative research. Journal of Health Services Research and Policy 1998:3:167-72.
1 Harding G, Gantley M. Qualitative methods: beyond the cookbook. Family Practice 1998;15:76-9.
2 Boulton M, Fitzpatrick R. Qualitative methods for assessing health care. Quality in Health Care 1994;3:107-13.
3 Faltermaier T. Why public health research needs qualitative approaches: subjects and methods in change. European Journal of Public Health 1997;7:357-63.
4 Blaxter M. Criteria for evaluation of qualitative research. Medical Sociology News 1996;22:68-71.
5 Secker J, Wimbush E, Watson J, Milburn K. Qualitative methods in health promotion research: some criteria for quality. Health Education Journal 1995;54:74-87.
6 Mays N, Pope C. Rigour in qualitative research. In: Mays N, Pope C, eds. Qualitative research in health care. London: BMJ Books, 1996.
7 Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology assessment: a review of the literature. Health Technology Assessment 1998:2(16).
8 Hammersley M. Reading ethnographic research. New York: Longman, 1990.
9 Lincoln YS, Guba EG. Naturalistic inquiry. Newbury Park, CA: Sage, 1985:84.
10 Popay J, Rogers A, Williams G. Qualitative research and the gingerbread man. Health Education Journal 1995;54:389-443.
11 Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of qualitative literature in HSR. Qualitative Health Research 1998;8:341-51.
12 Hammersley M. What’s wrong with ethnography? London: Routledge, 1992.
13 Kirk J, Miller M. Reliability and validity in qualitative research Qualitative Research Methods Series No 1. London: Sage, 1986.
14 Seale C, Silverman D. Ensuring rigour in qualitative research. European Journal of Public Health 1997;7:379-84.
15 Stone L, Campbell JG. The use and misuse of surveys in international development: an experiment from Nepal. Human Organisation 1986;43:27-37.
16 Meyer JE. Lay participation in care in a hospital setting: an action research study. London: University of London, unpublished PhD thesis, 1995.
17 Silverman D. Interpreting qualitative data: methods for analysing talk, text and interaction. London: Sage, 1993.
18 Bloor M. Techniques of validation in qualitative research: a critical commentary. In: Miller G, Dingwall R, eds. Context and method in qualitative research. London: Sage, 1997:37-50.
19 Silverman D. Telling convincing stories: a plea for more cautious positivism in case studies. In: Glassner B, Moreno JD, eds. The qualitative-quantitative distinction in the social sciences. Dordrecht: Kluwer Academic, 1989:57-77.
20 Dingwall R. Don’t mind him - he’s from Barcelona: qualitative methods in health studies. In: Daly J, McDonald I, Willis E, eds. Researching health care. London: Tavistock/Routledge, 1992:161-75.
21 Guba EG, Lincoln YS. Fourth generation evaluation. Newbury Park, CA: Sage, 1989.
22 Strong P. Qualitative sociology in the UK. Qualitative Sociology 1988;11:13-28.
23 Dingwall R, Murphy E, Watson P, Greatbatch D, Parker S. Catching goldfish: quality in qualitative research. Journal of Health Services Research and Policy 1998;3:167-72.