Critical Appraisal Questions for Evidence-Based Practice: Are the Results Important?

Critical Appraisal Question: Are the research results important? Photo credit (c) Darren Whittingham on Shutterstock.com

Critical Appraisal Question: Are the research results important? Photo credit (c) Darren Whittingham on Shutterstock.com

This month I’m talking about the questions to ask when critically appraising a research study for use in your evidence-based nursing practice. In this post, I’ll address the question of determining if the actual research findings/results are significant and meaningful – What are the Results? (AKA Are the Results Important?)

The major critical appraisal questions for evidence-based practice are: Is the research study necessaryAre the results of the study valid? What are the results?, and finally, Can I apply the results to my patient? 

If your answer is yes to the first question of necessity, your next question is Are the results of the study valid?  As I said in the last post — asking if the study results are valid is the MOST important critical appraisal question you can ask. The reason is that if the methods used to design and conduct the study are not rigorous or internally valid, then you CANNOT BELIEVE the research results!

Once you determine the results are valid, then you can move onto examining the research results for clinical and practical significance – the focus of this blog post.

So Where Do You Find the Research Results? 

In both qualitative and quantitative studies, the research findings are found in the Results section of the manuscript or journal article. 

Qualitative Research Results

In qualitative studies, the research results are commonly in the form of a thematic analysis. Qualitative data collection frequently consists of 1:1 or focus group interviews with the research participants. The interviews are transcribed word for word and the researcher reviews the transcripts and analyzes the data according to the process of the qualitative method directing the study. Common themes that “emerge” from the data are identified to describe the lived experience of the participants; the participant’s own words are excerpted from the transcript to support the identified themes. When critically appraising a qualitative research study, you want to see a justification for the themes or patterns identified and will look for congruence between the themes identified and the supporting excerpts. 

The results may also be in the form of a theory or a description of culture or patterns or process depending on the type of qualitative study that was conducted. Typical qualitative methodologies include phenomenology, ethnography, grounded theory, narrative, and case study.

Because quantitative researchers present their findings as numerical results, and these are typically harder to interpret than qualitative (narrative) findings, I’ll focus on how to interpret quantitative results for significance and importance for this post. In keeping with the previous post, let’s look at how to critically appraise research results from an intervention or treatment study. 

Quantitative Research Results

Therapy can be defined as interventions that are instituted to achieve a therapeutic outcome. Healthcare interventions, nursing interventions, and medical or surgical interventions are all types of therapy or treatments of disease. With any intervention we implement, we need to assess whether the desired outcome was achieved and whether any adverse events occurred as a result of the treatment (i.e., harm).  Assessing the outcomes of therapy is a question of treatment effectiveness (DiCenso & Guyatt, 2005).

Once you determine that a study is validonly then do you care about the actual results of the study. 

In studies focused on treatments or interventions, you want to understand and interpret what those numbers and stats really mean – so this question – are the results important? – is looking at the statistical and clinical significance of the actual results.

The quantitative results are presented to show the reader how large the treatment effect was (the magnitude) and how precise the estimate of the treatment effect was. The answers to both of these questions will help us determine if the intervention is likely to make a measurable and meaningful difference in the patient’s outcomes.

How Large was the Treatment Effect? is determined by looking at the measure of the effect being studied – what variables are being studied? Risk measures such as relative risk, absolute risk, odds ratio, relative risk reduction, and number-needed-to-treat (NNT) are presented.

How Precise was the Treatment Effect? is determined by assessing the confidence interval — how close is it to the probable true population value? and does it contain the line of no difference?

What Subquestions Help Determine, Are the Research Results Important?
Look at the data themes or results to determine Are the results important?

Look at the data themes or results to determine Are the results important? Photo credit:(c) Shutterstock.com

 

In these questions and subquestions, you will be looking at the results for the variables studied and make a determination of whether these results are (a) important – by looking at the magnitude and precision of the result; (b) statistically significant – by looking at the p values and confidence intervals (CIs); and (c) clinically significant – by deciding whether the result is important enough to change practice (regardless of whether the result was statistically significant or not).

 

Was the Result Important?

  • How large was the intervention effect? (What was the magnitude of the effect?) Depending on the type of study, we are looking at how big was the effect? How much difference did the intervention make between the groups? Means, standard deviations, and effect sizes are reported for group differences. Outcomes are reported as relative risks, absolute risks, odds, odds ratios, and risk reductions/risk increases or benefit increases. You’ll have to make a decision as to whether the effects were clinically significant or not (even if not statistically significant).
  • How precise was the estimate of the intervention effect? Precision is estimated using the CIs. The point estimate is the study result and is an estimate of the true risk of a population based on the sample of patients in the study. The point estimate may be reported with a 90%, 95%, or 99% CI. Once you see the range of the interval, then you have to make a decision as to whether the CI is wide or narrow. The narrower the CI, the more precise the results because we can be 90%/95%/99% confident that the point estimate is a reflection of the actual population value.

For example, a relative risk of 3.3 with a 95% CI of 2.2-4.1 is very narrow and therefore very precise.  A relative risk of 3.3 with a 95% CI of 0.2-14.1 is wide and therefore not very precise. The most common reason for a wide CI is a small sample size.  

Was the Result (the Point Estimate) Statistically and Clinically Significant?

Statistical significance has to do with the likelihood that a research result is true (i.e., a real effect of the intervention) and not merely a matter of chance.

Clinical significance is a subjective interpretation of a research result as practical or meaningful for the patient and thus likely to affect provider behavior. 

Once you have interpreted the magnitude and precision of the research results, you want to determine if the results are statistically significant and likely not due to random chance and if the results will matter to our patients? The answers to these subquestions will help you answer the critical appraisal question: Are the Results Important? 

I wrote about both of these topics in detail, so I’ll refer you back to that post where I discuss the differences between these two concepts, how to interpret each, and give you examples.

Bottom line: If the treatment effect is large enough to make a meaningful difference in patient outcomes and the treatment effect is precise, then you can conclude that the study results are important. 

Once you are satisfied that the research results are meaningful and important,  you can move on to seeing how you can apply the results to your patient or population of interest. Stay tuned — that’s the next post!

How to Cite this Blogpost in APA*: 

Thompson, C. J. (2017, November 21). Critical appraisal questions in evidence-based practice: Are the results important? [Blogpost]. Retrieved from https://nursingeducationexpert.com/critical-appraisal-results-important *Citation should have hanging indent

Reference

DiCenso, A., & Guyatt, G. (2005). Health care interventions. In A. DiCenso, G. Guyatt, and D. Ciliska (Eds.). Evidence-based nursing: A guide to clinical practice (pp. 48-70). St. Louis: Elsevier Mosby.