GRADE Methodology

What is GRADE?

GRADE is an acronym for Grading of Recommendations Assessment, Development and Evaluation. 

GRADE provides a systematic process for the development of CPGs. (1) The GRADE approach is a system for making judgments about the quality of evidence of research studies and strength of recommendations. GRADE was designed by an international team of researchers who were motivated to create a system that would improve upon, and ultimately replace, the many existing systems of evidence assessment and recommendation grading that guideline developers were then using. 

The AOM began using the GRADE methodology for guideline development in 2011. Prior to adopting the GRADE approach, AOM CPGs used the levels of evidence and grades of recommendations previously used by the Canadian Task Force on Preventive Health Care (CTFPHC). (2) The CTFPHC began using GRADE in 2010. A number of other organizations that evaluate research evidence and/or develop guidelines also use GRADE, including the World Health Organization, the National Institute for Health and Clinical Excellence (NICE), and the Cochrane Collaboration.

What’s different about GRADE? 

GRADE has a number of key benefits over the CTFPHC approach to rating levels of evidence and strength of recommendations:

  • It’s less dependent on study design.
    One of the major problems with the CTFPHC approach is that quality of evidence is determined without explicit consideration of factors other than study design: randomized controlled trials (RCTs) automatically score higher than observational studies, regardless of the quality of the RCT and its direct relevance to the clinical question being investigated. However, there are lots of clinical questions important to midwives for which evidence from RCTs won’t ever be available, for ethical and/or practical reasons. 

    GRADE requires that guideline developers consider fundamental study design AND additional methodologic factors when evaluating quality of evidence. The GRADE system allows guideline developers to “downgrade” RCTs with major methodological aws, and “upgrade” high quality observational studies (e.g., cohort or case-control studies) that show very strong effects. 

  • It encourages the consideration of client-important values and allows for a fuller weighing of benefits and harms.
    In deciding upon strength of a recommendation, GRADE encourages CPG working groups to consider the balance of benefits and harms from the perspective of midwifery clients.

  • It encourages transparent decision-making.
    GRADE specifies clear criteria for making judgments about quality of evidence and strength of recommendation. This permits the CPG Work Group to be explicit about how a guideline recommendation came about. This hasn’t necessarily been the case in the past – recommendations that come about by consensus often don’t reflect a transparent decision-making process.

The GRADE CPG development process GRADE flowchart

AOM CPGs use the GRADE methodology for guideline development, and also rely on the AOM’s Value-based Statement on Guideline Development to guide the work of AOM staff and CPG Work Groups. (3)

Interpreting CPG recommendations made using GRADE

Recommendations in AOM CPGs are graded as either strong or weak according to the GRADE system. 

The strength of recommendation reflects the extent to which the CPG Work Group is confident that the benefits of a recommended intervention outweigh its harms, or vice versa. The strength of recommendation is influenced by the quality of supporting evidence, the balance between desirable and undesirable effects, and the perceived variability or uncertainty in clients’ values and preferences with respect to the intervention. 

The CPG Work Group’s judgements about the quality of evidence reflect its confidence that available evidence correctly reflects the real world effects of the intervention for midwifery clients and is sufficient to support informed choice. 

Most recommendations also include a rationale, which explains the benefits, risks and burdens considered by the Work Group when deciding upon the strength of the recommendation. The rationale also clearly identifies the client-centred outcomes that the recommendation addresses.

GRADE recommendation sample

More about strength of recommendation and quality of evidence

STRENGTH OF RECOMMENDATION

The extent to which the CPG Work Group is confident that benefits of the recommended intervention outweigh its harms (or vice versa)

STRONG

Benefits clearly outweigh risks and burdens (or vice versa)

Can be interpreted as: 

  • Most clients should be offered the intervention, assuming that they have been informed about and understand its benefits, harms and burdens. 
  • Most clients would choose the recommended course of action and only a small proportion would not. 

WEAK

Benefits, risks and burdens are closely balanced

Can be interpreted as: 

  • The majority of clients would choose the suggested course of action, but an appreciable proportion would not.
  • Values and preferences vary widely.

Based on: (1,4-6)

QUALITY OF EVIDENCE

How certain we ought to be about an estimate of effect or association

HIGH

Further research is very unlikely to change confidence in the estimate of effect. 

  • Benefits clearly outweigh risks and burdens (or vice versa). 

Decision-making around quality of evidence is influenced by the following factors:

  • Study design 
  • Study quality
  • Consistency across studies 
  • Directness of population, intervention, or outcome 
  • Precision in estimates of effect
  • Effect size 
  • Presence of a dose-response effect

MODERATE

Further research is likely to have an important impact on confidence in the estimate of effect and may change the estimate. 

  • This evidence provides a good basis for decision-making. 

LOW

Further research is very likely to have an important impact on confidence in the estimate of effect and is likely to change the estimate. 

  • This evidence provides some basis for decision-making. 

VERY LOW

Any estimate of effect is very uncertain. 

  • This evidence does not provide much of a basis for decision making. 

Based on: (1,6,7)

References: 

(1) Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008 Apr 26;336(7650):924-926. 

(2) Canadian Task Force on Preventive Health Care. New grades for recommendations from the Canadian Task Force on Preventive Health Care. CMAJ 2003 Aug 5;169(3):207-208. 

(3) Association of Ontario Midwives. Collated Response: A Values Based Approach to CPG Development. 2006. 

(4) Andrews J, Guyatt G, Oxman AD, Alderson P, Dahm P, Falck-Ytter Y, et al. GRADE guidelines: 14. Going from evidence to recommendations: the signi cance and presentation of recommendations. J Clin Epidemiol 2013 Jul;66(7):719-725. 

(5) Andrews JC, Schunemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Coello PA, et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation’s direction and strength. J Clin Epidemiol 2013 Jul;66(7):726-735. 

(6) Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A, et al. Going from evidence to recommendations. BMJ 2008 May 10;336(7652):1049-1051. 

(7) Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011 Apr;64(4):401-406.