8 Conditions When Research is Put Into Practice: More Lessons From Health Care

In Health Care putting research into practice can be high stakes. Changing routines and practices as a result of new findings often has direct impacts on the health, treatment or recovery of patients. A significant contribution to the study of changing practices was made in 2003 by Richard Grol and Jeremy Grimshaw who published the study:

From best evidence to best practice: effective implementation of change in patients’ care”,  The Lancet, Vol 362, October 11, 2003.

Keeping up with the pace of research

The speed of medical research provided Grol and Grimshaw with an interesting opportunity to explore the intersection of research and practice. They found that although syntheses helped physicians stay current with findings and saved time from having to read the original research papers, physicians still had difficulty keeping pace with the rapid advances in health care knowledge. In addition to physician factors, the environment (attitudes and approaches of colleagues) also played a role in evidence impacting action.

The 8 conditions research was most often put into practice was when:

  1. You don’t think the current practice works: Physicians quickly changed practice when there was existing scepticism of the benefits of an established practice (review of treatment of otitis media).
  2. It’s an easy problem to address: physicians were more likely to change practices for acute care issues than for chronic care
  3. You feel the findings are more believable and compelling (better quality of evidence)
  4. The new practice matches current values
  5. The new practice makes your life easier: “less complexity of decision-making”
  6. You have a clearer understanding of what needs to be done: “more concrete description of the desired performance”
  7. The new recommendations don’t require a lot of new skills or changes to the organization
  8. The new practice is targeted at specific obstacles to change.

Although Grol and Grimshaw were unable to find a single strategy for knowledge mobilization that was effective in every setting and condition, they provided a very useful and thought-provoking summary of strategies with respect to impact.

Effectiveness of Knowledge Mobilization Strategies:

  • Limited Effects
    • Total quality management/continuous quality improvement (1 review, 55 studies)
  •  Mixed Effects
    • Educational materials (9 reviews, 3-37 studies)
    • Conferences, courses (4 reviews, 3-17 studies)
    • Use of opinion leaders (3 reviews, 3-6 studies)
    • Education with different educational strategies (8 reviews, 5-63 studies)
    • Feedback on performance (16 reviews, 3-37 studies)
  • Mostly Effective
    • Reminders (14 reviews, 4-68 studies)
    • Computerised decision support (5 reviews, 11-98 studies)
    • Introduction of computers in practice (2 reviews, 19-30 studies)
    • Mass media campaigns (1 review, 22 studies
    • Interactive small group meetings (4 reviews, 2-6 studies)

A further review of the strategies found:

  • Education and information had short-term effects
  • Reminders had modest and sustained effects
  • Performance feedback is effective but ceases if feedback is not continued.

Does the effectiveness of these strategies surprise you?

Would you have expected reminders to be more effective than opinion leaders, feedback, conferences or educational materials? One thing that shouldn’t be surprising is that the more strategies you employ, the more likely it will be effective. Grol and Grimshaw reflected that multifaceted interventions had “pronounced effects on practice and outcomes” involving the combined use of education, written materials, feedback and reminders.

It remains to be seen whether these strategies would result in similar levels of effectiveness in an education context but it provides an interesting look at knowledge mobilization efforts. The question now to consider is, how effective are the strategies you are using?

Posted in Research Resources | Tagged , | Leave a comment

A look at Twitter and #OERS14

This was the ninth year for the Ontario Education Research Symposium which is organized by the Ontario Education Research Panel.  As with previous years it was a wonderful venue to share research, explore new collaborations and expand networking opportunities.  Throughout the conference, attendees reflected and reacted to the keynote speakers and workshop presenters on twitter (#OERS14).  As with previous years, I used TAGS v5.1, a spreadsheet utility developed in Google Drive by Martin Hawksey, to collect and archive the conference tweets.

Over the course of the conference I had several people ask how I work with the twitter feeds. So, for those who are interested, here are the steps, premises and limitations of how I look at tweets: But first, some stats:

177 people engaged in #OERS14 through twitter with 1,280 tweets and 271 links shared.  This year, the top 5 tweeters were:

  • @abbaspeaks (100)
  • @ResearchChat (95)
  • @O3atORION (93)
  • @CarolCambell4 (76)
  • @KNAER_RECRAE (63)

Many thanks to everyone who contributed to the discussions.

If you would like to take a closer look at the comments and resources that were shared the tweets are available as a pdf, an excel sheet and an online link:

  • Click here for a pdf of the tweets (with active links)
  • Click here for the excel sheet that is produced by TAGS v5.1
  • Click here to see what the google spreadsheet looks like.

The following steps can be seen in the excel sheet linked above.

Identifying retweets:

Working on the premise that the letters “RT” followed by a space signifies a retweet:

  • In a new column (in this case, column T)  I use the excel formula @LEFT(D2,3) to return the first three characters of each tweet.  This formula is copied in every cell in the column;
  • In the next column (column U) I use the formula =IF(SEARCH(“RT”, T2),1,0) to evaluate the three characters that are returned in column T and then return a 1 if it contains “RT “  and a null value if it doesn’t;
  • Using a filter I can sort on column U and take a look at the retweets.

Limitations: this won’t pick up retweets that are embedded further into a tweet, for example the tweet “I love this!  RT You won’t believe what they said” would be excluded because the RT occurs further into the text.

Identifying the most retweeted comment:

  • The approach is similar to the previous method.  In a new column (column V) @Left(cell, 30) is used to identify the first 30 characters of a tweet;
  • A pivot table is created using the RT codes and the Tweet stems from column V.  The Pivot table is selected and then sorted by the Counts.  The 9 or 10 tweets with the highest counts are selected and the full tweets are compiled from the original list.

Limitations: if there are slight variations in text further into the tweet it isn’t picked up or distinguished.  For example my tweet about the EQAO Research Bulletin had the most retweets for the math bulletin (8) than the literacy bulletin.  I manually changed this as I compiled the tweets.

Number of Retweets – Tweet:

  • 12 – RT @Anniekidder: #oers14 OECD data show no correlation between hours in school and performance. Quality not quantity. #OntEd
  • 12 – RT @CarolCampbell4: Future PISA tests to include collaborative problem solving & global citizenship #oers14
  • 12 – RT @Anniekidder: #oers14 Dr. Bruce Ferguson says “our classrooms are emergency rooms for the social problems in our society.”
  • 10 – RT @Anniekidder: #oers14 Students of parents who have high expectations report higher perseverance, motivation, confidence and greater enga…
  • 10 – RT @drcathybruce: #OERS14 Student efficacy (belief that they can do well) is crucial to math performance – #PISA results 2012
  • 8 – RT @ResearchChat: Prodigy game – Math RPG built on Ontario math curriculum and EQAO http://t.co/nnFgZH7Boz did I mention it is free? #OERS14
  • 8 – RT @Anniekidder: #oers14 OECD research showed it’s crucial that one staff person – not the teacher – is responsible for reaching out to/eng…
  • 8 – RT @ResearchChat: EQAO Research Bulletin – Longitudinal study of mathematics achievement: http://t.co/QulEU2Sz1l #OERS14 #ResearchChatl
  • 7 – RT @KNAER_RECRAE: Looking for education resources in French on various subjects? Check out http://t.co/zlnPXIMNlH #oers14 @OISENews @wester

Identifying Tweets with links:

  • In yet another new column (Column W)  I use the search formula again to identify tweets that contain any instances of “http:” and code it with a 1:  =IF(SEARCH(“http:”, D2),1,0);
  • Going back to the pivot table and refreshing to include the new column, I summarize by the urls from column W and select the 6 urls with the highest frequencies;
  • To make it easier to read (and more useful), I add the full URL and title of the website for the 5 most tweeted links:

Website Title

Full Website

Tweeted URL

Count of url

KNAER Photo of Partnership


http://t.co/YigaizwuSP (6 retweets)

http://t.co/v8Gq4djTs8 (6 retweets)


Prodigy Math Game




EQAO Research Bulletin on Mathematics




KNAER website








Ministry of education FDK Infographic




As with many other areas, the same task can be accomplished in a variety of ways.  I’m always interested in learning more efficient and elegant solutions to these kinds of data requests so if you have any thoughts or suggestions, please share them in the comments.

Retweeted Text

Number of times Retweeted

RT @Anniekidder: #oers14 OECD data show no correlation between hours in school and performance. Quality not quantity. #OntEd


RT @CarolCampbell4: Future PISA tests to include collaborative problem solving & global citizenship #oers14


RT @Anniekidder: #oers14 Dr. Bruce Ferguson says “our classrooms are emergency rooms for the social problems in our society.”


RT @Anniekidder: #oers14 Students of parents who have high expectations report higher perseverance, motivation, confidence and greater enga…


RT @drcathybruce: #OERS14 Student efficacy (belief that they can do well) is crucial to math performance – #PISA results 2012


RT @ResearchChat: Prodigy game – Math RPG built on Ontario math curriculum and EQAO http://t.co/nnFgZH7Boz did I mention it is free? #OERS14


RT @Anniekidder: #oers14 OECD research showed it’s crucial that one staff person – not the teacher – is responsible for reaching out to/eng…


RT @ResearchChat: EQAO Research Bulletin – Longitudinal study of mathematics achievement: http://t.co/QulEU2Sz1l #OERS14 #ResearchChatl


RT @KNAER_RECRAE: Looking for education resources in French on various subjects? Check out http://t.co/zlnPXIMNlH #oers14 @OISENews @wester…


Posted in Blog | Tagged , , , | 2 Comments

In the Edlines…

“Edlines” is a new page that features links for current education research, resources, and discussions.  As new links are added, older links will move further down the page. A link to the Edlines page can be found at the top of the front page.

If you would like to have an education research or resource link featured in the Edlines, please feel free to submit it in the comments section.

Posted in Blog | Leave a comment

Data Vis … The Movie

A few years ago I learned 2 important tricks to navigating a movie theatre:

1)      Ask how full the theatre is when you purchase your ticket.  Finding a single seat is easy, two seats is usually no problem, but when you have a party of 5, finding seats in a theatre that is 80% full becomes an interesting social challenge.

2)      You can get a refund on your ticket if you ask for it before the first 15 minutes of the movie have passed.  This is useful if you find that either the “percentage full” info is misleading or your skills of persuasion aren’t up to the task of getting a group of friends to move further down the aisle.  If you want to see the psychological and sociological nuances of territory at play, a movie theatre is the place to be.

The problem underlying both of these issues is distribution – where people choose to sit at a movie.  Seat selection is far from an arbitrary decision: are you close enough to see?  Too close?  Do you have an empty “buffer-seat” protecting you from the incredibly young child that will be sharing the intense action movie experience with you? Do you need the extra leg room offered by an aisle seat?  Or perhaps the tub of pop was too large and you do not want to bother the entire row when you have to “sneak out” (usually when something important is happening in the story)?  Whatever the reason, people have preferences and will guard their choices (and territory) fiercely.

While on vacation, we were saved from this frustration when we stopped in for a movie at Empire Theatre (Saint John, New Brunswick) where we encountered this:

A visual display of the movie theatre with color coded icons of each seat that was taken.  With the addition of a swiveling flatscreen monitor, a straightforward picture of the seating arrangement helped navigate the decision making.  With the added information offered by this visual presentation, it is now possible to decide whether your group can stay together and, if so, anticipate whether the location is worth the price of admission.

The logistics of the seating process is equally straightforward.  When you purchase your ticket you are asked to select the seats you would like.  The selected seats are then printed on your admission ticket which guards, in some measure, against impromptu seat changing.  When I asked how faithful people were to sit in the seats they chose, the attendee said they had no problems and people work it out between themselves.   Printing the seat number on the ticket seems to eliminate the personal aspect of the request (“would you mind skooching over 2 seats so I can sit with my family?”) and transforms it to a formalized contractual experience (“yes, I would mind, which seats are printed on your ticket?”).

Now the only thing left to round out the movie-going experience would be some kind of signal to let you know when there is a lull in the movie so you could pop out for a minute and return without compromising the story or missing a pivotal moment…

Posted in Data Visualization | Tagged | Leave a comment

The 3,4,5’s of Informed Consent

In the midst of the demands and responsibilities of a new school year, education researchers are immersed in logistical considerations for projects starting (or continuing) in 2012-2013.  An important consideration for every research project is the issue of consent which, in education, is most often discussed in terms of active or passive consent.  Consent demands even more time for consideration in the health sector where it is not only a component of research activities but is also critical to the patient-physician relationship.  In Hall, Prochazka and Fink’s (2012*) article “Informed consent for clinical treatment”, they provide an excellent overview of the components and purpose of informed consent.  Although written from the health care perspective, their reflections and recommendations lead to interesting conversations when you consider them from an education research perspective.

Hall et. al. offer these reminders and warnings with respect to consent:

“The consent form should not be confused with the consent process; the form merely documents that the process has occurred.”

“Unfortunately, pressures for efficient workflow may shift the focus of the informed consent process from robust conversation to the mere requirement of getting a signature.”

“The standardized form best suited for documenting administrative compliance may not be ideally suited for documenting the goals of care or the type of discussion that builds trust.”

The 3 purposes of informed consent

Informed consent serves ethical, legal and administrative purposes:

  • Ethical: support for autonomous decision-making
  • Legal: protection of rights
  • Administrative: documentation of who is involved and that ethical and legal requirements were observed (which serves as a safeguard)

The 4 elements of informed consent

For consent to be truly considered to be “informed”:

  • The decision-maker should have the capacity to make decisions.
  • The decision-maker should be provided with sufficient details necessary for making a choice
  • The decision-maker should show his or her understanding of the disclosed information
  • The decision-maker should freely authorize the plan.

The 5 components to include in a discussion seeking to obtain informed consent

A conversation that results in obtaining informed consent includes a review of:

  • Why action is needed (diagnosis)
  • What action is recommended (treatment)
  • What risks and benefits associated with the action
  • What alternative actions are available (and risks and benefits of those alternatives)
  • What the consequences might be of not acting

Although this article is not available online for free, you can read the discussion that ensued here.   For those without access to the article, the American Cancer Society compares the difference (in health care) between consent for standard treatment vs. consent for participation in a clinical trial in their article “How is informed consent for a clinical trial or research study different from consent for standard treatment?”.

*Hall, D., Prochazka, A., & Fink, A. (2012). Informed consent for clinical treatment. Canadian Medical Association Journal, 20(184), 533–40.

Posted in Research Resources | Leave a comment

Revisiting the Healthcare Slopegraph – A Python implementation

Last summer I shared an adaptation of Tufte’s slopegraph using data from the  Institute for Clinical Evaluative Sciences‘ (ICES) 2011 Quality Monitor report in the post Education and Health Care – Using Slopegraphs to Understand Complex Systems.  While this initial visualization involved post-processing (with the hope of exploring code to generate something similar), Bob Rudis (@hrbrmstr) has developed a Python solution to generating slopegraphs and has used the 2011 Health Care slopegraph to serve as an example.  In his post Slopegraphs in Python – Slope Colors Bob has replicated both the slopes and the color coding, which I had used to visually reinforce the direction of the slopes.

Thanks to Bob for his willingness to share his code and for using my adaptation as one of his examples!

Posted in Data Visualization | Tagged , , , , | Leave a comment

Census 2011: Ontario Population Pyramid

Statistics Canada has released the Age and Sex data set for 2011.  A traditional visualization for this kind of data is the population pyramid.   The population pyramid is a modified version of a stacked bar chart with the division between categories centered around 0.   You may notice that this population pyramid for Ontario, using the 2011 census, looks more like a tree than a pyramid.  One of the reasons for this is that the youngest categories have been divided into 5 year ranges whereas the adult ages are divided into 10 year ranges.

Source: Statistics Canada, 2011 Census of Population, Statistics Canada catalogue no. 98-311-XCB2011017 (Canada, Code01)

These ranges are historical categories that have been used to describe Canadian populations as far back as 1921.
Although the difference between age ranges accounts for the narrow neck of the pyramid, the thicker bands in the older age ranges provides a glimpse of the movement of the baby-boomers through the distribution.  The impact of the baby-boomers on the population distribution becomes even more apparent when the historical data sets are presented as an animated time series:

An interactive population pyramid is also available from Statistics Canada that presents the data across smaller age ranges, giving a more detailed and refined view of the changes.  Where the population pyramid at the beginning of this post was created using data reported by the Census, Statistics Canada’s interactive visualization uses data that is “extrapolated using the annual rates of population growth from the Population Estimates Program.”

Posted in Data Visualization | Tagged , , , | Leave a comment