Getting Started with Reproducible Research in R: What you need and Where to get it

You have been persuaded that engaging in Reproducible Research is worth your time and effort and now you want to get started. This post is a quick overview of the steps you need to take to install LaTeX and make it available to RStudio.

The following steps work as of R version 3.2.2,  RStudio version 0.99.489 and the proTeXt page dated:  2014/04/22 20:51:39

  • Install latest version of R:
  • Install latest version of RStudio:
  • Download proTeXt (MiKTeX based):
    This file is very large (around a Gig) grab a coffee or start the download just before lunch.

    • Run the file and place the files on your Desktop – all of the files will be extracted onto your Desktop
    • In the directory the files were extracted to, click on the TEX Setup icon
      • Click on “Install” for MiKTeX
      • Accept the MiKTeX copying conditions
      • Click “Next”
      • Select “Complete MiKTeX”
      • Select:  “Anyone who uses this computer”
      • Select: preferred paper: “Letter”
      • Select: install missing packages on-the-fly: “Yes”
      • Click on “Next”
      • Click on “Start”
      • Leave the default installation directory and click “Next” – RStudio will look for it in the Program Files
    • Go to RStudio to set up the options:
      • Go to: Tools> Global Options >
      • Select: Weave Rnw files using:knitr
      • Select: Typeset LaTeX into PDF using: XeLaTeX
      • Select: Preview PDF after compile using: “Sumatra
Posted in Blog | Leave a comment

#AERA15 and Ontario School Board Research

The American Educational Research Association’s (AERA) 2015 Annual Meeting will be starting this Thursday, April 16th with more than 14,000 people participating in over 2,600 sessions. This year AERA is being held in Chicago Illinois with the theme

Toward Justice: Culture, Language, and Heritage
in Education Research and Praxis

As with previous years, researchers from Ontario will be sharing their work at this event:

  • (Thursday April 16, 12:00 to 1:30) 14.044 Teachers as Inquirers, Knowledge Generators – and Researchers? OISE, University of Toronto, Peel District School Board and George Brown College.
  • (Friday April 17, 4:05 to 5:35) 35.081 An Innovative University/School-Based Teacher Education Initiative: The Diverse School Initiative. Toronto District School Board.
  • (Saturday April 18, 10:35 to 12:05) 49.060 The Next Phase in Education Reform in Ontario: Excellence, Equity, Well-Being, and Public Confidence.
    • How a Province-Wide Public Consultation is Informing the Next Stage of Ontario’s System-wide Education Improvement Efforts. Ministry of Education
    • Aboriginal Education Evidence-Based Best Practices for Supporting First Nation, Metis, and Inuit Academic Achievement and Well-Being. Ontario Ministry of Education.
    • Changing the Educational Culture of the Home to Increase Student Success at School. OISE/University of Toronto and the Ontario Ministry of Education.
    • Success for All: Using Implementation Science to Build Conditions and Capacity to Support Student Mental Health in Ontario Schools. Ontario Ministry of Education.
  • (Saturday April 18, 2:45 to 4:15) 52.085 Building School District Capacity in Assessment for Learning: A Study on Teacher Learning in Assessment Through an Instructional Rounds Approach. Queen’s University and the Ottawa Carleton District School Board.
  • (Sunday April 19, 12:25 to 1:55) 63.027 The Listening Stone: Learning from the Ontario Ministry of Education’s First Nations, Metis, and Inuit Collaborative Inquiry. York University and the Ontario Ministry of Education.
  • (Sunday April 19, 4:05 to 5:35) 66.080
    • Equity and Accountability in a Major Metropolis: An Exploration of the Toronto District School Board. Toronto District School Board.
    • Deconstructing Intelligence-Based Special Education Exceptionalities. York University and Toronto District School Board.
    • Students’ Experience of Belonging and Exclusion across Toronto. Toronto District School Board.
    • Student, Family, School and Neighborhood Social Disorganization and Stratification and Educational Sorting of Postsecondary Pathways. Toronto District School Board.
    • Post Secondary Aspirations and Choices of Native and foreign-Born Adult Learners in the Toronto District School Board. York University and the Toronto District School Board.
  • (Monday April 20, 12:25 to 1:55) 73.061 Students’ Conscientious Technology Designs as Actions on Socio-scientific Issues. OISE and Peel District School Board.

You can explore the schedule in more detail with AERA’s online portal

If you are unable to attend these events in person, you can follow the discussions at #AERA15 on Twitter. As with previous years, an archive of tweets will be created and shared.


Posted in Blog, Twitter | Leave a comment

#OERS15 – Looking Closer at the Shared Links

Of the 2,654 tweets, 744 (29%) included a link. Of those 744, 204 links were retweeted 540 times.  Taking a closer look at the kinds of links that were posted, photos were the most frequent kind of link shared:

  • PhOERS Link typesoto (155)
  • Website (27)
  • Video (8)
  • Data viz (5)
  • Facebook (3)
  • PDF (3)
  • Spreadsheet (1)
  • OERS Agenda (1)
  • Cartoon (1)

Summarizing this information ended up being a more involved process than I had anticipated.  Before the links could be categorized the duplicates had to be removed, the links needed to be lengthened and the urls parsed.  Following is a description of the process:

  • Original tweets were compiled (retweets and truncated links were removed)
  • Links were lengthened:
  • Lengthened links were categorized according to the url content:
    • links indicate videos
    • Twitter/photo/1 links indicate photos
    • Links ending in .pdf indicate documents
    • links indicate facebook posts
    • Some links needed to be expanded a second time and were fed back into the Bulk URL Checker, lengthened and categorized
    • Remaining links were reviewed and categorized
  • Links and categories were put into a pivot table and summarized (a copy of the excel sheet is available here)
Posted in Twitter | 1 Comment

#OERS15 – Promoting Well-being – Evidence to support Implementation of the Renewed Vision

Last week people from across Ontario came toToronto to attend the 10th annual Ontario Education Research Symposium. Over 400 people from schools, boards, post-secondary institutions, communities and networks met together to learn and share about well-being. The event featured:

  • a fire side chat with Jean Clinton,
  • keynote addresses by Dominic Richardson (UNICEF – Day 2) and Stuart Shanker (Day 3)
  • 34 workshops
  • a provocative speakers series
  • 2 panels
  • poster presentations that were accessible over all three days
  • networking opportunities to learn more about the work of colleagues
  • a group of students sharing their experience as researchers

Following is a quick overview of the tweets, comments on some twitter utilities and a look at a non-traditional approach to twitter analysis. An excel version of the archive is available here (includes some additional summary fields) and the live archive can be accessed here (the archive will be left open for a while to get a sense of how long #OERS15 material circulates).

#OERS15 Summary

As of February 16th:

  • 2,564 tweets from 409 people (an average of 6 tweets per person)
  • 269 people only tweeted once (66% of those who tweeted)
  • 19% of tweets were shared before (4%) and after (15%) OERS 2015
  • 233 tweets (9%) were shared on the first day
  • 1,200 tweets (47%) were shared on the second day
  • 637 (25%) on the third day
  • Top 4 Retweets:
    • 73RT @ResearchChat: Saw this while in Toronto Canada at #OERS15 RT @ShiftParadigm: PLEASE RETWEET —  Dissolving Boundaries @bnighrogain http…
    • 59RT @CarolCampbell4: Students as Researchers! #oers15 @OISENews @KNAER_RECRAE
    • 22RT @CarolCampbell4: Interested in Ontario’s education research, evaluation, data & knowledge mobilization? Follow along #oers15 Tue-Thur @O…
    • 15  –  RT @DrJeanforkids: #oers15 john Dewey said “We don’t learn from experience, we learn from reflecting on experience ” have we learned that?
  • Top 4 Tweeters – Frequency of tweets before, during and after OERS 2015:

OERS15 - Top Tweeters


Visualizations – adding an element of gamification

OERS15 - TAGS tweet

The visualizations offered by TAGS 6.0 and Neoformix give two very different considerations to twitter activity. The visualizations in TAGS 6.0 give a cumulative view of twitter activity. The more you tweet over the course of the entire event, the higher you climb on the top tweeters, top hashtags and top conversationalists summaries.

However, activity on twitter is not uniform. People do not post 1.6 tweets every minute over the course of the entire event. Instead they post in fits and spurts as topics interest them and as their devices continue to be powered. At OERS one power tweeter was pushed off line when her device ran out of power (which gave the rest of us a brief moment of hope we might catch up).

Instead of an entire archive of tweets, the Neoformix Spot application visualizes the last 100 tweets which brings a different perspective to the top tweeters and topics lists. People and topics that might be overshadowed and overlooked in TAGS emerge on the Neoformix list.

Neoformix summary by timeline and by frequency of tweets:

OERS15 - Neoformix Top TweetersOERS15 - Neoformix Timeline

While the intention of twitter is to share information socially, the introduction of visualizations and ranking lists introduces an element of gaming. In this context, participants “win” solely by volume of tweets.  The more you tweet, re-tweet (RT) and modify tweet (MT), the higher you raise through the ranks.  However, nowhere in these visualizations does the quality of the information get addressed.

OERS15 - TweetReachAs I reflected on the reach of the #OERS15 after the symposia, I turned to TweetReach ( to explore how much exposure the tweets received. The image to the left shows the activity of #OERS15 the day after the symposia. Not only has the post-OERS conversation continued, it has reached over 50,000 user accounts and had over 70,000 impressions. It is interesting to see that the tweeters with the most impact after the event are not the same people who were most active during the symposia:

  • @BlessTheTeacher tweeted once and had over 40,000 impressions from that single tweet
  • @researchimpact tweeted twice, retweeted once and had over 11,000 impressions.
  • @ShastaCH tweeted 14 times and had 4,000 impressions
  • @DP_math tweeted 11 times and had over 2,000 impressions

Non-traditional approach

To explore the quality of tweets I used an online app developed by Healey and Ramaswamy of NC State University’s “Sentiment Viz – Tweet Sentiment Visualization” .

This app is easy to use with a single query box to enter a keyword/hashtag to be visualized. Once entered the application searches Twitter for tweets containing the keyword and processes the text according to how it relates to a sentiment dictionary. Each tweet is then placed in a circular grid which has been developed from Russell’s model of emotional affect (

Once visualized, the tweets create a scattergraph give an impression of the emotional content. An example of this graph can be found below which visualizes the tweets (373 tweets) on the last day of the Ontario Education Research Symposium. The dots on the left side that appear to have a negative affect were tweets that were shared during Dr. Shanker’s presentation which addressed the impact of stress on children.Sentiment Analysis Chart - OERS15 Feb 13

Geography of Twitter – still a difficult approach

Although OERS 2015 was held in Toronto I have been wondering about the geographic spread of #OERS15 tweets. Participants come from across the province to attend OERS but what does the tweet distribution look like and how far do the tweets travel beyond those attending?  It would be interesting to create a map showing the geographic distribution of participants that post to #OERS15. There is a utility called TweetsMap ( which will create a map of twitter followers but so far I haven’t been able to find anything that begins with a hashtag and ends with a map of those using it.  If you know of one, please share it in the comments section.

Qualitative Analysis – a call for additional analysis

There is a lot of information contained in the tweet archive and I expect that, like the previous years, I will not not have enough time to dig deeper into the content. For those who are passionate about qualitative analysis, active on Twitter and have the time to dig deeper, it would be interesting to see how the tweets relate not just in terms of content but also styles of use: social interaction (invitations for lunch etc.), note taking (posting points from powerpoint or speaker comments), invitation to discussion (questions to provoke discussion), conversations (responding to the questions), value added material (providing supplemental material to the discussions and resources), and trolling (it’s only logical there would be a few trolling comments).  Perhaps there is a teacher or professor with a group of students learning about qualitative analysis that might take this analysis on? If you do, please let me know I’d be interested to learn about your approach, experience and findings.

Finally, to the organizers of the 2015 Ontario Education Research Symposium, many thanks for the opportunity to attend and connect.

Posted in Data Visualization, Twitter | Leave a comment

6 Forms of Bias That Weaken Your Research

Dr. John Ioannidis’ 2005 article “Why Most Published Research Findings are False” is a provocative reflection on how vulnerable research can be to bias. With citation in over 1,400 papers over the last 9 years ( has inspired and incited a lot of discussion. Although written for the medical community, the observations and concerns have as much application and importance to education as it does to medical research.

Finding what you want to see

Dr. Ioannidis defines bias as a “combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced. Selective or distorted reporting is a typical form of such bias. ” Following are 6 aspects of bias that Ioannidis explores (with examples) in his article:

  1. The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
  2. The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. This is particularly interesting given the controversy over the last year with John Hattie’s work.
  3. The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
  4. The greater the flexibility in designs, definitions, outcomes and analytical modes in a scientific field, the less likely the research findings are to be true.
  5. The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
  6. The hotter the scientific field (with more scientific teams involved), the less likely the research findings are to be true.

To help reflect on whether bias is an issue to be concerned about in your context, here are three questions to consider:

  • How many articles or projects report insignificant findings? Knowing what doesn’t work is as important and valuable as knowing what does work. If there are few (or no) insignificant findings, this could suggest data-mining/dredging (conducting analysis until positive results could be found), negative results were withheld, observations were vulnerable to the Hawthorn effect or observations were vulnerable to confirmation bias.
  • How consistent is the use and implementations of strategies and metrics across studies? Many of these considerations may be modified to align with the culture of a board. While this may strengthen the face validity of a study, it can weaken its generalizability or comparability to other studies. It is Dr. Ioannidis’ contention that the more meaningful findings emerge from research initiatives that are large scale, have multiple independent teams engaged in the inquiry and where consistency is ensured in the design, metrics and analysis.
  • Are there any dissenting or critical voices for strategies or findings? Ioannidis reflects that research in popular fields of study may be more vulnerable to “rapidly alternating extreme research claims and extremely opposite refutations” as research teams build their reputations by promoting their most positive results and use negative results as a challenge to other research teams.
Posted in Blog | Leave a comment

8 Conditions When Research is Put Into Practice: More Lessons From Health Care

In Health Care putting research into practice can be high stakes. Changing routines and practices as a result of new findings often has direct impacts on the health, treatment or recovery of patients. A significant contribution to the study of changing practices was made in 2003 by Richard Grol and Jeremy Grimshaw who published the study:

From best evidence to best practice: effective implementation of change in patients’ care”,  The Lancet, Vol 362, October 11, 2003.

Keeping up with the pace of research

The speed of medical research provided Grol and Grimshaw with an interesting opportunity to explore the intersection of research and practice. They found that although syntheses helped physicians stay current with findings and saved time from having to read the original research papers, physicians still had difficulty keeping pace with the rapid advances in health care knowledge. In addition to physician factors, the environment (attitudes and approaches of colleagues) also played a role in evidence impacting action.

The 8 conditions research was most often put into practice was when:

  1. You don’t think the current practice works: Physicians quickly changed practice when there was existing scepticism of the benefits of an established practice (review of treatment of otitis media).
  2. It’s an easy problem to address: physicians were more likely to change practices for acute care issues than for chronic care
  3. You feel the findings are more believable and compelling (better quality of evidence)
  4. The new practice matches current values
  5. The new practice makes your life easier: “less complexity of decision-making”
  6. You have a clearer understanding of what needs to be done: “more concrete description of the desired performance”
  7. The new recommendations don’t require a lot of new skills or changes to the organization
  8. The new practice is targeted at specific obstacles to change.

Although Grol and Grimshaw were unable to find a single strategy for knowledge mobilization that was effective in every setting and condition, they provided a very useful and thought-provoking summary of strategies with respect to impact.

Effectiveness of Knowledge Mobilization Strategies:

  • Limited Effects
    • Total quality management/continuous quality improvement (1 review, 55 studies)
  •  Mixed Effects
    • Educational materials (9 reviews, 3-37 studies)
    • Conferences, courses (4 reviews, 3-17 studies)
    • Use of opinion leaders (3 reviews, 3-6 studies)
    • Education with different educational strategies (8 reviews, 5-63 studies)
    • Feedback on performance (16 reviews, 3-37 studies)
  • Mostly Effective
    • Reminders (14 reviews, 4-68 studies)
    • Computerised decision support (5 reviews, 11-98 studies)
    • Introduction of computers in practice (2 reviews, 19-30 studies)
    • Mass media campaigns (1 review, 22 studies
    • Interactive small group meetings (4 reviews, 2-6 studies)

A further review of the strategies found:

  • Education and information had short-term effects
  • Reminders had modest and sustained effects
  • Performance feedback is effective but ceases if feedback is not continued.

Does the effectiveness of these strategies surprise you?

Would you have expected reminders to be more effective than opinion leaders, feedback, conferences or educational materials? One thing that shouldn’t be surprising is that the more strategies you employ, the more likely it will be effective. Grol and Grimshaw reflected that multifaceted interventions had “pronounced effects on practice and outcomes” involving the combined use of education, written materials, feedback and reminders.

It remains to be seen whether these strategies would result in similar levels of effectiveness in an education context but it provides an interesting look at knowledge mobilization efforts. The question now to consider is, how effective are the strategies you are using?

Posted in Research Resources | Tagged , | Leave a comment

A look at Twitter and #OERS14

This was the ninth year for the Ontario Education Research Symposium which is organized by the Ontario Education Research Panel.  As with previous years it was a wonderful venue to share research, explore new collaborations and expand networking opportunities.  Throughout the conference, attendees reflected and reacted to the keynote speakers and workshop presenters on twitter (#OERS14).  As with previous years, I used TAGS v5.1, a spreadsheet utility developed in Google Drive by Martin Hawksey, to collect and archive the conference tweets.

Over the course of the conference I had several people ask how I work with the twitter feeds. So, for those who are interested, here are the steps, premises and limitations of how I look at tweets: But first, some stats:

177 people engaged in #OERS14 through twitter with 1,280 tweets and 271 links shared.  This year, the top 5 tweeters were:

  • @abbaspeaks (100)
  • @ResearchChat (95)
  • @O3atORION (93)
  • @CarolCambell4 (76)
  • @KNAER_RECRAE (63)

Many thanks to everyone who contributed to the discussions.

If you would like to take a closer look at the comments and resources that were shared the tweets are available as a pdf, an excel sheet and an online link:

  • Click here for a pdf of the tweets (with active links)
  • Click here for the excel sheet that is produced by TAGS v5.1
  • Click here to see what the google spreadsheet looks like.

The following steps can be seen in the excel sheet linked above.

Identifying retweets:

Working on the premise that the letters “RT” followed by a space signifies a retweet:

  • In a new column (in this case, column T)  I use the excel formula @LEFT(D2,3) to return the first three characters of each tweet.  This formula is copied in every cell in the column;
  • In the next column (column U) I use the formula =IF(SEARCH(“RT”, T2),1,0) to evaluate the three characters that are returned in column T and then return a 1 if it contains “RT “  and a null value if it doesn’t;
  • Using a filter I can sort on column U and take a look at the retweets.

Limitations: this won’t pick up retweets that are embedded further into a tweet, for example the tweet “I love this!  RT You won’t believe what they said” would be excluded because the RT occurs further into the text.

Identifying the most retweeted comment:

  • The approach is similar to the previous method.  In a new column (column V) @Left(cell, 30) is used to identify the first 30 characters of a tweet;
  • A pivot table is created using the RT codes and the Tweet stems from column V.  The Pivot table is selected and then sorted by the Counts.  The 9 or 10 tweets with the highest counts are selected and the full tweets are compiled from the original list.

Limitations: if there are slight variations in text further into the tweet it isn’t picked up or distinguished.  For example my tweet about the EQAO Research Bulletin had the most retweets for the math bulletin (8) than the literacy bulletin.  I manually changed this as I compiled the tweets.

Number of Retweets – Tweet:

  • 12 – RT @Anniekidder: #oers14 OECD data show no correlation between hours in school and performance. Quality not quantity. #OntEd
  • 12 – RT @CarolCampbell4: Future PISA tests to include collaborative problem solving & global citizenship #oers14
  • 12 – RT @Anniekidder: #oers14 Dr. Bruce Ferguson says “our classrooms are emergency rooms for the social problems in our society.”
  • 10 – RT @Anniekidder: #oers14 Students of parents who have high expectations report higher perseverance, motivation, confidence and greater enga…
  • 10 – RT @drcathybruce: #OERS14 Student efficacy (belief that they can do well) is crucial to math performance – #PISA results 2012
  • 8 – RT @ResearchChat: Prodigy game – Math RPG built on Ontario math curriculum and EQAO did I mention it is free? #OERS14
  • 8 – RT @Anniekidder: #oers14 OECD research showed it’s crucial that one staff person – not the teacher – is responsible for reaching out to/eng…
  • 8 – RT @ResearchChat: EQAO Research Bulletin – Longitudinal study of mathematics achievement: #OERS14 #ResearchChatl
  • 7 – RT @KNAER_RECRAE: Looking for education resources in French on various subjects? Check out #oers14 @OISENews @wester

Identifying Tweets with links:

  • In yet another new column (Column W)  I use the search formula again to identify tweets that contain any instances of “http:” and code it with a 1:  =IF(SEARCH(“http:”, D2),1,0);
  • Going back to the pivot table and refreshing to include the new column, I summarize by the urls from column W and select the 6 urls with the highest frequencies;
  • To make it easier to read (and more useful), I add the full URL and title of the website for the 5 most tweeted links:

Website Title

Full Website

Tweeted URL

Count of url

KNAER Photo of Partnership (6 retweets) (6 retweets)


Prodigy Math Game


EQAO Research Bulletin on Mathematics


KNAER website




Ministry of education FDK Infographic


As with many other areas, the same task can be accomplished in a variety of ways.  I’m always interested in learning more efficient and elegant solutions to these kinds of data requests so if you have any thoughts or suggestions, please share them in the comments.

Retweeted Text

Number of times Retweeted

RT @Anniekidder: #oers14 OECD data show no correlation between hours in school and performance. Quality not quantity. #OntEd


RT @CarolCampbell4: Future PISA tests to include collaborative problem solving & global citizenship #oers14


RT @Anniekidder: #oers14 Dr. Bruce Ferguson says “our classrooms are emergency rooms for the social problems in our society.”


RT @Anniekidder: #oers14 Students of parents who have high expectations report higher perseverance, motivation, confidence and greater enga…


RT @drcathybruce: #OERS14 Student efficacy (belief that they can do well) is crucial to math performance – #PISA results 2012


RT @ResearchChat: Prodigy game – Math RPG built on Ontario math curriculum and EQAO did I mention it is free? #OERS14


RT @Anniekidder: #oers14 OECD research showed it’s crucial that one staff person – not the teacher – is responsible for reaching out to/eng…


RT @ResearchChat: EQAO Research Bulletin – Longitudinal study of mathematics achievement: #OERS14 #ResearchChatl


RT @KNAER_RECRAE: Looking for education resources in French on various subjects? Check out #oers14 @OISENews @wester…


Posted in Blog | Tagged , , , | 2 Comments