Getting Started with Reproducible Research in R: What you need and Where to get it

You have been persuaded that engaging in Reproducible Research is worth your time and effort and now you want to get started. This post is a quick overview of the steps you need to take to install LaTeX and make it available to RStudio.

The following steps work as of R version 3.2.2,  RStudio version 0.99.489 and the proTeXt page dated:  2014/04/22 20:51:39

  • Install latest version of R:
  • Install latest version of RStudio:
  • Download proTeXt (MiKTeX based):
    This file is very large (around a Gig) grab a coffee or start the download just before lunch.

    • Run the file and place the files on your Desktop – all of the files will be extracted onto your Desktop
    • In the directory the files were extracted to, click on the TEX Setup icon
      • Click on “Install” for MiKTeX
      • Accept the MiKTeX copying conditions
      • Click “Next”
      • Select “Complete MiKTeX”
      • Select:  “Anyone who uses this computer”
      • Select: preferred paper: “Letter”
      • Select: install missing packages on-the-fly: “Yes”
      • Click on “Next”
      • Click on “Start”
      • Leave the default installation directory and click “Next” – RStudio will look for it in the Program Files
    • Go to RStudio to set up the options:
      • Go to: Tools> Global Options >
      • Select: Weave Rnw files using:knitr
      • Select: Typeset LaTeX into PDF using: XeLaTeX
      • Select: Preview PDF after compile using: “Sumatra
Posted in Blog | Leave a comment

#AERA15 and Ontario School Board Research

The American Educational Research Association’s (AERA) 2015 Annual Meeting will be starting this Thursday, April 16th with more than 14,000 people participating in over 2,600 sessions. This year AERA is being held in Chicago Illinois with the theme

Toward Justice: Culture, Language, and Heritage
in Education Research and Praxis

As with previous years, researchers from Ontario will be sharing their work at this event:

  • (Thursday April 16, 12:00 to 1:30) 14.044 Teachers as Inquirers, Knowledge Generators – and Researchers? OISE, University of Toronto, Peel District School Board and George Brown College.
  • (Friday April 17, 4:05 to 5:35) 35.081 An Innovative University/School-Based Teacher Education Initiative: The Diverse School Initiative. Toronto District School Board.
  • (Saturday April 18, 10:35 to 12:05) 49.060 The Next Phase in Education Reform in Ontario: Excellence, Equity, Well-Being, and Public Confidence.
    • How a Province-Wide Public Consultation is Informing the Next Stage of Ontario’s System-wide Education Improvement Efforts. Ministry of Education
    • Aboriginal Education Evidence-Based Best Practices for Supporting First Nation, Metis, and Inuit Academic Achievement and Well-Being. Ontario Ministry of Education.
    • Changing the Educational Culture of the Home to Increase Student Success at School. OISE/University of Toronto and the Ontario Ministry of Education.
    • Success for All: Using Implementation Science to Build Conditions and Capacity to Support Student Mental Health in Ontario Schools. Ontario Ministry of Education.
  • (Saturday April 18, 2:45 to 4:15) 52.085 Building School District Capacity in Assessment for Learning: A Study on Teacher Learning in Assessment Through an Instructional Rounds Approach. Queen’s University and the Ottawa Carleton District School Board.
  • (Sunday April 19, 12:25 to 1:55) 63.027 The Listening Stone: Learning from the Ontario Ministry of Education’s First Nations, Metis, and Inuit Collaborative Inquiry. York University and the Ontario Ministry of Education.
  • (Sunday April 19, 4:05 to 5:35) 66.080
    • Equity and Accountability in a Major Metropolis: An Exploration of the Toronto District School Board. Toronto District School Board.
    • Deconstructing Intelligence-Based Special Education Exceptionalities. York University and Toronto District School Board.
    • Students’ Experience of Belonging and Exclusion across Toronto. Toronto District School Board.
    • Student, Family, School and Neighborhood Social Disorganization and Stratification and Educational Sorting of Postsecondary Pathways. Toronto District School Board.
    • Post Secondary Aspirations and Choices of Native and foreign-Born Adult Learners in the Toronto District School Board. York University and the Toronto District School Board.
  • (Monday April 20, 12:25 to 1:55) 73.061 Students’ Conscientious Technology Designs as Actions on Socio-scientific Issues. OISE and Peel District School Board.

You can explore the schedule in more detail with AERA’s online portal

If you are unable to attend these events in person, you can follow the discussions at #AERA15 on Twitter. As with previous years, an archive of tweets will be created and shared.


Posted in Blog, Twitter | Leave a comment

#OERS15 – Looking Closer at the Shared Links

Of the 2,654 tweets, 744 (29%) included a link. Of those 744, 204 links were retweeted 540 times.  Taking a closer look at the kinds of links that were posted, photos were the most frequent kind of link shared:

  • PhOERS Link typesoto (155)
  • Website (27)
  • Video (8)
  • Data viz (5)
  • Facebook (3)
  • PDF (3)
  • Spreadsheet (1)
  • OERS Agenda (1)
  • Cartoon (1)

Summarizing this information ended up being a more involved process than I had anticipated.  Before the links could be categorized the duplicates had to be removed, the links needed to be lengthened and the urls parsed.  Following is a description of the process:

  • Original tweets were compiled (retweets and truncated links were removed)
  • Links were lengthened:
  • Lengthened links were categorized according to the url content:
    • links indicate videos
    • Twitter/photo/1 links indicate photos
    • Links ending in .pdf indicate documents
    • links indicate facebook posts
    • Some links needed to be expanded a second time and were fed back into the Bulk URL Checker, lengthened and categorized
    • Remaining links were reviewed and categorized
  • Links and categories were put into a pivot table and summarized (a copy of the excel sheet is available here)
Posted in Twitter | 1 Comment

#OERS15 – Promoting Well-being – Evidence to support Implementation of the Renewed Vision

Last week people from across Ontario came toToronto to attend the 10th annual Ontario Education Research Symposium. Over 400 people from schools, boards, post-secondary institutions, communities and networks met together to learn and share about well-being. The event featured:

  • a fire side chat with Jean Clinton,
  • keynote addresses by Dominic Richardson (UNICEF – Day 2) and Stuart Shanker (Day 3)
  • 34 workshops
  • a provocative speakers series
  • 2 panels
  • poster presentations that were accessible over all three days
  • networking opportunities to learn more about the work of colleagues
  • a group of students sharing their experience as researchers

Following is a quick overview of the tweets, comments on some twitter utilities and a look at a non-traditional approach to twitter analysis. An excel version of the archive is available here (includes some additional summary fields) and the live archive can be accessed here (the archive will be left open for a while to get a sense of how long #OERS15 material circulates).

#OERS15 Summary

As of February 16th:

  • 2,564 tweets from 409 people (an average of 6 tweets per person)
  • 269 people only tweeted once (66% of those who tweeted)
  • 19% of tweets were shared before (4%) and after (15%) OERS 2015
  • 233 tweets (9%) were shared on the first day
  • 1,200 tweets (47%) were shared on the second day
  • 637 (25%) on the third day
  • Top 4 Retweets:
    • 73RT @ResearchChat: Saw this while in Toronto Canada at #OERS15 RT @ShiftParadigm: PLEASE RETWEET —  Dissolving Boundaries @bnighrogain http…
    • 59RT @CarolCampbell4: Students as Researchers! #oers15 @OISENews @KNAER_RECRAE
    • 22RT @CarolCampbell4: Interested in Ontario’s education research, evaluation, data & knowledge mobilization? Follow along #oers15 Tue-Thur @O…
    • 15  –  RT @DrJeanforkids: #oers15 john Dewey said “We don’t learn from experience, we learn from reflecting on experience ” have we learned that?
  • Top 4 Tweeters – Frequency of tweets before, during and after OERS 2015:

OERS15 - Top Tweeters


Visualizations – adding an element of gamification

OERS15 - TAGS tweet

The visualizations offered by TAGS 6.0 and Neoformix give two very different considerations to twitter activity. The visualizations in TAGS 6.0 give a cumulative view of twitter activity. The more you tweet over the course of the entire event, the higher you climb on the top tweeters, top hashtags and top conversationalists summaries.

However, activity on twitter is not uniform. People do not post 1.6 tweets every minute over the course of the entire event. Instead they post in fits and spurts as topics interest them and as their devices continue to be powered. At OERS one power tweeter was pushed off line when her device ran out of power (which gave the rest of us a brief moment of hope we might catch up).

Instead of an entire archive of tweets, the Neoformix Spot application visualizes the last 100 tweets which brings a different perspective to the top tweeters and topics lists. People and topics that might be overshadowed and overlooked in TAGS emerge on the Neoformix list.

Neoformix summary by timeline and by frequency of tweets:

OERS15 - Neoformix Top TweetersOERS15 - Neoformix Timeline

While the intention of twitter is to share information socially, the introduction of visualizations and ranking lists introduces an element of gaming. In this context, participants “win” solely by volume of tweets.  The more you tweet, re-tweet (RT) and modify tweet (MT), the higher you raise through the ranks.  However, nowhere in these visualizations does the quality of the information get addressed.

OERS15 - TweetReachAs I reflected on the reach of the #OERS15 after the symposia, I turned to TweetReach ( to explore how much exposure the tweets received. The image to the left shows the activity of #OERS15 the day after the symposia. Not only has the post-OERS conversation continued, it has reached over 50,000 user accounts and had over 70,000 impressions. It is interesting to see that the tweeters with the most impact after the event are not the same people who were most active during the symposia:

  • @BlessTheTeacher tweeted once and had over 40,000 impressions from that single tweet
  • @researchimpact tweeted twice, retweeted once and had over 11,000 impressions.
  • @ShastaCH tweeted 14 times and had 4,000 impressions
  • @DP_math tweeted 11 times and had over 2,000 impressions

Non-traditional approach

To explore the quality of tweets I used an online app developed by Healey and Ramaswamy of NC State University’s “Sentiment Viz – Tweet Sentiment Visualization” .

This app is easy to use with a single query box to enter a keyword/hashtag to be visualized. Once entered the application searches Twitter for tweets containing the keyword and processes the text according to how it relates to a sentiment dictionary. Each tweet is then placed in a circular grid which has been developed from Russell’s model of emotional affect (

Once visualized, the tweets create a scattergraph give an impression of the emotional content. An example of this graph can be found below which visualizes the tweets (373 tweets) on the last day of the Ontario Education Research Symposium. The dots on the left side that appear to have a negative affect were tweets that were shared during Dr. Shanker’s presentation which addressed the impact of stress on children.Sentiment Analysis Chart - OERS15 Feb 13

Geography of Twitter – still a difficult approach

Although OERS 2015 was held in Toronto I have been wondering about the geographic spread of #OERS15 tweets. Participants come from across the province to attend OERS but what does the tweet distribution look like and how far do the tweets travel beyond those attending?  It would be interesting to create a map showing the geographic distribution of participants that post to #OERS15. There is a utility called TweetsMap ( which will create a map of twitter followers but so far I haven’t been able to find anything that begins with a hashtag and ends with a map of those using it.  If you know of one, please share it in the comments section.

Qualitative Analysis – a call for additional analysis

There is a lot of information contained in the tweet archive and I expect that, like the previous years, I will not not have enough time to dig deeper into the content. For those who are passionate about qualitative analysis, active on Twitter and have the time to dig deeper, it would be interesting to see how the tweets relate not just in terms of content but also styles of use: social interaction (invitations for lunch etc.), note taking (posting points from powerpoint or speaker comments), invitation to discussion (questions to provoke discussion), conversations (responding to the questions), value added material (providing supplemental material to the discussions and resources), and trolling (it’s only logical there would be a few trolling comments).  Perhaps there is a teacher or professor with a group of students learning about qualitative analysis that might take this analysis on? If you do, please let me know I’d be interested to learn about your approach, experience and findings.

Finally, to the organizers of the 2015 Ontario Education Research Symposium, many thanks for the opportunity to attend and connect.

Posted in Data Visualization, Twitter | Leave a comment

6 Forms of Bias That Weaken Your Research

Dr. John Ioannidis’ 2005 article “Why Most Published Research Findings are False” is a provocative reflection on how vulnerable research can be to bias. With citation in over 1,400 papers over the last 9 years ( has inspired and incited a lot of discussion. Although written for the medical community, the observations and concerns have as much application and importance to education as it does to medical research.

Finding what you want to see

Dr. Ioannidis defines bias as a “combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced. Selective or distorted reporting is a typical form of such bias. ” Following are 6 aspects of bias that Ioannidis explores (with examples) in his article:

  1. The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
  2. The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. This is particularly interesting given the controversy over the last year with John Hattie’s work.
  3. The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
  4. The greater the flexibility in designs, definitions, outcomes and analytical modes in a scientific field, the less likely the research findings are to be true.
  5. The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
  6. The hotter the scientific field (with more scientific teams involved), the less likely the research findings are to be true.

To help reflect on whether bias is an issue to be concerned about in your context, here are three questions to consider:

  • How many articles or projects report insignificant findings? Knowing what doesn’t work is as important and valuable as knowing what does work. If there are few (or no) insignificant findings, this could suggest data-mining/dredging (conducting analysis until positive results could be found), negative results were withheld, observations were vulnerable to the Hawthorn effect or observations were vulnerable to confirmation bias.
  • How consistent is the use and implementations of strategies and metrics across studies? Many of these considerations may be modified to align with the culture of a board. While this may strengthen the face validity of a study, it can weaken its generalizability or comparability to other studies. It is Dr. Ioannidis’ contention that the more meaningful findings emerge from research initiatives that are large scale, have multiple independent teams engaged in the inquiry and where consistency is ensured in the design, metrics and analysis.
  • Are there any dissenting or critical voices for strategies or findings? Ioannidis reflects that research in popular fields of study may be more vulnerable to “rapidly alternating extreme research claims and extremely opposite refutations” as research teams build their reputations by promoting their most positive results and use negative results as a challenge to other research teams.
Posted in Blog | Leave a comment