Online Annotations are the new Sticky Notes

Lu, J., & Deng, L. (2012). Reading actively online: An exploratory investigation of online annotation tools for learning inquiry learning. Canadian Journal of Learning and Technology. 38(3), 1-16.



Critical thinking is a difficult concept and students need to learn the skills necessary to accomplish that.  My research hopes to incorporate technology, critical thinking, and sound pedagogy in order to help students achieve the maximum benefits when learning.  This research study looks at how a specific piece of technology can be used to help students engage in critical thinking skills during reading.  The research was conducted in Hong Kong with students who were the equivalent of tenth graders in the United States.  Research was evaluated in two categories: a review of pedagogical annotations and a review of currently available annotating web programs.  The literature shows that the more annotations readers take, the more they increase their comprehension.  This is true for both for the frequency and quality of those notes.  The annotation process is helpful when done either individually or collaboratively and this information was factored into the study.  There were five online annotation tools available.  The literature detailed the differences between them and then explained its rationale for choosing Diigo (Digest of Internet Information, Groups and Other stuff).  Diigo provided a few features that allowed students to interact with the text in ways that some of the others did not. The authors believed it was best suited to support the critical thinking process.


The study consisted of two classes of students.  One class was an advanced level class.  That class began with 44 students but the study only assessed 42 students after accounting for factors such as excessive absences.  The other group of students was a class of regular education students.  That class began with 37 students but only accounted for 27 once also weighing for absences.  The differences between the two types of classes was purposeful.  One question the researchers examined was the difference between how the two groups’ behaviors and observations related to Diigo.  Other goals of the study were to find out how all of the students used the technology, perceived it, and how their actual use of it compared to their reported responses.  The research was conducted in four sessions.  First, the teacher gave the students material to read.  The next session required the use of Diigo.  On the third session, students worked independently and could take notes or interact with the text however they chose.  The fourth session was for students to share their notes with each other and synthesize information.


The researchers individually observed and assessed each note card entered it into the Diigo system.  Calculations were based on those results.  The notes were categorized into four sections: define, tag, record, and discuss.  A Likert scale assessment was also used to measure the students’ opinions.  MANOVA tests were performed.   Scores were adjusted to account for the differences in the number of students in the two classes. The results showed that Class A used, and reported liking, the sticky notes more.  They also used the define category the most.  Both groups reported enjoying the notes according to the survey results, however, Class B had so few notes that some categories could not be fully assessed.



Strengths and Critiques

This research compares two classes one of which had high achieving students.  One of the big problems is that the tool being assessed involves students’ ability to read critically.  The reason that Class A may have had more notes in Diigo may have had nothing to do with the product or its effectiveness but everything to do with the students’ skill in Class B to complete the assignment.  The research did not detail the reading level of the material presented to the classes nor did it specify if it was the same passage for both groups.  There were a few other problems as well.  Class A and Class B were very small making it unable to be generalizable even if they were both the same type of learners.  Another problem is that one class lost two students while another class lost ten.  There might be some dynamic or secondary issue going on (behavior, illness, etc.) that had an effect on the remainder of the students which could, in turn, effect the results.  The authors didn’t address that issue.  Finally, the design called for students to be grouped by their teachers.  Again, since both classes started out with larger numbers and ended smaller, the research did not explain how groupings were changed during the project as absences occurred.  Since Group B lost ten students and Group A only lost 2 that might have been another factor.


The researchers evaluated each note card themselves.  That left the opportunity for interpretation of the cards.  There was no independent party also evaluating the messages on the sticky notes so the breakdown of data could have been construed differently had someone without a bias been the arbiter.


There literature review was broken into two sections.  The first section evaluated the pedagogy value behind annotations.  The second section was not a review of literature at all.  Rather, it was a review of the products that are currently on the internet and available.  It explained to the readers the rationale behind the choice of using Diigo as the source for this research.


The layout of the paper was fine, however, the one typographical error was very noticeable and did create difficulty when reading.  At the end of the Research Question section, it stated that there were three questions they would be focusing on but then proceeded to list four questions.  I reread that several times as I was initially unsure if the mistake was that the three should have been a four (which is what I concluded) or if one of the questions on the list was the error. That mistake was very confusing and distracting.


Further Study

I think that the researchers had a very valuable idea by choosing to research how a specific piece of technology can assist in building students’ reading skills.  In order to extend beyond this study, the research needs to be repeated with several changes.  First, more students need to be involved.  Second, consistent academic levels need to be considered. Once this study has been repeated accurately, other studies also can be done to compare the other technology options that were presented at the beginning of the literature review section.  One more direction that this study could be taken is to compare the use of the online annotations to traditional annotations with paper and pencil or sticky notes. This study compared higher level students with average students but on a very small scale.  The next study could be done on a large level with groups of students at both high and average academic levels which would make the results generalizable.  The reading passages each group gets could match their abilities.  The results could be compared to each other after the fact thereby eliminating that variable as a factor.


Relate to Another Reading

The literature review and discussions in Effects of Technology on Critical Thinking and Essay Writing Among Gifted Adolescents (Dixon, F., Cassady, J., Cross, T., & Williams, D., 2005) argued that little research has been explored specifically regarding how gifted students learn.  Both studies look at how technology is used by high achieving students.  Neither study used large enough groups to make the results generalizable so neither were able to contribute much to the overall development of literature of the way gifted students acquire knowledge.


Brainstorms for My Area of Interest

This study had two of the three components that I am looking to use in my study; the technology and the critical reading development.  The pedagogy base was discussed in the literature review but not analyzed in the research so I do not consider it as fully part of the research.  I was completely unaware that this type of annotating technology existed until I read this research.  This seems like a simple, free, easily accessible piece of software.  The literature section described several of the options available and provided the sites to access them.  What it made me realize is just how much may be obtainable for my research that I have not even thought about.  I recognize now that I may have been limiting my options.  I am going to begin to trolling through many places to explore what else may be possible before I narrow my research decisions.



Dixon, F., Cassady, J., Cross, T., & Williams, D. (2005). Effects of technology on critical thinking and essay writing among gifted adolescents. The Journal of secondary gifted education, 16(4). 180-189.


Boys vs Girls vs Computers

Dixon, F., Cassady, J., Cross, T. & Williams, D. (2005). Effects of technology on critical thinking and essay writing among gifted adolescents.  The Journal of secondary gifted education, 16(4), 180-189.


Article Summary

My area of interest is critical thinking, technology, and pedagogy.  The pedagogy I hope to focus on is Bloom’s Taxonomy.  I want to create a study where all three meet in my fifth grade classroom.  This study interested me because it is about critical thinking and technology and part of the assessment that they used to measure the critical thinking focuses on analysis, synthesis, and evaluation which are the higher level of Bloom’s order so this research fits well with my classroom goal. The technology used in this study is trying to determine if a computer can help students become better critical thinkers when they write.  It is also trying to determine if the students’ gender makes any difference in the outcome.   Researchers decided to determine if technology would have an impact on the writing and critical thinking of gifted high school students.  The research was conducted at a residential, gifted high school.  99 incoming juniors wrote an essay.  39 of those students were males and 60 were females, all were sixteen years old.   One year later, the students were incoming seniors and they wrote a second essay.  The prompts for each essay were based off of the same poem.  Directions given for both essays were identical.  This second time the students wrote their essay, they were randomly assigned to one of two groups.  One group wrote the essay by handwriting it as they had done the first time and the second group used a computer to compose their thoughts.  These essays were also scored by the same two people using the same rubric they used the year prior.   The essays measured critical thinking using a five point scale.  The five point assessment measured the critical thinking skills of analysis, synthesis, and evaluation and did not focus on the mechanics of writing.  To score those, two people were brought in and trained until an interrater reliability was established.  A second critical thinking assessment was also used called the Watson-Glaser Critical Thinking Appraisal.  That was an 80 question test that measured critical thinking a different way than the essays.  The results compared the two scores.   There were two sets of results that this study examined.  One was the comparison of the critical thinking scores and the basic writing indicators.  As far as the first comparison, the conclusions of the study led to the analysis that critical thinking was significantly related.  The other evaluated if gender and the computers were interconnected which was the primary focus of the study.  “A…2 (male, female) by 2 (word process, handwrite) repeated measures multivariate analysis of variance was employed, examining four dependent variables at two points in time (WS-1, WS-2). That revealed statistically significant main effects for gender, method of writing at WS-2, and the repeated factor (time)” (Dixon, F., Cassady, J., Cross, T. & Williams, D., 2005 p.185).  The study found that boys did better using the computer.  It found no difference for girls.



Strengths and Critiques

This study had several limitations that were not addressed.  First, the sample size was very small which makes the results not generalizable.  Not only does it begin with 99 students, but it looks at the results based on gender and the gender is not split evenly to start with so only 39 boys are part of the study out of the 99 total.  That is before dividing the students for the second half of the study.  The researchers also said that the students were randomly assigned to write either by hand or on the computer.  What they did not specify is if the calculations were purposefully made so that the students were evenly divided down the middle or if the random assignments were made by gender.  If they were not, then there is no way of knowing how many male students used the computer and how many completed their second essay by hand. In that case it is possible the results could be very skewed.  Even if the students were evenly divided by gender, that would have left only 19 boys in one of the groups and 20 in the other which is a very small group.   Another issue examine is that this research specifically states that it is done with gifted students.  The authors cite the lack of research in the field of gifted education.  Doing research specifically for the gifted community is a valuable contribution to the field.  However, in addition to repeating this study within the gifted community, because of the small sample size, it might also be a good idea to have another study in which this is attempted in the non-gifted population as well to see what those results show.  A additional study with special education students might also prove worthwhile.


One more issue that was not addressed was the amount of word processing skills or familiarity that any of the students had with the computers.  We don’t know how often these students had access to the computers they used for the second essay and if that could account for any of the disparity.  If, by chance, the males had access to the computers more often that might be a contributing factor to their increased scores.  We also don’t know if they were ever given word processing classes, how often, if they had the same access to it as the females, etc.  Other unknown factors that could have impacted the study were the students’ connection to or interest it either of the prompts or to the poem.


The overall organization of the article was good and the literature study was detailed.  Several articles were given to support the connection between critical thinking and writing as well as articles supporting the use of computers to assist in writing.  There were no editorial errors.



Connecting to Past Research

As a classroom teacher who teaches writing, I think that this study has some interesting promise.  Writing fluency is an important component to being a competent writer.  More research needs to be done to explore the effects of using the tools that are available.  Not only should larger sample sizes be used but other types of research could be explored.  For example, what types of hardware (tablets, personal computers, etc.) would help versus hinder the writing process.  Will word processing programs get in the way of students’ writing because they become too encumbered with the minutia of spell-checking and editing rather than focus on the bigger picture of concepts? If technology is available and can help students with the writing process then it is definitely something that should be explored.  If it is something that is prone to help one gender more than another, that is worth examining as well.  If the findings from this research that males are able to write significantly more fluently by using computers is proven valid then the ramifications could have an enormous impact on the way we help students learn.  The technology is readily available in many classrooms and if the expectations become that we allow boys the access to do their writing on technology, and as a result, they become better able to communicate their thoughts, it could have a great impact on their ability to reach academic excellence.


Furthering My Area of Interest

This connects to my research in that it opens my mind to the type of technology I might use with my students.  This study made me realize that I don’t have to use something extravagant in order to be effective and have an impact on student learning.  I had been searching for a specific technology “tool” to use but it is now something I am starting to rethink.  In one of the research articles I read, they named a precise Microsoft program (Hubbard, J. D. & Price, G. 2013) that they used. After reading this research, I am reconsidering the direction to take. My goal is to use technology to help students become better critical thinkers based on sound pedagogy.  The background piece for this research began by explaining that laptops are in students’ hands every day which is why they chose to do this study.  The impact of their study alone, even with its issues, has made me consider letting some of the students in my classroom use computers to see if it will help them become more fluent writers.  This has made me contemplate that when I choose my study it would be more helpful to my students, and to anyone who reads my study, to have technology that is a part of their daily live rather than an isolated piece.



Hubbard, J. D. & Price, G. (2013). Cross-culture and technology integration: Journal of the Research Center for Educational Technology (RECT) 9(1).

Active Learning in Health Professions

McLaughlin, J. E., Roth, M. T., Glatt, D. M., Gharkholonarehe, N., Davidson, C. A, Griffin, L. M., Esserman, Denise A., Mumper, R. J. (2014). The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Academic Medicine : Journal of the Association of American Medical Colleges, 89(2), 236–43.

The article, The Flipped Classroom: A Course Redesign to Foster Learning and Engagement in Health Professions School (McLaughlin, J. E., Roth, M. T., Glatt, D. M., Gharkholonarehe, N., Davidson, C. A, Griffin, L. M., Esserman, Denise A. & Mumper, R. J., 2014) is about how the University of North Carolina (UNC) Eshelman School of Pharmacy redesigned the course, Basic Pharmaceuticals II, using a flipped classroom model. The course redesign was “inspired by a desire to transform the educational experiences of our students and to meet students’ requests for enhanced in-class active learning experiences” (p. 237). The article discussed what changes were implemented in the course redesign; they include, replacing in-class lectures with on-line videos to watch outside of class and then spending valuable class time on active learning exercises. The three main element focal points include, offloaded content (recorded videos, etc.), student centered-learning and appropriate assessments.

In trying to determine if implementing a flipped-classroom model would be effective the researchers obtained approval from the UNC institutional review board in order to administer pre- and post-surveys regarding demographic information, students’ perceptions of active learning activities, preferred curriculum delivery format and engagement. In addition, they collected data on exam scores and additional assessment tools and compared the outcomes of traditional classroom format (class of 2011) to those that participated in the flipped classroom format (class of 2012). The overall findings validated that overall student learning increased after participating in a flipped-classroom format.

Review of Strengths and Contributions

Organization – The organization of this article was well constructed. I particularly valued that the authors compared the traditional lecture and course design to the newly implemented student-centered pedagogy.

Contribution to Field – The authors acknowledge that there have been many significant changes to how healthcare is delivered and discussed the increasingly complex healthcare system, yet state, “little has changed in the way that education is structured and delivered to aspiring health professionals” (p. 236). This articles contributes to the field that by incorporating active learning into a classroom, it can enhance learning, improve outcomes, and fully equip students to address 21st-century health care needs.

Literature Review – In my review of the article what I found most interesting is what is happening in traditional classrooms. Some of those happenings included, “students’ attention declines substantially and steadily after the first 10 minutes of class and that the average attention span of a medical student is 15 to 20 minutes at the beginning of class. Although students’ attention returns in the last few minutes of class, they remember only 20% of the material presented during that time. Furthermore, passive learning in hour-long lectures often bores students and can deprive them of rich educational experiences” (McLaughlin et al., 2014, p. 236).

Analysis/Finding – The authors compared pre- and post-course survey responses, course evaluations responses and final exam scores between the traditional and flipped-classroom cohorts. The finding conclude that the students in the flipped classroom evaluated the overall class better in areas such as, comprehension of material, engagement during class, preparedness, etc.

Discussion/Conclusions – The authors discussed in detail how the course was being redesigned but more importantly, in my opinion, honestly discussed the time commitment on both the instructor and the teaching assistants (TA). While the initial time commitment by the faculty is significant, it will decrease in subsequent years, however for the TA the time commitment will remain static. By showing the time commitment implications, I feel that future teachers will feel motivated to incorporate active learning techniques and feel confident that in subsequent classes or years they will not need to devote so much time on planning for the same class material.

Miscellaneous – What I particularly found valuable was some of the next steps and changes that will be implemented for the spring 2013 class. Some of those changes include: no longer considering the textbook to be required reading, replacing the student presentations and discussion with a new 30-minute active learning exercise, and creating “an online 411 Pharmacopedia to be used as an information portal for expanding concepts, new technologies, breaking news, current clinical trials, new drug products, and Web links” (McLaughlin et al., 2014, p. 242). This showed that the authors were incorporating ways to improve the course.


In my blog post from last week, I reviewed the article Does Active Learning Work? A Review of the Research (Prince, 2004). While I am in the infant stages of researching IF and HOW active learning works, I happily find myself being drawn into wanting more information. Some of my curiosity revolves around how students balance their in-class time with their out of class responsibilities and what are the long-term material retention statistics for those who participate in an active learning setting versus a traditional lecture classroom setting.

I am interested in implementing more active learning sessions for a course that I co-direct for fourth year medical students. During their final year of medical school, the students are in their elective rotations locally and across the country. In the spring, prior to graduation, we bring them back for a two-week course that is designed to help prepare their transition into residency. There are some active learning sessions during these two weeks, but approximately 80% of the course sessions are lecture based. In working with the director of the course, we are trying to develop sessions that involve more student involvement and particularly enhance ways to assess their clinical skills. I feel this article (McLaughlin et al., 2014) and the study described within can help persuade administration to allow us to achieve our goals of designing more active learning sessions and move away from the traditional lecture-based sessions.


McLaughlin, J. E., Roth, M. T., Glatt, D. M., Gharkholonarehe, N., Davidson, C. A, Griffin, L. M., Esserman, Denise A., Mumper, R. J. (2014). The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Academic Medicine : Journal of the Association of American Medical Colleges, 89(2), 236–43.

Prince, M. (2004). Does active learning work ? A review of the research. Journal of Engineering Education, 93(July), 223–231.