Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework (O’Cathain, 2010) provides valuable insight and information on qualitative and quantitative research methods separately, in order to introduce and describe the leading framework for analyzing mixed method research. The article closes with an invitation for researchers to apply the outlined model to their own work, in hopes of further developing the framework and understanding of how to assess the quality of mixed methods research.
It is important to note that a potential limitation of the mixed method framework presented is that the author, Alicia O’Cathain, builds upon the framework of &
The discipline of research methodology is new to me, and although I take time to scrutinize procedure, bias and context of research, I have been limited in my approach to analyzing research. As a high school science teacher, I am further limited in my experience as my focus is often on quantitative research, hence I have chosen a research article that uses qualitative methodology. In the article, Using a Project Blog to Promote Student Learning and Reflection (Worthington, Reniers, Lackeyram, Dawson, 2018) the role of blogging in student learning and reflection is analyzed. I am keenly interested as this is a practice I plan on implementing in my Environmental Science 11 class in next school year.
My teaching philosophy and practice is evolving from that of traditional direct instruction to an inquiry based model. My experience in school was during the 80’s and 90’s when conventional teaching methods dominated. During my undergrad I began to experience a more authentic learning environment, and throughout my 20 year teaching career have thoroughly enjoyed experiencing the opportunities and enthusiasm inquiry based teaching and learning affords. My intention for teaching Environmental Science 11 is to:
- inspire students to participate in caring for the natural world through passion projects mentored by local environmental groups;
- instill confidence in their abilities to contribute to a larger community of environmentally responsible individuals;
- use the inquiry based teaching model to promote critical thinking;
- encourage students to motivate others to be environmentally aware through social media and community interactions.
My goals for this class are not conducive to traditional teaching practices, hence students will be assessed on reflections and projects rather than traditional tests. My optional article to review and consider in light of Assessing the Quality of Mixed Methods of Research: Toward A Comprehensive Framework, is Using a Project Blog to Promote Student Learning and Reflection. My purpose for choosing this article for reflection is to further my understanding of qualitative methods and research, and consider the effectiveness of using blogs to promote student learning in senior Environmental Science classrooms.
The three research questions focused on in Using a Project Blog to Promote Student Learning and Reflection (Worthington, et al. 2018) are:
- How frequently did students write blog posts?
- What content did students write about?
- What functions did the students’ blog posts serve?
The qualitative research undertaken in Using a Project Blog to Promote Student Learning and Reflection is designed to inform instructor practices (Worthington, et al. 2018). Three undergraduate students were instructed to use blogging to reflect, communicate, brainstorm, evaluate literature and document team creations throughout a curriculum mapping project as a co-op work placement. Little direction as to how to use the blogs was given to the three students, however several weeks into the project they were requested to add a task list describing their weekly activities in their blogs.
Worthington, et al. (2018) give evidence from previous studies that suggest blogging increases connectedness among students (Miceli, Murray, Kennedy, 2010), improves knowledge integration outside of class (Halic, Lee, Paulus, & Spence, 2010), and encourages cognitive, meta-cognitive, and affective learning (Chu et al., 2012). The author also refers to research that highlights the benefits of grading blogs (Chu et al., 2012) and providing structure for blog engagement (Divitini, Haugalokken, and Morken, 2005).
The results of this study suggest that the use of blogs helps guide projects and promotes learning. Using the comprehensive teaching of O’Cathain, I will investigate the research methods of a qualitative analysis of blog posts undergone in the study by Worthington, et al. (2018) and suggest how a subsequent study might address any shortcomings.
Summary of Findings
Research Question 1: Frequency of Blog Posts.
The frequency of blog posts were analyzed by counting the number of blog posts, and using Microsoft Word’s word count to determine the length of the post. The average post length was 568 words, and an average of seven posts per student per month were written.
Research Question 2: Content of Blog Posts.
The content of blog posts included curriculum-related issues such as pedagogy and Bloom’s taxonomy, and personal reflection.
Research Question 3: Function of Blog Posts.
The functions of blog posts included organization, communication with their faculty supervisor and one another, summarizing learning outcomes, share links to resources and upload documents for future reference.
Analysis of Qualitative Methods for This Case Study Using the Quality Framework for Mixed Methods Research (O’Cathain, 2010)
Planning Quality.
Care was given to be transparent in situating the study and shaping the research questions, methods, analysis, and reporting. The feasibility of the study was good.
Design Quality.
The design was appropriate for addressing the research questions. Strengths and weaknesses were not considered in depth, and weaknesses were not addressed in the design. Specifically, the primary author, P. Worthington, was one of the participants in the study and also developed the codebook used to analyze the results. At the time of the co-op, the participants were not aware that the blogs would be analyzed retrospectively, but it does bring in to question whether the participant/author may have had an idea for future research during her participation of the co-op. Also, the time that each student participated in the co-op varied greatly in duration and time of year. Overall, I would rate the design strength of this study as poor as the sample size is too small to infer reliable conclusions and the controls of the design were inadequate.
Data Quality.
Admittedly, I struggle to give an informed opinion on analytic adequacy and analytic integration rigor as I am unfamiliar with data analysis techniques for qualitative research. That being said, the analytic adequacy seems to be compromised due to the author’s participation in the study. As noted above, the participants were not aware that the blogs would be analyzed retrospectively, however, the author may have personally experienced opportunities for reflection and learning due to blogging, and thereby developed the study to verify her conclusions. If this were the case, she could not have been objective in her analysis. Furthermore, there are many quotes in the article to evidence higher level thinking, communication, organization, and reflection, and now that I know that one of the authors is a participant it brings in to question the motivation for the article. Was it an opportunity to bring attention to the quality of the author’s own use of blogging? I don’t believe it completely negates the data, the research, or it’s implications but it definitely raises concerns and my stance is that the conclusions warrant further investigation. The data transparency and rigor appears to be sufficient, and sampling adequacy was poor.
Interpretive Rigor.
Interpretive transparency and consistency were weak, and theoretical consistency was good as the inferences of the study were consistent with current knowledge. Interpretive agreement was weak as the sample size was too small and the objectivity of the author is questionable.
Inference Transferability.
From the findings of the study I have sufficient reason to believe that blogging will assist with student organization, learning, and reflecting in senior secondary environmental science classes. However, I am not confident these findings will be consistent across all my students as not all participants will be highly motivated, as they were in the study, therefore ecological transferability is good and population transferability is poor.
Reporting Quality.
Reporting availability was good as the study was successfully completed within time,money and staffing constraints. The reporting transparency was also sufficient as many key aspects of the study were described thoroughly. The yield of the study was good as the conclusions could have far reaching implications.
Synthesizability.
The study has poor design (due to small sample size) and data quality which limits the reliable application of this study.
Utilitity.
Utility of this study was good as prior research supports the research conclusion that blogging promotes student learning and reflection.
Conclusion
In conclusion, the research conducted in the study Using a Project Blog to Promote Student Learning and Reflection is important for informing the teaching practices of educators. I believe the implications of this study can extend beyond the Post Secondary environment in which it took place and has value for teachers, students, software companies, education researchers, ministries, and school districts. These stakeholders would benefit from further research involving a larger pool of more diverse students to warrant widespread support of blogging to enhance student learning and reflection practices. The authors of the article would have benefited from reading O’Cathain’s work, supported by Tashakkori and Teddlie, as their work could have contributed to the planning, design quality and data quality of the study, hence improving the validity and reliability of the findings.
Author Biographies
Paisley Worthington is currently an undergraduate science student at the University of Guelph.
Jennifer Reniers (PhD) is the Educational Analyst in the Office of Teaching and Learning at the University of Guelph
Dale Lackeyram (PhD) is the Manager of Educational Development, Centre for Open Learning and Educational Support at the University of Guelph
John Dawson is a Professor and Director in the CBS Office of Education Scholarship and Practice (COESP).
References
Chu, S. K. W., Chan, C. K. K., & Tiwari, A. F. Y. (2012). Using blogs to support learning during internship. Computers and Education, 58(3), 989-1000. http://doi.org.ezproxy.library.uvic.ca/10.1016/j. Compedu.2011.08.027
Divitini, M., Haugaløkken, O., & Morken, E. M. (2005). Blog to support learning in the field: lessons learned from a fiasco. Proceedings of the 5th IEEE International Conference on Advanced Learning Technologies (ICALT 2005), 219-221. http://doi.org.ezproxy.library.uvic.ca/10.1109/ ICALT.2005.74
Halic, O., Lee, D., Paulus, T., & Spence, M. (2010). To blog or not to blog: Student perceptions of blog effectiveness for learning in a college-level course. Internet and Higher Education, 13, 206-213. http://doi.org.ezproxy.library.uvic.ca/10.1016/j.iheduc.2010.04.001
O’Cathain, A. (n.d.). Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework. Sage Handbook of Mixed Methods in Social & Behavioral Research, 531-556. doi:10.4135/9781506335193.n21
Miceli, T., Murray, S. V., & Kennedy, C. (2010). Using an L2 blog to enhance learners’ participation and sense of community. Computer Assisted Language Learning, 23(4), 321-341. http://d0i.0rg/10.1080/09588221.2010.495321
, & (2003). The past and future of mixed methods research: From data triangulation to mixed model designs.
Tashakkori, A., & Teddlie, C. (2010). Sage handbook of mixed methods in social & behavioral research. SAGE Publications.
Worthington, P., Reniers, J., Lackeyram, D., & Dawson, J. (2018). Using a project blog to promote student learning and reflection. The Canadian Journal of Higher Education, 48(3), 125-140.