Dr. Colin Fredericks
Title: Patterns of Behavior in Online Homework Systems
Group Meeting Date & Time: Monday, June 4, 2012 @ 2:00 pm
View the slides from the group meeting
Dr. Colin Fredericks is working in MIT’s RELATE group as a postdoctoral researcher in physics education. Dr. Colin Fredericks’ work in the RELATE group includes the creation of online course materials, the analysis of homework data, and work with multimedia production. Dr. Fredericks received his Bachelor’s and Master’s degrees in Physics from Rensselaer Polytechnic Institute (RPI) and his Doctorate in Physics Education from the University of Massachusetts at Amherst.
In his group meeting, Dr. Colin Fredericks presented his doctorate work in a talk entitled “Patterns of Behavior in Online Homework Systems.” Dr. Fredericks strove to quantify student behaviors in online homework systems, determine how students complete their homework assignments, the effectiveness of homework assignments on student learning, and how much we can learn from homework data. Additionally, Dr. Fredericks inquired whether different types of problems can enhance student learning.
Dr. Fredericks utilized the extensive online homework data from two different physics courses, a non-majors course and a majors course. This data was obtained from OWL (online web learning), which is the electronic homework system at University of Massachusetts at Amherst. Specifically, he analyzed students’ behavior while completing homework assignments, as well as their performance on their assignments, quizzes and exams. A large amount of data were collected on each homework assignment, including student identifier information, details on the question itself, the student’s score, the date and time when the question was answered, and the length of time to answer a question. In one Physics course alone, there were 140,000 lines of data!
To make sense of the immense data set, he analyzed the data using Excel and IGOR, employing correlation factors to analyze linear relationships and Principal Component Analysis (PCA) to analyze multi-linear, complex relationships. The data were first sorted by removing students who had an incomplete grade. Students were then grouped into either the engaged or disengaged categories. Students who completed more than 85% of homework assignments, lecture preparation questions, surveys, and quizzes as well as completed all exams were deemed “engaged”.
Dr. Fredericks and his thesis advisor, Dr. Gerace, were interested in whether students’ behavior on different type of questions correlated with overall performance in the course and with performance in specific areas. To analyze this, they categorized each problem into one of four categories: i) analysis (several concepts are covered), ii) conceptual (only one concept is covered), iii) multiple-choice / definition or iv) traditional (textbook).
Using all of this data, Dr. Fredericks calculated specific measurements of student activity, called gauges, which were used to analyze counts or averages of data. Examples of gauges include the number of seconds a student took to respond to a question, and the number of attempts a student used to solve a problem. When the gauges were compared across problem type, he found that the problem types were separated nicely by the data. Dr. Fredericks found that some, but not all, of the performance-based and time-related gauges, as well as nearly all of the attempt-related gauges, were able to predict student performance. Specifically, students who started working on their assignments early tended to do well on their assignments.
The gauges were then normalized, and measurements of student behavior were created via multilinear combinations of gauges. Dr. Fredericks found that behaviors were a better predictor than gauges in predicting student performance. By weighting specific data values and grouping them into categories, such as “Uncertainty”, “Tenacity” and “Efficiency”, Dr. Fredericks found that one such combinatorial behavior, termed “Tenacity and Efficiency”, which measured students’ ability to answer a problem correctly in a short length of time and with minimal failed attempts, was able to accurately predict course grade (R2 = 0.82268) in both physics courses (see Figure 1).
Figure 1. Dr. Fredericks found that students who exhibit the tenacity and efficiency behaviors when completing their online homework assignments did better in the course.
Additionally, students were asked to complete surveys on the online homework assignments. Interestingly, Dr. Fredericks found that the length of time students reported to spend on a problem did not match the actual amount of time it took to answer a question. Also, while disengaged students reported that they wanted to understand the material, engaged students preferred to improve their existing knowledge of the subject. Disengaged students were more likely to give up or fruitlessly “keep trying”, while engaged students sought out extra help and utilized multiple resources when stuck on a problem.
Dr. Fredericks’ group meeting presentation clearly detailed all of the information that can be obtained from online homework systems. While the student behaviors surrounding completing homework assignments can be complex, his work is easily applicable to the development of online courses developed on new online learning platforms, such as MITx. In particular, the need to determine whether practicing solving problems can augment grading for homework assignments, and the types of data that need to be collected on student behavior, are perfectly suited with the current development of online courses on MITx.
Prior to joining the RELATE group, Dr. Fredericks worked with MIT’s Teaching and Learning Laboratory (TLL) as a specialist in physics education for the SUTD collaboration, and with the UMass Physics Education Research Group, assisting with teacher training and delving into statistical analysis of online homework systems. Dr. Fredericks also has extensive teaching experience in both high school and college settings. He has taught introductory physics and mathematics at the university level at RPI, University of Massachusetts at Amherst, and Siena College as well as at the high school level at Northfield Mount Hermon and the Hyde School in Woodstock, CT. Dr. Fredericks has won two teaching awards: the Walter Eppenstein Teaching Assistant Award at RPI and the Learning Triangle at the Hyde School. Dr. Fredericks’ writes about education and his work on his website: dontstoplearning.wordpress.com.
