•  
  •  
 

PEDAGOGY IMPROVEMENT PROJECT WITH TECHNOLOGY SUPPORT AND ITS EVALUATION

Abstract

This project utilizes technologies to overcome challenges in implementing research-supported pedagogical interventions in an open-access institution. The target course is an introductory programming course (CS1) with large enrollment, though the principles can be generalized to other ITEC and STEM courses. CS1 is widely recognized as a barrier course, with national failure rates often exceeding 30%. To address both cognitive and motivational challenges, we developed the Synergy Approach, a multi-component instructional model. For each chapter, three core components address major steps in learning a programming construct: (1) concept formation through Process-Oriented Guided Inquiry Learning (POGIL), (2) semantic understanding through auto-graded visual tracing exercises, and (3) problem-solving strategy formation through a guided framework. The web-based tools—Amplify/Desmos for team-based guided inquiry and TracingQuiz for interactive visualized code tracing—enable scalable coordination, immediate feedback, and consistent implementation. This presentation reports evaluation results from a pilot study across Fall 2024 and Spring 2025 involving 11 instructors and 20 sections in varying modalities, using a pre/post-test and pre/post-survey control/treatment design. To ensure fair and consistent measurement of learning gains, we implemented a rubric-based, question-focused double-grading protocol for written-response coding questions. Intraclass Correlation Coefficient (ICC) analysis identified items needing rubric refinement; after clarification and adjudicated re-grading, ICC values improved to 0.892–0.967, demonstrating high inter-rater reliability. This scalable assessment framework provides a robust method for evaluating written programming work in large courses. Students in treatment sections demonstrated statistically significant improvements over control sections, outperforming by 10.21 points (~8.5% of maximum) in the full dataset and by 5.15 points (~4.3%) among instructors who taught both conditions. Analyses indicate that the approach raised the performance floor for lower- and middle-performing students. Attitudinal data showed mixed but encouraging patterns: confidence and perceived usefulness declined slightly, but persistence and affective engagement increased, and students rated both tools positively for supporting learning and interaction.

Acknowledgements

NSF Improving Undergraduate STEM Education: Hispanic-Serving Institutions (HSI) Program, Grant No. 2345097

This document is currently not available here.

Share

COinS