Asher, Catherine Armstrong, Ethan Scherer, and James S. Kim. (2020). Using a Factorial Design to Maximize the Effectiveness of a Parental Text Messaging Intervention. (EdWorkingPaper: 20-305). Retrieved from Annenberg Institute at Brown University: https://doi.org/10.26300/86na-3y47
Abstract: Parental text messaging interventions are growing in popularity to encourage at-home reading, school-attendance, and other educational behaviors. These interventions, which often combine multiple components, frequently demonstrate varying amounts of effectiveness, and researchers often cannot determine how individual components work alone or in combination with one another. Using a 2x2x3 factorial experiment, we investigate the effects of individual and interacted components from three behavioral levers to support summer reading: providing updated, personalized information; emphasizing different reading views; and goal setting. We find that the personalized information condition scored 0.03 SD higher on fall reading assessments. Test score effects were enhanced by messages that emphasized reading being useful for both entertainment and building skills compared to skill building alone or entertainment alone.
Kim, J. S., Burkhauser, M., Mesite, L., Asher, C.A., Relyea, J.E., Fitzgerald, J., Elmore, J. (Forthcoming in the Journal of Educational Psychology). Improving Reading Comprehension, Science Domain Knowledge, and Reading Engagement through a First-Grade Content Literacy Intervention.
Sample: 674 first-grade students from 38 classrooms at 10 Southeastern schools.
Target Group: Grade-1 students
Outcome of Interest: science knowledge, reading engagement, reading comprehension, basic literacy skills
Intervention: curriculum (content literacy intervention)
AEA RCT Registration Number: AEARCTR-0002746
Data: We are in the process of creating a replication toolkit for researchers
Kim, J. S., Asher, C. A., Burkhauser, M., Mesite, L., & Leyva, D. (2019). Using a Sequential Multiple Assignment Randomized Trial (SMART) to Develop an Adaptive K–2 Literacy Intervention With Personalized Print Texts and App-Based Digital Activities. AERA Open. https://doi.org/10.1177/2332858419872701
Sample: 273 kindergarten to second-grade students from 1 Southeast school.
Target Group: K-2 students, parents
Outcome of Interest: take-up and usage of an educational literacy app, reading comprehension, basic literacy skills
Intervention: educational literacy app and targeted messaging to parents
AEA RCT Registration Number: AEARCTR-0002983
Kim, J. S. (2019). Making Every Study Count: Learning From Replication Failure to Improve Intervention Research. Educational Researcher, 48(9), 599–607. https://doi.org/10.3102/0013189X19891428
Why, when so many educational interventions demonstrate positive impact in tightly controlled efficacy trials, are null results common in follow-up effectiveness trials? Using case studies from literacy, this article suggests that replication failure can surface hidden moderators—contextual differences between an efficacy and an effectiveness trial—and generate new hypotheses and questions to guide future research. First, replication failure can reveal systemic barriers to program implementation. Second, it can highlight for whom and in what contexts a program theory of change works best. Third, it suggests that a fidelity first and adaptation second model of program implementation can enhance the effectiveness of evidence-based interventions and improve student outcomes. Ultimately, researchers can make every study count by learning from both replication success and failure to improve the rigor, relevance, and reproducibility of intervention research.
Wantchekon, K., & Kim, J. S. (2019). Exploring the Relationship Between Reading Engagement and Reading Comprehension by Achievement Level. Reading & Writing Quarterly. DOI: 10.1080/10573569.2019.1594474
The present study examined potential synergistic relationships between reading engagement and reading comprehension among 3,689 third and fourth graders across 59 schools in North Carolina. Using hierarchical regression analyses, we replicated previous findings that reading engagement explains unique variance in reading comprehension. Our results indicated that reading engagement explained an additional 4% of variance in end-of-year reading comprehension above and beyond initial skill, student demographics, and school membership. We then utilized multilevel modeling to examine the tenability of two common hypotheses in the literature: that reading engagement is more strongly related to the reading comprehension of below-average readers (the compensatory hypothesis) and that reading engagement is more strongly related to the reading comprehension of above-average or competent readers (the cognitive-constraint hypothesis). Students were broken into above-average, average, and below-average skill groups based on their beginning-of-year score on a nationally normed assessment of reading comprehension. Results supported the cognitive-constraint hypothesis that the relationship between reading engagement and reading comprehension is attenuated for below-average readers. The strength of the relationship for average and above-average readers did not significantly differ, suggesting homogeneity in the strength of the relationship among these students.
Book retelling has been frequently used as an indicator of children’s reading proficiency. However, how children’s performance varies across retelling narrative and expository texts and whether that has different implications for reading proficiency remains understudied. The present study examined 85 high-poverty second- and third-graders’ retelling of narrative and expository books. A parallel coding scheme was developed to evaluate children’s performance on retelling fluency, content, and language complexity. Children’s retelling performance was compared across text types and analyzed in relation to reading proficiency. Findings revealed similarities and differences in retelling across text types, with narrative retelling containing a higher proportion of content-matched T-units, whereas expository retelling contained a higher proportion of inference generation and more complex syntactic structures. Moreover, indicators of reading proficiency were found to vary across text types. Findings highlight the distinct cognitive and linguistic demands posed by reading narrative and expository texts and provide implications for effective instruction and assessment.
Troyer, M., Kim, J. S., Hale, E., Wantchekon, K., & Armstrong, C. (2018). Relations Among Intrinsic and Extrinsic Motivation, Reading Amount, and Comprehension: A Conceptual Replication. Reading and Writing, An Interdisciplinary Journal. https://doi.org/10.1007/s11145-018-9907-9
Children's motivation to read is a strong predictor of their reading comprehension. However, some recent research has suggested that the relationship between reading motivation and reading comprehension may be mediated through the amount that students read. This study attempts a conceptual replication of several existing models that explore the relationship among children's reading motivations, out-of-school reading amount, and reading comprehension, using a large sample of over 4000 third- through fifth-graders in 59 U.S. elementary schools. Consistent with prior research, several control variables, including children's prior reading comprehension ability, gender, and socioeconomic status, directly contributed to later reading comprehension. Results also replicated positive associations between intrinsic reading motivation, reading amount and reading comprehension, and negative associations between extrinsic reading motivation, reading amount and reading comprehension. Using structural equation models, our analyses found no evidence that the relationship between children's intrinsic and extrinsic reading motivation and later reading comprehension was either partially or fully mediated by reading amount. This suggests that it is critical to attend to context-specific determinants of motivation and reading amount, including students' background characteristics and quality of texts read. Furthermore, this study underscores the importance of replicating methods used by original researchers to confirm and disconfirm hypotheses, and of conducting research with large and diverse samples that enhance the generalizability of results.
Quinn, D. M., & Kim, J. S. (2018). Experimental Effects of Program Management Approach on Teachers’ Professional Ties and Social Capital. Educational Evaluation and Policy Analysis, 40(2): 196-218. DOI: 10.3102/0162373717742198
Theory and empirical work suggest that teachers’ social capital influences school improvement efforts. Social ties are prerequisite for social capital, yet little causal evidence exists on how malleable factors, such as instructional management approaches, affect teachers’ ties. In this cluster-randomized trial, we apply a decision-making perspective to compare a literacy intervention managed under a “fidelity-focused” approach, in which teachers were expected to implement researcher-designed procedures faithfully, versus a “structured adaptive” approach, in which teachers collaboratively planned program adaptations. In the short term, the adaptive approach increased teachers’ accessing of intervention-related social capital, but decreased their accessing of social capital unrelated to the intervention. Short-term effects varied based on participants’ role in the intervention. No group differences were found on social capital measures one year later, suggesting that the structured adaptive approach did not make teachers more likely to form ties that would be useful outside of the intervention.
Quinn, D. M., & Kim, J. S. (2017). Scaffolding Fidelity and Flexibility in Educational Program Implementation: Experimental Evidence from a Literacy Intervention. American Educational Research Journal, 54(6): 1187-1120. DOI: https://doi.org/10.3102/0002831217717692
In a common approach for scaling up effective educational practice, schools adopt evidence-based programs to be implemented with fidelity. An alternative approach assumes that programs should be adapted to local contexts. In this randomized trial of a reading intervention, we study a scaffolded sequence of implementation in which schools first develop proficiency by implementing the program with fidelity before implementing structured adaptations. We find evidence supporting the scaffolded sequence: A fidelity-focused approach promoted learning and instructional change more so for teachers inexperienced with the intervention, while a structured adaptive approach was more effective for teachers experienced with the intervention. Students benefited more from the structured adaptive approach but only when their teacher had prior experience with the fidelity-focused version.
Kim, J. S., Burkhauser, M. B., Quinn, D. M., Guryan, J., Kingston, H. C., & Aleman, K. (2017). “Effectiveness of Structured Teacher Adaptations to an Evidence-Based Summer Literacy Program.” Reading Research Quarterly, 52(4): 443-468
The authors conducted a cluster-randomized trial to examine the effectiveness of structured teacher adaptations to the implementation of an evidence-based summer literacy program that provided students with (a) books matched to their reading level and interests and (b) teacher scaffolding for summer reading in the form of end-of-year comprehension lessons and materials sent to students’ homes in the summer months. In this study, 27 high-poverty elementary schools (75–100% eligibility for free or reduced-price lunch) were matched by prior reading achievement and poverty level and randomly assigned to one of two implementation conditions: a core treatment condition that directly replicated implementation procedures used in previous experiments, or a core treatment with structured teacher adaptations condition. In the adaptations condition, teachers were organized into grade-level teams around a practical improvement goal and given structured opportunities to use their knowledge, experience, and local data to extend or modify program components for their students and local contexts. Students in the adaptations condition performed 0.12 standard deviation higher on a reading comprehension posttest than students in the core treatment. An implementation analysis suggests that fidelity to core program components was high in both conditions and that teachers in the adaptations condition primarily made changes that extended or modified program procedures and activities in acceptable ways. Adaptations primarily served to increase the level of family engagement and student engagement with summer books. These results suggest that structured teacher adaptations may enhance rather than diminish the effectiveness of an evidence-based summer literacy program.