Galloway, F. & Shea, M. M. (2009). Does your organization welcome participants with disabilities? A new assessment tool. Afterschool Matters, 9, 12–19.
With an increase in the enrollments of youth with disabilities in afterschool programs, organizations must evaluate if their programs truly welcome children and youth with disabilities. The authors of this study developed a valid and statistically reliable instrument, Organizational Developmental Model of Inclusion for Individuals with Disabilities (ODMI-IWD), to assist the program providers in developing policies to improve on perceived weakness in the areas of inclusion: diversity, differential treatment, congruency, motivational imperative, and experience.
Bohnert, A., Fredricks, J., and Randall, E. (2010). Capturing unique dimensions of youth organized activity involvement: Theoretical and methodological considerations. Review of Educational Research, 80(4), 576–610.
This study reviews the literature regarding current approaches to measuring participation in organized out-of-school-time (OST) activity settings and their effects on learners. The paper examines learners’ participation in terms of the dimensions of breadth, intensity, duration, and engagement, discussing the theoretical foundations and methodological approaches for each. The researchers note the dialectical nature of each of these dimensions. For example, participation is likely to become more intense (frequent and lengthy) as it endures over time, and as it endures over time it is more likely to intensify. This study provides a comprehensive overview of relevant measurement issues and approaches.
Jakobsson, A., Mäkitalo, Å. & Säljö, R. (2009). Conceptions of knowledge in research on students' understanding of the greenhouse effect. Science Education, 93(6), 978–995.
This study suggests that the assessment of students’ understanding of scientific vocabulary, concepts, and reasoning associated with the greenhouse effect may be better accomplished by observing and understanding learners’ developing language use over time. The indication of previous research that students hold tenacious misconceptions may be an artifact of the questionnaires used. The authors argue that listening to student conversations is the key to better recognize learning. This paper can help ISE educators think more deeply about how and when to assess for student understanding, including considering most appropriate and informative methods.
Sandoval, W. (2014). Conjecture mapping: An approach to systematic educational design research. Journal of the Learning Sciences, 23, 18–36. doi:10.1080/10508406.2013.778204
Design-based research (DBR) is a method for testing educational theories while simultaneously studying the process of creating and refining educational interventions. In this article, Sandoval proposes “conjecture mapping” as a technique to guide DBR processes. Conjecture mapping responds to critiques that DBR lacks clear standards and methodological rigor.
Van Schijndel, T. J. P., Franse, R. K., & Raijmakers, M. E. J. (2010). The Exploratory Behavior Scale: Assessing young visitors’ hands-on behavior in science museums. Science Education, 94, 794–809.
The authors of this paper were interested in knowing how parents can support exploratory behaviors of their preschool-aged children at museum exhibits. They developed a quantitative instrument based on psychological literature on exploration and play in order to describe and quantify young children's increasing levels of exploration of their environment. They then tested the measurement tool with parents and their preschool-aged children to investigate what types of adult coaching would achieve high-level exploratory behavior at various exhibits.
Palmer, K. L., Anderson, S. A., and Sabatelli, R. M. (2009). How is the afterschool field defining program quality? A review of effective program practices and definitions of program quality. Afterschool Matters, 9, 1–12.
This study is a summary of the review of the research literature of afterschool quality frameworks. It presents the debates on program effectiveness to help organizations, policymakers, funders, and evaluators make decisions about afterschool programming. This review is of help to ISE educators and program directors in understanding the current trends in outcomes-based programming, while grounding the conversation in the complexity and range of relevant developmental tasks.
Pekarik, A. J. (2010). From knowing to not knowing: moving beyond “outcomes.” Curator: The Museum Journal, 53(1) 105–115.
In this paper, Pekarik challenges the conventional approaches that institutions use to monitor success. He argues that outcome-based evaluations simply record impact in a set of predetermined categories and do not document the many and varied effects that participants may experience. This paper may be of interest to informal educators seeking new ways of thinking about program evaluation.
Siaca, J. L. (2010). High-impact afterschool for all. Afterschool Matters, 11, 1–5.
In this article, the author describes the process and result of a statewide effort in New York to develop an afterschool quality framework and assessment tool that can be used to guide program design, reflection, and assessment. One benefit of the tool is that it can be used by program stakeholders without the participation (i.e., the expense) of an external observer. This article might be of interest to ISE educators seeking to adopt existing program evaluation tools that do not require the participation of external evaluators.
Lyon, G., Jafri, J., & St. Louis, K. (2012). Beyond the pipeline: STEM pathways for youth development. Afterschool Matters, 16 , 48–57.
This article critiques the concept of the “STEM pipeline,” an analogy commonly used in education and policy discussions to describe the academic progression of students from K–12 through higher education in STEM. The authors’ new conceptual framework supports youth development goals in addition to STEM learning and reflects the experience of urban youth in out-of-school time settings.
Krantz, A., Korn, R., & Menninger, M. (2009). Rethinking museum visitors: Using K-means cluster analysis to explore a museum's audiences. Curator: The Museum Journal, 52(4), 363–374.
This paper presents a quantitative strategy (K-means cluster analysis) for exploring museum-motivated ideas that can be helpful in resource allocation, marketing, event planning, and designing exhibits. Cluster analysis provides a potentially useful way of knowing and understanding visitors, especially when the rating statements used in the questionnaire and in the analysis represent the museum's intentions.