Assessing learning in OST settings: A review of the research

By Bronwyn Bevan - June 2011

This study reviews the literature regarding current approaches to measuring participation in organized out-of-school-time (OST) activity settings and their effects on learners. The paper examines learners’ participation in terms of the dimensions of breadth, intensity, duration, and engagement, discussing the theoretical foundations and methodological approaches for each. The researchers note the dialectical nature of each of these dimensions. For example, participation is likely to become more intense (frequent and lengthy) as it endures over time, and as it endures over time it is more likely to intensify. This study provides a comprehensive overview of relevant measurement issues and approaches.

This study includes an extensive bibliography that would be useful to designers and evaluators of OST programs. The researchers explain that many reviews of OST programs have failed to effectively articulate the ways in which participation may change by the time of year, by particular phases in the students’ lives, and across a broad range of activities. For example, they point out that researchers need to understand the number of hours spent in OST programs not as a flat number, but by the amount of time spent in specific programs: 6 hours a week in one program may lead to different outcomes than one hour a week in 6 programs. One hour in a high-quality program may result in stronger developmental effects than 10 hours in a low-quality program. Furthermore, what a student reports during a specific time of year may be radically different from what he/she would report just a few months later as he/she gets exposed to particular programs or opportunities seasonally. For example, a student might report attending only one sports program a week in the fall, but in the spring he/she may be engaged in a number of sports programs.

The paper cites scores of studies attempting to understand OST participation and its relationship with learning and developmental outcomes. This summary shows that the current research is decidedly mixed on making connections between participation and learning outcomes, with some studies finding strong positive relationships and some finding none. Despite discrepant findings, the researchers contend that the studies at present show that greater breadth of involvement in OST activities may be important during early- to mid-adolescence; and that more intense levels of participation facilitate better developmental (but not academic) outcomes among children and adolescents. This conclusion notwithstanding, the review makes clear that a much better theoretical and methodological grounding is necessary in order to compare and learn from various studies.

An important consideration for ISE educators is the authors’ focus on the duration of participation. The researchers explain that it takes several years to develop high-quality and supportive relationships, and that it takes time and practice to develop many skills, as well as to support identity formation. From a theoretical perspective, providing youth with opportunities to engage over time may be critical to their development and learning (although these opportunities may occur in different settings). They conclude that there is evidence indicating that the longer one has been involved in activities the greater the benefit in terms of academic success. They point out that youth seem to benefit most from participation that endures for two years or more.

The researchers conceptualize engagement in terms of behavior, emotion, and cognition. They cite Larson (2000) and Weiss et al. (2004) to note that “high engagement is characterized by relatively high attention, interest, enjoyment, and effort to master new skills, whereas low engagement is identified by apathy, boredom, inattentiveness, and passivity” (p. 593). They distinguish between measurement of individual engagement and program engagement. The latter, which may indicate program quality, is usually assessed through observations. The former is usually assessed through retrospective interviews or, more recently, though experience sampling methods (ESM), where students carry pagers or alarms and are asked to note their levels of engagement at set times, thus capturing in situ (as opposed to retrospectively) how interested and involved they are at various times of the day.

While discussing future directions, the researchers suggest that developing a single measurement of involvement, combining breadth, intensity, duration, and engagement, could be of use (NB: scholars employing more interpretive methods of inquiry, such as phenomenology, might be skeptical about such a unitary measurement.) In developing models and methods that might predict behavior and outcomes, the researchers state that it is important to account for barriers to participation, such as the availability of programs, family circumstances that affect participation, competing programs, academic requirements, etc. The researchers claim that bioecological models, which argue for the dynamic relationship between person, experience, and context (see Bronfenbrenner & Morris, 2006), may be useful in trying to understand how OST programs support engagement and participation.

References of note include:

Bronfenbrenner, U., & Morris, P. (2006). The bioecological model of human development. In W. Daman & R. M. Lerner (Eds.), Handbook of child psychology. Vol. 1: Theoretical models of human development (pp. 793–828). Hoboken, NJ: John Wiley.

Larson, R. W. (2000). Toward a psychology of positive youth development. American Psychologist, 55, 170–183.

Mahoney, J. L., Larson, R. W., & Eccles, J. S. (Eds.). (2005). Organized activities as contexts of development: Extracurricular activities, afterschool, and community programs. Mahwah, NJ: Lawrence Erlbaum.

Weiss, H. B., Little, P. M. D., & Bouffard, S. M. (2005). More than just being there: Balancing the participation equation. New Directions for Youth Development, 105, 15–31.