November 22, 2022
LRDC's Brian Galla and colleagues sought to study an intervention that could accurately assess student self-regulation.
School is where young people spend most of their waking hours outside the home. Recently, policymakers have urged schools to reach beyond the traditional academic coursework into the domain of social-emotional skills by identifying contexts that cultivate self-regulation. Adolescence may be particularly important for supporting self-regulation because of the rapid growth, learning, adaptation, and neurobiological development that mark this period of life. It is also where they experience a number of factors, such as adult role models and peers, that can affect the development of self-regulation. Impulsive choices in adolescence (e.g., to start smoking, to drop out of school) can have life-long effects.
Collecting data on self-regulation often entails using self-report questionnaires — the most common way to assess self-regulation — and comparing groups of individuals (e.g., from different schools). Self-report questionnaires — the most common way to assess self-regulation — are prone to reference bias. Reference bias is the error that arises when respondents "refer" to different standards when answering the same questions. Reference bias can distort inferences when there are comparisons of self-regulation across different groups — for example, schools with very different peer cultures with respect to effort, or even subcultures within a school.
A team of researchers across the United States, including Learning Research and Development Center's Research Scientist Brian Galla, sought to study how to accurately assess student self-regulation. The researchers gave senior-year high school students two assessments. The first was a traditional, subjective self-regulation questionnaire. The second was the Academic Diligence Task, a behavioral task in which students pay attention to either "good-for-me-later" math problems or "fun-for-me-now" games and videos.
In three studies involving more than 220,000 students, the authors found evidence for reference bias. In Study 1, high school seniors rated themselves lower in grit (the ability to focus on long-term goals) when their schoolmates earned higher GPAs and standardized achievement test scores. In Study 2, the authors replicated this effect using self-report questionnaires of conscientiousness and showed that it was driven by near-peers rather than by far-peers. Further, the authors showed that the GPA of near-peers (but not far-peers) correlates positively with self-regulation standards. In Study 3, they found that using self-report questionnaires of grit and self-control to predict college graduation six years later produced paradoxical results: Within a high school, students with higher self-reported self-regulation were more likely to graduate from college six years later, but across schools, average levels of self-regulation negatively predicted graduation. In contrast, an objective task measure of self-regulation — which indexed performance directly and did not ask students to judge themselves — positively predicted college graduation both within and across schools.
These findings suggest that standards for self-regulation vary by social group, limiting the policy applications of self-report questionnaires. Although the importance of personal qualities such as self-regulation is undeniable, the real possibility of reference bias argues against relying on self-report questionnaires when comparing students attending different schools, citizens who live in different countries, or indeed any of the members of any social group whose standards could differ from one another.
Are you a hard worker? Responding to such a question requires looking within to identify the patterns of our behavior.
The evidence for reference bias presented here suggests that knowingly or not, we also look around when we decide how to respond.
Lira, B., O'Brien, J.M., Peña, P.A., Galla, B.M., D'Mello, S., Yeager, D.S., Defnet, A., Kautz, T., Munkacsy, K., & Duckworth, A.L. (2022). Large studies reveal how reference bias limits policy applications of self-report measures. Scientific Reports.