Technology-Enhanced Language interactions for Learning (TELL)

Summary

This project has three goals: elucidate our understanding of peer learning, by analyzing what constitutes effective knowledge sharing and effective explanations devise a computational model of peer learning interactions, and embody it by building an artificial peer dialogue agent contribute to Computer Science education by developing the peer dialogue agent in the domain of basic data structures and algorithms, and pushing students to develop abstract, generalizable knowledge about data structures and algorithms. The dialogue agent will provide a vehicle for studying different strategies to support and assess peer learning. The development of the dialogue agent will be informed by a detailed language analysis of peer interactions that the project will collect. The dialogue agent will use natural language processing technology, but to avoid the possibly confounding effects of imperfect natural language understanding and generation, the experiments will include a human interpreter who will review and correct as necessary the system's interpretation of the student's inputs and the system's replies back to the student. By first using the dialogue agent, KSC-PaL, (KSC stands for Knowledge Sharing and Construction) as a peer in computer science education, we hope to increase retention in undergraduate computer science programs, especially among female students. Peer collaboration on programming tasks in classrooms and laboratory settings has been shown to benefit students of both genders. This project is a collaboration between the University of Illinois at Chicago and the University of Pittsburgh.

Web site: https://sites.google.com/site/pittkscpal/

Summary

The goal of this project is to take a step towards meeting President Obama’s challenge to produce “learning software as effective as a personal tutor.” We will do this by building an enhanced version of a natural-language dialogue system that engages students in deep-reasoning, reflective dialogues after they solve quantitative problems in Rimac, a natural-language (NL) tutoring system for physics. Enhancements to this system will focus on addressing a key limitation of NL tutoring systems: although these systems are “interactive” in the sense that they try to elicit explanations from students instead of lecturing to them, automated tutors do not align their dialogue turns with those of the student to the same degree, and in the same ways, that human tutors do. In particular, automated tutors often fail to reuse parts of the student’s dialogue turns in their own turns, to adjust the level of abstraction that the student is working from when the student is over-generalizing or missing important distinctions between concepts, and to abstract or specialize correct student input when doing so might enhance the student’s understanding. Empirical research shows that these forms of lexical and semantic alignment in human tutoring predict learning. The main outcome of this project will be a fully working reflective dialogue system that can carry out these functions and serve as a research platform for a future study that compares the effectiveness of the enhanced NL tutoring system with a baseline version which lacks these alignment capabilities—thereby allowing us to test the hypothesis that it is not interaction per se that explains the effectiveness of human tutoring, but how it is carried out.

The enhanced version of this reflective dialogue system will be developed through an iterative process of preparing a prototype for experienced physics teachers and students to try out using the “Wizard of Oz” paradigm, identifying cases in which the system does not work as intended (e.g., the tutor prompts the student to generalize or make distinctions when this is not warranted by the discourse context), refining the software to correct these problems, and testing the revised software in a subsequent field trial. The subject pool for these trials will be students enrolled in a first-year physics course at the University of Pittsburgh and high school students taking physics in Pittsburgh urban and suburban schools. During the third (final) year of the project, we will collect pilot data that addresses the feasibility of implementing the system in authentic high school physics classes, and the promise of the system to increase students’ conceptual understanding of physics and ability to solve physics problems. The latter will be determined by comparing students’ pre- and post-test performance on measures of conceptual understanding and problem-solving ability in physics, and by comparing the performance of students who use the current and enhanced version of the system on these measures.

Web site: https://sites.google.com/site/rimacsite/

Summary

An important line of research in the past few years indicates that the key to developing highly effective, adaptive ITSs is to derive policies that can guide an automated tutor in making pedagogical decisions such as when to initiate a hint and what kind of hint to provide; whether to tell the student a particular piece of domain knowledge, or guide the student in co-constructing that knowledge. Several policies that have been discovered using data-driven, machine learning approaches have been implemented in ITSs, and these systems outperform counterpart tutors that carry out random or fixed (non-adaptive) tutoring policies. However, automatically derived policies tend to be cryptic and domain specific, and the process of finding them is costly and technically challenging.

The main goal of this project is to formulate hypotheses and an associated set of preliminary policies for several tutorial decisions, with respect to particular types of students—for example, whether to address conditional relations in physics didactically or interactively; whether to address concepts and principles with reference to the particular physical situation or in abstract terms; whether to end a dialogue with a summary of the main principle discussed, or just end it; whether to support such a summary with an analogy or example, etc. Using conventional experimental methods, instead of data-driven automated techniques, we will examine which options are best for students at different ability and motivation levels, as measured by student learning outcomes. Specifically, through a series of carefully controlled experiments, we will manipulate automated dialogues in a computer-based tutoring system that we will use as a research platform, in order to determine if some tutoring decisions are better than others, for particular types of physics content, and particular types of students. Participants in our experiments will include high school physics students from several socio-economically diverse schools in the Pittsburgh, PA area.

Web site: https://sites.google.com/site/rimacsite/

Summary

Even though several tutorial dialogue systems have proven to be nearly as effective as human tutors, these systems nonetheless typically fail to outperform control systems that provide students with canned text as feedback instead of dialogue. Nor have tutorial dialogue systems achieved the effect sizes of the best human tutors: 2 standard deviations relative to classroom instruction. One possible reason why tutorial dialogue systems have not yet reached their full potential is that they can’t teach to a high mastery standard, as the most effective human tutors do. This might be because automated tutors lack the ability to track students’ level of mastery about particular curriculum elements (concepts, skills, etc.) during tutoring and this, in turn, is due to their lack of a student modeling engine. In particular, a student modeling system would enable the tutor to assess student ability and determine the right level of support to provide as students engage in conversations with the automated tutor. Because nearly all tutorial dialogue systems lack a student model, they can’t dynamically tailor instruction to the student’s ability level on particular knowledge components. Consequently, automated dialogues are often ineffective (failing to provide sufficient scaffolding) and inefficient (providing too much scaffolding). Since several studies have demonstrated the value of incorporating student modeling within non dialogue-based ITSs, and a few recent studies hint at the promise of student modeling to enhance tutorial dialogue systems, we will investigate how to rigorously incorporate student modeling within a tutorial dialogue system for physics, Rimac, and then conduct a pilot study to assess the value of doing so.

This project will be conducted through an iterative cycle which consists of alternating phases of software development, field testing, and refinement. Field trials and the culminating pilot study will take place in urban, suburban, and parochial schools in and near Pittsburgh, PA. The pilot study will be a randomized controlled trial which compares the student model enhanced version of Rimac with an identical system that differs from the first only insofar as it lacks the student modeling engine. Using statistical methods (mainly ANOVA and multiple regression), the two versions of the system will be compared with respect to effectiveness in improving student learning; potential aptitude-treatment interactions will also be considered. User satisfaction with each version of the system will also be compared, through analyses of student responses to items on a user satisfaction survey. The outcomes of this research will have broader implications for developing tutoring systems for various subject areas, building domain-neutral tutor authoring environments, and resolving the “assistance dilemma”—that is, striking the right balance between helping students and encouraging autonomy.

Web site: https://sites.google.com/site/rimacsite/