STUDENT ACHIEVEMENT RESULTS

The SimCalc mission is to define an innovative, integrated curriculum and technology that enables extended and widespread access to a robust, integrated, multifaceted mathematical understanding of rate and variation concepts. SimCalc addresses the connection between "how fast?" (velocity) and "how much?" (position) descriptions of both familiar phenomena and more formal mathematical systems, using multiple representational forms, including graphs, linguistic descriptions, and algebra.

SimCalc's core target is urban, low SES 7-9th grade students, with access to affordable yet powerful technology. Equitable access to powerful mathematics, such as the mathematics of change, has been a defining concern of all SimCalc work (Kaput & Roschelle, 2000). By clarifying the conditions under which mainstream children can learn these concepts, we can lay the groundwork for continuing elevation of national and state standards and key assessments over the long term, which will lead to systemic conditions in which all students have an opportunity to master these powerful but challenging ideas.

Over the course of approximately 8 years and two major NSF grants, we have refined and tested our innovations. This work has involved deep research on student cognition, technology designs, and alternative curricular sequences (1993-97), resulting first in a proof of concept largely detached from systemic factors. Subsequent work tackled systemic issues of curricular integration, teacher professional development, and assessment (1997-2000). We published and presented our work widely in more than 50 publications and hundreds of presentations (significant SimCalc publications are included in the reference section and are marked with "*"). Throughout, we have contributed broader impacts via the involvement of graduate students, support for early-career investigators, extensive work with teachers, and service on policy bodies informing standards development. Over time, we have systematically addressed three barriers to wide-scale adoption: access, scope, and student achievement results. With respect to access, SimCalc originally required software and hardware that cost $5,000 per student. With Texas Instruments (TI) as market partner, we now can deliver SimCalc on the technology most widely used in mathematics classrooms, the "TI-83 Plus" graphing calculator, which costs less than $100 per student (Hegedus & Kaput, 2001). With respect to scope, SimCalc was initially an addition to an already crowded curriculum that would attract few teachers. We now have refined it into replacement units intended to be adopted easily and widely (Kaput, in press).

We have achievement data from extensive field trials. SimCalc has been tested in interventions ranging from 15 to 45 hours in over 30 different trials, in over 15 school settings, with over 1,500 students, in contexts ranging from New Bedford, MA, Newark, NJ, Syracuse, NY, and San Diego, CA. Students included at-risk middle and high school students, gifted middle school students, classes of remedial first-year college students, AP Calculus students, and pre-service and in-service teachers. The form of the intervention has varied from replacement units to after-school settings to summer or weekend enrichment settings. Different trials also varied the content significantly, with a trend to increasing integration with grade-level-appropriate algebraic content. All trials collected extensive data, usually including video, field notes, and student work. In at least 20 of the trials, we collected pre-post data, albeit with no control, since the content addressed was not generally included in existing school curricula. Note that these field tests were designed to produce formative data rather than summative results. Formative studies are often smaller, given that they are designed to produce information to refine an innovation (Brown, 1991; Lesh, 2002) and not to answer questions of effectiveness of the mature intervention.

We highlight two notable findings from this formative work.

1. In fifteen studies, we have worked with 7th, 8th and 9th graders. Much of the early and ongoing work involves SimCalc staff working directly with students. However, in nine (9) studies, we have worked with classroom teachers. In a recent example, a teacher offered a 17-hour after-school SimCalc class to local 7th, 8th and 9th grade students. Dartmouth is a small town that includes a wide variety of income levels. The nine 7th graders and five 8th graders that signed up for the class were high-performing students taking the class as an honor, while the twenty-four 9th graders were low performing students, taking the class because they had failed or nearly failed the 8th grade Massachusetts Comprehensive Assessment System achievement exam. Twenty-four students completed the course. No attempt was made to retain those who dropped out because the classroom was sized for 25. No systematic differences appear between the group that persisted and the group that dropped out on either pre-test achievement or grade level. Reasons given for dropping the class included that it was too hard (2), that is was too easy (2) and conflicting activities (9).
   
   

   Figure 1: Performance in math of change of twenty-four 7th, 8th, and 9th grade students before and after SimCalc.

   As shown in Figure 1, the class made significant as well as important gains from the pre-test to the post-test (t(23)=11.68; p<.0001;d=1.7). The gains within each grade-level were also statistically significant. The 9th graders, the most disadvantaged group, paralleled the gains of the more advantaged students, and, by the end, matched their incoming state.
   
   
2. In the summer of 2000, thirteen High School and four Middle School math teachers participated in a two-week summer institute on SimCalc, conducted by Jim Kaput and SimCalc staff. It met for 6 hours a day for 4.5 days, with a half-day given over to testing and administrative matters. They extended their mathematics understanding, analyzed videos of students and teachers using the same lessons, and planned for integration of the technology into their own classrooms. Participants were recruited from a local area, which is one of the worst performing districts in Massachusetts as measured by the MCAS examination. Seventy percent of the students that these teachers work with are on free and reduced lunch. The teachers who participated had a range of backgrounds, including at the high end, some who had taught AP Calculus. As illustrated in Figure 2, teachers showed statistically significant and important improvement in performance on the post-test compared to the pre-test (t(16)=6.16;p<.0001;d=1.1).
   
   

   Figure 2: Performance in math of change and variation of 17 Middle and High school teachers before and after SimCalc.

   From interviews, we also know that these teachers were able to make use of the materials in the following year. For example, one teacher wrote:
   
   • I have just begun the chapter on slope and on Thursday introduced the slope as a rate. I had wonderful results as far as the understanding was concerned. I have never introduced slope in this way and saw how well they grasped the concept. This was an average group of Freshmen in Algebra I. They were able to come up with many examples of slope as a rate of pay, distance, ratio of students to teachers. They were also able to connect this idea to graphing linear functions….
   
   Although these results support the hypothesis that SimCalc can be implemented by middle school teachers, we have not yet measured the robustness of implementation outcomes with a probability sample of teachers who vary on critical variables and conditions. We are in exactly the position the designers of the IERI program anticipated: we have converging innovation research results from ROLE and similar funding sources, which are not yet robust, rigorous implementation research results. Thus, we applied for and received an IERI planning grant to help us conceptualize a transition (Confrey, Castro-Filho, & Wilhelm, 2000) from formative design experiments (e.g., Lesh, 2002) to a more rigorous methodology for implementation research (Cook, in press; Torgerson, 2001). Within the modest scope of that grant, we made important headway on building a partnership among innovation and implementation organizations and on three critical issues:
   
   1. Conceptualizing our innovation framework.
   2. Developing a framework and plan for implementation
   3. Defining an assessment blueprint based on a review of all previously available data.