Report on the After-School Algebra Enrichment Experiment
Conducted by the SimCalc Project
March 4th - April 5th 2002

Executive Summary
What Happened and Who Were Involved?
With the enthusiastic cooperation of staff from high school and middle school , the SimCalc Project engaged in an extended after-school teaching experiment on Tuesdays-Thursdays, 2:30-3:45PM from March 5th to April 4th, 2002 in an iMac Lab. Approximately 35 students enrolled in the course, roughly a third being volunteers from middle school(7th and 8th graders) and the remaining students being 9th graders who were low-scoring on the 8th grade MCAS. The course was most ably taught by a high school mathematics faculty member, and assisted by SimCalc staff as well as several other mathematics teachers. The curriculum focused on core ideas in algebra: slope as rate-of-change, linear functions, modeling, and simultaneous conditions. Classes were videotaped, copious field notes were taken, and pre-/post-tests were administered, yielding a comprehensive picture of the event and its consequences.

Methods and Goals of the Intervention
The SimCalc approach to these ideas exploits tightly linked motion simulations, graphically definable and editable functions, and algebraic formulas running on both computers and graphing calculators. While this approach has been widely tested across the U.S. and in other countries, this particular teaching experiment, for the first time, examined impacts and opportunities of an added ingredient, classroom connectivity. In particular, most classes involved students working in small groups to create functions to satisfy various conditions (usually modeling some given situation), and then sending their functions to the teacher who systematically aggregated their work and displayed it on a common classroom display. In this way, each student's work becomes quickly available for discussion, but more importantly, can interact with other students' constructions in the public display. Due to the shared nature of the students' work, their personal identity becomes intimately engaged in their mathematical activity. A central goal of the investigation was to determine how to tap this engagement for optimally productive mathematics learning and to determine the pragmatic conditions for successful use of these methods and technologies.

Results Significant Improvements for the Students
The primary means by which success was measured was through pre- and post-tests, where approximately two thirds of the items were chosen from released 10th grade MCAS test items. We are pleased to report that both sub-populations of students made significant gains on all items but one (where ceiling effects were seen). The mean test scores increased from 44% on the pre-test to 66% on the post-test. Aggregate results, as well as the tests themselves, can be found in the full report, and individual student scores have been submitted separately. We regard the data as worthy of note because the items were either challenging 10th grade items or even more challenging AP Calculus and SimCalc items, and the students were either 7-8th graders or low-performing 9th graders. Test scores of the Middle School students exceeded those of the 9th graders, but the High School students showed somewhat larger gains than did the 7-8th graders from pre- to post-test. Furthermore, on some MCAS items, the students' post-test means exceeded both the High School and statewide means on the respective items. One last positive result of the intervention was de facto faculty development which will likely lead to further use of these advanced technological resources by the math faculty in the coming years. We have promised full support of these efforts.

Classroom Activities
Classroom activities were designed to exploit the computer connectivity of the lab network by utilizing a version of MathWorlds that enables students to send their mathematical constructions to the teacher's computer from which the teacher could display any or all of the students' constructions as needed for the activity at hand. This sharing of constructions led to mathematically intense interactions by the students and engaged them in public discussion about what they saw in terms of graphs and the motion of dots controlled by the graphs.

Here we illustrate one such activity and how it offers an alternative method for understanding (a) the idea of slope as rate-of-change, velocity in this case, (b) the meaning of y-intercept as the initial position, and (c) the variation of parameters, the "m" and "b," in the formula for a linear function, Y=mX+b.

Staggered Start–Staggered Finish:
In this activity, we have six groups of students, with 3 to 5 people per group, where the groups are numbered from 1 to 6. Each student within a group must travel at 3 feet per second for 5 seconds but start at a position equal to their group number. Thus in this example, each student travels at the same rate for the same amount of time. Once their functions have been aggregated using the network on the teacher's computer and displayed, the variation between groups is apparent and becomes the focal point of a lively class exchange. The variation is provided by parameterizing the "b" in Y=mX+b, which in this context is now more than just the "y-intercept"–it is a student's starting position and group number. Discussion topics include "Which dot am I and how do I know?", "Which group is mine and how do I know?", and when a student does not construct the motion according to the given directions, "Who is the outlier?", "How do we know?", and "What needs to be done to make him/her fit in?".

Figure 1: Aggregated data being projected by the teacher for classroom discussion.

Figure 1 shows the aggregated motion on the teacher's display, the result of the teacher collecting the functions that the students have produced over the network. Note that, as expected, overlap occurs in the six position graphs as students' functions fall into Groups, but nonetheless, each student is represented by a dot in the upper half of the screen. Here the Groups are still visible in the vertical alignment of the dots. Animating the motion leads to all of the dots moving at the same speed but off-set in Groups, as in a parade, because all the persons in a given group travel side-by-side. Individuals in Group 1 are farthest to the left and those in Group 6 are farthest to the right. Displaying the work publicly via projection of the teacher's computer screen leads to intense class discussion centered around where an individual student is in the aggregate. Students are asked to identify themselves, the "Which dot are you?" question, in the aggregate and present convincing arguments to the class justifying their choice (which is fairly simple in this case, but much more demanding in later activities). We also emphasize the generic formula that describes all the groups, i.e. Y=3X+b where "b" is the Group number.

Summary Performance Data
This section reports on the overall results of the study, highlighting certain questions that showed particular significant gain from the pre- to the post-test. Included are some results from the attitude/background questionnaire administered at the time of both tests.

Pre- and Post-Tests
We administered a pre and post-test of 20 items to assess the effect of the teaching experiment. The majority of these items (11) were taken from the set of released MCAS items with the remaining items extracted from AP Calculus exams (2) and SimCalc test items (7). The latter were selected from a pool of items developed by the SimCalc Project and refined over several years of use. Sixteen of the assessment items were multiple choice and the rest were open response. We adopted a scoring rubric for the MCAS items that scored multiple choice questions as 0 or 1 and open response questions on a 0 to 4 scale. The SimCalc and AP Calculus items followed the same rubric. Two questions from the SimCalc items were scored with 2 points as they were multiple response questions (Items 14 and 19). The maximum score for the test was 31 points. Additional items were included on the post-test, but are not included in this report.

Importantly, the test used challenging items from the Grade 10 MCAS whereas the students were of varying ability from grades 7 through 9. The Middle School students from grades 7 and 8 showed a higher ability in both the pre- and post-test. The weaker 9th grade students had an average of 218 on their 8th grade MCAS test. This mixture led to interesting social interaction in our non-standard classroom setting. We also administered a questionnaire on both the pre- and post-tests to investigate differences between the backgrounds of the middle and high school students, and how their mathematical beliefs and the environment in which they are taught affect their performance. We are correlating this data with other sources of mathematical background for our sample of students. The group of students as a whole increased from pre-to post-test with statistically significant gains.

Results
The five-week teaching experiment had a positive effect on the mathematical behavior of both groups of students as presented in Table 1:

GROUP	PRE-TEST MEAN	PRE-TEST VARIANCE	POST-TEST MEAN	POST-TEST VARIANCE	p1
ALL	13.6 (44%)	19.04	21.1 (66%)	21.07	<0.0001
7th & 8th	16.2 (52%)	23.96	23.8 (77%)	14.62	<0.0001
9th	11.7 (38%)	8.06	19.2 (62%)	17.72	<0.0001
Table 1:  Pre- and Post-Test Means and Variances.
1This p-value corresponds to a paired t-test that was completed to see if a statistically significant increase had occurred in the means of the student before and after the 5-week session.

Even though the group of 7th and 8th grade students (n = 10) showed higher test averages than the 9th grade students (n = 14) the latter group exhibited a higher gain in their group mean (64%) with less variance. This illustrates that while the 5-week session had a very positive effect on both groups it had a more positive effect on the 9th grade students, a fact that we regard as significant because these students were chosen on the basis of weak 8th grade MCAS performance and represent the more challenging students to teach.

In addition, we have shown that it is possible to increase the ability of students to complete some 10th grade MCAS items in a short period of time before they reach the 10th grade.

Figure 2: Percent correct on each of the twenty test items.

Figure 2 highlights the pre-/post-test percentages of correct responses by item. Modest to excellent gains were achieved in every item of the test except one (Item 2) and there it was a statistically insignificant change, probably due to ceiling effects.

We now examine two questions that showed the largest pre/post changes and compare these scores with those of High School students and their statewide peers.

The first example we examine in detail is Item 5.

5. Based on the graph, which organization showed the most growth in membership over the 10-year period? A. The Math Club B. The Hiking Club C. The Drama Club D. The Drama Club and the Hiking Club are tied for the most growth.
Figure 3:   Item 5 from the Pre-/Post-Test.

The content of this item was not explicitly taught during the experiment and served as a face-validity item in our test. For this reason, it was evident that the students acquired skills that were applicable to a wider set of mathematical tasks not directly addressed in the 5-week session. The question tests graphical interpretation and comparison of starting and ending point differences, skills which were only implicitly used in our activities, as illustrated in our example activity Staggered Start–Staggered Finish.

The scores below show that our sample of students scored very similarly on the pre-test as both their High School and statewide peers in the previous year (about 46% correct). The post-test showed significant relative gain (almost 100%) in their performance.

Figure 4: Comparison of participant scores on Item 5 with larger populations.

Another large-effect example is Item 10. Again, the content of this question was not explicitly taught and involved interpreting a linear function with Pi(?) as a coefficient.

10. The circumference, C, of a circle is found by using the formula C = ?d, where d is the diameter. Which graph best shows the relationship between the diameter of a circle and its circumference?
Figure 5: Item 10 from the Pre-/Post-Test.

The scores show that the performance of the participants was very similar on the pre-test to their High School and statewide peers in the previous year (about 33%). Once again the post-test figures show significant gain in their performance.

Figure 6: Comparison of participant scores on Item 10 with larger populations.

Summary of Attitude/Background Questionnaire

Our attitude/background questionnaire investigated students' responses to questions regarding their present math classes, whether they liked working in small groups, their beliefs regarding their ability in and attitudes towards both Math and English, and their reasons for taking the course. We used a 5-point Likert "agreement scale" (1 = Not at All to 5 = Very Much).

The results show no significant impact in students' response to fundamental affective issues such as "Do you like math." We did not expect a change in such a small amount of time–although there may also be a damping effect because some students may not have regarded much of our non-traditional activities as "math." The frequency graph exhibits a standard normal distribution of response by the group of students both before and after completing the course, with an average score of 3.40. Also evident is that our groups of middle school students have a more positive attitude towards math (3.60) than our group of high school students (3.23) – not surprising given the nature of the populations.

One notable result, which is illustrated in the frequency charts below, is the

Figure 7: Student responses to the statement "I like math in general."

Figure 8: Student pre-test responses to the statement "I like to work in small groups."

difference between the middle and high school groups of students. It is clear that the 9th graders liked working in groups after the program more than they did before. The average response increases from 3.80 to 3.92 from pre- to post-test. Compare this with the group of middle school students whose average response scores dropped from 3.40 to 3.20. The program did not have any notable affective results for the middle school students although they increased their overall performance. This result speaks to how the two populations of students differ in their response to instruction and how this correlates with the improvement of their work.

Figure 9: Student post-test responses to the statement "I like to work in small groups."

Conclusions
Conclusion, Reflections, and a Look Ahead
The participants showed very positive results on core math topics for two very different populations of students in an intense after-school program exploiting new technologies and new curricular approaches. Importantly, the results include substantial gains on 10th grade MCAS Items despite the fact that one sub-population was comprised of middle school students and the other was comprised of the 9th graders who had the lowest district scores on the 8th grade MCAS.

While the results are very positive, we expect that (a) improvements can (and will) be made in subsequent iterations of the course and (b) that the high level of classroom support provided to an extremely competent teacher helped lead to the positive results. Hence, while (a) suggests further improvement and broader applicability, (b) suggests that comparable performance may be more difficult to achieve in less well-supported circumstances with more typical teachers under typical classroom circumstances. Overall, however, we are confident that the approaches used with this class are widely applicable given access to the technology and the curriculum materials.

The SimCalc Project stands ready to provide support to both middle and high school teachers who wish to use either the network-based materials or the previously constructed and commercially available materials (at no cost). We would also support the mathematics teachers whom provided their services to the program should they wish to train other teachers in the use of any parts of our technologies and resources — including the use of motion detectors in combination with SimCalc software.

We are particularly interested extending our research by offering a brief but intense follow-up course next September to certain students who took the Spring course. This course would focus on ideas underlying calculus and be somewhat more intellectually challenging than the Spring course, which deliberately held to linear function ideas that occur mainly in Algebra I.