Gerd Kortemeyer

November 2003

Background on LON-CAPA

In the fall of 1992, CAPA (a Computer-Assisted Personalized Approach) was piloted in a small physics class of 92 students. CAPA provides students with personalized problem sets, quizzes, and exams. “Personalized" means that each student sees a slightly different computer-generated problem: different numbers in numerical problems, different choices in multiple choice-type problems, different graphs, images, simulation parameters, etc.

Students are given instant feedback and hints, and may correct errors without penalty until the assignment due date, unless the instructor chooses a different setup. The system records the students' participation and performance, and the records are available online to both the instructor and the individual student.

The LectureOnline project was started to serve physics course material over the web in the fall of 1997 with 770 students. With only the web-browser as interface, LectureOnline enables instructors to seamlessly put together a presentation of material gleaned from all over the Internet, and to create different types of individualized online homework. Grading, communication, groupwork and enrollment are also handled by the system.

In 1999, the CAPA and LectureOnline groups joined efforts in the creation of the LearningOnline Network with CAPA, which provides a superset of the CAPA and LectureOnline functionalities. The LearningOnline Network with a Computer Assisted Personalized Approach (LON-CAPA) is an integrated system for online learning and assessment. It consists of a learning content authoring and management system allowing new and existing content to be shared and re-used within and across institutions, a course management system, an individualized homework and automatic grading system.

LON-CAPA is serving over 13,000 course enrollments per semester at MSU alone, and well over 23,000 course enrollments system-wide, ranging from middle school to graduate level courses. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology.

CAPA, LectureOnline, and LON-CAPA have directly or indirectly attracted research funding from the National Science Foundation, the Howard Hughes Medical Foundation, as well as the Sloan and Mellon Foundations, totaling approximately four million dollars at MSU.

Research Themes

Previous research indicates that in general learners do not necessarily like LON-CAPA, but judge it to be helpful in their learning progress. When asked if learners would like to use LON-CAPA again in a coming semester, the answer is generally yes. However, studies on the actual learning gain achieved by the use of online homework and instruction usually come out with no significant performance difference on the whole, e.g., [Bonham01,03;Russell01; Pascarella02]. When observing a course in session, though, it is hard to believe that such a significant change in instructional design could make no difference. The instances of overall non-significant difference are more likely than not the result of averaging effects, while the effect for specific classes of learners and of specific classes of curricular material may well be significant.

The impact of LON-CAPA feedback and help on learners with different epistemological beliefs

Learner epistemological beliefs (students’ views about the nature of knowledge and also their views about the nature of learning [Elby00]), expectations, subject domain specific attitudes, and metacognitive skills can have significant impact on their learning outcome, in spite of all external factors being the same [Redish03]. Indeed, in the case of CAPA, Pascarella [Pascarella02] found evidence of different learning impact on students with different epistemologies and problem-solving strategies: Students with expert-like views of learning judged traditional homework to be the better learning tool. As they were observed both while solving traditional homework, and while solving online homework, it became apparent that in the online setting they neglected to apply and exercise some of their metacognitive skills.

Pascarella attributed the undesirable impact of online homework on metacognition to the immediate “correct/incorrect” outcome feedback and the availability of multiple penalty-free tries: learners do not see the need for development and implementation of mechanisms to check their own results themselves by for example deriving the result in a different way, testing their solution in limiting cases, doing dimensional analysis, etc.

Yet, immediate feedback is widely recognized as an important benefit of online exercises [Bransford99]: in traditional homework, if graders are available, learners receive their results with a time delay, while the course itself moves on. The learner might not be able to make sense of the new material due to an undetected and uncorrected misunderstanding of the earlier material. Once the corrected homework gets returned to the learner, it might have lost relevance in light of the new material, and there is no way for learners to correct their mistakes. It is very common for learners to state that LON-CAPA keeps them from “falling behind.”

Beyond simple “yes/no” outcome feedback, already available in CAPA, feedback can consist of hints, which in their simplest form are supposed to help the students find the correct solution. Pasceralla found that students in general did not find these hints helpful, but it is both unclear for what reasons (and these reasons might well be different for students with different beliefs and expectations), and how much this evaluation was due to the particular set of homework problems and instructor-provided hints that were used. Since in CAPA and LON-CAPA material is provided by a large number of authors, hints vary strongly in their approach to being “helpful:” some are procedural (“you should multiply the acceleration with the mass to …”), some are background-providing (“read up on the relationship between force and acceleration”), some are attempting to rhetorically invoke certain thought processes (“what would happen if the force is doubled?”), etc. 

Feedback in LON-CAPA currently is automatically presented (system-initiated), whether the learner seeks it or not. Aleven, Stahl, Schworm, Fischer, and Wallace (at MSU) [Aleven03] studied the effect of on-demand help in interactive learning environments, and point out that active help-seeking is a metacognitive skill, namely self-regulation. There is currently no study of the relationship between epistemological beliefs and help-seeking behavior, but Aleven et al. argue that epistemological beliefs are likely to influence both when learners decide to seek help, and what type of help they might be looking for: will they seek help to “get the answer,” or proof and logical deduction leading them to resolve the problem?

At the time Pascarella conducted her study, our systems only had a very limited feedback and hint mechanism. Today, LON-CAPA allows for hints to be adapted to pre-anticipated wrong solutions, hints and feedback also on correct solutions, follow-up questions, the disabling of any correct/incorrect-feedback, as well as far more flexible control over the number and reward-structure of multiple tries. Since LON-CAPA is a complete learning content management system, hints can link back into targeted tutorial material. Thus far, no study has been conducted on how to adequately make use of this functionality to keep the benefits of the immediate feedback while at the same time not turning “Thinkers” into “Guessers” [Pascarella02].

• How do students with different epistemological beliefs interact with the current online homework mechanism?
• What changes can be observed in student problem-solving strategy if
• the immediate outcome feedback is switched off
• the number of tries are limited
• a penalty/bonus scheme is imposed based on the number of tries
• follow-up questions instead of hints are used
• help is available on-demand, rather than system-initiated

?

• How will the students’ evaluation of LON-CAPA’s usefulness as a learning tool change as any of the above changes are introduced?
• How will the students’ evaluation of the helpfulness of hints as any of the above changes are introduced?
• What is the impact of students’ achievement orientation on help-seeking behavior and the use of system-initiated help and feedback?
• What is the correlation between help-seeking behavior, as well as the use of volunteered help, and learning outcomes?
• Where is the best balance between system-initiated and the enabling of on-demand feedback and help?
• What type of hints (procedural, background-providing, etc) are most effective in which scenarios and for which learners?

The impact of gender

D. Kashy et al. [KashyD01a] found indications of gender-specific differences in learning gain, and in particular, that implementing CAPA in several aspects of a course (homework, quizzes, exams, exam corrections) appears to help close the gender gap. The initial evidence suggesting that CAPA technology may increase gender equity comes from a two-semester introductory physics course in which the first semester was taught using traditional methods and the second semester of the sequence was taught using technology. In a comparison of the grade distributions for male and female students, and a shift indicating that female students’ grades tended to show greater improvement during the second semester (using technology) relative to male students’ grades is clearly evident.  In a second analysis, data from two years of the first semester of an introductory physics course for scientists and engineers taught using technology were examined. This analysis looked at the progress of men and women over the semester by comparing men’s and women’s exam performance across the four exams administered. Across both years women began the semester considerably behind men but the difference between men and women decreased over time and by the final exam men and women did not differ in their performance.

Kotas [Kotas00] in his study of the use of CAPA in introductory physics at Central Michigan University found that women significantly outperformed men in two of three observed semesters, but he found no significant difference in the third. In an analysis of the log files, it turned out that in the semester where women outperformed men, women started to work on their homework earlier than men, and did not work up to the last minute like their male counterparts did. In the semester where male and female students performed similarly, there was also no difference in the homework access patterns. Across the semesters, females reported to be working more frequently in study groups than men. At MSU using LON-CAPA, in Spring 2002, females outperformed men in the completely online phy232c, and showed equal performance in online phy231c.

L. McCullough and C. H. Crouch conducted a study of introductory physics courses across eight different institutions of higher education [McCullough02] and found that while women showed a higher pre-/post-test gain, a significant gender gap still existed at the end of the semester, both in performance on the Force Concept Inventory [Hestenes92a] and in the overall course grades.

Based on the previous investigations, hardly any claims can be made yet regarding differential benefits for women using the online systems. A necessary first step would be to go back to pre-1992 (pre-CAPA) assessment and performance data and look for gender differences there.

Definitely no claims can be made regarding possible reasons for a differential benefit. It has been speculated that as the multiple attempts allow learners to persevere when responses are incorrect, they might enable learners to better overcome differences in preparation (across all studies, women’s pretest or early formative assessment scores were lower than men’s) – Kashy found that women frequently used more attempts than men to get problems correct. It has also been argued that the generally disproportionate number of male teaching staff might discourage female learners from seeking remedial assistance in person, but not from doing so online or directly from the computer system.

The impact of LON-CAPA personalization on student collaboration and “cheating”

Most faculty members would state that the purpose of the personalization feature is to prevent students from “cheating (the instructor).”  In addition, it is frequently claimed that personalization prevents students from cheating themselves, and that it enables students to solve problems in groups without the temptation of simply exchanging solutions, and instead to discuss the problem on a more “conceptual” level – in the end, the students will have to solve their own version of the problems.

Anecdotal evidence for some of these claims have been found when observing student groups in the helproom, but none have been systematically researched and verified ­– which, given the effort needed to prepare this material and code the relevant system functionality, is surprising.

In most group work scenarios toward solving a problem, participants quickly assume or are assigned different roles, both active and passive. It can be hypothesized that personalized assignments will reduce this tendency, especially that of assuming a more passive role, since it is not enough for the group to solve one problem – every participant in the end has to solve their own.

In addition to the official collaborative venues, such as the helproom and the online moderated discussion groups, a culture of “cheat sites” has been developing, most notably http://www.allmsu.com/, which currently advertises “73,408 class discussion entries.” Cheat sites like allmsu are frequently mentioned by faculty as a serious concern, and their existence is for some instructors reason enough not to use LON-CAPA. Kashy et al. [KashyD03] found that statistically students using the “cheat site” did worse in the course than those using the sanctioned venues, but it is unclear what is cause and what is effect.

Frequently what students and faculty would define as “cheating” differs – it should be kept in mind that if the presented problems are purely algorithmic in nature, students will not perceive it as “cheating” when they solved the questions by exchanging algorithms rather than discussing concepts – faculty do. In fact, especially students with less expert-like epistemologies will argue that they did exactly what was expected of them. It can be assumed that the nature of the presented material and questions has a strong influence on the quality of collaborations – by the reverse token, even on the declared “cheat site” allmsu, there are anecdotal occurances of valuable conceptual discussions, but these do not happen by accident.

• Do student discussions while working on their homework in groups indeed happen on a more conceptual level when having personalized instead of uniform assignments?
• How does the students’ and the instructors’ definition of “cheating” differ?
• How do different types of problems influence “cheating” behavior?
• Is there evidence of a more expert-like approach to group work and problem-solving as a team when having personalized instead of uniform assignments?
• How does student group work differ between face-to-face helproom, sanctioned online discussion, and “cheat site” online discussion?
• Under what circumstances do students use “cheat sites,” and how is decision to seek help at these sites related to their epistemological beliefs and expectations?

JiTT and classroom use of interactive problems

While certain immediate feedback may or may not be beneficial to certain learners, feedback is extremely important to the educator. Just-in-Time-Teaching (JiTT) has shown encouraging attitudinal and cognitive results [Novak03]. JiTT is based on the interaction between web-based study assignments and an active learner classroom. Students respond electronically to web-based assignments which are due shortly before class, and the instructor reads the student submissions "just-in-time" to adjust the classroom lesson to suit the students' needs. Thus, JiTT relies on a "feedback loop" formed by the students' outside-of-class preparation that affects what happens during the subsequent in-class time together [Novak03]. Physics JiTT assignments are available through Project Galileo [Mazur03].  Several years ago, an instructor in chemistry implemented JiTT strategies in chemistry using LectureOnline and her own study assignments, and reported that both she and her students were better prepared for lecture.

At Westshore Community College, also using LectureOnline, an instructor has extended JiTT into the classroom, which one time a week is actually a computer lab. Lecturing, problem solving, and discussions are interwoven. The instructor reports that he is better able to tailor his instruction, does detect individual learner problems better, and that “you can visibly see the different rates of uptake of math and physics knowledge” [Houk02].

• What impact will adaptation of JiTT study assignments and implementation in LBS physics courses have on learning gain?
• Does Westshore’s approach of using personalized assignments in class scale to class sizes at MSU, for example using customized PRS functionality or wireless PDAs?

Hybrid courses

Educators have been experimenting with hybrid courses, where classroom time has been replaced with online components. For example, at Hebrew University Jerusalem, G. Ashkenazi compared students who self-studied chemistry material with the aid of CAPA with those attending traditional lecture, and thus far found no significant difference in learning outcome [Askenazi03]. S. Riffell and D. Sibley at MSU taught a hybrid ISB course (containing LON-CAPA online assignments) simultaneously with a traditional lecture course.  They found that completion rates of LON-CAPA homework were greater than attendance rates to traditional lectures, and this difference increased with higher class rank.  Students in the hybrid class read and referenced their textbooks frequently each week, but students in the traditional course used their book rarely if at all.  Their results suggest that hybrid course formats might be effective for increasing student attendance and frequency of textbook use in introductory science courses [Riffell03a]. They also assessed student perceptions of this hybrid class, and most students felt that student-instructor interaction was better than in traditional courses and that online homework aided time management and learning. This suggests that online homework may replace traditional lectures when coupled with in-class, active learning activities [Riffell03b].

For several years now, phy231c and phy232c were run in a completely online mode out of LectureOnline and LON-CAPA, and given that the instructors show online presence and involvement, no significant difference in student performance compared to the lecture-based sections was found (for several semesters, both sections used the same final exams). This suggests that it may no be necessary to “cover” or even “touch on” all subject material in class – these “transmission” functions might frequently as well be carried out online. Instead, alternative uses of some of the regular physics lecture times should be explored.

Analysis of access patterns

When providing a web-based learning environment, author expectations of its use often do not match actual learner access patterns. While author might have invested significant time and effort into organizing and presenting their materials, learners often are driven by critical project tasks and assessments, and do not engage with the site beyond what is necessary to accomplish these goals [Sheard03].

For physics, a large number of LON-CAPA content resources are provided in addition to assessment problems. The material is still organized in back-of-the-chapter fashion, with the exercises following a chapter of content material. A statistical analysis of the web logs has shown that unfortunately these learning content resources are rarely used, and if at all, only after the learners attempted and failed to solve homework problems.

The First-Year On-Line group took a different approach to the relationship between content and assessment resources [McGroarty03]: they distinguished between so-called “Level 1” questions, which are embedded early on into the learning content and only require minimal interpretation of the presented content (corresponding to the Knowledge Level of Bloom’s taxonomy [Bloom56]) and “Level 2” questions (Comprehension and Application), which appear later in the units. Merrill [Merrill02] reported that even in this setting, a number of learners were still mostly driven by getting points: they browsed and worked only on the pages which had embedded assessment with a point value assigned, and neglected pages were zero-point “Level 1” questions were presented.

Kotas [Kotas00] conducted an analysis of online problem solving behavior using CAPA log files. Procrastination on homework was found to be the best negative predictor of overall success in the course, followed by the positive predictor of interacting with the material on a daily base. Students who worked on the homework immediately following the lectures also outperformed other students. In a more recent study, data mining technologies were employed to analyze individual access paths though the material in LON-CAPA [Minaei03]. Classes of different online behavior were identified and successfully used as a predictor of overall course success as measured in terms of final grade.

It has been hypothesized that there may be gender differences in these access patterns, i.e., that women might more frequently read the material before attempting the problems.

• How will individual student paths through the material differ between the end-of-the-chapter and the embedded assessment courses?
• How will changing the physics courses to the embedded assessment model change student online behavior?
• Will this change have an impact on learning gain?
• Can online student behavior be used reliably as an early indicator of learners at risk?
• Are gender and online access patterns indeed correlated?

Impact of different classes of curricular material

LON-CAPA is a tool, not a curriculum, and as such does not impose a certain teaching philosophy – it is not surprising that effectiveness strongly depends on how it is used.

For example, Kashy [KashyD01b] found that what the authors call “interactive problems,” namely those where learners need to read relevant values from graphs or observe simulations, are better predictors of overall success in the course than other problem types. Learners have to identify and “measure” relevant data themselves, and draw non-quantitative conclusions. More recent developments even allow for more than one correct solution for a given scenario along the lines of “if this happens next, then …”

These findings appear to correspond to Jacobson and Spiro (now at MSU) [Jacobson95], who found that learners with expert-like epistemological beliefs were better able to learn and apply their knowledge after using a complex hypertext system than those with non-expert-like beliefs. In their experiments, students had to respond to a variety of scenarios, and were given hyperlinks to “investigate” the aspects of the scenario, and answers were not unique.

This kind of problems is different from the majority of problems in the system, which are quantitative in nature and would be characterized as “zero friction” problems by Redish [Redish03]: highly idealized scenarios, all and only the needed data are given. “Problems with these problems” are pointed out by Mazur [Mazur96] and acknowledged by most physicists. Experience with having LBS honors students write instead solve online problems indicates how little transfer appears to occur for “zero friction” problems – students are largely utterly unable to construct such scenarios, even though they are successful at solving them.

Many of the problems in the so-called “interactive” category cannot currently be technically implemented in systems other than LON-CAPA, except through the application of custom programming.

In the study of problem types, the “interactive problems” represented only a small fraction of the two hundred assigned problems – this is understandable given the amount of effort required at that time to write such problems. Currently, it is still unclear what exactly it is about those problems that leads to the observed correlations, and what exactly is cause and what is effect: did students who successfully solved this problem class “learn more” from them, or were it simply the “better” students who were able to solve these (in general) more difficult problems?

Also, the level of difficulty of the problems used in the study was increased not only by their interactive character, but also by the requirement to draw more than one correct conclusion at the same time to get credit (as opposed to coming up with for example just one number) – this additional requirement should be dropped to achieve better comparison.

Interactive problems could easily be taken one step further to include estimations – the ability to estimate and do quick “back-of-the-envelope” calculations is highly valued by most physics, but with few exceptions not reflected in problems. It was also suggested to include in the answer some check boxes that help define the principles or laws, which the students identify as applicable to the problem.

Methodology

Survey instruments

Pre-/Post-Test instruments

The qualitative Force Concept Inventory [Hestenes92a] and the quantitative companion Mechanical Baseline Test [Hestenes92b] have been used in a large number of studies connected to the teaching of introductory mechanics, however, not the only ones available. The Foundation Coalition has been developing a number of relevant concept inventories [Foundation03], namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields).  Since these were designed from an engineering point of view, some adjustment might be necessary.

Interviews

To start addressing some of the mentioned “cause and effect” questions, as well as for formative studies, students will be invited to interviews. This will typically be a smaller sample of the student population in a given course.  Pascarella developed some frameworks for these interviews, which can be built upon.

Interaction analysis

Online discussions are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns. Discussion contributions and states can be linked to online transactions, such as submission of homework attempts, browsing of content material, and hint usage. Raven Wallace at MSU [Wallace03] reviewed existing research on such online interactions, however, some adaptation of several of the existing conceptualizations will be necessary to account for the nature of science and in particular physics courses.

Helproom discussions between students can be taped and analyzed in a similar way to the online discussions, and linked to online behavior if absolute timing and learner identify is preserved.

Control

An issue at MSU is that control for several of the research questions is hard to obtain. All undergraduate physics classes at MSU are using LON-CAPA, and given the fact that students generally desire to use LON-CAPA and as a majority feel that it does them help to learn, it will be hard to establish control groups that do not use the system at all. So control will likely need to be established by varying LON-CAPA functionality for the whole course between different parts of the semester.

The current introductory physics courses at LBS are divided into thirds by the two midterm exams. A viable research setup may be to divide the course into two groups with treatments A and B, then switch the groups after the first midterm, and have students select treatments according to their preferences after the second midterm.

Research-Based Curriculum Development

Insight in the above research topics will enable informed and research-based curriculum development. This will include homework and exam problems, as well as content material. Curriculum developed within other projects will be evaluated, selected, adapted and implemented. Curriculum development efforts will take advantage of LON-CAPA’s inherent cross-institutional content sharing mechanism, which facilitates the dissemination of material, as well as the existing gateway to the National Science Digital Library.

References

[Aleven03] V. Aleven, E. Stahl, S. Schworm, F. Fischer, R. Wallace, Help Seeking and Help Design in Interactive Learning Environments, Review of Educational Research, Vol. 73, No. 3, pp 277-320 (2003)

[Artus99] N. N. Artus, K. D. Nadler, A Computer-Assisted Personalized Approach            in an Undergraduate Physiology Class, Journal of Plant Physiology, Vol. 119, pp 1177-1186 (1999)

[Ashkenazi03] G. Ashkenazi, private communication (2003)

[Bloom56] B. S. Bloom, M. D. Engelhart, E. J. Furst, W. H. Hill, D. R. Krathwohl, Taxonomy of Educational Objectives, Handbook 1: Cognitive Domain, Edward Bros, Ann Arbor (1956)

[Bransford99] J. D. Bransford, A. L. Brown, and R. R. Cocking (editors), How People Learn: Brain, Mind, Experience, and School, National Research Council, ISBN 0-309-06557-7 (1999)

[Bonham01] S. W. Bonham, R. J. Beichner, D. L. Deardorff, Online Homework: Does It Make a Difference?,  The Physics Teacher (2001)

[Bonham03] S. W. Bonham, D. L. Deardorff, R. J. Beichner, A comparison of student performance using web and paper-based homework in college-level physics, submitted.

[Chi81] M. T. H. Chi, P. J. Feltovich, R. Glaser, Categorization and Representation of Physics Problems by Experts and Novices, Cognitive Science, Vol. 5, 121-152 (1981)

[Elby00] A. Elby, D. Hammer, http://www2.physics.umd.edu/~elby/papers/epist_substance/Substance.html (2000)

[Elby01] A. Elby, J. Fredriksen, C. Schwarz, and B. White, Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument, http://www2.physics.umd.edu/~elby/EBAPS/EBAPS_items.htm (2001)

[Foster00] T. M. Foster, The Development of Student’s Problem-Solving Skill from Instruction Emphasizing Qualitative Problem-Solving, dissertation, University of Minnesota (2000)

[Foundation03] Foundation Coalition Key Components: Concept Inventories, http://www.foundationcoalition.org/home/keycomponents/concept/index.html (2003)

[Hestenes92a] D. Hestenes, M. Wells, and G. Swackhamer, Force Concept Inventory, The Physics Teacher, 30 (3), 141-151 (1992)

[Hestenes92b] D. Hestenes and M.Wells A Mechanics Baseline Test, The Physics Teacher, 30 (3), 159-166 (1992)

[Houk02] D. Houk, private communication (2002)

[Hunter00] P. W. W. Hunter, The Use of a Computer-Assisted Personalized Approach in a Large-Enrollment General Chemistry Course, University Chemistry Education (Royal Society of Chemistry), Vol 4, No. 2, p. 39 (2000)

[Jacobson95] M. J. Jacobson and R. J. Spiro, Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: An empirical investigation, Journal of Educational Computing Research, Vol 12, pp 301-333 (1995)

[KashyD01a] D. A. Kashy, G. Albertelli, E. Kashy and M. Thoennessen, Teaching with ALN Technology: Benefits and Costs, Journal of Engineering Education, 89, 499 (2001)

[KashyD01b] D. A. Kashy, G. Albertelli, G. Ashkenazi, E. Kashy, H.-K. Ng, and M. Thoennessen, Individualized Interactive Exercises: A Promising Role for Network Technology, Proceedings, Frontiers in Education Conference (2001)

[KashyD03] D.A. Kashy, G. Albertelli, W. Bauer, E. Kashy, M. Thoennessen, Influence Of Non-Moderated And Moderated Discussion Sites On Student Success, Journal of Asynchronous Learning Networks, Vol.7, No. 1 (2003)

[KashyE93] E. Kashy, B. M. Sherrill, Y. Tsai, D. Thaler, D. Weinshank, M. Engelmann, and D. J. Morrissey, CAPA, an integrated computer assisted personalized            assignment system, Am. J. Phys. 61 (12), pp. 1124-1130 (1993)

[KashyE95] E. Kashy, S. J. Gaff, N. H. Pawley, W. L. Stretch, S. L. Wolfe, D. J. Morrissey, and Y. Tsai, Am. J. Phys. 63 (11), pp. 1000-1005 (1995)

[KashyE96] E. Kashy, M. Thoennessen, Y. Tsai, N. E. Davis, and S. L. Wolfe, Using Networked Tools to Enhance Student Success Rates in Large Classes, IEEE Transactions on Education (1996)

[Kluger96] A. N. Kluger and A. DeNisi, The Effects of Feedback Interventions on Performance: Historical Review, a Meta-Analysis and a Preliminary Feedback Intervention Theory. Psychological Bulletin, 119, 254-284 (1996)

[Kluger98] A. N. Kluger and A. DeNisi, Feedback interventions: Toward the understanding of a double-edged sword, Current Directions in Psychological Science, 7, 67 (1998)

[Kotas00] P. Kotas, Homework Behavior in an Introductory Physics Course, Masters Thesis (Physics), Central Michigan University (2000)

[Mazur96] E. Mazur, The Problem with Problems, Optics and Photonics News, Vol 6 , pp. 59-60 (1996)

[Mazur03] E. Mazur, Project Galileo, http://galileo.harvard.edu/

[McCullough02] L. McCullough and C. H. Crouch, American Association of Physics Teachers Conference (2002)

[McGroarty03] E. McGroarty, J. Parker, M. Heidemann, H. Lim, M. Olson, T. Long, J. Merrill, S. Riffell, J. Smith, J. Batzli, D. Kirschtel, Supplementing Introductory Biology with On-Line Curriculum, draft, Michigan State University (2003)

[Merrill02] J. Merrill, Colloquium, Michigan State University (2002)

[Minaei03] B. Minaei-Bidgoli, W. F. Punch, Using Genetic Algorithms for Data Mining Optimization in an Educational Web-based System, Proceedings, Genetic and Evolutionary Computation Conference (2003)

[Morrissey95] D.J. Morrissey, E. Kashy, Y. Tsai, Using Computer-Assisted Personalized Assignments for Freshman Chemistry, Journal of Chemical Education, Vol. 72, pp 141-146 (1995)

[Novak03] G. Novak, Just-In-Time Teaching, http://www.jitt.org/

[Pascarella02] A. M. Pascarella, CAPA (Computer-Assisted Personalized Assignments) in a Large University Setting, Ph.D. (Physics) dissertation, University of Colorado (2002)

[Redish98] E. F. Redish, J. M. Saul, and R. N. Steinberg, Student expectations in introductory physics, Am. J. Physics, Vol 66, pp 212-224 (1998), survey available on The Physics Suite CD in Teaching Physics, ISBN 0-471-39378-9 (2003)

[Redish03] E. F. Redish, Cognitive Science and Education: A theoretical framework, draft, University of Maryland

[Riffell03a] S. K. Riffell and D. F. Sibley, Increasing Attendance and Textbook Use in Undergraduate Science Courses: An Experimental Test of a Hybrid Course Format, Computers and Education, in review (2003)

[Riffell03b] S. K. Riffell and D. F. Sibley, D.F., Student perceptions of a hybrid learning format: can online exercises replace traditional lectures?, Journal of College Science Teaching, vol XXXII, Number 6, pp. 394-399 (2003)

[Russell01] T. L. Russell, The No Significant Difference Phenomenon, A Comparative Research Annotated Bibliography on Technology in Distance Education as Reported in 355 Research Reports, Summaries, and Papers, ISBN 0-9668936-0-3 (2001)

[Schoenfeld85] A. H. Schoenfeld, Mathematical Problem Solving, Academic Press (1985)

[Sheard03] J. Sheard, J. Ceddia, J. Hurst, J. Tuovinen, Inferring Student Learning Behaviour from Website Interactions: A Usage Analysis, Education and Information Technologies, Vol 8:3, p. 245 (2003)

[Thoennessen96] M. Thoennessen and M. J. Harrison, Computer-Assisted Assignments in a Large Physics Class, Computers Educ. 27, 141 (1996)

[Thoennessen99] M. Thoennessen, E. Kashy, Y. Tsai, and N. E. Davis, Impact of Asynchronous Learning Networks in Large Lecture Classes, Jour. Group Decision and Negotiation 8, 371 (1999)

[Wallace03] R. M. Wallace, Online Learning in Higher Education: a review of research on interactions among teachers and students, Education, Communication and Information, Vol 3, No. 2, 241 (2003)