Research Topics

1 Resource Pool Infrastructure

We plan to implement and field test Dublin Core [18], as well as elements of the IMS educational metadata standards [19] as parts of our library functionality. The applicability of these standards will be tested both in the cataloging of "legacy" material, as well as in the situation where authors catalog their own newly created resources — what tools and "wizards" need to be provided to make this feasible? Are other educators finding resources based on this metadata?

Royalty schemes would be relatively straightforward if resources were always used as-is: The original author sets a royalty price for the usage of a resource per course per student, the system keeps track of resource usage and enrollments, deducts a service-fee, and eventually pays the author.

However, to further encourage resource re-usage and improve the quality of resources, resource consumers should be able to modify the resources for their own use, or even be "value adders". Adding value to a resource (while certainly also preserving the resource in its original form) in a way similar to Open Source Communities involves additional royalty issues in this scenario. We intend to for example explore innovative licensing schemes such as the Intellectual Capital Appreciation License put forward by the Educational Object Economy EOE [20]. How can it be assured that the material still remains affordable to the learner, in fact, that the price is comparable or lower than that of a traditional course pack? What is the business plan?

Regarding the functionality aspects similar to an instructional management system, we plan to investigate issues surrounding adaptive curricula and multiple knowledge representations. How can the generation of such non-linear course sequences be facilitated using metadata? How can it be insured that learners do not "get lost" between these options and branchings?

What is an acceptable review process for sub-libraries of the resource pool, out of which educators can randomly generate "standardized" online exams similar to the Force Concept Inventory [21] to objectively evaluate learning success, while insuring that one is not "teaching to the test"?

What feedback, survey and statistical mechanisms do have to be implemented for educators to evaluate the effectiveness of their resources?

2 Learner-Centered Environments

Learning is a complex and highly individual process, whose rate of success cannot be reduced to a function of just a few variables for all learners. Rather, the variables are numberless, and their relative weight will differ from learner to learner [29,30]. While on the one hand we do seek to gather statistical data to gauge the overall effectiveness of our tool and its contents, we believe that there is data in the statistical noise generated by each individual learner - data, however, which applies only to that particular learner.

We believe that the build-in adaptivity of LON-CAPA is best utilized in connection with the system "getting to know" the individual learner, and see it as a research goal to move from single-variable pretest-posttest measures of learning success to studies of ongoing learning processes. The system can easily gather floods of data on individual usage patterns, especially as learners go through multiple steps towards solving assignments, and as they along the way choose between multiple representations of the same knowledge elements (for example, video lecture demonstrations, a derivation, a worked example, case-studies, etc). As the resource pool grows, an increasing number of such multiple content representations will be available to the learner. The system can log the nature of the representation chosen by the learner based on the resource's metadata, as well as sequence and frequency of access in relation to successful completion of an assignment.

However, in the study of such patterns, we are immediately confronted with the chaotic nature and high dimensionality of the problem. To overcome this, we propose to apply Genetic Algorithms (GAs) in combination with a k-nearest neighbor (knn) classification technique to identify salient features which discriminate preclassified data. GAs are an artificial adaptive system based on a biological model of both Mendelian genetics and the theory of Darwinian evolution [31]. They have been successfully used at MSU on large feature selection problems [32,33], and have shown the ability to search very large feature spaces efficiently and effectively. Knn is a means to classify new features into groups of known samples. The hybrid GA-knn approach gauges the "importance'' of each feature for the task of discrimination, and thus identifies the features which should be used for subsequent classification.

Internal to the system, each learner will have a "chromosome" representing the weights assigned to each feature. A weight of 0 effectively removes the feature from consideration, while other values modify the importance of each feature for classification. In this manner, the system is now searching for a relative weighting of features that gives optimum performance on classification of the known samples. Those weights that move towards 0 indicate that their corresponding features are not important for the discrimination task. Essentially, those features "drop out'' of the feature space and are not considered. Any weight that moves towards the maximum weight indicates that the classification process is sensitive to small changes in that feature. That is, the feature's dimension is elongated, allowing better class resolution along this feature axis. The end result of running GA feature extraction is a scaling of each of the features such that an optimal class separation can be achieved between the known classes. This technique has been used at MSU on various data sets, for example using a thousand features with many thousands of patterns, and it has been found that as few as 35 features that can discriminate the predefined classes [34].

We hope to find similar patterns of use in the data gathered from LON-CAPA, and eventually be able to make predictions as to the most-beneficial course of studies for each learner based on a limited number of variables for each individual student based on their "chromosome." Based on the current state of the learner in a learning sequence, the system could then make suggestions to the learner as to how to proceed. In system science, this type of creation of structure via selection rules and feedback loops is known as self-organized criticality.

Beyond this, by comparing individual learner "chromosomes" with each other, we in addition hope to discover classes of users with similar patterns of use and similar sets of success-determining variables. We claim that the "no-significant difference phenomenon" between traditional and non-traditional teaching methods is not due to an actual lack of a difference, but rather due to inappropriate averaging over different classes of learners and the resultant canceling-out of effects: what works for one class of students does not work for another class, and might even be counter-productive for a third class.

3 Evaluation of Effectiveness in the Use of Technology for Teaching and Learning

Evaluation of resource material quality and effectiveness of materials for learning

One of the most challenging aspects of this project is to provide resource users with information concerning the quality and effectiveness of the various materials in the resource pool in terms of their effects on student understanding of concepts and their knowledge of procedures. These materials will include web pages, demonstrations, simulations, and individualized problems designed for use on homework assignments, quizzes, and examinations. Methods for both objective evaluation and subjective evaluation of material quality and effectiveness are proposed.

• Objective evaluation of problems designed for use on homework assignments, quizzes, and examinations: The proposed system will generate a range of statistics that can be useful in evaluating the degree to which individual problems are effective in promoting formative learning for students. To evaluate whether individual problems designed specifically to generate formative learning are successful, these problems can be paired with examination problems that are designed to be summative and the correlation between performance on the two types of problems can be computed. Performance on individual problems can also be correlated with measures of overall success in the course.

• Objective evaluation of all materials: We propose to construct an automated exam engine that will produce randomized and/or individualized tests without the instructor having the need or even the opportunity to select the problems. This can be done by providing a large pool of exam questions via an open-source database. Each exam problem contains attached metadata that catalog its degree of difficulty and discrimination for students at different phases in their education (i.e., introductory college courses, advanced college courses, and so on). In this way, faculty cannot "teach to the test", and their teaching innovation receives a fair evaluation that is standardized and can be compared to national data [22,23]. While we intend to use this tool for the formative and summative evaluation process in our own project, we will also make it available to the national education community. Fundamental research on teaching and learning needs solid evaluation tools, and we plan to provide one of them.

• Subjective evaluation of all materials: To evaluate resource pool materials a standardized format will be required so that materials from different sources can be compared. This will help resource users in selecting the most effective materials available. Two questionnaires will be developed:

1. A standardized measurement instrument that will be completed by faculty who use the material. The questionnaire will assess faculty evaluation of resource quality and efficacy.
2. A standardized measurement instrument that will be completed by every student who uses the material in a course. The questionnaire will assess student evaluation of resource quality, helpfulness in learning the material, clarity, and will probe for student suggestions for ways in which materials can be improved.

Evaluation of the LON-CAPA project

There are several studies that report positive impact on student achievement when using technology [e.g., 36], while many indicate no significant difference [23]. We need to better understand when, where, and how the use of technology enhances education as opposed to be simply mediating it.

• To evaluate the success of the networked technology with respect to student learning, historical data concerning student performance in comparable courses prior to implementation of the technology will be obtained from LON-CAPA users. Such data, in combination with data concerning student performance using the technology, will allow us to compare student achievement with and without technology and some insight into the effects of the technology on actual student outcomes can be obtained. To this end, educators who use resource materials will be requested to share information concerning student performance in their courses prior to implementing the technology. They will also be asked to provide information concerning student outcomes when the technology was used in the course. (All student data will be anonymous.) In addition to student performance data, instructors will be asked to provide copies of their course materials (e.g., hand-outs, assignments, examinations) from their pre-technology courses as well as from their courses using LON-CAPA. These materials will be examined by a panel of faculty experts to determine whether and to what degree the content of the course was affected by the implementation of the technology.

• A comparison study for four different approaches to teaching introductory physics content will be conducted. The different approaches will include a conventional large lecture format with hand-graded homework assignments, a conventional large lecture format with computer-graded homework assignments (using the LON-CAPA system for homework, quizzes, and examinations), a completely web-based format with weekly online homework assignments, and a completely self-paced format with on-demand exams. Although all of these models are in current use, either at MSU or at other universities, there presently is no study that can directly test the relative effectiveness of the four formats. To make such a direct comparison, a subset of identical problems must be included on examinations so that there is equivalence in the outcome measure. Although it will not be possible to randomly assign students to the four formats, possible confounding variables such as intelligence/ability will be measured and controlled statistically using analysis of covariance.

Evaluation of content representations and their effectiveness in different subject categories

A large diverse resource pool, together with standardized measuring mechanisms, provides a unique opportunity to study the relative effectiveness of various media types (static web pages, graphics, audio-files, simulations, problems, …) and formats (textual, mathematical, audio-visual, …). Because a large number of instances of each type of content representation will be available (the type being identified by metadata), the investigators will be able to pool information while averaging over the specific instances. Thus, we will be able to determine whether some types of media are most effective in conveying information overall, irrespective of their specific educational content domain. In fields for which a wide range of materials have been posted into the resource pool, the question of which representation is most effective can be tested separately by content domain

Evaluation of content representations and their effectiveness for different types of students

Educational specialists suggest that the effectiveness of content representations varies substantially from individual to individual [29,30,35]. The current proposal will allow us to study several questions concerning what types of students benefit most from various media types and formats.

• A relatively simple yet important individual difference that is likely to predict the effectiveness of a particular content representation is the gender of the student [37]. Because men tend to show increased spatial skills relative to women on average, resource materials that are more visually oriented should be more effective for men than for women. On the other hand, women may be more successful when material is presented in the form of verbal text on a web page. However, the literature also indicates that when women have experience work in environments that draw on their spatial skills, their spatial abilities can improve. The evaluation system we have proposed will allow us to test whether such gender differences do indeed exist.

• When information concerning student disabilities is available, we will also be able to study the effectiveness of different content representations for students with learning disabilities or other disabilities such as visual impairment or hearing impairment. Although the representation of such students tends to be very small in individual courses, across many courses and many universities, a reasonably sized sample of students with disabilities should be available. The technology may be especially effective in promoting learning for such students.

The data collected from these evaluations will be made available on both a resource specific (as dynamically generated part of the individual resource metadata) and general level (as part of the general dissemination) to instructors, and will eventually facilitate the selection of appropriate material from the resource pool. At a more general level, the information gathered by these various forms of evaluation will also be published to contribute to a better understanding of the role of technology in education.

4 Sustainable Business Entity

The success of LON-CAPA involves 1) enhancing the technological capabilities of the existing CAPA and LectureOnline environments and 2) devising a strategy that encourages stakeholders (e.g., faculty, students, educational institutions, commercial publishers) to actively participate. While a number of current CAPA and LectureOnline users have indicated an interest in the enhanced LON-CAPA environment, designing a system that increases the likelihood of generating a critical mass of users is crucial. Thus, concurrent with enhancing the technological environment of the LON-CAPA system, we will conduct "proof of concept" research to understand the expectations of the LON-CAPA stakeholders. This "proof of concept" undertaking will ensure that the necessary stakeholder expectations are designed and implemented into LON-CAPA while simultaneously formulating a strategy for creating a sustainable non-profit business (lon-capa.org). It is important to note that while this proof of concept is essential for creating a sustainable entity, the explicit functions (e.g., structure of royalty mechanism) that will be developed are unclear at this time necessitating research investigation. Thus, in order to conduct this proof-of-concept research, a LON-CAPA advisory board will be created made up of a diverse set of stakeholders whose needs must be addressed to create a critical mass of users.

The creation and involvement of an advisory board is crucial for two reasons. First, commercial software currently exists [5-7] to support web-based delivery of materials. While this software has been purchased by a growing number of institutions, we believe the LON-CAPA environment can provide significant enhancements. Beyond the technical enhancements (e.g., individualizing assessments, adaptivity, resource pooling and sharing, etc), the non-proprietary and open LON-CAPA environment creates the potential for a community of educators to synergistically develop and deliver the richest set of materials available to support learning. The advisory board members can provide ongoing feedback regarding the necessary and desired capabilities of LON-CAPA to ensure that it meets/exceeds expectations. Secondly, creating a learning community that can flourish necessitates embedding mechanisms into LON-CAPA that encourage educators and publishers to participate actively in the environment. Thus, the development and integration of a dynamic royalty mechanism that facilitates the creation of a critical mass of faculty, academic institutions, and publishers is critical. Further, a royalty mechanism is likely to provide added incentive for faculty/institutions to generate high quality products - a marketplace is created where different materials compete.

In order to understand the stakeholders' expectations associated with these and other issues, on-site visits, interviews, and surveys will be conducted. Initial focus will be on collecting data from stakeholders located at higher educational institutions as well as the textbook publishers. LON-CAPA must satisfy the needs of at least four different stakeholders - below are some of the issues that will need to be successfully addressed in order to satisfy these needs; the PI team and the LON-CAPA advisory board will bear responsibility for making recommendations in each of these areas:

1. Faculty

• LON-CAPA system ease of use expectations/faculty technological competency required for adoption
• Royalty expectations for different types of content
• Content control expectations
• "Packages" of content that would encourage LON-CAPA adoption
• Expectations regarding LON-CAPA process and outcome capabilities

1. Students

• LON-CAPA system ease of use expectations/student technological competency required for effective use
• Acceptable price range for "course pack"
• Content delivery: what content is most preferred electronically versus paper

1. Educational Institutions

• Technological infrastructure and technology support capabilities available for implementing/administering LON-CAPA
• Desired centralized versus decentralized hosting of content, etc.
• Institutional pricing models

1. Commercial Publishers

• Content/Royalty payment structure (textbook, ancillary materials, videos, etc.)
• "Chunking" of material. The degree to which 2 chapters from one textbook can be combined with 3 chapters from another publisher’s text