Instructional Technology Conference 2008

Title: Proposed model for evaluating course management system usage by faculty in higher education institutions

Name: James Janossy

Audience Level: Faculty with some contact with course management system usage, administrators in college departments using course management systems

Audience: Faculty, instructional technology specialists, distance learning professionals, deans, operations managers, and institutional research personnel

Length: One hour

Abstract:
Course/learning management (C/LMS) software accessed by students via the internet is now common. Yet assessing the extent of usage of this software by faculty often relies on incomplete data, ad hoc surveys, or estimates based on guesswork. We propose a usage model and summary metric that assesses the extent of C/LMS usage for each course and software methods to determine it. The derivation method can be applied to different C/LMS software to provide a reliable means of comparing usage across different systems.

Description:
We propose a model for course/learning management system (C/LMS) usage and derive a metric expressed as two numbers, based on established practice in a different discipline. This metric expresses in a standard summary form the extent of course management system usage by a course and the "richness"; of feature usage for imparting learning beyond traditional "chalk and talk"; lectures. The model and metric are proposed as a starting point for development of software to extract relevant measures directly from the database supporting a C/LMS with the intent of refining both the metric and the software to produce it. This effort would begin with analysis of the database supporting a generally-available and widely used product such as Blackboard, as implemented and currently utilized in a major institution of higher education. Methods to produce the metric will be constructed based on content analysis of database tables, in comparison with the known feature usage of the top 10% of current courses for which the database contains the most entries, on the assumption that these courses will represent the greatest extent of C/LMS feature usage. Known feature usage will be examined by actual inventory of the web sites of these courses. The metric produced will be judged as to its efficacy and accuracy in portraying C/LMS based on this criteria.

The goal of this effort is to construct a pilot version of a workable metric derived automatically from the C/LMS that would ultimately to allow administrators to unambiguously and accurately assess its usage across the entire institution. The provision of such a metric could alleviate the high degree of imprecision that currently exists in making such usage assessments and facilitate more accurate correlation of learning outcomes to C/LMS usage for purposes of administrative and academic resource allocation decision-making, helping to provide answers to questions such as these:

• is the C/LMS being used consistently in the same course as taught by different instructors? If not, what courses show the greatest degree of disparity, and why?
• is the C/LMS being used to a consistent degree across various courses with seemingly similar types of subject matter, for example, between a chemistry course and a biology course? What factors might justify this difference, or should some greater degree of consistency reasonably be expected to enhance the student learning experience?
• is the C/LMS used to the desired degree of consistency between major units of the university, such as colleges? If significant differences in usage are revealed, are they reasonable or should some greater degree of consistency be expected?
• is there any correlation between greater usage of the C/LMS, as indicated by the proposed metric, and student performance (grades) and retention rates? Knowing the metric for each course taken by a given student across the student's career could produce the data needed for longitudinal studies, supplying a variable heretofore unavailable.

Session Type: Lecture/Presentation

Contact information/affiliation:
James Janossy
DePaul University School of Computer Science
243 S. Wabash Avenue, Room 400
Chicago, Illinois 60604
(773) 325-8629
e-mail: jjanossy@depaul.edu

Equipment: Venue must provide a computer projector and screen. Note: the presenter will bring a laptop to connect to the provided computer projector.

---

Return to Proposals