The designer of instructional multimedia needs to be aware of currently accepted practices to keep appraised of the array of methods and applications from which he or she will want to choose, and also to be aware of practices that appear to be expected, appropriate, and functional. Most experts agree that several measures and methods should be used to determine course effectiveness

Current computer-based instruction (CBI) and Web-based instruction (WBI) assessment practices include a variety of methods such as instructors evaluations of students, students’ own self assessments, peer to peer assessments, exams, project based assessment, and assessments that are tied into log in and usage data. Bergman and Moore (1990) suggest three questions that designers should answer about the evaluation process: Is the evaluation voluntary or required? How much user activity is being assessed? What type of feedback is provided? Bergman and Moore believe that voluntary self-assessment "is a sound strategy for adults" (p. 63). However, they also list three more traditional categories, common in assessment, that they feel are appropriately used for specific circumstances: single items -- to test knowledge of single concepts; grouped items -- for assessing knowledge of "logical units" (i.e., on chunked material); and multiple groups of items -- for a final test

Current evaluation procedures range from "one shot grading," such as taking a final or semester test, to mastery or "perfection-based" grading, which allows students to redo assignments, within an acceptable time frame, until they are perfected (Cox, 1999). Counter to traditional design theory is the growing need to specify instructional goals and learning outcomes to accept what is termed as a "diversity of outcomes." Duchastel explains that this can best be achieved by, "aligning evaluation not with knowledge, but with task accomplishment that utilizes knowledge" (1997, p. 225).

The most critical issues for instructional designer are those regarding how best to plan, present, and evaluate objectives and outcomes for CBI/WBI instructional materials. Bannan and Milheim (1997) describe the issue in this manner:

These issues are also concerns of the designer of instructional programs delivered by means of CD-ROMs or other computerized, stand-alone tutorial materials. In traditional instructional design, objectives are the critical, conceptual foundation on which the instructional designer constructs the framework of the instruction to be. The instructional objectives are usually the starting point, and serve as the basis for all other instructional-design decisions made during the instructional design and development process. (Dick and Carey, p. 5). As explained by Dick and Carey, the objectives establish the purpose of instruction in terms of: desired learner outcome behavior or performance, conditions under which the behavior is to be evaluated, and the standards for evaluation and assessment.

In terms of learner assessment, then, objectives tell the learner, at the outset, what he or she is expected to learn in the course of the lesson. For the instructional designer, identifying the objectives of the instruction to be delivered is a critical step in choosing the best cognitive and presentational strategies for the instructional material, and involves awareness of the learner's needs as well as issues such as cognitive styles, need for learner interaction, and other related concerns. (p. 126)

For the instructional designer who is developing CBI or WBI, instructional objectives appear to be as critical as they are to the designer of "traditional" instruction. That is, for the designer, objectives are a critical step in (1) the designer's choice of the best cognitive and presentational strategies for the instructional materials to be developed as well as (2) in the choice of best media for instructional delivery other, similarly critical decisions in instructional development (Pernici and Casati, pp. 246-247). Basically, the designer of computer- or Web-based instruction has to know the outcome intended -- learning desired or task to be achieved --and the assessment method that will be used to evaluate that outcome before the designer chooses the best way to inform the learner about those objectives.

The instructional designer has to plan so that the initial objectives provide, for the learner, clear and reasonable expectations about the assessment phase of instruction. In general, because multimedia instruction is usually at least one step removed from face-to-face teacher-to-student interaction, the statement of objectives at the beginning of a lesson plays a critical role in learner orientation. For distance learners, the stated initial objectives of an on-line course of study or a CD-delivered tutorial may be the chief means of allowing the learner to self-advise in selection of one course of instruction over another. (Benson, 1994).

In addition, for "remote" or CBI instruction delivered as tutorials or via "automated" computer-tutorial, initial stated objectives have proved to be important as "advance organizers" for learners' outcome performance. For example, in an experiment by Beasley and Waugh (1996), groups of college students studying computerized units of instruction showed remarkably different test scores, and significant differences in content retention, that were affected by only one experimental factor. The experimental group of students was told that, after viewing the instructional materials, they would be asked to draw a concept map of how the topics of the lesson related to one another. Knowing the objective -- and the means of assessing what they were to study -- acted as a powerfully effective "advance organizer" in allowing the students to view the instructional materials in the context of the performance goal and to perform the target task far better than the control group, who were not told anything other than that they were to "cover" the unit of study. Not only did the experimental group -- the ones told the objective -- perform better than the control group, these differences in assessed performance were still present a week later when the two groups were given a "pop quiz" to check their retention of the information. Again, the group that was informed of the assessment objective performed significantly better than the "unknowing" control group.

The role of learning objectives as indicating performance outcomes and being critical in deciding on a course of study is also related to the issue of "just-in-time" learning, which is an important source for Web-based learning and instructional design. Just-in-time learning is information delivered "so that each individual may learn just what he or she needs at the time when he or she needs it" (Romiszowski, 1997).

The "just in time" learning concept is related to the idea of both goal-oriented learning and to performance-based assessment. It is therefore similarly related to the requirements that objectives and assessment outcomes be clearly established from the beginning of the instructional development process so that the instructional designer can choose which cognitive strategies and which presentation and assessment methods are most probably going to result in the desired learning.

In addition to a final assessment, in the sense of an overall "grade" for the learner who completes the unit of multimedia instruction, Bergman and Moore discuss three types of more sophisticated feedback categories for learner interaction that they believe should be considered by the instructional designer: Informative feedback, that simply identifies a correct or incorrect response; Reinforcing feedback, that praises or critiques (both of which should be limited); Diagnostic feedback, that involves comments or suggestions about the learner's response.

Iuppa, on the other hand, distinguishes between "testing" and "exercises," saying that "tests are not designed to teach," (p. 125) and therefore, in the case of learners who do not perform well on tests, no remediation needs to be provided. In terms of Bergman and Moore's categories, Iuppa seems to be saying that only informative feedback is required for the learner. Others -- including Dick and Carey (1996) -- disagree, believing that learners need a variety of types of feedback during the course of the lesson in order to improve their performance on the final examination for the instructional unit (p. 192), which applies whether the instruction is delivered in the electronic or the traditional classroom. Even more important, then, is feedback to the learner using multimedia instructional materials in lieu of an instructor's providing this type of interaction.)

Iuppa makes a useful distinction between test styles, specifically concept tests and simulation tests. Concept tests are used to assess learners' understanding, while simulations test learners' performance. In discussing how to assess concept and performance via digital media, Iuppa cautions that digital assessment has its limits. He says, for instance, that tactile skills are "among the very few" types of skills that digital media cannot assess and points out digital media cannot, for example, test how smoothly the student manages to screw in a spark plug, which is an important part of the process.

In the Evaluation Cookbook, Jen Harvey (1998) lists, as possible evaluation methods, the following options: checklists, concept maps, confidence logs, cost effectiveness methods, designing experiments, ethnography, focus groups, interviews, nominal group techniques, pre and post testing, questionnaires, resource questionnaires, split screen video, supplemental observation, system log data, and trials.

Two basic types of assessment for online college-level courses prevail: (1) those with CBI and "automated" or computer-administered types of instruction, the most typical final assessment method is the proctored exam, and (2) those with instructor-monitored distance learning types of instruction, with the most typical assessment method being criteria-based evaluation, with a final grade assigned by the instructor (Rasmussen et al, 1997).

Instructor-monitored distance learning courses tend to follow one of two models: (1) the "portfolio" or "contract" model, with students turning in frequent, small reports or completing a scheduled, sequence or series of assignments, and (2) the project model, in which the student turns in, at the end of the course, one big project or term paper (Hudspeth, 1997).

An example of typical statements about proctored tests and exams is this one from Indiana University's online Chemistry 101 course:

Clearly, age-old concerns about ethical practices in assessment (i.e., cheating) take on new twists in the distance-learning environment. The issue of authorship of student work has always been one that is difficult to resolve, even when the course is taught with traditional methods. A statement about expectations for online students, again from Indiana University, illustrates a common philosophical approach to the problem of authorship:

Online or computer-based multimedia instruction can utilize instructor-based evaluation in a multitude of forms, just as the face to face classroom setting affords multiple methods of evaluation. One method of evaluation that is unique to CBI and WBI is the option of having the computer score the student’s work and record a grade "automatically."

As more and more emphasis has been placed on behavioral objectives and "statements of explicit behaviors that learners must demonstrate, it has been increasingly obvious that a fair and equitable evaluation system is one that measures those specific behaviors. After learners have been told what they have to do to the type of test that evaluates these kinds of specifically stated objectives is called a criterion-referenced test. Although it is possible to create computer-based testing and scoring, there are many concerns related to this type of evaluation. Grades are grades, and students are sometimes ingeniously determined to improve their grade point averages or acquire credit where credit is definitely not due.

One solution to the academic honesty issues appears to be administering "final" exams in proctored computer labs. Other possible approaches to the issues of academic honesty and reliability raised in relation to assessment of student learning in CBI/WBI instructional modules may involve use of the technology that has introduced the problem in the first place. For example, some virtual colleges and online universities are discussing use of security and identity-verification methods, such as voice recognition or fingerprint-recognition systems.

During the Web Based Training Online Learning ’99 conference, Brad Cox, author of the award winning Taming the Electronic Frontier, was asked how he evaluated instruction in his online course. He replied:

Cox's approach recommends using a series of small, sequential, individualized tasks and student-centered personal responses to provide multiple checkpoints during the online course and ensure that students, in order to complete the assignments, have to keep up with the class readings and respond to class assignments themselves. Multiple, individualized tasks are harder to counterfeit, if only because the coordination involved in the student's arranging for someone else to do the work in a timely and appropriately specific manner becomes prohibitive to intellectual dishonesty.

Cox suggests a sequential, portfolio-style approach to the student's progress through a course and assessment of that progress. Another, somewhat similar, approach is recommended by DeLayne Hudspeth (1997) in discussion of "just in time" learning programs. Hudspeth (p. 356) recommends a "contract" approach, which he describes as:

Hudspeth points out that the benefit of using a contract-type approach is that the course assignments and the criteria for their completion can be explicitly described for the student, but the student can use considerable flexibility in exploring issues and resources that have personal interest and that, presumably, encourage the student to see the benefit of actually doing the learning rather than trying to find a stand-in (p. 356).

Another method of online course assessment, particularly for adult learners, is the completion of one large end-of-course project (Mood, 1995). For example, in a business course, students may submit, at the end of instruction, a fully developed business plan or, as the final project in an online course, the student may be required to submit a substantially researched term paper.

With project-based assessment, the dangers of having the student simply purchasing a pre-packaged term paper are diminished the more individually the project is tailored to the resources used in the course, the student's individual interests, and the use of intermittent "checkpoints" -- such as weekly, brief, e-mailed "progress reports" that are individual and specific enough to reduce the likelihood of students' thinking they can submit someone else's work and avoid being found out (Hudspeth, p. 354).

Others suggest using Internet-based videoconferencing or online synchronous class discussions or similar performance-on-demand type class activities as part of authentic or performance assessment of student achievement. In addition to encouraging participatory learning, enhancing student communication and/or technology skills, and interaction between isolated learners or between learners and the distant instructor, these "real-time" performance activities also tend to ensure, incidentally, that the actual student be present and accountable for course content.

In formative evaluation, program components are evaluated for the purpose of revision and modification. This process should take place as soon as objectives, outcomes, and a program of instruction have been specified and should involve "the collection of data and information during the development of instruction which can be used to improve the effectiveness of the instruction" (Dick and Carey, p. 256).

There are varying interpretations of the specific phases of a formative evaluation, but most instructional design texts recommend the same basic process: Evaluation of a program in development should be a continuous process, and if time is required to revise and redo, it is usually time well spent. The best strategy to ensure that the project is on course is to check early with a formative evaluation, then recheck early to ensure that the program of instruction is working as intended. Only through early, effective revision can the instructional designer confidently determine whether the assessment of a learner measures the objectives correctly and whether learner outcomes indicate actual mastery of a subject.

For the multimedia designer, formative evaluation takes on special significance. The multimedia instructional package is generally expensive to produce, and it will probably be delivered to learners that the designer will find difficult to observe in the same manner in which they are likely to be using the instructional materials. There should be more learner feedback and more interaction with the learner than just drill-and-practice exercises. The ideal is for multimedia instruction to "immerse the learner in activities, promoting interaction, involvement, and engagement (Hill, 1997, p. 75)" The reality, however, is that far too often "most course-based or learning sites simply post course materials. Use of the Web as merely an 'electronic book' falls far short of the potential the medium affords. As with other distance learning initiatives, the Web has substantial potential for moving instruction away from a repository model to one where active learning can occur." (Hill, p. 75)

Alessi and Trollip, in Computer-Based Instruction: Methods and Development (1991), discuss in detail a variety of concerns relating to formative evaluation that are of particular importance to the instructional designer. They divide the process of formative evaluation into three phases: quality review, pilot testing, and validation (p. 385). These three phases correspond to Dick and Carey’s one-to-one, or clinical, evaluation: small group evaluation, and the field trial (Dick and Carey, p. 257), and to the first three of Newman and Lamming’s methods (Newman and Lamming, p. 193).

Whether all of these described methods or phases of formative evaluation are used by the designer or only a few, there must be a revision process where the program is reviewed to see if it truly assesses a learner’s mastery of the objectives. If features of the program impede or conflict with the learning delivered -- or with the cognitive style of the learner, or with the learner's engagement or on-task involvement with the learning provided -- then formative testing should reveal this flaw (Dick and Carey, p. 256).

Alternatively, viewed from another angle, if only certain learners can succeed at a program, then the learning is not suited to the needs of the actual learning audience (p. 257). The process of formative evaluation is one that allows the designer to make sure that what has been intended to be provided as a learning environment for the learning presented in the program is actually being experienced by the learners who are using the program. Again, this is particularly critical in instructional design for CBI /WBI instructional programs, because the designer will most probably only have the opportunity to check and "fine tune" a program delivered remotely during the formative evaluation stage.

In attempting to describe the difference between formative and summative evaluation, Harvey (1998) says, "When the cook tastes the soup, it is formative evaluation; when the dinner guest tastes the soup, it is summative evaluation."

Some instructional designers do not consider summative evaluation as primarily the responsibility of the designer. Dick and Carey state that "Since the summative evaluation usually does not involve the designer of the instruction but instead involves an independent evaluator, this component [of the instructional design process] is not considered an integral part of the instructional design process per se." (p. 7) According to Dick and Carey's model summative evaluation is concerned with evaluation of the total instructional package, which is essential to sound instructional design. Instructional designers use lessons and tests as basic building blocks for instructional media. Nicholas Iuppa (1998) says "the very minimum you can do when teaching something is to demonstrate it and then ask one or more questions about it in a test." (p. 89 In order to insure that the test measures what the learner was supposed to learn, the Dick and Carey Model as well as many other instructional design models stress the importance of developing assessment instruments prior to developing the instruction.

Multimedia computer based instruction (CBI) and Web based instruction (WBI) are not unique in using the final evaluation instrument as a building block for instructional design. However, the use of multimedia as a means of instruction delivery, particularly the distance from the learner that generally accompanies use of the media, does raise some unique issues in relation to learner assessment. (Hudspeth, 1997) Since the multimedia designer plays such an important role in establishing the outcomes of an instructional lesson, the designer may be called upon to play a more active role in the summative evaluation of a multimedia instructional unit than is common in "traditional" instructional design.

"Final" or summative evaluation for certification or course credit is unlikely to be eliminated anytime in the near future. However, the advent of the technology that has provided the means to offer online courses of study has also caused the re-evaluation of assessment issues, such as what should be assessed in a virtual, for-credit course. In addition, distance delivery of instruction has raised issues about how to assure academic honesty and reliability.

In discussing learner assessment, the role of the instructional designer is growing. Because the instructional designer is intimately involved in establishing the instructional objectives for instruction as well as the assessment instruments by which the student's learning will be judged, the instructional designer's role is critical to insuring that CBI/WBI instruction works as expected. Since the instructional delivery is remote, and results are sometimes difficult to evaluate in terms of instructional effectiveness, the unique position of the instructional designer is more critical to the development of technology-based instruction than ever.

Bannan, Brenda, and Milheim, William D. (1997) Existing web-based instruction courses and their design. In Badrul H. Khan (Ed.), Web-based instruction, (pp. 381-387).Englewood Cliffs, NJ: Educational Technology Publications.

Beasley, Robert E. and Waugh, Michael L. (1996) The effects of content-structure focusing on learner structural knowledge acquisition, retention, and disorientation in a hypermedia environment. Journal of Research on Computing in Education, 28(3), 271-281.

Benson, G.M., Jr. (1994) The lifelong learning society: Investing in the new learning technology market sector. Stephentown, NY: Learning Systems Engineering. ERIC Document ED375809.

Bergman, Robert E., and Moore, Thomas V. (1990) Managing interactive video/multimedia projects. Englewood Cliffs, NJ: Educational Technology Publications.

Borich, G. (1999) How to observe. In G. Borich, Observation skills for effective teaching (Third Edition). Columbus, OH: Merrill.

Dick, Walter and Carey, Lou. (1996) The systematic design of instruction, (Fourth Edition). New York: HarperCollins College Publishers.

Duchastel, Philip. (1997) A web-based model for university instruction. Journal of Educational Technology Systems, 25(3), 221-228. New York: Baywood Publishing Company.

Hill, Janette R. (1997) Distance learning environments via the world wide web. In Badrul H. Khan (Ed.), Web-based instruction, (pp. 75-80.). Englewood Cliffs, NJ: Educational Technology Publications.

Hudspeth, DeLayne. (1997) Testing learner outcomes in web-based instruction. In Badrul H. Khan (Ed.), Web-based instruction, (pp. 353-356). Englewood Cliffs, NJ: Educational Technology Publications.

Iuppa, Nicholas. (1998) Designing interactive digital media. Boston: Focal Press.

Laffey, James M., and Singer, Jon. (1997) Using internet-based video conferencing tools to support assessment. In Badrul H. Khan (Ed.), Web-based instruction, (pp. 357-360). Englewood Cliffs, NJ: Educational Technology Publications.

Marzano, R.J. (1992) A different kind of classroom: Teaching with dimensions of learning. Alexandria, VA: Association for Supervision and Curriculum Development.

Mood, T.A. (1995) Distance education: An annotated bibliography. Englewood, CO: Libraries Unlimited.

Newman, William M. and Lamming, Michael G. (1995) Interactive system design. Wokingham, England: Addison-Wesley.

Pernici, Barbara and Casati, Fabio. (1997) The design of distance education applications based on the world wide web. In Badrul H. Khan (Ed.), Web-based instruction, (pp. 245-254). Englewood Cliffs, NJ: Educational Technology Publications.

Rasmussen, Karen, Northrup, Pamela, and Lee, Russell. (1997) Implementing web-based instruction. In Badrul H. Khan (Ed.), Web-based instruction, (pp. 341-345). Englewood Cliffs, NJ: Educational Technology Publications.

Romiszowski, Alexander J. (1997) Web-based distance learning and teaching: Revolutionary invention or reaction to necessity? In Badrul H. Khan (Ed.), Web-based instruction, (pp. 25-37). Englewood Cliffs, NJ: Educational Technology Publications.