Distance Education at a Glance Guide #4 October 1995

Engineering Outreach College of Engineering University of Idaho

Guide #4
Evaluation for Distance Educators

Why Evaluate?	Evaluation Methods	Evaluation Tips	Information
Types of Evaluation	What to Evaluate	References

Why Evaluate?

Effective teachers use a variety of means, some formal and others informal, to determine how much and how well their students are learning. For example, to formally evaluate student learning, most teachers use quizzes, tests, examinations, term papers, lab reports, and homework. These formal evaluation techniques help the instructor to evaluate student achievement and assign grades.

To evaluate classroom learning informally, teachers also use a variety of techniques. For example, teachers pose questions, listen carefully to student questions and comments, and monitor body language and facial expressions. Informal, often implicit evaluations permit the teacher to make adjustments in their teaching: to slow down or review material in response to questions, confusion, and misunderstandings; or to move on when student performance exceeds expectations.

When teaching at a distance, educators must address a different teaching challenge than when teaching in a traditional classroom. For example, instructors no longer have:

• A traditional, familiar classroom.

• A relatively homogeneous group of students.

• Face-to-face feedback during class (e.g. students' questions, comments, body language, and facial expressions).

• Total control over the distance delivery system.

• Convenient opportunities to talk to students individually.

For these reasons, distance educators may find it useful to not only formally evaluate students through testing and homework, but to use a more informal approach (see Angelo and Cross, 1993) in collecting data to determine:

• Student comfort with the method used to deliver the distant instruction.

• Appropriateness of assignments.

• Clarity of course content.

• If class time is well spent.

• Teaching effectiveness.

• How a course can be improved.

Types of Evaluation

Evaluation can be either formative, summative, or a combination of both.

Formative evaluation:

• Is an on-going process to be considered at all stages of instruction.

• Will enable the instructor to improve the course as he/she proceeds.

• Facilitates course and content adaptation.

• Will identify major gaps in the instructional plan or the need for minor adjustments.

Some strategies that educators can use to collect formative data from their distant students include:

• Post cards - provide each student with prestamped and preaddressed poscards. On a weekly basis, have students use the postcards to share their concerns or respond to questions during the last three to five minutes of class.

• Electronic mail - Can be a very effective way for instructors and students to communicate. Another plus, while the instructor is eliciting information about classroom learning, students become familiar with the use of electronic mail, a valuable skill.

• Telephone - Call students often. Ask them open ended questions (e.g., "What snags did you run into on the second writing assignment?") to let students voice their concerns. Follow with probes (e.g., "Then, will you need more information sources?"). Set phone-in office hours but be sure to welcome calls at other times.

Summative evaluation:

• Assesses overall effectiveness of the finished product or course.

• Can be a springboard in developing a revision plan.

• Can be a baseline of information for designing a new plan, program, or course.

• Will not help current students since it is conducted upon course completion.

Some questions that educators may want to ask students when collecting summative data include:

• List five weaknesses of the course.

• List three (or five) strengths of the course.

• If you were teaching the course, what would you do differently?

• Student background information: age, level in school, number of distance delivered courses taken prior to this one.

• What would you recommend to a friend planning to take this course?

• What did you think would be covered in this course but was not?

• Would you recommend this course to a friend? Why or why not?

Evaluation Methods

Within the context of formative and summative evaluation, data may be collected through quantitative and qualitative methods.

Quantitative evaluation:

• Involves asking questions which can be statistically tabulated and analyzed, frequently using a scale, check list, or yes/no responses.

• Limits students to responding to the categories made available to them.

• Needs a large student sample for relevant statistical analyses.

Quantitative methods may be most useful for gathering information on large numbers of respondents for whom more in-depth, personalized approaches are not feasible. However, they do have some significant drawbacks:

• Many distance education courses have relatively small class sizes with students from various backgrounds. These small, stratified populations typically defy relevant statistical analysis.

• Quantitative surveys typically result in a rate of return of under 50 percent. A low rate of return often suggests that only those feeling very positively or negatively about the course responded to the evaluation.

• By definition and design, forced choice surveys offer respondents a limited number of possible response options. Therefore, fresh insights and unique perspectives falling outside the provided response categories go unreported.

• The cumbersome and often tedious nature of quantitative data collection can discourage formative evaluation, and often results in an over-reliance on summative evaluation.

• Statistical analysis often results in an illusion of precision that may be far from reality.

Qualitative evaluation:

• Is typically more subjective.
• Involves gathering a wider range and depth of information.
• Is more difficult to tabulate into neat categories.
• Will be less affected by typical small class size.
• Is a more flexible and dynamic method.
• Is not limited to pre-conceived topic of inquiry.
• Allows for student output of topics.

Can use:

• Open ended questioning -- with respondents asked to identify course strengths and weaknesses, suggest changes, explore attitudes towards distance delivery methods, etc.

• Participant observation -- with the distance educator observing group dynamics and behavior while participating in the class as an observer, asking occasional questions, and seeking insights regarding the process of distance education.

• Non-participant observation -- with the distance educator observing a course (e.g., an audioconference, interactive television class, etc.) without actually participating or asking questions.

• Content analysis -- with the evaluator using predetermined criteria to review course documents including the syllabus and instructional materials as well as student assignments and course-related planning documents.

• Interviews -- with a facilitator or specially trained individual collecting evaluative data through one-on-one and small-group interviews with students.

What to Evaluate

Consider the following areas:

• Use of technology - familiarity, concerns, problems, positive aspects, attitude toward technology.
  

• Class formats - effectiveness of lecture, discussion, question and answer; quality of questions or problems raised in class; encouragement given students to express themselves.

• Class atmosphere - conduciveness to student learning.
  

• Quantity and quality of interaction with other students and with instructor.

• Course content - relevancy, adequate body of knowledge, organization.

• Assignments - usefulness, degree of difficulty and time required, timeliness of feedback, readability level of print materials.

• Tests - frequency, relevancy, sufficient review, difficulty, feedback.

• Support services - facilitator, technology, library services, instructor availability.

• Student achievement - adequacy, appropriateness, timeliness, student involvement.

• Student attitude - attendance, assignments submitted, class participation.

• Instructor - contribution as discussion leader, effectiveness, organization, preparation,enthusiasm, openness to student views.

Evaluation Tips

• Check out and adapt already published questionnaires; there�s no need to re-invent the wheel.

• Draft and revise questions; change if necessary.

• Make use of follow-up probes:

• Alternate between instruction and interaction.

• Sequence your questions for best effect - go ahead and ask for suggestions for improvement before asking for what is good. This will help convey sincerity for seeking improvements.

• Place open ended questions after quick answer questions. This gives students built-in thinking time.

• On summative evaluation, assure anonymity. This can be accomplished by having all questionnaires sent to a neutral site where they would be removed from their envelopes and forwarded to the instructor without a postmark.

• Establish rapport by being interested and supportive. Withhold judgmental responses.

• Adapt to the student in degree of formality and pace of communication.<

• Use evaluation as a method for understanding teaching and learning.

• Try to get both positive and negative feedback. It is important not only to know what is not working, but also what is working.

References

Angelo, T. & Cross, P. (1993). Classroom assessment techniques: A handbook for college teachers. San Francisco: Josey-Bass Publishers.

Willis, B. (1993). Distance education: A practical guide. Englewood Cliffs, NJ: Educational Technology Publications

For Further Information

#1 Distance Education: An Overview
#2 Strategies for Teaching at a Distance
#3 Instructional Development for Distance Education
#4 Evaluation for Distance Educators
#5 Instructional Television
#6 Instructional Audio
#7 Computers in Distance Education
#8 Print in Distance Education
#9 Strategies for Learning at a Distance
#10 Distance Education: Research
#11 Interactive Videoconferencing in Distance Education
#12 Distance Education and the WWW
#13 Copyright and Distance Education
#14 Glossary of Distance Education Terminology