Evaluation

**media type="youtube" key="CBoI0wBo4vw" height="219" width="290" align="right"Welcome to week 5,** In the ADDIE model, evaluation is a systemic process that determines the quality and effectiveness of the instructional design as well as the final product. Evaluation is an ongoing activity conducted at each phase of the ADDIE model. Evaluation consists of two parts: formative and summative. Formative evaluation is part of each proceeding phase and determines effectiveness and quality of each stage. Summative evaluation consists of tests for criterion-related referenced items and provides opportunity for feedback from the users and assesses learner outcomes.
 * Evaluation Definition **

**Formative Evaluation**  Formative evaluation involves gathering information during the early stages of the design process with the focus on finding out whether efforts are unfolding as planned, uncovering any obstacles, barriers or unexpected opportunities hat may have emerged, and identify mid-project adjustments and corrections which can help insure the success of the project. The feedback gathered during formative evaluation is designed to fine-tune the implementation of the program, gather reaction and identify what is not working. **Six Stages of Formative Evaluation** **Summative Evaluation**  Summative Evaluation is the process of collecting data following implementation of the project in order to determine its effectiveness and satisfies the instructional objectives. The summative evaluation is instructionally congruent, allows for complete analysis, and provides feedback. Summative evaluation may measure knowledge transfer, learner outcomes, cost factors, and learner attitude.
 * 1) **Evaluation Goal Specification**Specify the goals of the stakeholders for which the evaluation is performed
 * 2) **Preparation**Arrange necessary personnel and measurement instruments
 * 3) **Data Collection**Hire and train data collectorsElicit feedback from target audience and experts
 * 4) **Data Analysis**Tabulate statistical data
 * 5) **Revision**Modify product to improve effectiveness and efficiency based on data collected above the stakeholder goals
 * 6) **Recycling**Retest product, move to summative evaluation and dissemination after removing weaknesses.

he Evaluation Phase of the ADDIE Model calls for a plan which addresses both formative and summative evaluation. Information will be gathered to ascertain learning effectiveness, learner motivation, content and technical quality, and implementability. Remember that. . . <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Formative Evaluation** is an ongoing process designed for use at each phase of the ADDIE Model. Formative evaluation directs the project and allows for ongoing improvement and adjustment. Developers collect data and information at each stage of development to improve the effectiveness of the product. **Summative Evaluation** evaluates the product in its final form and is conducted to determine whether or not the learning objectives have been met.
 * <span style="color: #ffa100; font-family: 'Times New Roman',Times,serif; font-size: 120%;">Samples **

<span style="display: block; font-family: 'Times New Roman',Times,serif; font-size: 110%; text-align: center;">**Product** <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Product format:** Text with CD.**Target Learners:** Middle School Teachers in Idaho**Process:** Four Hour hands-on Training for middle school social science teacher in Idaho.**Learner Objective:** Given the Oregon Trail CD, Teacher’s Guide and four hours of Training, the learner will be able to navigate the Oregon Trail CD and demonstrate it to middle school students. <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">//A pre-test and a post-test will be given to be used in both formative and summative evaluation. A student attitude questionnaire will also be given.//

<span style="display: block; font-family: 'Times New Roman',Times,serif; font-size: 110%; text-align: center;">**Formative Evaluation Plan** <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">There are three stages of formative evaluation, however, prior to the implementation of these stages, the Oregon Trail CD and Teacher’s Guide will be reviewed by both a //Subject-Matter// and //Learner Specialist// to ensure accuracy in both areas. <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Stage 1: One-to-One Evaluation**

<span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;"> The evaluator will work individually with three teachers, two with average ability and one with below average ability. The evaluator will gather information to see how the learner reacts to the instruction and interacts with computer technology. The following elements will be evaluated by gathering descriptive data in the following areas: <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Clarity:** Is the information presented on the Oregon Trail CD and Teacher’s Guide clear to the target audience? <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Impact:** How does the instruction on the Oregon Trail CD and Teacher’s Guide impact the learner? <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Feasibility:** How is the instruction in the Oregon Trail CD and Teacher’s Guide managed?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Message:** How clear is the message?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Links:** How clear are the examples, illustrations, and demonstrations?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Procedures:** How clear is the sequencing, segmentations, and transitions?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Is the information personally relevant?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Can the information, quizes and problems be accomplished?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Is the instruction interesting?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">How will the independence and motivation of the learner impact the amount of time needed to complete the instruction?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Is the equipment necessary for the instruction easily operated?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Is the learner comfortable with the computing environment?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Is the cost of delivering instruction via the CD reasonable?

<span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Stage 2: Small Group Evaluation** <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;"> Stage 2 evaluation expands on Stage 1. Small group evaluation determines if changes made in the first stage were effective and whether or not the instruction is effective without the aid of a monitor. Fifteen teachers, representative of the target population, will be selected. In contrast to One-on-One evaluation, the evaluator will present the Oregon Trail CD and Teacher’s Guide; explain that the materials are in the developmental stage, allow the learners to forge through the instruction as if in its final form, and request feedback. Quantitative data will be added to the descriptive data collected in Stage 1. A student attitude survey will be given. The attitude survey will take into account, ease and familiarity of equipment, instructor competency and helpfulness, interaction with instructional module as well as teacher and other learners, and the learning environment.

<span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">**Stage 3: Field Trial** <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;"> The final stage of formative evaluation encourages the close simulation of the actual learning environment and instructional materials. At this juncture, the Oregon Trail CD and Teacher’s Guide have been revised and recommendations from the previous two stages have been incorporated.

<span style="display: block; font-family: 'Times New Roman',Times,serif; font-size: 110%; text-align: center;">**Summative Evaluation** <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Use of well-designed pre and post tests to determine knowledge gain is recommended. The evaluation tool must:
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Accurately reflect the instructional goals to ensure content validity.
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Be based on multiple observers to ensure reliability.
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Be consistent among different classrooms, teachers, and rotations

<span style="display: block; font-family: 'Times New Roman',Times,serif; font-size: 110%; text-align: left;">**Suggested Means for Evaluation**
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Test for criterion-related referenced items
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Provide means for anonymous feedback as well as face-to-face
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">The evaluator will discuss, one-on-one, with the learner particular responses and reactions of the questionnaire and pre-test, post test as well as the above mentioned questions to search for mistakes and reasons for mistakes.
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">The various evaluation instruments will be evaluated for clarity, reliability, and accuracy.
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Amount of time needed to complete the instruction will be calculated
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">A Computer/learner interaction check list for consistency, matching of instructional goals, and coding errors will be used.

<span style="color: #ec776f; font-family: 'Times New Roman',Times,serif; font-size: 15.6px;">**Activity (5)** <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">The Oregon Trail design team has completed the prototype of a multimedia product. They are three weeks behind the completion of the design phase. Two members of the design team have completed one on one evaluations with three students. The individual students collectively were able to follow 85% of the instructions and wrote down approximately 15 words that were unclear or confusing. The descriptive data indicates that students are somewhat motivated but were having some difficulty operating the program independently. Encouraged by the initial results of the one on one evaluations, the design team decided to cancel the small group evaluation and field test to make the production deadline. The team made this decision partially because the production coordinator insistence on complete production on time and the design team anticipated that they could partially complete the small group and field test during the final performance-context evaluation. The design team worked on the Oregon Trail project an additional week to clear up some of the ambiguous instructions and develop a few more questions relevant to a forth grade students. At the end of the week, the design team felt very confident that they had an effective product and were ready to conduct a performance context evaluation.

<span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">The design teams used the students from the one on one evaluations and twenty additional students from the local forth graders classroom to begin the performance context evaluation. The student and teacher incorporate the Oregon Trail multimedia program into the Idaho history, math and regional geography curriculum. At the end of three weeks, the design team returns to interview 5 students and the teacher. During the interview, the teacher confessed to her disappointment with the program. The students required one on one supervision to operate the program, 68 % of the students had difficulty understanding the instructions and 30% refused to use the Oregon Trail. The initial data confirmed the teacher’s impression, 43% of the math problems and 57 % of geography problems did not transfer to other areas in the curriculum. Discouraged by the results yet aware of a production meeting with the project coordinator and grant supervisor, the team reassessed the evaluation data. The design realized the evaluation was limited and it was difficult to tease out all the problems discovered during performance context evaluation. <span style="font-family: 'Times New Roman',Times,serif; font-size: 110%;">Help the design team to re-evaluate the formative evaluation process. Determine the errors in the formative evaluation process. How would you correct the plan and what formative evaluation process would you use? Include information like what is being evaluated, what evaluation methods, what procedures will be used, and what are the expected outcomes. Also, address the criteria used to evaluate clarity of instruction, impact on the learner, feasibility, interactive process and instructional strategies and the learner’s setting.