With the end of the educational year upon us (for the Southern Hemisphere at least) it’s a good time to talk ‘evaluation’.
All educational and training courses require evaluation of both content as well as monitoring procedures. In the case of education and training using elearning as the delivery method, there are additional reasons to evaluate. For instance:
- To justify the mode of delivery
- To make sure that everything is working properly
- To ensure that online students get a learning experience that is just as good, if not better, than traditional students.
In the context of the design and development of elearning material, evaluation refers to the measuring of outcomes against predetermined criteria that can be broadly grouped as processes and products.
The most obvious product is the elearning material, whether it is a small interaction or a complete online course, and it is the effectiveness of the end product that most people relate to when they think of evaluation.
A number of aspects of your elearning content will need to be monitored and evaluated:
- The effectiveness and efficiency of learning
- The attitudes of both the learners and staff to elearning
- The reliability and usability of the technology used to deliver the content
- The effectiveness of the support systems (administration, technical etc.).
Types of Evaluation
Just as there are many components of the development and delivery process that can be evaluated, there are a variety of evaluation methods that can be used. These can take the form of questionnaires, interviews, data analysis and so on and fall into two basic categories – formal methods and informal methods.
Formal methods of evaluation usually involve some type of statistical analysis. Depending upon at what stage you conduct the evaluation, and the product you use to deliver the elearning content, you may be able to use some of the tools within the delivery program (eg learning management system) to collect some of the statistics. For example:
- Tracking tools
- Participation rates
- User logs
- Learning measures.
Informal methods involve getting a ‘feeling’ for the course.
- Did the course feel right?
- Did the facilitator get hassled (more than normal) by the students?
- Did the students complete the course requirements?
- Did the students perform as well as expected?
You will also need to consider how to collect the data of your evaluations. Again, there are several options and the option selected will depend on two factors:
- The type of evaluation to be conducted
- The phase in the development and delivery process at which the evaluation will occur.
Data collection options include:
- Electronic – access logs, tracking, mail, assessment/quiz results, online surveys and questionnaires
- Paper – surveys, questionnaires
- Face to face – interviews.
When to evaluate
Evaluation can be conducted at various phases during the development and delivery process.
Formative evaluation is concerned with measuring the progressive development of a project as it proceeds through its various phases. It is usually descriptive and qualitative. Its purpose is to ensure the project remains ‘on track’ and is meeting the goals laid down in the educational and project plans.
If other team members are involved, regular reports help evaluate how closely a project is to its timeframe/budget/desired outcomes etc. Encouraging critical analysis from colleagues and ‘consultants’ as each part of the elearning course develops is another way of progressively evaluating the quality of the course.
Summative evaluation is concerned with the effectiveness of the elearning course after it has been used by the learner. It is usually quantitative as well as qualitative and should consist of direct and indirect measures. Direct measurement of outcomes includes the use of questionnaires and assessment results. More indirect measures of success can also be used such as verbal feedback from students or from feedback from facilitators implementing the elearning course.
Evaluation needs to occur at all stages of the development and delivery process.
- At what stages within the course development and delivery process will you conduct some type of evaluation?
- What specifically will you be evaluating in each stage?
- What methods of evaluation will you use?
- What means will you use to collect the data?
- What will you do with the data?