Brighton-based e-learning developer Epic and the University of Birmingham have used good old Donald Kirkpatrick’s four levels of evaluation to demonstrate the worth of their Skills4uni study skills programme. Skills4uni "gives prospective students an idea of what to expect from their degree programme and university life in general. The course includes tools to help students identify what kind of learner they are, advice on critical and objective thinking, how to make best use of the library and practical tips on taking notes/asking questions in lectures". The course was considered to be especially important because it focused on the University’s Access to Birmingham programme (A2B) which is geared to attracting students from under-represented backgrounds, who are likely to experience more of a culture shock than their peers, as their friends and family have little or no experience of higher education.
I’m a believer in the value of study skills training, so I was interested to see how well this programme performed at each level:
Level 1 (reactions): There was almost universal agreement amongst those taking the course that the material was useful, and more than 95% said they’s recommend the course to their friends. Now I know Kirkpatrick knockers don’t value level 1 evaluations (perhaps because that’s as far as most evaluations get), but when a course of study is voluntary, as this is, you’ll be running it for an audience of zero if students don’t like it.
Level 2 (learning): According to the report, despite a challenging pass mark, only 3% of A2B students failed the assessment. Obviously this isn’t enough to prove the worth of the programme, because for all we know at this stage the learning content may have had little relevance in the real world and the assessment may have been flaky (which is why there are more than two levels). Nevertheless, a study skills programme that a high proportion of students fail is not going to be good psychological preparation for the more onerous studies that will follow and so this result should be viewed positively.
Level 3 (application of learning): The University saw an increase in applications from A2B students of 14% compared to a national increase of 8%. Also interesting were the results of a survey of students and their parents taken one year after they started their courses. Some 77% of students who completed the Skills4uni course felt confident learning independently compared to 52% of those who had not, and 60% of students felt their first year met their expectations of university life compared with 40% in the control group.
Level 4 (results): The measure which the University chose to use here was drop-out rates. It seems that, over the past two years, the number of students failing to complete the A2B scheme has halved from 12% to 6%
What does this all mean? Well for me it shows (1) that study skills training is worthwhile, (2) that e-learning seems to do a good job of providing this training, and that (3) Kirkpatrick’s four levels of analysis seem to yield useful evaluation data. Good result all round I would say.