|
Assessment or evaluation?The literature review for the Value & Impact project was based mainly on literature from the United States; most of the US writers identified through the literature review tended to use ‘assessment’ in relation to performance measurement. However, in the UK ‘evaluation’ is a more commonly used term. As Markless and Streatfield (2006) note, the term assessment tends to be used to describe judgements about people’s knowledge or skills, whereas evaluation is used to refer to judgements about systems and services. As such and wherever possible, the value and impact toolkit uses ‘evaluation’. EvaluationEvaluation is specific to a programme or project and its purpose is to ‘improve’ by determining the merit, worth or value of something. It focuses on process or outcomes. One definition of evaluation is that it is ‘the provision of information about specified issues upon which judgements are based and from which decisions for action are taken’. It is often characterised as an ‘inherently political enterprise’ (Cohen et al, 2003). ResearchResearch is designed to produce results and contribute to theory that go beyond individual programmes and projects with the intention of making them generalisable to other populations, conditions or times. Evaluation or research?Evaluation and research are closely related. They share similar methodological characteristics and therefore the skills that evaluators and researchers use are very similar. However, they serve different purposes. Commentators suggest that the barriers between research and evaluation are being broken down as research (through its funding and the conditions attached, such as demonstrating impact) becomes increasingly politicised. ValueValue can be both qualitative and quantitative. In qualitative terms it is the regard, importance, worth, or usefulness that something is held to deserve. In quantitative terms, it is the material or monetary worth of something, and the worth in terms of its cost. The toolkit focuses on the concept of ‘value for money’. As noted by the Audit Commission (2009), value for money is about obtaining the maximum benefit over time with the resources available. Decisions about value for money are a daily reality in all our lives. We are constantly choosing which items or services to buy, and judging the right balance for us between quality and cost. It is about achieving the right local balance between economy, efficiency and effectiveness – the 3Es: spending less, spending well and spending wisely. This means that value for money not only measures the cost of goods and services but also takes account of the mix of cost with quality, resource use, fitness for purpose and timeliness to judge whether or not, together, they constitute good value. In a student services context, value for money measures might include:
ImpactThere is a tendency to confuse impact with customer / student satisfaction. Customer satisfaction focuses on measuring whether or not students like or are happy with the educational experience and services they receive. Impact, however, is aimed at measuring whether or not the educational experience / service is making any difference to what they do and how. Customer satisfaction is relatively easy to measure; for example, the following statements are aimed at measuring student satisfaction:
Impact, however, is more difficult to measure. Impact is about change, which implies that a situation needs to be evaluated before an action to stimulate change takes place, and after to determine whether indeed change has taken place. Impact might also be evaluated in terms of the effect of an activity on different groups; for example, students might attend a particular programme on a voluntary basis, so impact might be measured after the programme takes place in relation to the knowledge levels of those who attended against those who did not attend. Markless and Streatfield (2006) in their work on the impact of university library services in the UK, use the following definition of impact: "…any effect of the service (or of an event or initiative) on an individual or group." Impacts can be positive / negative, intended / unintended and direct / indirect. The impact of an event or initiative will affect different people – students, student services professionals, senior managers, academics – and in different ways. Impact can show, for example, through shifts in the ‘quality of life’ (self-esteem, confidence), and in academic outcomes. Impact is also difficult to measure because of issues of causality and timescale. Because the measurement of impact is about identifying and evaluating change, it is not easy to evaluate the impact of an intervention or activity independent of other factors that may potentially influence outcomes. The best that can be achieved is to find ‘strong surrogates’ for impact that provide a close approximation (Markless and Streatfield, 2008). Value and impactValue and impact are closely related. Clearly, if something is of value then it has an impact, although questions would need to be asked about whether that impact is measurable. Conversely, if something has an impact, it may not necessarily have value, because impacts can sometimes be negative and unintended. ‘Value for money’ is often defined in terms of the three Es (economy, efficiency and effectiveness) and effectiveness is ‘a measure of the impact that has been achieved and is essential to demonstrating the achievement of value for money’. The two terms are, therefore, inextricably linked. |