Value & Impact toolkit data instruments

Background

The purpose of the evaluation will determine whether quantitative, qualitative or mixed methods are best suited. Generally, if information is needed on what is occurring, quantitative methods (for example, surveys) are more appropriate. If information is needed about why something is happening, then qualitative methods (interviews, focus groups, observations) are better. Thus, the method will determine what types of tools are required for gathering information.

The main focus of this section is on instruments used for qualitative evaluations. The summary focuses on three specific instruments: surveys, discussion / focus groups, and interviews. For each of these, uses, considerations, advantages and possible disadvantages are highlighted.

Top

Summary

Here is a summary of the instruments that might be used in evaluations of value and impact: Summary of instruments (PDF 91 KB).

Top

Questionnaires

A questionnaire is a multi-stage process starting with a clear definition of what is being examined through the interpretation of the data that is collected. Every step requires careful design - the results are only as good as the weakest link in the design process.

Questionnaires might be seen as a relatively cheap method of data collection, but they can be exhausting (and therefore expensive) in terms of the time required for their design and the interpretation of their results.

Pay considerable attention to potential respondents and to see the questionnaire through their eyes and how they will regard it (for example, with hostility, suspicion, apathy, grudging compliance, as easy or difficult, motivating or boring, straightforward or complex).

The objectives

• A questionnaire requires well-defined objectives - what are you trying to achieve/measure?
• If the objectives have been well-defined, the questions should flow in a logical sequence.
• If the writing of the questions becomes difficult, this probably means that not enough time has been spent in defining the objectives of the questionnaire.

The questions (1)

• Questions can measure both qualitative and quantitative data, but qualitative questions require more careful crafting to avoid ambiguity.
• Decisions will need to be taken about what demographic data to collect.
• Demographic data is used to correlate responses between different groups of people and to see whether they are consistent across groups. Ask only those questions that are relevant.
• Decisions will also need to be taken about the most appropriate types of questions – dichotomous (yes/no, male/female), multiple choice, rank order, rating scales (for example, Likert), closed, open.
• Open format questions ask for unprompted opinion and they are useful for gathering subjective data. However, the main drawback is that responses need to be read and it is not possible to perform any statistical analysis on them (without knowledge of qualitative analysis software). They are therefore costly and responses can be open to the interpretation of the reader.

The questions (2)

• Closed format questions are easy to complete, responses can be statistically analysed and they are not costly. Furthermore, opinion can be tracked over time and extreme responses can be filtered out.
• For ease of analysis, particularly of a large sample, more closed than open questions are recommended.
• Questions must be clear, succinct and unambiguous. The chance that a question might be interpreted in different ways by different respondents needs to be eliminated.
• Ask only one thing at a time in a question.
• Leading and hypothetical questions must be avoided.
• It should not be assumed that respondents will always know the answer. Therefore, include where relevant, ‘don’t know’, ‘not applicable’, ‘unsure’, ‘neither agree nor disagree’ and ‘not relevant’ categories.
• Be aware of ‘prestige bias’ where respondents provide answers to make themselves look or feel better. This is difficult to prevent, but the main way is to assure respondents that their replies are confidential.

Quality testing

To ensure that a questionnaire is clear and understandable, it needs to be tested out on some pilot users (for example, by using a specially convened focus group). Issues arising should be discussed and the questionnaire revised and tested once more if possible.

General

Include a covering explanation about the purposes of the questionnaire, how anonymity and confidentiality will be addressed, who is doing the evaluation / research and who will see the results.

Remember to thank the target sample / population for their anticipated co-operation.

Top