Unit Author: Professor Nick J Fox
Learning objectives
Having successfully completed the work in this unit, you will be able to:
- Describe the different methods used in surveys and list their advantages and disadvantages.
- Design a simple self-completion questionnaire (postal or online).
- Design a simple administered questionnaire/quantitative interview (face-to-face or telephone).
Introduction
In Unit 3, we noted a wide range of different quantitative social research designs, including randomised controlled trials and cohort studies. However, the quantitative design most commonly used in social research is the survey, and this will be the concern of this unit, in which we consider methods of quantitative data collection.
In any piece of research, the method of data collection is clearly important. Without appropriate and rigorous methods, any conclusions drawn from the data will not stand up to critical appraisal. In this unit, we will consider the different ways that surveys can be undertaken, focusing on two data collection methods: questionnaires and structured interviews. We will look at the different ways to collect data using these methods, and offer practical tips for using them.
We begin with an overview of the survey approach, and then look in detail at designing and delivering a questionnaire, and then designing and using a structured interview approach.
1. An overview of surveys
Surveys gather a ‘snapshot’ or cross-sectional picture of a phenomenon, and can be used to provide descriptive data on some aspect of human behaviour or attitudes. They can also be analytical, and responses can be coded to allow sophisticated inferential statistics to be used. Data from different questions can be correlated to identify associations, or multivariate analysis can identify the effects of a range of independent variables (e.g. social class, age, gender) upon a dependent variable (e.g. voting behaviour).
The methods of data collection that can be used in surveys may be categorised as:
- Self-completion methods, such as online or postal questionnaires.
- Researcher-administered methods, such as face-to-face questionnaires or telephone structured interviews (unstructured, qualitative interviews are considered in Unit 8).
The choice of how to collect survey data depends upon a number of factors, including:
- the nature of target respondents (such as their literacy levels, professional status and motivation to respond);
- access to potential subjects/respondents;
- the subject matter;
- resources (such as your budget and timescales);
- researcher skills needed.
Each of the following methods has particular advantages and disadvantages in relation to these factors, and you need to weigh these up when choosing a method of data collection.
1.1 Face-to-face questionnaire/ structured interview
The face-to-face questionnaire or interview is the gold standard for surveys, but is expensive and time-consuming, and may not be the best way to approach some groups.
Target audience: This method is effective in gathering data from respondents of all types, including those with low literacy levels or language skills, as it enables the researcher to mediate the delivery of the survey. It may be the most appropriate when there is a need for trust between researcher and respondent to gather valid data.
Access: Face-to-face interviews/questionnaires may not be appropriate with hard to reach groups (for example those who may be suspicious of ‘authority figures’, sub-cultures or those at the margins of society), and may also pose issues of researcher safety.
Subject matter: Face-to-face interviews are preferable when the subject matter is very sensitive, if the questions to be coded are very complex, or if the interview is likely to be lengthy.
Resources: Face-to-face or personal interviews are very labour intensive (and hence the most expensive means to conduct data collection). However, they can be the best way of achieving high quality data.
Researcher skills: This method requires specific researcher skills. These are dealt with in more detail later in this unit. (See also Unit 8 for details of qualitative interviewing.)
1.2 Telephone interviews
Telephone surveys are a quick means to approach a larger number of respondents than face-to-face. However, people are highly resistant and sometimes suspicious of telephone surveys, due to widespread marketing and ‘scam’ calls.
Target audience: Appropriate for targeting ‘mainstream’ respondent groups with fixed residences; not suitable for hard to reach or transient respondents, or for some professional groups who may be unwilling to answer a survey over the phone.
Access: Provides reasonable access using telephone number databases, but may miss those without landlines, and may not be appropriate in developing countries with low levels of telephone access.
Subject matter: Suitable if the questionnaire or interview schedule is limited in length. Inappropriate if there is a need for visual aids or prompts. Respondents may be unwilling to give sensitive or personal information by phone.
Resources: Telephone interviews can be a very effective way to access respondents, particularly when respondents are widely geographically distributed. However, there may be a low response rate to ‘cold-calling’. A prior appointment and covering email or letter may enhance the response rate and length of interview.
Researcher skills: Lower skills required, as researcher can use a set ‘script’ to ask questions and prompts.
1.3 Postal questionnaires
Postal questionnaires are a tried-and-tested means to administer a survey. Response rates can be very low. Design is very important and there is a limit to how long a questionnaire can reasonably be.
Target audience: Response rates may be very low, so it is only really appropriate for respondents who have been primed to respond (for example, as part of a cohort, or offered payment) or those with an interest in the survey subject.
Access: Good access to those with a fixed residence.
Subject matter: Some respondents may be willing to answer personal or sensitive questions in a postal survey, while for others, such content will reduce response rates.
Resources: A postal survey is a cost-effective method if respondents are widely distributed. Costs need to build in follow-up mailings, as response rates are often very low.
Researcher skills: No face-to-face skills required. However, due to the lack of personal contact between the respondent and the researcher, the design and layout of the questionnaire is all important. The questionnaire must be totally self-contained and self-explanatory. We look at this in detail later in this unit.
1.4 Online questionnaires
Questionnaires are more and more being administered online, either from e-mail lists or online fora. Public websites can also be used to recruit respondents, though this carries risks to the honesty of responses or even respondent identities. Software such as ‘Survey Monkey’ and ‘Smart Survey’ can be used to design online questionnaires, supply an Internet portal for questionnaire completion, and provide basic analysis of data.
Target audience: May be appropriate for hard-to-reach groups that otherwise are not easily approached, and may now be the medium of choice for younger respondents. The Internet poses problems in terms of knowing who your respondents are, and you may need to exercise especial caution about the identities of respondents if solicited via public websites or online fora.
Access: Excellent access, but only to those with internet connections.
Subject matter: As for postal surveys.
Resources: Very cheap to produce and circulate but response rates will be very low, requiring follow-up mailings and potentially a very large sample.
Researcher skills: As for postal surveys. The software packages include some basic analysis techniques.
Having considered the advantages and disadvantages of differing survey methods, we will now turn to specific issues in designing questionnaires and structured interviews.
2. Questionnaire design
Both self-completion and researcher-administered methods use structured questionnaires.
2.1 Pre-existing questionnaires
The simplest way to design a questionnaire is to use one based upon a ready-made index such as the European Social Survey or SF36 health survey. These instruments have the added advantage of already having been validated, ensuring that questions are well-designed to elicit the information that the researcher is seeking. In addition, there may already be data from other groups who completed such a survey, with which you can compare your results.
Before deciding to design your own questionnaire, you should spend time investigating what surveys already exist. Even if a ready-made survey is not available, you may learn something from inspecting how they have asked certain questions.
2.2 Question design
The style and content of a questionnaire will depend very much on your research question and your aims and objectives. A structured interview or postal questionnaire will contain a greater proportion of closed questions with pre-coded answers, whereas a questionnaire or topic guide for use in a semi-structured interview may also contain some open-ended questions.
A closed question is one where the possible answers are defined in advance and so the respondent is limited to one of the pre-coded responses given. So, for example, you might ask a respondent to a survey on media consumption:
‘On average, how many hours of TV do you watch each day?’
(INTERVIEWER TO READ OUT)
[table id=34 /]
This is an example of a closed question, where the possible pre-coded answers would be read out or shown to the respondent on a card. The choice of answers is limited to those shown on the card. The codes on the right are for use during analysis.
An open-ended question allows respondents to offer their own response. For example, you could ask the same respondent:
‘How does watching TV affect your daily life?’
In this open-ended question, respondents can interpret the question in their own way. They could, for instance, choose to talk TV as relaxation, as a distraction from other tasks, or about the quality of the programmes. The answers you get back will be very rich in details but it may be difficult to compare the responses over a large number of subjects because the question is not very directed. We have not specified to the respondents, in this case, which areas of their lives we are interested in.
The problem with asking many people open-ended questions is that the answers require content analysis and post-coding. This is very time-consuming, or ‘resource heavy’ – you need to ensure that you have enough time, or enough co-researchers to handle the data produced.
2.2.1 Pre-coding
This can be partially solved by using an open-ended question with partial pre-coding. There may be questions that we wish to ask in an open-ended manner so that we are not leading the respondents in any way, but nevertheless we can anticipate some of the possible responses. Potential responses can be anticipated by carrying out a pilot study, assessing previous studies, or informed guesswork.
The pre-codes listed do not need to be exhaustive; we can always allow for an ‘other, please specify’ option to catch the response which we had not thought of in advance. An example of an open-ended question with partial pre-coding would be:
‘How does watching TV affect your daily life?’
DO NOT READ OUT (INTERVIEWER INSTRUCTIONS) CIRCLE ALL THAT APPLY
[table id=35 /]
In this case we have anticipated some of the most likely answers, but we have also allowed for the respondents to give answers that we had not thought of in advance. So, although some content analysis is still required, the overall amount is still reduced. This will save time in the analysis. This type of open-ended question can be asked in an interview situation, either face-to-face or over the telephone. (With self-completion questions the respondents would be able to see the pre-codes that could influence their choice of answer.)
2.2.2 Post-coding
Sometimes, it is not possible to predict what answers will be offered to an open-ended question, and so it is not possible to offer pre-coding. Consider the following question in the same media survey:
‘What is your favourite TV programme?’ (INTERVIEWER TO FILL IN RESPONSE)
While it is theoretically feasible to create a code for every single programme, this would run to thousands of codes and would need a huge database of codes, and would be of little use during analysis. However, if we wanted to get a broad understanding of what kind of programmes are popular, respondents’ choices can be post-coded during the first stage of analysis into various categories, as follows.
[table id=36 /]
For each completed questionnaire, the response would be post-coded and allocated the relevant code.
2.3 Question wording
In order to maximise the validity of data, it is important to ensure that questions you use in a questionnaire are easy to understand, and that it is clear what is being asked. Follow these rules to improve the questions in your questionnaire.
- Whatever your target sample of respondents, use plain English.
- Do not use multi-syllable words when simple words will do.
- Use language that is appropriate for your target respondents. For example, if targeting supermarket shoppers, use ‘lay’ language to explore aspects of their shopping experience, and avoid any technical or business terminology. In contrast, if you are targeting retail managers, you should use the terminology they use themselves in their daily work to describe how they present, market and sell their goods.
- Questions should not be too long and should not be ‘complex’, containing several questions in one sentence. For example,
‘Do you find TV relaxing or is it a waste of time?’
If the answer was ‘yes’, what would that mean?
- Avoid asking leading questions, which could bias your findings. Do not, for example, ask:
‘We know that cigarette smoking can cause lung cancer. How many cigarettes do you smoke a day?’
- Avoid patronising questions based on your own (or commonly accepted) social prejudices.
Please complete the following SAQ to consider these issues further.
SAQ 6.1 Survey questions
What problems might you encounter if you used each of the following questions in a questionnaire?
[table id=37 /]
2.4 Question order
As a rule, it is best to move from the general to the particular when designing a questionnaire.
Try to start with general questions that are easy to answer. These can act as a warm-up. Then move onto the questions which focus on your area of study. Any potentially sensitive questions may be best left until nearer the end of the questionnaire.
The questionnaire should ‘flow’. The order of the questions is a particularly important issue to consider when planning a self-completion or postal survey. Although respondents may choose to look ahead, you must consider the cumulative impact of each question upon the next.
Because you cannot assume anything about the respondent, it is inevitable that the questionnaire will need to contain certain filters and instructions to either the respondent or the interviewer as to where to go next. For instance:
Q3 Do you smoke cigarettes?
| Yes | 1 | If Yes, go to Q4 | |
| No | 2 | If No, go to Q5 | |
Q4 How many cigarettes do you smoke a day?
| 1 | 1 |
| 2 – 5 | 2 |
| 6 – 10 | 3 |
| 11-20 | 4 |
| 21-30 | 5 |
| 31 – 40 | 6 |
| 41 or more | 7 |
Avoid over-complicated filtering, especially in postal questionnaires, because respondents may tire of following it.
2.5 Asking sensitive questions
Apart from leaving sensitive questions to near the end of the questionnaire, there are other ways of trying to elicit an honest answer.
It is sometimes possible to introduce a question by reference to the activity of others, such as:
‘Some people try many different diets in an attempt to lose weight. Have you used different diets?’
2.6 Using scales in questionnaires
Scales provide a range of easy-to-understand ways to ask a question which may have a range of possible answers. Scales will thus assist in measuring the strength of attitude or feeling rather than simply ‘Yes’ or ‘No’, ‘Agree’ or ‘Disagree’.
It should be remembered however that a scale is not a precise measure of an attitude, merely a way of assessing relative measures. There are a variety of different scales to choose from.
2.6.1 Likert Scale
The Likert scale is one of the most commonly used scales. It is essentially a 5-point scale that ranges from ‘Strongly agree’ to ‘Strongly disagree’. Respondents are presented with one or more attitudinal statements and asked to score each statement on the scale. For instance:
‘To what extent do you agree with the following statements?’
| Strongly agree | Agree | Uncertain | Disagree | Strongly disagree | |
| 5 | 4 | 3 | 2 | 1 | |
| TV programmes are a good source of information | |||||
| People watch too much TV each day | |||||
| I watch TV in the evening to relax. |
2.6.2 Differential scale
Developed by Osgood in 1957, semantic differential scales are used to select their subjective position concerning an event or a statement, on one or more bi-polar dimensions such as adequate/inadequate; useful/useless; interesting/boring. For example, a questionnaire might ask respondents to mark their responses to this question:
‘Do you think that adding calcium and vitamins to yoghurt in order to improve health … ?
2.6.3 Visual analogue scale
As an alternative to a verbal scale, a visual analogue scale is simply a way of asking respondents to indicate their choice visually or spatially. For instance:
2.6.4 Ranking
A further way of getting respondents to express attitudes is by ranking different objects or concepts. One simple way to do this is by using sort cards. Respondents are given a deck of cards, each of which has a word or phrase on it. They are asked to sort them from most to least important or most to least preferred.
For example, a respondent could be asked:
‘Which of these items are the most important benefits of Internet access?’
They are provided with sort cards labelled:
- Entertainment
- National Security
- Gathering Information
- Communicating with friends
- Checking up on others
and invited to sort these. This provides a way for respondents to visualise a larger number of concepts than if one simply presented them with a list.
2.7 Other design issues
Other points to consider when putting together your questionnaire are:
- Title: the title of the questionnaire needs to be appealing and inviting, not academic
- Identifiers: Each copy of a completed questionnaire will require a confidential unique identifier. Names and addresses of respondents should not be written on the questionnaire itself, in order to retain confidentiality during analysis, though the identifier will enable the principal investigator to cross-reference back to the respondent if necessary. If your survey is fully anonymous (for instance, if you have agreed not to obtain any personal information from respondents, or they have replied to an Internet advertisment for respondents), the identifier will simply be a means to keep track of each completed form during analysis..
- Instructions: If you are doing a postal or online survey you will need clear instructions for the respondent. Alternatively, if using interviewers you will need to provide them with instructions in terms of filtering and what to read out etc.
- Layout – the questionnaire should be produced to a professional standard, designed to be easy to read (use type of at least 12 point size and a clear typeface), attractive to look at, and avoid brightly coloured type or paper. Do not try to cram in too much information on each page.
- Length: The length of the self-completion questionnaire has to be carefully considered when conducting a general survey, but if the subject matter is of sufficient interest to the respondent, then length is less important.
2.8 Maximising response rates to postal and online surveys
The validity of a survey will depend on its representativeness (see Units 4 and 5), and postal and online self-completion questionnaires pose challenges for representativeness due to low response rates, thereby threatening the validity of findings.
It is rare to get a response rate above 70 per cent for a postal questionnaire (apart from compulsory exercises such as national censuses or elections in some countries), and often the rate can fall as low as 20 per cent. Online questionnaires struggle even more than postal surveys to reach acceptable response rates. It is possible to improve response rates by specific actions:
- The subject matter must be of interest to the respondents, and not of too sensitive a nature.
- The sampling frame (e.g. a mailing list) must be up to date.
- The questionnaire must be well laid out, not too complicated, and self-explanatory.
- Consider offering an incentive for responding, such as entry into a prize draw, a shopping token or other small gift.
- Reassure the respondents of confidentiality in a covering letter, if possible addressed to each respondent in person.
- For postal surveys, provide a pre-paid self-addressed envelope and perhaps even a pen for replies.
To maximise response rates, allow sufficient time for the responses – ideally four to six weeks. Carry out a second wave of mailing/e-mail reminders to non-responders after this time, this may encourage more responses. While it may be tempting for online surveys to keep harassing non-respondents, this is unlikely to be productive, and may cause bad feeling towards future surveys. You will need to cost in the time and money involved in chasing non-respondents when planning your survey.
3. Quantitative interviewing
Interviews can be classified as follows:
- Structured (effectively a list of questions, usually with a limited number of possible answers).
- Semi-structured (more flexible, with some open ended questions).
- Qualitative (in-depth or focused, rarely using standard questions and tailored to each respondent).
The choice of interview very much depends upon your overall approach. An in-depth interview is only applicable to a qualitative approach, where you explore your subjects’ views and thoughts in depth. We explore qualitative interviewing is detail in Unit 8.
Quantitative interviews have much in common with the other survey methods already discussed, and administering face-to-face and telephone questionnaires can be regarded as structured interviewing. These methods are used (as with other surveys) when you need to ensure that the findings are comparable across the sample, and if you plan to use statistical analysis. A semi-structured interview can produce richer findings, but can limit comparability and quantitative analysis. Such interviews may also be longer and more complex to analyse, and thus more expensive.
While face-to-face and phone interviews have advantages over self-completed surveys in terms of response rates, they require specific skills (for more on interview skills, see Unit 8).
3.1 Structured interviews
Structured or standardised interviews use an interview schedule which is effectively a questionnaire comprising a fixed sequence of questions that have been planned in advance, sometimes refined with the help of a pilot study. Many of the questions will be pre-coded, thus forcing the respondent into a limited number of responses, which will be noted or written down on the questionnaire. As noted earlier, open-ended responses may be either pre- or post-coded.
When carrying out a structured interview, it is important that the interviewer adheres closely to the interview instructions:
- only interview those subjects who fit the sampling criteria;
- follow the correct order and filtering throughout the questionnaire;
- read out pre-codes and prompts where instructed, and write down open-ended responses in full;
- ask all the questions;
- keep personal opinions to oneself.
3.2 Semi-structured interviews
Semi-structured interviews are similar to structured interviews in that the topics or questions to be asked are planned in advance, but instead of using closed questions, semi-structured interviews are based on open-ended questions. Semi-structured interviews are useful when collecting attitudinal information on a large scale, or when it is not possible to draw up a list of possible pre-codes because little is known about the subject area. However semi-structured interviews are much more time-consuming than structured interviews, because of the requirement to draw up coding frames and carry out content analysis on a large number of interviews. Responses can either be recorded or written down by the interviewer.
Because open-ended questions reduce uniformity across respondents, it is all the more important that the interviewer refrains from influencing the respondent in any way, and maintains a neutral manner.
3.3 Problems in quantitative interviewing
When preparing a face-to-face or phone interview, the same guidance on questionnaire design and question wording applies. However, face-to-face interviewing is a social encounter, and this brings with it some additional and specific problems that may affect the data collection.
- Social desirability: in a face-to-face situation, interviewees may offer answers that they believe the interviewer will favour, or which they consider more ‘socially desirable’. (Sometimes referred to as the ‘halo effect’.)
- Characteristics of interviewers: interviewees’ responses may be affected by the personal characteristics of the interviewer.
- The problem of meaning: it cannot be assumed that the interviewee will interpret and share the same meanings of the questions as those of the interviewer and the research team.
3.4 Practical issues in quantitative interviewing
The following practical tips provide some further suggestions for quantitative interviewing.
- Dress appropriately for the setting where you are interviewing.
- Carry out the interview at a time that is convenient to the respondent, and in private if possible.
- Be aware of your own personal safety, carry a charged mobile phone, and make sure someone in your office knows where you are conducting the interview and when you should be expected back.
- Make sure you introduce yourself at the start of the interview, where you are from, why you want to do the research, what the interview will entail and how long it will last.
- Reassure the respondent of the confidentiality of the interview and research process.
- Have a pen that works.
- Listen to the respondent’s answers, even if you are only coding their responses.
Having looked at the two main approaches to surveys: self-completion questionnaires and interviews, please complete the following SAQ.
SAQ 6.2: Comparing survey methods
[table id=38 /]
Summary
In this unit we have considered the main methods in surveys: self-completed questionnaires and face-to-face and phone quantitative interviews. You should by now be able to describe the advantages and disadvantages of these methods, and also have an understanding of some of the issues involved in designing survey instruments.
Further reading
Gillham, B. (2008) Developing a Questionnaire. London: Continuum.
Oppenheim A. (1998) Questionnaire Design, Interviewing and Attitude Measurement (2nd Edition). London: Continuum.
The European Social Survey can be accessed at: http://www.europeansocialsurvey.org/methodology/questionnaire
To complete your study in this unit, answer the following exercise for your logbook.
Reflective Exercise 6.1
Think about how you could use a survey to answer a research question that interests you.
[table id=39 /]
Answers to SAQ 6.1
None of these are appropriate questions, because:
Q1 does not offer adequate choices and assumes the respondent drives.
Q2 restricts the choice of answers, and may be seen as patronising as these are all expensive foodstuffs.
Q3 is four questions in one.
Q4 is a loaded question inviting a particular response.
Q5 is ambiguous and vague.
Q6 used jargon that may not easily understood.
Q7 is a leading question.
Q8 encourages a specific answer.
Answers to SAQ 6.2
| Main advantages | Main disadvantages | |
| Face-to-face interview |
|
|
| Telephone interview |
|
|
| Postal survey |
|
|
| Online survey |
|
|
[box type=”info”]This unit is part of our course on Social Research Methods. You must be registered and logged in to access course content. Back to courses Welcome page.[/box]