Surveys

Surveys are structured queries to a large group of people. The goal is to draw conclusions about an entire population of users.

Properties

• Large samples
• Most preparation
• Provide qualitative and quantitative results
• Data collection can be automated
• Some analysis can be automated
• Indirect method

Sources of error

• Selection bias
• Chance error

We have mathematical formulas for estimating chance error. Bias is more difficult to estimate. Kuniavsky uses the terms target audience, sampling frame and sample to discuss sources of bias.

Conducting the survey

• Phone
• Email
• Web site

Analysis

Both quantitative and qualitative data can be collected with surveys. Open-ended questions contribute to qualitative results.

Qualitative analysis

1. Group responses into themes
2. Write summary statements
3. Report number for each theme

Quantitative

Quantitative results are often displayed with tables, frequency bar charts or pie charts (for categorical responses).

Estimating the sampling error

• Standard error is calculated by taking the standard deviation and dividing it by the square-root of the sample size.
• Standard deviation of the proportion of a particular response is calculated as sqrt(P * (1 - P)), where P is the proportion of the particular reponse.
• An approximate 95% confidence interval is obtained by doubling the standard error.

Note that the population size does not figure into calculating sampling error. It's not clear how the numbers are calculated in the table on p. 332 in the Kuniavsky book.

Items for discussion

• Sources of bias in sampling
• Strategies for choosing and editing questions