HCI 460: Evaluating Human Computer Interaction
Fall 2007
Assignment 4
Comparison Study and Report
Due Tuesday November 13
Overview
For this assignment, your group will conduct a simple comparison study
and report its findings.
Requirements and Guidelines
Objective
Unlike your usability test, your group will focus on only one
objective. Ideally, the objective should only require a test with two
conditions and one dependent variable. The dependent variable should
be quantitative (e.g. error count, timing or sum of user ratings).
While not required, you will likely choose an objective that addresses
a design decision early in the development process, which allows for a
simple paper-and-pencil experiment.
Method
The method should be short and simple and directly address the
study's objective. Ideally, the process will last only a few minutes
for each session and can be conducted by one experimenter. This way,
each person in the group could run 10 participants within an hour and
possibly yield a total of 30-40 data points. At minimum, your group
needs to produce 8 data points per condition, although more is
preferred. Because your study involves human participants, you will
need their informed consent documented with a consent form.
The study may have a between-groups design or a within-groups
design. For a between-groups design, make sure that each
experimenter runs the same number for each condition. For a
within-groups design, make sure that the conditions are properly
counter-balanced.
Statistical Analysis
Statistical tests are required. Because of the simplicity of the
study, a T-test should suffice in most cases. Talk to
me if you need advice or help in choosing and conducting your
analysis. Also, you should report confidence intervals for the
dependent variable's average(s).
If you choose to use a subjective user rating for your dependent
variable, you are encouraged to use one of the standard scales discussed
previously in class.
The Report Format
Rubin doesn't provide any specific format for a comparison study.
Here are some guidelines for putting your report together.
Executive Summary
Create a short paragraph, perhaps just two or three sentences, that
describes your findings.
Purpose and test objective
Provide a paragraph that presents the background motivation for the
comparison test, followed by the test objective.
Method Section
This section presents your procedure for conducting your study.
The description should be specific enough so that others could read
your report and successfully replicate the procedure. You may want to
create subsections for describing the materials and participants used
in the study.
Because you test will use human participants, you will need to
ensure informed consent. Make sure you describe your procedure in the
method section and supplement it with whatever notice you give to your
participants.
Results Section
This section provides a written summary of your results, including
the statistical outcome and possibly a table or graph. Raw data
usually is not presented in the results section. For your report,
provide the raw data in the appendix.
Findings and Recommendations
This section should provide an interpretation of the results. If the
statistical analysis failed to show any significance, provide possible
reasons why the results were not significant. Remember that a failure
to produce a statistically significant effect does not necessarily
mean that no real effect exists.
Submission
Final submissions should be posted through the course online (COL) web site.
Again, provide a short description of how
the individual members in your group contributed to conducting the
test and writing the report.
Grading
I will use this
document to review your report.