For this assignment, your group will be preparing a usability test on an application of your choice. Your test should approximately correspond to what Rubin calls an assessment test. On a working system, you will be identifying usability problems and determining the amount of user satisfaction and interest for the product. The test plan should be short and simple, thus allowing you to better reflect on its goals, design and limitations. For each participant, you should plan on approximately fifteen minutes of interaction time that, when added to greeting, instructions and debriefing, provides for a half-hour session.
During the week that the test plan is due, all teams will conduct a pilot test using the test plan. The live section will meet as a team and conduct their test on October 9 at a scheduled time. Online teams will conduct a pilot test at their own convenience but will need to submit a recording of the test, along with a commentary outlining issues and changes to the plan. The pilot recording for online teams must be submitted by Tuesday October 14.
Choosing the application from your first project might be a good idea, especially if you have some unresolved questions from your evaluations. If you choose a new application, give it an informal evaluation to see if you can identify some usability issues that could be resolved by testing. Again, I must be able to run your chosen application. While it is not required this time, you are welcome to send me a message asking about the suitability of your chosen application.
In chapter 5, Rubin provides a model format (outline on p. 83; new edition p. 67) for a usability test plan. You should also follow this format in creating your plan. The following provides additional requirements, guidelines and suggestions for completing each stage.
This statement may be one or two sentences that addresses the need to determine the application's usability problems and interest to potential users. You are welcome to invent some story as to why your group would want to do this (e.g. your company might want to buy and use this product, or the application's developers have asked you to evaluate it).
In this section, you provide the specifics of what your group will be looking for. The heuristic evaluation and the cognitive walkthrough can help you provide these specifics. For your plan, list five to eight test objectives in the form of questions, one to two sentences each. At least one of these objectives needs to address the user satisfaction of the product.
Identifying good test objectives is critical for planning and conducting usability tests. See Figure 5.1 (new edition 5.2) for examples. I strongly encourage you to discuss your proposed research questions with me before you proceed with the rest of your planning.
Normally, the needs of the client dictate who the targeted users are. However, for this project, you may invent a context that targets users whose profile fits easy-to-recruit participants (e.g. other students and friends). In any case, your plan should describe the user profile and motivate it given the context of your project.
Keep the design simple. When describing your procedure (which includes greeting, instructions, debriefing, etc.), refer to scripts and checklists, but append these materials to the end of the test plan. Start with a short overview of your design (a few sentences) before writing out the details (a few paragraphs). Make sure you specify the testing roles of your group members.
In addition to one pilot run, your group should plan to run tests with at least four participants. Ideally, some of your recruits will come from outside of class.
The set of tasks should last approximately 15 minutes. It may be one long task or a set of small tasks. If it is several tasks, state and motivate their order.
A short paragraph should be adequate here. Consider what space will be conveniently available to your group. In all likelihood, you will describe something similar to Rubin's simple single-room setup, possibly without the video camera. For the live section, we are planning to conduct the pilot test in CS&T 804, which is a small conference room. All other tests can be conducted in a location convenient to your group.
Here your group determines and explains how much the test monitor interacts with and helps the participant.
Decide what data your group will record in order to address the test's objectives. The data may be qualitative or quantitative. Creating and submitting a blank log sheet would be a good addition to your test plan, which you could place in the appendix and summarize in this section. Evaluating the user's satisfaction with the product will probably involve a post-test questionnaire.
This can be a summary paragraph anticipating what conclusions your final report will address. It includes explaining how the evaluation measures will address the test objectives.
These include consent forms, scripts, questionnaires, checklists and blank log sheets. They need to be prepared and turned as an appendix to the test plan.
Bring a hard copy of our test plan to the pilot test. Final submissions should be posted through the course online (COL) web site.
The grading principles from the last project still apply. Note that the guidelines and requirements listed above are in addition to the instructions and example in the Rubin text.
I will use this grading sheet to help me review the project.