Task Your firm of interaction design consultants is trying to build up a portfolio of impressive work, to enable it to pitch for business convincingly in the future. Your task is to produce a usability evaluation of an interactive system, plus a presentation of your results. You have a completely free choice of what interactive system you evaluate, except that it must not be one that you have evaluated for a previous assignment. You may get people to help you with tasks that are easier or better done by more than one person, but you need to be very clear about what you did and what others did. Choice of Interactive System Possibilities include software applications such as CASE tools or games or e-commerce websites or photo editing systems; electronic devices such as remote controls for televisions or DVD players, or digital cameras, or car radios; or control panels for appliances such as microwave ovens or home heating systems; or a self-service system such as an automatic train ticket vending machine. You may, if you wish, choose to evaluate two very similar and directly competing products, and assess ways in which one is superior to the other. This works well. It’s perfectly okay to decide to evaluate a part of a big or complicated system, or consider a limited set of use cases. The one piece of advice we can give is to choose something that is complicated or difficult to use, or is used to carry out complicated tasks, and preferably has obvious usability problems. Studying more complicated and less frequently used features of a system is likely to be more fruitful than focusing on the standard functions people use all the time. If you choose the standard or basic features of very popular systems you’ll find it very hard to dig out interesting results. You may choose to interpret ‘interactive system’ very broadly and present a usability evaluation of a static information display, but this would require a sophisticated and detailed analysis of how people use it for practical tasks, and these tasks would need to be complicated enough to give you something to analyse. Ask advice if you consider this. ? The Usability Evaluation Producing the usability evaluation will involve 1. Identifying the use cases or aspects of the functioning of the system to be considered, and briefly describing them in your report. (These don’t need to be a complete set of use cases; for very complicated systems focusing on one part of what they do is just fine. However you should give a clear indication of what subset of the functionality of the system you are considering, and what you are not considering. If in doubt, cover less functionality in more detail.) 2. Choosing an evaluation methodology. You should apply a standard evaluation methodology such as user testing, cognitive walkthrough, or heuristic evaluation. (You need to show that you know what the evaluation methodology is; not knowing what evaluation methodology you have been asked to use or are using is a common cause of assignment failure. If you want to do something non-standard, ask advice from your tutor.) 3. Defining an evaluation procedure. This will include stating one or several user tasks to be tested or considered with exact descriptions of the scenario and the goal the user is trying to achieve, as well as what the evaluator will do to collect results and produce an evaluation. The evaluation procedure needs to be described in full, separately from the description of the results. For a user trial, this involves describing the exact wording of the instructions given to the subjects. For a heuristic evaluation this involves being clear about the heuristics being considered; using a set that is more detailed and concrete than Nielsen’s set of ten principles is a good idea. 4. Carrying out the evaluation. This will involve applying the procedure and documenting what happens, and what the procedure finds. (If applying your procedure looks like an excessive amount of work, or is producing an excessively large volume of documentation, ask advice; we would prefer an evaluation giving detailed insight into part of the functionality to an evaluation with broad coverage but a thinner or more superficial analysis.) Aim to be as detailed and exact as possible about what the problems are and when and where they turn up. 5. Deriving findings about the usability of the interactive system from the results of the usability evaluation. This should include consideration of how strong and how general the conclusions are. Written Submission Your report should comprise the following elements: • Part ONE: The interactive system and its users. A brief statement of what the interactive system is and what it does – sufficient to make the rest of the report comprehensible; plus a description of the user populations and the assumptions it is reasonable to make about the capabilities of the users. • Part TWO: The use cases. Brief accounts of the use cases considered, plus a statement of what you are not considering, if you are only looking at part of the system. You need step by step descriptions of the correct procedures for completing the use cases you are considering in a heuristic evaluation or cognitive walkthrough, or asking people to perform in a user trial. However if the tasks are long and complicated with a variety of possible paths, something briefer might be needed; if you think this applies to your system, ask advice. • Part THREE: The evaluation methodology. An exact description of the evaluation procedure to be followed, including what the methodology is, exact descriptions of user tasks being considered, instructions to be given to users in user testing, or the set of guidelines used in heuristic evaluation. • Part FOUR: The evaluation. The results of applying the evaluation procedure: what you saw test subjects doing, measurements of their performance, answers to questions and so on; or evidence for violation of particular design guidelines; or descriptions of how and why beginning users might go wrong in particular places, etc. • Part FIVE: The findings of the evaluation. The findings of your evaluation of the usability of the interactive system. Include comments on how the findings relate to the results of the evaluation procedure, and ideally about how strong the evidence is, as well as judgements of how serious you think the usability problems are. An itemized bullet point structure is likely to be easier to read than long paragraphs of text. This should also include an appraisal of the strengths and weaknesses and successes and failures of the evaluation process. • Your notes made during observations of user trials, while conducting a heuristic evaluation, etc, should be included in an appendix. Handwritten notes should be scanned or photocopied.