User Experience Research: Myths, Data, Metrics, Usability, and Learning

Task 1: UX Research Myths and Sources

1. a) Tullis & Albert (2013) list ten myths about user experience (UX) metrics or aspects thereof:

• The time it takes to collect metrics
• It cost too much money
• Not useful when focusing on small improvements
• Don’t help us understand causes
• The data collected is too noisy
• You can trust your gut – no need for testing
• Does not apply to new products
• There are no metrics available for all types of product
• Management is not interested
• You cannot collect data with a small sample

 

Read section 1.6 in the prescribed handbook with attention. Make sure you understand exactly what each myth entails. Then use the general internet to search for five further (i.e., not the same as Tullis & Albert, 2013, nor formal research articles) myths about user experience research.

 

These 5 myths must be sourced from 5 different websites, and must not be specific to a product (website, mobile phone, software product etc.)
Complete the table as required. For the U or UX row, Tullis & Albert (2013) notes that some people distinguish between usability and user experience (p. 5). In your view, does the myth you have sourced relate more to U (usability), to UX (user experience), or to both? Motivate your answer. 

 

2. Chapter 2 is about knowing your data. The specific type of data you collect will dictate what statistics you should perform, and how you can present your results graphically.
Consider the flowchart below, which offers a high-level summary of Chapter 2. The orange boxes (types of data) link to the grey boxes (types of analysis given the data types), which link to the green boxes (statistics or statistical tests) that can be performed, which is dependent on the white boxes (the types of samples/users).
As you will note, some shapes (numbered from 1 to 10 on the chart) are missing information. Write down the numbers 1 to 10, and next to each number, provide the correct information from the list that should appear in a specific shape.

 

3. The International Organization for Standardization (ISO) defines usability as "the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (BS EN ISO 9241-11:2018). In the definition, keywords have been boldfaced for you to take note of.
a) Using only Google Scholar as your search engine, identify any 5 formal research articles or publications (not a website) that define usability. Using the table below (repeat for 5 articles), and for each article/publication definition:
• Enter the full reference to the article/publication.
• Enter the definition as provided in the article. It should not be the same ISO definition.
• Enter keywords from the article definition that are similar (or have the same meaning) as the ISO definition
• Enter keywords from the article definition that are different (not found) in the ISO definition 

Task 2: Knowing Your Data

 

b) In a single paragraph, what have you learned from Task 8 (a)? 4 marks

c) In 1991, long before the current ISO standard, four classes of usability measures were identified from previously published usability studies:
i. Goal achievement
ii. Work rate
iii. Knowledge acquisition
iv. Operability
Consider all the individual metrics that are covered in Chapters 4 – 7 in Tullis and Albert (2013). Reproduce the table below, and for each row, in Column B, enter a single metric that would fit the class in Column A. 

 

d) What other classes of metrics (presented in Tullis and Albert (2013) as chapters), are absent from this early list of usability measurements? 3 marks

3.In Assignment 2, you were required to source 5 online sources which cover sample sizes in usability testing and to summarise what you have learned as it relates to the number of participants to use. The table presented here below lists how many usability issues were detected by 10 participants in a study conducted on a new Automatic Teller Machine (ATM). 

 

a) Using p. 117 as a guide, extend and complete the table with a proportion column and row.

b) Using the exact format provided below, show all calculations in providing the percentage of issues that would be identified with a sample of 2, 4, 6 and 8 users, respectively. Percentages
2 users: Calculations
4 users: Calculations
6 users: Calculations
8 users: Calculations 8 marks

 

c) Report on what you notice and conclude by stating how the results contribute to the debate on the number of participants required in usability studies to reliably identify usability issues.
What I have noticed
Your answer
Conclusion

d) Consider the table below. It contains more data collected with the ATM usability test. Rather than reporting results separately for each task, one can combine all the results into one "overall" score to answer the question - is the new ATM good enough or not? Note: It does not mean one cannot report the results separately. Such information may assist the developers to gain an understanding of where exactly they need to focus to improve the ATM. 

 

Copy the table and add 5 columns (Tasks, Accuracy, Satisfaction, New functions and Average) to the right of the table (see Tables 8.5 and 8.6 for examples). Convert each of the metrics to a percentage. Once that is done, calculate an average percentage for each of the five columns. 10 marks
e) Assume the following goals have been set before the usability test:
Fifty percent of users will be able to complete at least 5 tasks
No user should make more than 5 errors for all the tasks
More than 70% of users should agree that they are satisfied with the new ATM
More than 50% of users should "somewhat agree" that they find the new ATM functions useful
Based on these goals (i.e., have they been met?), how would you rate the usability of the new ATM? 

 

f) Based on the overall score (the far-right column average presented in the bottom row), provide an interpretation of the usability of the new ATM. 4 marks

 

g) Assume Participant 1 and Participant 2 made two errors that were the same. What is the total number of unique errors they made? 2 marks

4. Consider the combination column and line chart for the 10 tasks below.

 

a) Which tasks were most problematic? 2 marks
b) Which tasks had the most disparity between task rating and task success?
c) Why is the finding here above in b) troublesome? 2 marks
d) Assume three metrics are available. What type of chart would you use to represent the summary data? 2 marks

 

5. Assume you are interested in the amount of learning that is required to perform the following series of tasks that can be performed on an ATM.
Step 1: Insert ATM Card:
Step 2: Select Language
Step 3: Enter 4-Digit ATM Pin:
Step 4: Select the type of Transaction:
Step 5: Select the Type of Account:
Step 6: Enter the withdrawal amount
Step 7: Collect the ATM Card
Step 8: Collect the cash 

 

Explain in detail how you will collect, analyse, and present the data to the developers. 12 marks