Experimental Design for Measuring Task Success and Time on Task

Selecting Two Systems and Hypotheses

1.Select two systems as the target of analysis. Develop two hypotheses involving two performance-based metrics, task success and task time. For instance: 1.System A has a higher task success rate in comparison to System B 2.System A requires less time on task in comparison to System B Explain a specific feature or features that you aim to test 1.List the tasks/subtasks that a user needs to perform to test those features 2.Explain the specific scenario that you will test 3.State your experimental subject design: within or between. 2.Identify the independent and dependent variables from your hypothesis 3.Classify the data type of your dependent variables, e.g., nominal, ratio, interval 4.Shortlist the participants for the study (3-5 participants). 1.State their age, gender, experience with the system, and frequency of use 5.Explain the type of research (remote vs. in-person) Recommended that you conduct remote via skype or zoom t's ok to conduct in-person with a family member or roommate Experiment#1 Measuring Task Success

1.Run an experiment to measure task success

2.Explain how task success was measured? Task success is measured in many ways, such as: 1.verbally asking the user whether they were successful or not 2.asking users for feedback on a multiple-choice questionnaire

3.asking users to write their responses 4.observing the user perform the task 3.Explain how task failure was categorized. Types of task failure include: 1.Giving up. The user discontinues the task. 2.Moderator ending the session. If the moderator feels that the participant is not making any progress, they may end the session. 3.The participant is not able to complete the task in a given amount of time.

4.List and define different levels of task success? An example of a six-level approach is given below: 1.Complete success 1.With help 2.Without help 2.Partial success 1.With help 2.Without help 3.Failure 1.The user gave up without completing the task 2.The user thought it was complete, but it was not. 5.OR explain if a difficulty scale, such as the one below, was adopted for tasks? 1.1 = No problem. The user completed the task most optimally without encountering any problems. 2.2 = Minor problem. The user made a few mistakes but ended up completing the task successfully. 3.3 = Major problem. The user faced major difficulties but ended up completing the task eventually. 4.4 = Failure/gave up. The user did not complete the task successfully or gave up while trying to complete the task. (OR 4 Points) 6.Explain what level of assistance was provided to the participant? Justify? 1.The moderator may help the user by resetting the task if the user is getting confused. 2.The moderator may ask follow-up questions or explain the task again to prompt the user to think differently to complete the task. 3.The moderator may answer questions that help the user complete the task. 4.The participants may consult other sources, e.g., support teams, online guides, or user manuals. 1.Run an experiment to measure time on task 2.Explain how time on task data was collected and measured? 3.How was time data analyzed and coded? 4.Explain additional experimental detail 1.Did you count the time data for all tasks or just the successful ones? 2.Were the users in the study notified that their time is being measured? Explain your reasoning? 1.Run appropriate statistical tests (T-test, ANOVA, Chi-squared test) to determine if statistically significant results were reached between the two systems for the following dependent variables 1.Task Success 2.Task Time 2.Use appropriate visualizations (e.g., stacked bar graph) to present your results 3.Run a post-session interview (after all two experiments) to uncover additional insights into why a particular system performed better? Discuss possible design changes or refinements that could improve the system from performance standpoints