Usability Test Plan Template

Usability Test Plan Section

Test Plan for: Product/Application Component

Persona Type Tested: Primary

Context: at home, updating playlists
Duration: 1 minutes 50 seconds

Complete the following before starting: Greeting and Consent & User Background Information.

Overall Objectives

  • Assess the overall effectiveness of the U play3D application for users who are interested in creating "smart" playlists;
  • Identify obstacles to completing the creation of a new radio - "smart" playlist;;
  • Evaluate the time it takes users to perform essential tasks such as playlist creation, songs addition, playlist sharing and listening;
  • Test if the terminology and the labels make the application easy and clear to use;
  • Use the References to guide you in writing your objectives.
Scenario
  1. The user wants to create a new "smart" playlist.
  2. To accomplish this, he accesses the application via his Google account. He enters his credentials in the login page.
  3. After authentication, the main menu of the application is visible, including the "Radio" section.
  4. The user selects all the available default criteria for radio creation.
  5. He adds "Pink Floyd" as a custom search as well.
  6. He performs a new custom search for "The Beatles".
  7. He deletes the "Recent albums" criteria.
  8. Plays the new generated radio.
  9. Deletes the first three songs from the global player.
Tasks
  1. Authentication: 30 s;
  2. Selects all default criterias: 20 s;
  3. Performs a custom search for "Pink Floyd": 15 s;
  4. Performs a custom search for "The Beatles": 15 s;
  5. Deletes the "Recent albums" criteria: 5 s;
  6. Plays the "smart" playlist: 5 s;
  7. Deletes the first three songs from the global player: 20 s;
Total time: 1:50
Post-test Questions

These questions could be presented to the user as a questionnaire or verbally asked.

  • Question 1: How easy was it for you to choose the criteria for a new "smart" playlist?
  • Question 2: How easy was it for you to generate a new "smart" playlist?
  • Question 3: What did you think about the manner of deleting songs from the global player? Anything you would change about that?
  • Question 4: Was the terminology understandable for you?
Notes for Test Coordinator
  • Remind the user that you would like them to 'think aloud' so that you can understand what they are thinking, about the task or the product/application;
  • Read the user the scenario for the test;
  • Read the user the tasks he has to complete. If the tasks are complex, you may want to put a sheet of paper in front of the user with each task written out;
  • Record the user's actions, comments, questions, and body language (write them in the Test Observations section accompanied by video/audio).
  • Don't offer help; let the user attempt to perform the task. If they ask for help reply with: "What do you think you/that would do?" "What do you think that means?".
References

Test Results for: Product/Application Component

Success Criteria

A successful design has been achieved when:

  • 50% of users complete the tasks before the allocated time.
  • 80% of users complete the tasks without the need of any external input.

Summary

  • As for the previous test, the radio section was the most confusing.
  • The users depend too much on the tutorial.
Demographics
User Number Gender Age Range Persona Type Technical Level
User 1 male 20-25 primary medium
User 2 male 20-25 secondary medium
User 3 female 20-30 primary medium
User 4 male 20-30 primary medium
Interaction Notes
User Completion Time General Observation
User 1 4 minutes The most difficult task for the user was finding how to reset the radio to recommended criteria.
User 2 5 minutes The user had no major difficulties completing the tasks, apart from finding how you delete a song from the "now playing" list.
User 3 2 minutes The user managed to complete the test with ease, the most time consuming task being that of deleting a song from the player.
User 4 2 minutes 45 seconds The most problematic task was that of deleting a song from the player. As in general, some aspects regarding radio criteria were not clear enough.
Post-test Questions Answers
User 1
  1. The most confusing operation was resetting the radio to the default settings.
  2. No problems had.
  3. The manner in which the options for the songs is displayed (clicking and holding over the desired image) is confusing and hard to find without first reading the tutorial.
  4. All terms were clear for these tasks.
User 2
  1. Struggled a bit with finding the reset option for the radio, but generally no issues had.
  2. Very easy.
  3. It isn't an intuitive gesture if you do not read the tutorial beforehand
  4. Default criteria for a radio was a term that was not understood.
User 3
  1. One main issue was adding a custom criteria. After writing the criteria the user pressed "Enter", which was not the manner of addition.
  2. Extremely easy.
  3. Deleting songs was not very intuitive and is too dependent on the tutorial.
  4. The label given to the button responsible with criteria addition was not suggestive.
User 4
  1. The user described this task as a medium task, because the custom criteria did not support the "Enter" key action.
  2. Generating the "smart" playlist raised no obstacles. It had a clear and straightforward manner of execution.
  3. This task was frustrating for the user, since it is not intuitive at all.
  4. The criteria for radio generation were not very clear.
Test Observations

As for the previous test, the general workflow of the radio section is confusing and some terminology such as criteria or default criteria is not intuitive. The users also depend too much on the tutorial, as some gestures are not intuitive and not easily discoverable.

Potential Design Improvements

We should try to adopt more conventional gestures and idioms, to make the general workflow more intuitive and discoverable even if the user has partially read the tutorial or skipped it entirely.

A suggestion from a user is to change how the options menu is displayed for a song in the "now playing" bar, by replacing the long press gesture with a hover effect (displaying the song options - delete and delete all only when a user hovers the song). Another possible improvement suggested was that of dragging the song outside the player.

Another suggestion from a user was to slightly change how we focus on certain elements of the page during the tutorial, by changing from the current verison of a red highlight to a zoom effect over the desired regions. We feel this might cause confusion to some users if the shape of some elements is changed, and most likely will not be implemented.

Another suggestion from a user was to modify the way a custom criteria is added to the list, by enabling the action to be performed when pressing the "Enter" key.