Start Date:
January 2021
End Date:
March 2021
Role:
User Researcher
Our team investigated the usability of Armoire’s item discovery navigation system, focusing on the site’s search functionality and browsing options. We explored usability of the search bar, filtering, and browsing clothing categories. The purpose of this usability study was to build upon the previously conducted heuristic evaluation. We honed in on identified areas of concern through usability testing as well as gathered global metrics on user satisfaction and overall website usability.
.jpg)
A representative from Armoire suggested that our graduate student team explore the search & discovery process on Armoire's desktop website due to concerns that users, particularly new and inexperienced users, were having trouble finding clothing that they wanted to rent. Our team set out to investigate this concern and determine both where and why Armoire users struggled, as well as come up with suggestions as to what could be improved on the desktop website to streamline the user experience.
The results of this usability study were analyzed and then conveyed to the Armoire UX team, along with potential suggestions for improvement.
The results will be used by Armoire’s engineering, user experience, and website development teams to iterate upon the website’s existing search functionality and to improve the overall user experience.
Before beginning this usability study, Armoire relayed their business goals of new user acquisition, increasing value of service, and improving user satisfaction. Our research centered around these stakeholder goals. Because of their dedication and support, our team was able to use real Armoire users, recruit through Armoire's official communication channels, and provide incentives to participants. The specific user behavior explored in the usability study was predetermined through a heuristic evaluation. Our team was also given access to a participant persona to keep in mind when designing the usability study kit.
Due to our focus on the item discovery process, there are several major areas that were out of scope for this usability study. For example, while all users interacting with the Armoire website must complete login and logout processes, these processes were out of scope in this study due to a lack of corresponding user experience violations found during the initial heuristic evaluation. Furthermore, the conversion process was not tested due to time and resource limitations. Additionally, this usability study did not explore the social experience of the Armoire website due to it being explored by a different student team. Additionally, this research was conducted during the COVID-19 pandemic, so our methods were exclusively remote.
The purpose of this research was to determine areas of improvement in product discovery navigation on the Armoire website. A usability study was the optimal choice to fulfil this goal as it offers maximum results for minimum investment. This is because a usability study offers the ability to collect both qualitative and quantitative data on the experiences of actual product users and their individual website navigation patterns. Additionally, usability studies are an example of “lean research” in that they place low burdens on participants while still offering high value information for stakeholders, in this case Armoire. Usability studies are rigorous, ethical towards all involved, provide relevant information to stakeholders, and are optimally sized to maximize usefulness while minimizing costs. Furthermore, as this usability study focuses on interaction with an existing product, the proposed study can be categorized as a summative usability study.
Our core research questions focused on efficiency, usability, and satisfaction.
For this study, we first conducted a heuristic evaluation of the Armoire website. Potential project scope was then directly discussed and determined with stakeholders from Armoire. Participants were recruited exclusively through emails and the pool of potentially eligible users was identified by the Armoire UX team. Potentially eligible users were requested to complete a short survey to better identify eligibility and collect participant demographics. This study was successfully completed with 8 participants, each of whom participated in a 20-40 minute Zoom interview with at minimum one moderator and one notetaker present at all times.
Sessions took place through Zoom, a video conferencing software. Before each session, each participant submitted a consent form on Google Forms as well as logged in to a staging account. 7 of 8 participants were able to use their personal accounts on the sandbox website, but one 1 user ran into technical difficulties and therefore used a mock account.
These remote usability study sessions consisted of open-ended verbal questions along with follow up questions in Google Forms. Each session had three primary participant tasks: Searching for items, favoriting and unfavoriting items, and rating items using he Upcoming Styles feature. During each session, participants were instructed to think out loud. The moderator also asked pre-scripted open-ended questions as well as impromptu exploratory questions inspired by the 5 Whys method of interviewing. After the completion of each task, participants completed a short section of Likert scale questions on a Google Form. After completing all 3 tasks and corresponding Likert scale questions, participants completed the SUS and NPS scales and submitted their Google Form. Additionally, a notetaker was present during each interview session to record notes on a Google Sheet. Notetakers took notes both during the session itself as well as post-session due to the availability of session recordings.
This was a mixed-methods usability study: therefore, both qualitative and quantitative data was collected and analyzed. Qualitative data, namely participant quotes and behavior, was moved to a team Miro board and organized using affinity diagramming. Especially important reactions were generated into short, impactful reels using Reduct. Quantitative data was pulled from both the participant’s Google Form as well as from the team notes sheet. This data was copied to a shared Google Sheets for compilation and analysis.
Each metric relates directly to a research objective. Metrics lean towards more towards quantitative and behavioral areas due to this usability study being a summative usability study: the product already exists, but Armoire needs to know what's working or what's not working. We get the "why" and "how" from the open-ended questions and follow up questions with participants, and the numbers to back up our findings through quantitative data. It's important to note that since Armoire has a relatively small UX team, our results are actually used to make decisions and evangelize throughout the company: we also wanted to have an abundance of quantitative data (even if it's not actually statistically significant!) because we want our findings to be trusted and supported by non-UX team members and leadership.
Armoire’s target user group is established professional women between 30 and 50 years old who are looking for great clothes to wear every day. This core group of subscribers are loyal to Armoire not only because they love the service that Armoire provides, but also because they have an emotional connection to the overarching company culture and mission. Because of this, a representative for Armoire suggested that we focus on new/slightly experienced subscribers to gain insight onto the user experience of someone new to the product. This is reflective of the company’s current interest in new user acquisition.
Our team performed usability testing with 8 participants. Before starting recruitment, we identified the ideal participant sample size range to be 5-8 participants based on factors such as the project scope, single-group participant sample (all relatively novice users), resource needs, and number of overall Armoire subscribers. Participants were offered a bonus item as compensation for their time. We focused on new & slightly experienced subscribers to gain insight into the user experience of someone new to the product. We recruited current women-identifying subscribers with accounts less than a year old. It is also important to note that the participant sample subscription plan breakdown reflected the overall actual subscription plan breakdown.
After data analysis, findings were first tagged to categorize areas of success and areas of improvement. Then the severity of the findings was categorized using Jakob Nielsen's 4-step scale. Each finding, whether it be success or room for improvement, was relayed to Armoire with the severity rating and tags, number of users impacted, description, supporting data from the study, and potential recommendations to address the uncovered issues. Findings were described in a presentation as well as in a finding summary document.
In general, participants were satisfied by the Armoire user experience. We identified the following areas as being particularly successful with our participants:
To relay our findings, our team created and presented a slide deck that provided more detail on the findings.
