The “Wizard-of-Oz” method - testing users’ interaction with Swiss Post’s new consignment lockers

Creating a fully functional prototype can be time-consuming and resource-intensive. Using the Wizard-of-Oz method might solve that issue. It offers a cost-effective alternative, allowing designers to simulate system behavior and interactions without complex development. In this article, we are looking at this method, how we applied it to The Swiss Post case and what we have learnt from it.

The last decade witnessed a tremendous surge in e-commerce, with online shopping becoming the norm for many people. Companies like Amazon, Alibaba, and others expanded their reach, offering various products and services. This change profoundly impacted logistics as companies such as the post have to deal with high quantities of parcels. However, when delivering those parcels, mail carriers are often confronted with the issue that the recipients are not home. This meant the recipients had to pick the parcel up at the post office. Moreover, business hours could be restrictive if the recipients had an inflexible work schedule. The Swiss Post’s automated consignment lockers were designed to address the problem of undeliverable parcels and to offer recipients more flexibility. The idea was to let clients retrieve their parcels within a predefined timeframe or post them by using one of the consignment lockers. Those lockers were well-perceived by the customers due to their convenience. Therefore, the post planned on installing more such lockers. However, those new lockers differed from the original lockers in one aspect. They would no longer have a touch screen. Instead, they were operated by an application on the customers’ and mail carriers’ mobile devices. For this project those new consignment lockers were tested to ensure the new solution’s user-friendliness.

To ensure structured testing, firstly an evaluation plan was developed with three research questions:

• Would the customers accept the interaction with the locker using their own devices?
• Would the customers be able to operate the locker with the web application?
• Would the customers understand the communication between the web app and the lockers?

To answer these questions, the proposition was to do usability testing and apply the Wizard-of-Oz method. In this method, a human facilitator pretends to be an automated system. The user interacts with the facilitator as if interacting with the designated system. This allows the tester to observe and learn. The benefit is to find out whether something works or does not work early. Furthermore, all concerns about user requirements, such as user needs and pain points, are addressed well before implementation.

The relevant use cases were already known thanks to the other lockers, which had a touch display. Only those use cases for the web app on the customer’s device had to be adapted. The main use case was the customer dropping off and posting a parcel:

1. The customer brings the parcel to the locker
2. The customer scans the QR code on the machine with the mobile phone camera
3. The landing page opens on the mobile phone
4. The customer uses the mobile phone to scan the parcel’s tracking number
5. The customer specifies the needed compartment size of the parcel
6. The customer confirms by clicking on the button “Open compartment”
7. A free locker compartment opens
8. The customer puts the parcel in the compartment and closes the door
9. The application asks if the user wants a receipt
10. The application shows the completion message and asks for a rating

Based on that, a clickable prototype was built in Axure powered by custom JavaScript to display the phone’s live camera for simulating scanning interactions. Also edge cases were created that deviated from the standard case to gain more insights. An example would be when part of the system would not be used as intended, e.g. when there was a technical error, when someone would want to drop off multiple parcels, or when the locker would be intentionally misused.

As there were two UX designers working on the project, the roles were split as follows: my colleague would moderate the tests, and I was the one who would operate the simulations, do the technical setup, and do the observing. For the test, the client provided us with a demo locker with an accompanying app to open specific locker doors. Also needed were a test phone, a laptop for the operator, an app to open the lockers, parcels, and a printed introduction with a QR code. The latter was stuck onto the demo locker like in a real-life situation.

Six testers were recruited from a testing website since the specifications had a broad range. To carry out the tests and bring the experience to life, the test phone had to be mirrored via the open source mirroring tool “scrcpy” by @Genymotion to the laptop to click the hidden interactions on the prototype to show system messages. A fixed procedure was developed, e.g., opening the locker compartments at the right time. To get the execution just right, the entire was rehearsed test during a dry run. After partial recaps and adaptions, we felt confident and ready. During the tests, the execution worked quite well. The test users truly believed in interacting with the application. A couple of challenging moments occurred when test subjects acted differently than what had been anticipated. However, my colleague and I decided to play along and execute the system’s feedback. Pretending to be the system worked very well and let us gain further insights.

By applying the Wizard-of-Oz method, valuable insight into user behavior could be gained through using the mobile application for managing parcel lockers. Here are the learnings:

• Timing is key: to create a convincing experience, you need a script and carefully planned sequences.
• When executing the tests, be precise. Therefore, rehearse the process to ensure smooth testing.
• Test subjects sometimes react unexpectedly. Try to play along, as you can greatly benefit from such unexpected situations by getting a new perspective.
• Operating simulations during the testing requires a great deal of concentration. Remember this when the same person also has to observe and take notes. When in doubt, involve another person.
• Sharing the gained insights with the project team helps everyone better understand how users might interact with the final product.

The parcel lockers of Swiss Post that are operable by the users’ own mobile phones had to be tested. The aim was to gain valuable insight into the test users’ behavior with the application and their acceptance of the new way of operation. Therefore, the Wizard-of-Oz method was applied. Thanks to choosing this method, practical knowledge of the users’ behavior with the application and parcel lockers was gained. Specifically, it could be shown that most of the test users were open to using the mobile browser on their own devices to interact with a parcel locker. The users had little to no problem interacting with the simulated solution. However, it could be realized that it is further needed to rework some of the system’s messages displayed and the timing they would have to appear. Finally, new insights into the application could be gained by improvising when users reacted differently than anticipated. After this positive experience with the Wizard-of-Oz method, it can only be recommend considering for testing a digital or non-digital system.

Read the article also on medium.com here: The “Wizard-of-Oz” method — our way of testing the users’ interaction with Swiss Post’s new consignment lockers