Applying heuristic evaluation across the squad remotely

During the Covid Pandemic

This article doesn’t focus on how to perform a Heuristic Evaluation but rather on how the method was successfully adapted to an online format during the Covid pandemic.

The “new normal” marked by social distance and working remotely made people much more aware of the online experience. UX effects are much more noticeable now and so are the effects of not completely “tuned” designs. In the pre-covid world, some of these designs and their issues would go under the radar. Now, when users are much more aware, they result in loss of conversion (various degrees).

It is not for nothing that Intuitive interfaces, good usability, fluid experiences are themes that always guide the stakeholders’ minds. However, in the Agile development process, UX and usability often don’t get the attention they deserve. The development sprints try to make constant adjustments to meet the needs of users, however, development time is never enough, and usability tests and even interface validations cannot be performed at all times. In some cases, the product designer does not have enough time move forward with them, and in other cases the UX researcher has other urgent demands.

In addition, usability testing might take some time, as it includes: planning, defining tasks, recruiting users, running a test with at least five users, analyzing the results and writing the report.

It is worth mentioning that living in the Covid 19 Pandemic moment, which terrifies the world (especially Brazil), makes the face-to-face Usability testing almost impossible due to the social distance measures. Fortunately, there is the option of remote usability testing. However, it is not as reliable and informing as the face to face test, and still requires all the preparations mentioned above.

In summary, ‘lack of time’ and ‘limited budget’ are usually the main reasons for producing faulty/untuned/too raw designs.

Facebook Usability Testing

Given all that, how can I as a user researcher, help Products Designers deliver easy, delightful interfaces? How can I ensure that UX research a is given part of the Design process?

The task was to look for a UX research method that had minimal scientific rigor, that was also tested and proven, and could be made flexible enough to an agile development environment. After reviewing several design research methods, I chose the method called heuristic evaluation. This method is based on a usability inspection of the digital interfaces and identifies out-front and possible usability problems.

Heuristic evaluation definition
A heuristic evaluation involves having a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the ‘heuristics’).
Jakob Nielsen, The Nielsen Norman Group

The main reason that led me to choose the heuristic evaluation method was the fact I could easily adjust it to the agile environment (it is usually performed by one or two experts off-line) -NM addition. Another main reason was the fact that I didn’t need users to perform it. A lot of time is spent at the recruitment stage (almost always so time consuming) and since I didn’t need users for this method, I saved a lot of time and money. Other factors that led me to my choice were:

• Ease of execution, no need for special preparations of tools
• Easily applied Remotely (pandemic moment)
• Efficient in both the results stage and the costs ratio
• Flexibility — can help teach the organization new usage heuristics
• Can be applied at any stage of project development

Once the method has been established, the first thing I had to do was launch a pilot project, that included the following activities:

1. Heuristic evaluation training for stakeholders
2. Using heuristic evaluation to Remotely
3. Defining UX and usability criteria
4. Defining the studied design, screen or work/user flow

Let’s look at these activities closer:

Training — this activity was applied to a team of evaluators that was comprised of product designers, product managers and developers. All of them had little to zero knowledge about the heuristic evaluation method. I prepared a presentation containing UX Concepts, usability principles, some history about the field, and of course, how the dynamics work.

Evaluating remotely — The idea here was to adjust the method so it can be executed remotely. For this purpose, we used “Miro”. the team was already familiar with the platform, and it allowed physically separated people, to work together in a remote collaboration.

According to the method, the experts evaluated the interface individually. Afterwards, we’ve gathered to review the identified problems and reach a consensus on their severity levels.

An interesting point was that the participants acted alone (offline) and only later connected in order to aggregate the findings online. This combination of “offline” + “online” helped alleviate the usual arguments and conformations associated with group dynamics.

The entire documentation was easily imported into Miro. The usability issues f automatically turned into sticky notes, which made data analysis much easier.

After that stage, we set up a board where it was possible to follow the heuristics that were being violated with various frequencies and severity levels. This was very valuable information to help us improve the UX, and the usability of the system.

This adaptation of the heuristic evaluation method to a remote format was dubbed in the company as the “Heuristic lovers”.

Definition of UX and Usability Criteria — there were several principal sets we could have used:

• The 8 golden rules of Ben Shneiderman
• The 5 ergonomic criteria of Dominique Scapin and Christian Bastien
• The 7 dialogue principles of the ISO 924: 10 standards.

In order to simplify the experts’ analysis and bring speed up the project, we chose to use Nielsen’s heuristics. They are sufficient to guarantee efficiency and effectiveness in digital solutions. One of the most striking principals of the heuristic method is the possibility to add your own heuristic, which can be your company’s visual design guideline or other principles that you consider important. In our case, we created two new heuristics: transparency and User empowerment.

Here is the list of heuristics we’ve used in our pilot:

1. system status visibility
2. system compatibility with the real world
3. user control and freedom
4. consistency and standards
5. error prevention
6. recognition rather than remembrance
7. flexibility and efficiency of use
8. aesthetics and minimalist design
9. assist users to recognize, diagnose and correct errors
10. Help and documentation
11. Transparency
12. User empowerment

Screen or flow definition- It was a general decision, we chose to evaluate the scheduling flow, where leads (aka the user) fills out a quick form to schedule a visit to a property.

After we’ve completed these activities, the evaluation was carried out. The 3-day schedule was followed vigorously. Many interface issues were identified by all evaluators, some were unique to some of the specialists. For this reason, it was important that the team was multifunctional, belonging to different teams, different departments. Different points of view enriched the analysis, encouraged people to cooperate and also brought visibility to the projects that were happening at the Company.

Attention point

It is important to note, that in addition to so many positive characteristics of the heuristic evaluation method, there is one issue that is very important. Although the experts identified the usability issues based on the UX and usability principles, subjectivity is always present, and that can lead to some biases.

In order to avoid these biases impact on the reliability of the results, I proposed collecting feedback from users. This collection would be through a Usability testing where only the most critical problems raised in the heuristic evaluation will be investigated. In some of the projects we’ve realized that the problems mapped in the heuristic evaluation were the same ones pointed out in the usability testing. Therefore, the execution of the usability testing would not be crucial, because they would function more as a validation. Evidently this recommendation will depend on the level of experience of the specialists, topics investigated, level of maturity of the organization’s design System, etc.

Heuristic Lovers combo

In general, the Pilot Project was very successful. We’ve learned quite a bit. A valuable finding at the team / department level, was the generation of a consistent Backlog, which helped us in prioritizing projects. Stakeholders and designers have broadened their knowledge about HCI. And in a high-level view, the experiment stimulated the collective work and the sense of cooperation that provided an exchange of context between the teams and departments.

It is also worth mentioning that, if performed periodically, the heuristic evaluation also makes it possible to monitor UX metrics.

Results Heuristic Evaluation

Anyways, everything is a matter of transformation and adjustment. From a biological perspective, adaptation is the dynamic evolutionary process that adapts organisms to their environment, increasing their evolutionary aptitude “. We are able to live in humid tropical jungles, deserts, arctic wastelands and even cities with considerable amounts of pollution.

In other words, adaptation is the key to human evolution.

In User Experience Research the dynamics are similar. However, to truly know your users or how they interact with your product, it is not enough to view their interaction through a computer screen. Remote user research still has its challenges and restrictions, however, UX researcher’s methods, perception, and intuition can create new strategies to overcome these obstacles.