I’m a huge believer in usability testing. Even after all this time of day after day for years working on interfaces, I’m always stunned by how much I don’t understand. We get very very very little of what pisses users off. Sometimes it’s simple things. Sometimes it’s obvious, sometimes it’s not. You just don’t know. No matter how much experience you have, you always miss a lot of little things which together can aggregate into what turns into a “pissed off user”. But when you watch the faces of your users or hear their voices, you have a lot more information to make a real choice. It’s really important.
So for Breezi, we did usability across 50 users, remotely. There are a few services available for this. I’m not going to list them because my experience was quite horrible.
The huge benefit of remote user testing is that users are at the comfort of their own homes with their own computers. Those are two very important points.
Anyone that has done usability testing in person knows that users act a bit differently when they are at a “lab”. I would go as far as to say that they are definitely a bit more rude on average. They tend to give more “raw” opinions. Sometimes this is not very helpful but if you can catch pain points much more accurately. When they are in a lab, they can’t be rude, say what they really think or trash your product.
Also, the fact that the user is using her own computer means that you can test performance. That’s where everything failed for us.
It turns out the recording technology that they use has a massive impact on your computer. More specifically your ram and graphics card. I started getting suspicious when all the videos that were coming in were users saying “this is so slow…” So we went and got the worst computers we could find and started testing on them. The experience was fine… I couldn’t figure out for the life of me what the issue was. It really frustrated me because it seemed like we were acquiring users and sending them to a horrible experience.
So yesterday I finally got a chance to act as a user. I used MacBook Pro with 2.66 ghz core 2 duo / 8 gigs of ram with a 100 mbps connection. I figured I start with a “good” computer and work my way down.
So I started the test, load their requirements… I loaded our site ( http://breezi.com ) and immediately noticed all the animations were loading really slow. It was like everything was in slow motion. I went on to create a site and actually login… now my fan started going nuts and everything was laggging. There was at least a 2 or 3 second delay on everything I was doing. The experience was horrible. Just absolutely, horrible. Just.. pure.. horribleness.
It’s amazing that the usability testing software can ruin the usability of your software. I didn’t write this post to bash them (although they should be bashed). I wrote it because I don’t want other people to go through the same pain. If you want to usability, I think that doing them in person is still the best.
Update: this is an awesome table by boltpeters comparing various remote ux testing tools