Testing for Learnability - the quantitative side

A question was posed by a good friend and colleague a while back about testing for learnability. The discussion was quite good in terms of defining possible qualitative data to compare. However, what I'd like to know, is there a simple, quantitative way of testing for learnability?

Here's my take, and unfortunately, it's only a theory for now because I haven't practiced it:

There's one metric that hasn't been mentioned and that is time-to-task. Now, what if time-to-task can be measured several times on the same task as opposed to just once? If the user is suppose to learn about how to use an application, shouldn't the time-to-task be reduced on subsequent tries? Isn't that an indication that the user have learned how to use it? When we play video games, the second or third time around seems easier to get through a level, doesn't it? So in that sense, how do we test for it?

After determining what kind of learnability you are testing for, a possible usability session may consist of the following guidelines:

1. Have the participant perform a repeated task.

2. In the series of tasks, mix in another task that could be something completely different (or something related). This disjointedness could test the user on how well they remember to perform the previous task again.

3. Increase the time between the first time and the subsequent times of performing the repeated task.

This method can be a sure way of quantifying the learnability of a particular function or process because what will happen with a successful design is that the time-to-task should be decreased with each repeated tries. Something that is very learnable may takes significantly less time than the initial stab. If there is no difference on subsequent tries, well, you know it isn't learnable.

The idea is to reduce the learning curve (dictated by time) and with this method, the time-to-task metric can be used to its full potential.

Are we speaking Alien?

I have a good feeling - and I'm sure many other usability practitioners have come across this - that we tend to speak a different language from developers. (Developers meaning the true sense of "coders" for lack of a better term.) Let me explain.

We recently had an educational series about UCD principles and it was well-received. It was for the developers and apparently, they learned quite a bit about what people in the UX field are most cognoscente about. And of course, everything that was taught was new to them. How could they know?

And this brings me to the point of what I dislike about about some people in our profession. There's this "we" and "them" mentality - we, being the people in UX, them, being developers or anyone else who may be labeled as ignorant. While we do know it exists, sometimes it's also all in our own mind. Yes, I used to be one of them many years ago - but that's only because I didn't truly understand what this game was all about.

It's not about battling each other because of egos (even though that's the case sometimes), and it's not because "they" didn't want to understand. It's because "we" as people in the field of UX aren't speaking a language that ordinary people can understand. (Yes, we are weird.)

We talk of wireframes and task analysis and layouts and personas, user profiles, U.I. standards, U.I. patterns and card sorting, usability testing, heurisitics, VIMM models and it goes on. So what really is this language? It's all jargon that we use to explain ourselves, but it doesn't get the message across. So what to do?

We in the UX field need a translator. It can be in the form of another person or ourselves. Either we learn the language (of ordinary English) or the Developer needs to learn our language. Which do you think is easier?

Ok, so we opted for the latter, but it wasn't easy. And we had to put it in plain english or it wouldn't be understood. Sometimes we get so caught-up in our research and our own mental models that we're not doing an effective job.

The point is, we really need to be more sensitive to our developers. We're sensitive to our users, so what makes development any different.

If we're here to serve our users, let's also serve everyone who's related to them either directly or indirectly. We're looking to make our users' life easier and so it starts by talking to development properly. It also takes some education, on both parties.

So, how about it?

Be definitive in your recommendations

...especially when you know a principle you've learned and you need to share that information.

There are times when recommendations are needed because others' expertise are not sufficient. What I'm finding with most people in UX is that they use the word "depends" almost like it's their second breath. I believe there are times that the word "depends" doesn't always work. It can sometimes be misconstrued that we in UX do not know what we're talking about - and because of that, it becomes a credibility issue.

One of the things I very much dislike in people is wishy-washiness, much like how the cartoon Charlie Brown used to be. I'd rather have a definitive answer with options than to just say "depends". I think we owe it to the people we're serving because they don't think like us. To the developers, to the managers out there whom we've been confusing all those times, we need to communicate in such a way that brings home the point.

One thing I like to do is to give the people I serve, options. Options meaning ways to solve an issue or problem because ultimately, it's to the developer or the manager's judgment to go about how they will accomplish the usability objective. Their code and their management makes the product work behind the scenes, so we UX practitioners need to be definitive because the developers and managers are definitive.

No one ever got anywhere being on the fence - they usually get speared by the tip sitting on it too long.

Empower your team

Do you know what kind of impact you have when you've included your project development team into your user testing? I'm sure not many realize the benefits of this. While I'm sure most researchers, kind of like the scientific type would rather hide in a dark room to do their tests and only give out findings and recommendations later on so that there are no "biases" or outside forces co-mingling to create skewed results, there are better more efficient and effective ways to get the message across.

The goal for me has been to find ways to empower our team. Find ways so that the development team create their own message from what they have seen and experience by watching user testing. No hearsay reports, no convincing, and very little person to person influencing to design decisions.

By watching user testing, the team gain a few things:

1. Perspective - It's no longer by the developers or sales, it's all about the user and how they go about using the application. While some may say the user is wrong, or they're unfamiliar with it, or they're doing things wrong, one cannot argue with their actual mental patterns, their previous experience, make up all their user behaviors up until the time of testing.

2. Thinking towards a solution - the instant the team sees a user have a problem, the team sees ways in their mind to improve the program concurrently with what just happened in testing. By doing this, it takes the pressure off the usability professional and empowers the team with instant information - a catalyst to improvement.

3. Individual contribution to masterminding - This is the last point to this cascade. We all know - hopefully, that masterminding is one of the most effective tools to get to where we want to be instead of where we don't. By masterminding our design solutions and understanding the catalyst in its different facets, the team no longer designs in a vacuum. Also as individuals, there is some self-gratification to a better job done well.


By empowering my team, I've pushed-out the knowledge to create other ideas, not from myself. The team takes ownership and makes these kind of battles more personal. And with it, the combined effort applied is greater.

3 more Principles

So you're finally out! You've just completed your usability training from a commercial organization like HFI, or you've taken a program in Bentley College or Carleton University. Now what? Does the learning stop? Quite obviously, no. In fact, the real learning has just begun. While courses and programs give a basis and fundamentals from which to work on, the application of this knowledge is where you really start to learn. Please note, I said "Start". Even applying this knowledge is not enough. So really, when is enough really enough? From what I've learned over the past years are three timeless principles that contribute to the success of any businessperson:

1. Read
2. Listen
3. Associate

Read - Read books and manuals for reference so that when you come across a situation you need more help in, refer to your materials. Also look for articles on the web that relate to UX that may give you a larger perspective on something you are encountering or may encounter in the future. It has been known that when a person reads, they retain more than 90% of the material because the brain is more active in producing visuals that interpret words to pictures.

Listen - I know for me, I have yet to do this. I have yet to plug myself into an audio book or a podcast. What I have done in fact is connect to some of HFI's webcasts on some of the topics I find interesting or relevant to what I'm currently doing. Video is just as effective a learning tool as audio as visual examples can really hit home what is being explained.

Associate - We have a local usability community that meets every month. Unfortunately, I was told by a colleague that I missed the entire year and I am guilty for not following this principle. It is in a physical forum that we learn the most from. There's nothing like telling a story and listening to them to impact others around you whether you're in a job or in your own business. This goes for success-thinking - if you want to be successful, associate with like-minded successful people. I also find it to be quite fun.

As a lesson then, continuous learning and growth is crucial to anyone looking to become successful. As an example, I heard that the GE corporation has been known to be one of the most advanced companies because of their expenditures on human resources and learning. Most people that come into the company are in fact retrained so they can "unlearn" what they've learned in traditional schooling.

Talk about assimilation to the extreme.

Usability Findings is like an episode of CSI

I'm a big CSI (Crime Scene Investigations) fan and I'm starting to come across some parallels on how the usability profession is very much like solving a crime - or in this case, the crime of un-usability. So here's my take:

Where this parallel starts is when product support finally tells the development team that users can't take this frustration any more. "Do something about it!" they would cry. So the manager comes to the usability professionals after determining it might have something to do with usability - usually because it wasn't covered before.

So, as usability professionals, we start investigating. We first look at the data given to us by support - do some supposition and hypothesizing, some reflection, some research. We also use observational techniques, question those suspects that may be violating the usability laws.

We pour some heuristic phenol into the samples and find out there is blood - instant proof that someone has bled from the unforgiving usability issue. Let's just hope the user doesn't have carpel tunnel because of it!

We conduct more tests to prove or disprove our hypotheses. "Let the data tell the story" is our mantra. Through this, we can find the culprit - the issue that's holding us back to release 6.0 from version 5.9.0.3.

The blue UV lights of remote testing tells some more, uncovering and unraveling more than we wanted. The task flow DNA tells us little molecular and granular stories that something is fundamentally wrong.

"So what did you find?" Grissom Neilsen would ask.

An eyebrow goes up in surprise as the team hands him the final paper, the usability finding.

The issue is then resolved by confronting the developer user offender. He/she sweat in their pants for minutes in the interrogation room before he/she finally gives it up. They can't take it anymore!

The offender confesses.

The difference is, no one goes to jail. No one gets reprimanded. Usability on anything can always be improved.

So please, feel free to confess before going through an episode such as this - unless of course you want such entertainment.

Usability Training

I'm writing this at MicorTek right now, a common place where companies can use the facilities to train their people. At this instance, I'm in the course for application design held by Human Factors International. even though I have an extended background on many industries, I felt the need to brush-up on some of the principles. Not only that, since the material is quite heavy, it's a great precursor to knowing the material for the Usability Analyst Certification Exam. It's just another one of those "nice to haves" under your belt when you're a professional.

One thing I've already learned today was to bring this kind of information back to our developers so that they get a perspective as to how involved this process can be. Well, HFI has a course for that too, where in just 6 modules, you can be a certified trainer and teach this material to basically anyone. I'm up for that!

Okay, back to class. I've got a week before I head back.

Let the Interface tell the story

One of the most dreaded things being a user of any software application is the training and learning process one has to go through. The company pays for your courses. But if you're an independent, you have to go out there to learn it yourself.

Being and interface designer isn't any different either, especially if you're looking to redesign any interface. From my experience and my opinion, if the documentation or training course has more than 25 pages, the application is probably too complex to learn quickly.

There are many times where I come across a current design ready for revamping, and the interface tells me very little, or rather a lot - too much in fact because at that point, I've already assessed it to being too complex to learn in 30 minutes or less.

Some factors that prevent quick learning:

1. The documentation is the tutorial. They are usually too long and too complex to understand;
2. The interface was designed poorly. This is usually a case of when usability is still in at its infant stage;
3. The application is not focussed. Usually is the case when a set of target users is too broad or inappropriate.

I could go on, but then like some long documentation, I'd lose your interest.

So to solve all this, why not let the interface tell the story? Have the development processes integrate such a way that enables pre-development work. This also includes focusing your target users and developing mental models - a translation between the human being and software system. So that once actually concepts are being developed, the end in mind is always letting the interface tell the user how they want to be operated upon.

Of course, a lot of past experiences would have to be harnessed. This is what we call "intuitive". (That could also be another debate.) At least in this case, documentation is lessened, the training is made easier and simpler to follow. It would definitely make life a lot easier.

Usability Practitioner's Code of Honor

Every person needs a Code of Honor - that is, rules that govern how a person operates whether in their daily life professionally or not. This is a great way to keep track of how you're operating under certain circumstances. It keeps your behavior "in-check" especially if you're looking to improve or reach some of the goals for the year or lifetime.

I have my Usability Practitioner's Code of Honor. It has also adapted some of the principles from the UPA website I feel need to be addressed or kept in mind as a third-party ruler, thus removing my own opinion about whether something should done in a different way or not.

Here it is in full:

Usability Practitioner’s Personal Code of Honor

• Be on Time – apologize when late;
• Act in the Best Interest of society, your client and employer;
• Be Honest and Kind with everyone;
• Act with Integrity – do what you say. If something cannot be met, arrange for alternative ways for completion;
• Be Responsible to my actions – take ownership;
• Ask for help when I don’t know – I am not suppose to know everything;
• Honor intellectual property rights including copyrights, trademarks, patents, trade secrets, contracts, and licenses;
• Respect the privacy of your colleagues and participants;
• Honor promises of confidentiality, and anonymity;
• Strive to increase your competence every day and empower those you work with;
• Encourage others around me – others will return the favor when it’s time;
• Take initiative – find a way, make a way, no obstacle thinking;
• Anchor and celebrate all wins – especially when we reach milestones.

Imparting the user experience knowledge

Where I'm working now, we've implemented the User Centered Design processes starting a couple of years ago. We're currently a two person team as I have replaced another whom had left just last year. In terms of where we are in UCD maturity, we're on the brink of consistently hitting Stages 4 and 5 especially with some new initiatives. The reality is, we're not at the point of having enough people in our usability team to really expand our activities, yet we also want to make enough of a difference to empower our working colleagues. So how do we do this?

A great way to impart knowledge of what we know and ways to implement the user experience design into current processes is to develop an Intranet. This way, anyone who is in need of our help has access to us, not only in our database of UE knowledge, but also through constant feedback mechanisms.

This is a great way to serve those around you who have little or no exposure to what the UCD/UED discipline really entails. The transition is always constant and always progressive, little by little - not one big gigantic step. People cannot handle that much change all at once.

The key is also to have a plan, a strategy with a timeline of where you want the corporation to be (at what stage) and when.