Listen Up: User Research on a Shoestring

Last time, I wrote about why you need to do user-centered research for your website.

This post is about how to do it. It doesn’t require a lot of money, but it will require a lot of time. (Sorry, there’s no short cut – it will either cost you time or money.)

Interview: Get a general sense of the condition of your current site by talking informally to a few volunteer leaders, senior staff, and some members.  Ask general questions about their experience of the site. Ask them to perform a few basic transactions or tasks on the site and watch what happens.

Study: If your site logs are any good, study them. Look for trends in visitors and the time they spend on the site. What are the top 20 or 30 content pages? What transactions are being used most often? What are the most common inbound search terms? Are there any patterns to the error messages? This is hard data that will ground you in reality.

Prioritize: Presumably, your senior staff is already on the same page with regard to business priorities. If not, stop NOW and do that first! If so, then you need to prioritize how the site will fulfill those business priorities. For example, if your membership department is targeting young members, how will your site recruit new, younger members? If your business is using education to generate revenue, how will the site deliver knockout online courses?

You’ll end up with between one and three dozen site objectives. Have your staff put each site and business objective into one of three categories: Must have, need to have, and nice to have. Be careful; your team must be ruthless in doing this. Often, they will put everything in the top two categories in an effort to reach consensus. Bad idea. If too many things are a “must have,” your site will (continue to) be a bloody mess.

Force your team to reserve only a handful of things for “must have.” And make them put an equal number of things in the lowest priority. If you call the categories “critical,” “very important,” and “important,” no one will feel invalidated, and your staff may find it easier to do this.

Listen: Go to your members and talk to them about your site. There are dozens of ways to do this. We recruited five focus groups, segmenting them by age.

We limited committee and volunteer leaders to only 25% of the focus group members, because we want the site to be of interest beyond our core leaders. This took some work, and some time. It’s worth it. Don’t go for the usual suspects. If you do, you’ll get the answers you’ve always gotten.

We offered a $200 cash incentive for coming to the session. For medical students and practicing physicians driving for more than one hour, we paid for their mileage. This helped make our groups more diverse.
At the start of the session, they filled out a short questionnaire about how they use the Internet professionally and personally. The questionnaire had nothing to do with our site. Then we talked with each group about our site and how they use it. Some of the answers were quite humbling.

Then we did a paper exercise with them. We created two big sheets of paper with about three dozen boxes representing major content areas of the home page that we guessed they would be interested in. Some were old ideas, some new. We gave them scissors and glue sticks, and had them arrange the boxes in order of priority on another ledger size sheet of paper. The question they were answering was: What would they want on the new site?

Each session lasted about two hours. When it was all over, we compared our staff’s priorities to their priorities. Most times, we had a close match. On others, we were miles apart. Almost always, when there was a difference, we followed what the members said.

Design: Now you’re ready to draw your site map. From that, you can develop your wireframes. There’s both art and science in this. My key advice is, don’t organize your site by departments. No one knows your org chart but you, and no one cares. Organize it by your users’ goals, because that’s how users think when they go online.

Many designers go through a complex persona development exercise. These are valuable, if you know how to do them. However, most of us require outside expertise to develop them, and that requires money. If you have it, it’s worth considering. If not, thinking strictly and rigorously from the vantage point of your users’ goals will get you almost all the way there.

Test: Show your wireframes to at least a dozen members in a one-on-one setting. Ask them to accomplish at least two dozen different tasks, such as renewing their membership; learning about the association’s position on a legislative issue; taking a course; registering for a meeting; finding another member; reading a news release; finding a staff person.

At least 60% of your tasks should be successfully completed on the first click. If another 10% to 20% find it on the second click, it’s not a huge problem; it means you have a little tweaking to do.  If more than 10% to 15% of the tasks are utter failures, step back and take a look at where they failed. There’s probably a pattern in there. Go back and look at your focus group notes; you probably didn’t really listen to what they said.

Stakeholder management: I took the site map, wireframes and the usability test results to my organization’s major stakeholders. I showed them how the data proved that the material they cared about was easily found by our members, despite the new content structure. The results were clear, and no one could argue against success.

If you do this right, you will get three results. Some of your ideas will be validated. Others will be proven dead wrong. And, you will learn something absolutely new about you and your members. That’s really, really cool.

The ability to learn these new and exciting things entirely depends on you having open ears and an open mind.

And that’s doesn’t cost a single penny.

Recommended reading: The User Is Always Right, by Steve Mulder


You Are Not Your Member

Seth Godin had a great post a few months ago about a toaster maker who lost sight of its customers. He described a toaster that requires 10 steps just to make toast. He said, “The opportunity online is to fix your toaster. When you want to make toast, the site should get out of the way and let you make toast.”

The same applies to us. Last year, we knew it was time to update our site. The technology was tired. The visual design was dated. The user experience was not right. How did we know? Random comments like:

“ I can’t find what I want.”
“I just call when I want to find out something.”
“The search box $%&*#.”

I was talking to one of our volunteer leaders one day. It was a casual kind of interview as we were scoping out what was right, and wrong, about the site.

I asked him to find the dates for the next meeting of our House of Delegates. He didn’t go to the calendar; he went to the menu section for House content. I asked him why he didn’t go to the calendar. He said, “There’s never anything there. It looks ugly too.”

As we sometimes say in Massachusetts, light dawns on Marblehead.

So it was time to go back to square one, ask our members how they use the site, and ask what they want from it.

We did not know our members as well as we thought. One big surprise was about clinical content. I had always assumed that with all the good clinical content out there (including from our own medical journal, The New England Journal of Medicine), they didn’t need our help.

Hoo boy, was I wrong!

There is too much content out there. They are drowning in it. Our docs said very clearly that they want our help finding the right content to help them get through their day – fast. Relevant by specialty, interest, whatever; doesn’t matter. Just clear out the junk that clutters their day. (Including our own “junk”!) So we’re working on widgets that will pull targeted content from our own publications, and eventually from trusted outside publications too.

That was helpful. But sometimes research can mislead you. When we asked the focus group people if they used RSS feeds, most of them answered, “What’s that?” So we began to think that RSS shouldn’t be a focus. But in another survey, we asked them if they used iGoogle or MyYahoo. More than half said they use it daily or weekly. So guess what – RSS is back in. As my friend Andy Steggles has said, how you ask the question matters.

If you drink the user-centric Kool-Aid, there will be pushback. Your volunteer leaders may insist that they know their members, and tell you how to build your site. Maybe they know, but probably not. Do the research and show them the data.

Your staff leaders may be worried that their content can’t be found. Maybe, but probably not. Do your research, design the wireframes well, test it with users, so you can demonstrate that your members will find it. Show your fellow staffers the data.

A final caveat. Research should guide, but never determine, our decisions. The art of communications and marketing is knowing when to trust your brain, and when to trust your gut.

Using either alone will really mess you up.