PinPoint Performance Solutions Blog (Legacy Content)

What may be obvious to some is not so clear to others.  Quality assurance and usability testing are not same.

My current project work has placed me in the role of usability test participant rather than usability test moderator or usability test developer.  Much of this has been by choice because I think it is good for usability professionals to take a step back and place themselves in the shoes of the test participant.  There is much to learn from being a test participant.  The knowledge I've gain from being in this current role has created topics for future blog posts, but today I want to talk about one important thing I learned in regards to test methodology - the form of testing implemented plays a role in the kind of test data that is received.

I'm not sure if the folks I'm working with are using different terminology (where they say "usability testing", but they really mean "quality assurance" or vice versa) or if there is confusion over the definition of the two terms.  Quality assurance and usability testing are two completely different forms of testing and by defintion they produce two totally different forms of test data.

It breaks down to this:

• Quality assurance tests the suitability of a product.  In other words, is it ready for use and free of bugs or errors (OK, minimal bugs and errors).
• Usability testing tests the usefulness or performance of a product as it applies to the end-user.

Please note, when I say "products" it can also mean software applications, web applications or websites.

I'm sorry to say that the test I'm participating in is a quality assurance test, not a usability test, despite what the moderators say.  We're running test scripts and tasks to record results, but there is very little study into what the end-user's emotional response is to the application or overall performance related to basic tasks.  We're simply finding bugs and reporting them to the developers to fix.

If you are in the process of setting up your own usability test, be clear about your objectives and the data you want to record.  Do you want to measure the suitability of your product or its usefulness?  The type of test you implement will produce different forms of data which you can act upon.

A reader took me to task on a recent blog post I made where I ranted about how bugs and workarounds shouldn't be acceptable or "a way of life".

Patrick Neeman at Usability Counts made two valid points in his response to my post that resonated with me:

1. Test early and test often
2. We live in a beta test world

These two things, among other strategies, will help mitigate usability issues. 

The best example, currently, of living in a beta test world is Google Chrome.  Companies like Google make no bones about the condition of their applications as they are developing them and preparing them for a final release.  It's clearly in Beta - it even says so on the download page.  I'm even using a beta version of the blogging platform I'm using to write this very post.  Believe me, it's far from perfect.  However, the strategy is to let the users be the testers and give them the opportunity to take a form of ownership in the application's further development.

There's nothing wrong with this strategy and I often encourage it among the people I work with.  My point is if the business conditions allow it, do upfront due diligence to make sure the application you are putting out there has a minimal number of bugs, errors and usability issues.  Having these problems only hurts end user performance, if you are expecting them to use the application right away.  A buggy application only slows people down and keeps them from the work they intend to do leveraging the application.

I appreciate Patrick's feedback and I appreciate his willingness to let me comment to his post.  Lesson learned for me is to not vent and blog at the same time!

I was happy to see a post by Peter Abrahams at IT-Director.com on the Future of Usability Testing. The reason I was pleased to see this post was because it backed up my assertion from what I feel to be a credible source that usability and user-centered design during the conceptual and development phases of software, website and web applications, should be a holistic process.

By holistic process, I mean to say that the usability of the application should be linked to the business need. The business need can mean two things - one for the developer and the other for the customer.

As an advocate for usability, I find that user-centered design and usability applied throughout the development process lowers costs both before and after the application is released. During concept and development, user-centered design and usability under an agile methodology will lead to shorter time frame for the development phase, thus lowering development costs.

If there's one thing businesses love, it's lowered costs!

For the customer, a usable application means less support. Ease of use will allow them to be productive with the application and spend less time trying to train themselves how to do certain tasks. If an end-user can be self-sufficient in their use of the application, acceptance and productivity increases thus making the technology purchase a worthwhile investment. If they are able to complete their tasks quickly and efficiently with the application this can translate into lowered costs for the customer. The same is true for the developer. Money going toward support resources to handle "how-to" questions can be diverted to resources that tackle tough technical issues.

Since launching this blog I have written about this topic extensively. Here are some past posts on this subject:

• Maintaining User-Centered Design in Agile Software Development
• User-Centered Design and Agile: Why Can't We Be Friends?
• User Centered Design, HCI and Agile Development Compatibility

I think it will take some time for business leaders to have a meeting of the minds with developers on the need for usability, as demonstrated by the panel discussion that Abrahams reported on in his article. When that will happen it is hard to say, but the trend appears to be moving in the right direction.

Not often do I take on the role of participant in a UAT. Most of the time I am either an administrator or a moderator.

I'm deep into a project with a client and I decided it would be fun and informative to see the other side of things in their environment and I volunteered to take part in the testing of an application they are soon to launch. I also chose to participate because it would give me the end-user's perspective when it comes time to develop the training for the application.

The group administering the test is doing a good job, but I noticed a few things that could have made the UAT, overall, produce better results and be more objective.

1. Audience - The team did a good job picking the right people to fill the role of participant. These people, after all, are the target audience for the application. The only trouble was the limited schedules of the participants. The test was to be conducted over the course of a week, however it wasn't until the end of the week did most people begin participating in the test. When conducting a UAT, make sure the test participants have the time to conduct the tasks. You want to have a consistent, steady stream of test results that you can react to as soon as possible or uncover trends that may point to problems that can be resolved. If the issues or bugs discovered are "showstoppers", you can alert your developers so they can prepare to focus their attention to these issues and bugs once the test concludes.
2. Monitoring testing and recording results - As you probably guessed, the test participants were left to conduct testing on their own schedule, unmonitored. I feel most UAT's should be monitored to pick up on difficulties the test participants are experiencing. Most go undocumented by the participants. They may see them as minor inconveniences and not worthy of documenting, but in the long term they can cause usability issues that diminish the overall user experience. Watch the participants as they go through the test and solicit feedback from them. What you see and what you hear may reveal more insightful data.
3. Outline of test objectives and expectations - I think test participants should have some general idea of what is expected of them during the test. They don't need to know the overall test objective. If they do, it could be leading and might skew your test results. However, give them some checkpoints or milestones they need to reach for their tasks. This way, just like what was mentioned earlier, the test results are consistent and you can see trends emerge that you can alert your developers to ahead of time.

In all it was a a good UAT, but the lesson learned is that it is necessary while conducting any kind of testing to make sure you are consistent, objective and working toward specific goals. This way you can identify gaps in performance and establish a course of action to resolve the problems quickly and efficiently.

John Dvorak, columnist for PC Magazine, published an interesting article about Microsoft and the reason as to why he thinks there are many flaws in their software.

I like the title of his article: The Problem with Eating Your Own Dog Food.

I'm not here to say what's right or wrong with Microsoft or add fuel to the fire in the Mac vs. PC debate. I'll leave that to other people to battle.

What I found interesting about Dvorak's column was that it highlighted a point I've been advocating to my clients and the readers of my blog: When testing your products, don't recruit your own people to be your usability testers.

It's almost like preaching to the choir or a parent telling their child they are a talented singer when in reality they are not. Dvorak attributes this to "Eating Your Own Dog Food". If you have ever wondered why a company like Microsoft tends to have so much trouble with their products, Dvorak offers his theories as to why. Most of the problems are cultural and indicative of a large company who has been a dominate player for many years.

I know it may seem natural to have your own people test and use your products. After all, they have a vested interest in its success, don't they?. With success comes the guarantee they'll have a job tomorrow. That may be true, but what about the opposite side of the coin? If a company with a product that has cornered the market for several years and become its bread and butter has employee usability testers detect flaws, would rocking the boat be in the employee's best interest?

Remember, your product testing will be better served if you put it in the hands of actual end-users and work to document their feedback. If you are a company that become established in your field and have become a consumer of your own products, it may be time to stop and ask yourself, "Are we eating our own dog food?"

If you keep up with the latest news and trends in technology, it's hard to avoid anything being reported on the Apple iPhone. Since its release it has captured the hearts and minds of many hard-core Apple users and mobile technophiles.

Recently some news has sprouted about usability and quality issues associated with the Apple iPhone. The device is touted as being very user-friendly and intuitive, but some performance issues leave much to be desired. You can see what one group reported in their own usability study.

A recent article on the web reported how use of the Apple iPhone on the Duke University campus has caused issues with their wireless network. Subsequently, users see poor performance in their web browsing with the device.

This leads me to some important questions:

1. Did Apple test the iPhone in the field to make sure web browsing on the device would meet user expectations?
2. Did they anticipate what the performance would be when more phones are on a given network?
3. If Apple is fully aware of the issue, are they using the device as a tipping point to force the infrastructure it resides on to adapt to them versus them adapting the device to the infrastructure?

It's hard for me to believe that Apple didn't anticipate or test for this condition. However, stranger things have happened. Besides, Apple is in the process of gaining considerable market share in the mobile computing area. Once they do they can quite possibly call the shots, hence the reason for asking question number 3.

The lesson we can learn from this is usability testing and field testing is crucial to the success of any piece of consumer software, website, web application or hardware device. Testing the product under actual conditions an end-user would use it can yield a wealth of information that may not have been considered in the design phases. Once you learn this information you can apply what you learn and make the appropriate fixes to the final release of the product.

What's a technophile to do with a device that's not fully meeting their expectations? Word is a new version of the iPhone is already in the works.

PinPoint Performance Solutions has published a white paper on Search Engine Optimization (SEO) and Usability (PDF, 75K).

Abstract
The design and functionality of a website does not have to suffer just to receive a high ranking on a search engine results page. Search engine optimization, the practice of optimizing the code and content of a website to make it easy for search engines to index and for end-users to find and usability, the practice of making something easy to understand and learn are actually two sides of the same coin. Both practices are a lot more similar than you may think and lend to enhancing the user experience for websites and web-based applications.

On a few occasions (like the one in this post) I have written about how best to train people on new technology once it is implemented in organizations. The subject has come up recently in discussions with potential clients looking for performance support systems and e-learning and it has reminded me of a model of learning theory I learned about years ago that best explains how to present training. It is especially interesting when you look at it through the prism of using multimedia to train end-users on software or web applications.

Edgar Dale was an educator who developed what was called the Cone of Experience Theory. The main idea of the theory is this -

The closer to the actual experience you can get your end-user to while they are learning will increase overall retention.

Think about it. Why do pilots train in flight simulators? Why do city, county, state and federal emergency personnel run disaster preparedness drills? It's a safe environment to learn in and reflect upon because they are doing the actual tasks that are as close as you can get to a real situation with little or no threat of causing critical errors.

It means much more to the learner if they can actually experience the situation they would be facing than to sit in a classroom, read books, listen to a lecture and watch videos or people demonstrating how to use a product. So, don't be afraid to put some prototypes or demos in the hands of your end-users. This is probably one of the easiest, most fundamental forms of usability testing you can do for your application.

More importantly, go back to your support materials and build some simulations. Let your end-users see, touch and feel (i.e. - experience) your applications. You'll be thankful you did because you'll provide your end-users a safe environment to practice in and as an added bonus you'll lower those support calls that are "how-to's", which can become time consuming for you and your business.

Note: Content for this blog post has been drawn from Usability First: Consulting.

Usable systems such as software and web applications should seem intuitive to the end-user, but getting there is not always as intuitive for programmers and developers.

What Do Usability Consultants Do?
Usability consultants work with designers and developers to analyze how an application meets end-user needs so that they can improve upon its concept to enhance productivity, increase overall end-user satisfaction, and minimize end-user errors.

If a usability consultant is brought in before a website or software application is created, they can help developers achieve consistent, usable results. This is the preferred model for development, as usability is built into the application rather than as an afterthought. Usability is not about making a user interface attractive. It encompasses issues such as information architecture, design, writing, and quality assurance. Usability goes way beyond the surface elements.

If usability is brought in after production, it can evaluate the usefulness of a website or software application by using one or more usability evaluation methods such as focus groups, job / task analysis and end-user testing.

Usability consultants usually offer such services as needs analysis, heuristic evaluation (often conducted through expert critique or usability testing), user interface design, and usability training.

What Are The Benefits Of Hiring A Usability Consultant?

• Increasing the usability of a website or software application results in increased productivity for the end-user.
• Bringing a usability consultant into the design and production process will reduce overall costs as it is more cost effective to build a usable application than it is to repair an unusable one.
• Doing a less than thorough job can cost more than money. It can cost time and customers.
• Usability consultants can evaluate systems more efficiently because they know what to look out for based on their experience with a wide range of systems.
• They can bring perspectives that those close to a project may not have considered. It is often difficult to be objective about your own work when you are closely tied to it.
• Consultants are trained in a range of usability evaluation methods. The advantage comes with knowing which method is best for each individual project.
• Having a usable application will result in satisfied, productive users with lowered support costs.

Some of you are aware that in addition to my work with PinPoint Performance Solutions I also run a business networking organization in the Boise area called Treasure Valley Consultants' Network that focuses on promoting the knowledge and expertise of area consultants. We held a meeting last night, which you can read more about at the TVCNet Blog, on the topic of search engine optimization. The discussion took a brief but interesting turn to usability testing of website.

A guest at the meeting mentioned that a usability test doesn't have to be complex. Just get some people together and ask them to go through the website while you observe. With all due respect to my guest, I would have to disagree with the comment to a certain extent. While it's true usability tests do not have to be complex, it really depends on the nature of the website, web application or software. By that I mean its intended end-user and the tasks it was designed to help the end-user accomplish.

If we're talking about website, for example, that is developed for information purposes then it's perfectly fine to recruit people that are easily accessible to you to be your test participants. They may or may not be the intended audience for your website, but if you are out to learn just basic information - what works with the website and what doesn't work - then you should be OK. On the other hand, if the website is linked to important business processes, functions, or perhaps safety, then you need to recruit actual end-users who are likely to be using the website.

Why is it important to find actual end-users if the product is linked to something vital? Actual end-users have a vested interest over those who are just casual observers. They are closely tied to the situation the product is intended for and have history and intrinsic knowledge as to what will work and what won't. Consider this: if you designed a new snow shovel, would you give it to a guy in Florida to test out?

I find that some developers take a path of least resistance when it comes to usability testing. They choose not to conduct them because they are thought of as too costly, too time consuming or yield little to no results. Perhaps they don't know what they are looking for during a usability test? It's a question I've wondered about for quite some time.

A challenge I face in my business is demonstrating the overall value of a usability test. The decision as to whether or not to conduct a usability test rests in what I feel is the critical nature of the product. If it is critical then I would recommend conducting a formalized test with actual end-users. If it is not, then it is perfectly fine to find people easily accessible to you, actual end-users or not.

Either way, conduct a usability test to measure the value and effectiveness of your application, be it website, web app or software. Just make sure your test participants have a vested interest in what they are testing. In the end you'll get better results.

When businesses invest a lot of time and money on usability testing and analysis for their website, only later to find it's not generating the traffic or accomplishing the goals they were expecting, they often turn to search engine optimization (SEO) techniques to boost their ranking in search engine results pages. Unfortunately, some of the techniques learned fall into the black-hat category and end up doing more harm than good.

SEO is more than building reciprocal links, keyword choice and quality content. It has as much to do with those techniques as it does good information architecture, a facet of user-centered design. It also has much to do with a sense of place (end-users understanding where they are in a given website) and the scent of the information (the accuracy of the information queried).

Let's examine this further as it relates to information architecture and a sense of place.

Information architecture is what I like to call the framework, or structure, of a website or web application. It supports a sense of place in the mind of the end-user by providing organization and labeling of content. Your website will be user-centered provided the organization and the labeling match your intended audience's mental model.

One way to provide a sense of place is through breadcrumbs - textual links that demonstrate the strucutre of the content.


Remember, in the end it's all about the user experience - making it easier for your users to seek out information, retrieve that information and be able to use it right away. When developers of websites and web applications begin with information architecture over visual design, generally accepted practices of SEO should come easy. Logical sitemaps, labeling of content and keyword choice will fall into place.

Searches conducted by end-users on search engines will generally take them directly to the page with the content they are looking for (or as close a match as possible). If your end-users are landing on a page deep within your website, wouldn't it make sense to give them a "you are here" sign before they start looking for the nearest exit?

A post from last week on the topic of selecting users for usability tests briefly touched on the use of personas to help develop usability case studies or test scenarios. In reality, it goes much more beyond that. Personas are used in the beginning stages of software or web application development, during the user-centered design process and carried on through to usability testing. Personas are even used in the development of large scale websites or for products like household appliances.

If you don't know what a persona is, think of it as a detailed description of an end-user. It's much more than demographics such as age, gender or occupation. It's the basis for design and provides the context and rationale an end-user has when using a proposed product all told in story form. Whitney Queensbury, a usability expert, demonstrates the range personas can represent themselves using the following chart:


Notice how it flows from Evocative to Perspective. Essentially, it illustrates what evokes the need for the application or product to the why to the how it can help accomplish goals and tasks for the end-user from their perspective.

I had mentioned in another posting how if you don't know who you are developing for, much of your work will be in vain. Personas give you a firm idea of who your end-users are, better than if you did not use them at all or just relied on demographic data. An article from the Usability Professionals' Association (UPA) provides many points as to how personas are useful in making design decisions that ultimately tie to development. Also, a blog posting I found on this topic makes a pretty strong case for personas, namely when you should use them.

As you can tell, I'm pretty high on using personas in user-centered design and usability testing. While they may seem kind of phony, I've always found the best way to overcome the feeling that they are made up is to use actual people to represent the persona. By the very definition of persona we shouldn't be able to do that, but if we had a name and a face to connect with the profile, it makes it more personal. Imagine that person getting upset and giving you a piece of their mind because you didn't provide them with a needed feature in the application.

If persona is not the word we can use, perhaps profile is better?

Recently our hometown newspaper, The Idaho Statesman, debuted their new website to the public. The site is a radical shift from their previous website which I and many other visitors have become familiar with and have seen many minor, yet progressive changes made to it over the years.

When it comes to websites, change is a good thing. It's one way to keep visitors engaged and interested, however, changes that feel more like a disruption tend to turn off visitors rather than give them a reason to return.

Take a look at the two versions of The Idaho Statesman website - big difference!
















January 14, 2007
















January 16, 2007

In the past couple days I'm sure some visitors had to search around to find the information they were accustomed to finding quickly on the old website. Some have probably voiced their opinions in phone calls or e-mails.

Whenever a change like this is made, it's wise to assemble a panel of end-users that represent your visitors to solicit feedback from. Conduct some user analysis sessions to identify their true needs when it comes to visiting your website and do some usability testing on some mock-ups before a commitment is made to go live. Find out what the user experience is and make your changes based on that. This way, your change is predicated on the needs of the end-user and not done just for the sake of change.

This website is, after all, for the end-users, a means to promote the newspaper, increase subscriptions and do a little advertising online. Still, I wonder if the changes were made to help visitors by keeping them informed and encouraging them to subscribe to the paper or if the changes were in response to a new website launched by a rival paper, The Idaho Business Review. Hard to say.

No, that's not the set up to a bad joke. It's a legitimate question many people ask when developing and administering a usability test.

When I speak with people on this subject, I often refer to an article written by Jakob Nielsen called Usability Testing With 5 Users. It's not necessary to get 20 or 30 users to test an application when 5 will actually do the job. Statistically, 5 users will do and he explains how in his article.

I sometimes feel this is what scares most developers away from conducting objective, valid usability tests. Recruiting 20 to 30 users can be quite an undertaking. Administering a usability test with as many users can be even more difficult if time is of the essence. Instead, they rest their faith in the application and hope that their up front analysis (user research and task analysis) covered what was needed to build a usable application. Often, this is not the case.

In a future blog post I will be expanding on this subject by discussing personas in the development of usability case studies or scenarios. If the application being developed is complex and has many different users who use the same application in many different ways, a usability test following experimental rigor needs to be developed.

The Idaho Business Review had a terrific feature article in this week's edition on how Idaho technology firms are coping with the shortage of software engineers.

For those who don't know, in addition to my company I run a business networking organization in the Boise area called Treasure Valley Consultants' Network. In recent blog posts for this organization, I have been calling for member and non-member consultants to think about their respective businesses and find ways to specialize and focus in on specific industries. After all, clients need to be drawn toward something. If they are presented with an "all things to everyone model" they will have a difficult time finding and validating a need you can fill.

The reason I've been almost evangelical of specialization is because I discovered that this is the only way to survive in our economy. While this epiphany isn't groundbreaking and I'm sure any first-year marketing student can tell you the same, the challenges consultants and technology firms face are no different.

The article essentially reported that with most of the lower-level IT jobs being outsource overseas, technology firms are demanding the engineers that walk through their doorway have specialized skills that will support innovation. The printed edition of the Idaho Business Review included an interview with an executive of local IT firm. He said that the new software engineer must do more than just be good at coding. In order for them to succeed, he or she needs to understand the business drivers behind the technology and not implement technology for technology's sake.

In my own company, I knew that in order for me to survive and remain relevant in this economy I must specialize and focus on a specific industry. That is why I have chosen the IT field, more specifically software and web application development. The growth potential is there as well as the need as indicated in the article (i.e. - the business drivers behind technology). I feel that if I can support the industry with user-centered design and usability testing services, I can be a part of the innovation that is home-grown here in the states while other IT services can be outsourced.

Maintaining user experience and overall usability in software or web applications developed using the agile software development methodology can be a challenge. By it's very nature, agile development doesn't deliver a complete application or a working prototype that can be ran through a usability test evaluating all its features and functions. Applications developed under the agile methodology breaks the total project down into manageable "chunks" focusing on developing the most important features and functions that will have the greatest impact.

To adjust ourselves to this development methodology, user-centered design and usability professionals must adopt a similar process. Rather than looking at the application and its human-computer interaction model holistically we must work closely with developers, researching user needs and testing the usability of the features and functions that are created and delivered at the present moment.

The key is to avoid a cascading effect where the research conducted and the results of a specific test feed into the next. Be centrally focused. Just like agile development where key features and functions that have the greatest impact need to be developed first, test for usability based on just a few objectives that are supposed to be accomplished with the application.

For further insight into the alignment of these two practices Jeff Patton, an agile software developer, was recently interviewed on the subject of user-centered design and agile programming.