Set up a "Listen in on a power user" program for developers

Often software companies set up in-house usability labs to test out prototypes and observe how newbie users are struggling with their products so that they can improve the design of the features. That would be a great idea for Logos if they're not doing it already, but I digress from my main point:

Sometimes it surprises me to hear Logos devs being amazed at what
we're trying to do with their product, or why we might want
such-and-such a feature (this thread
is a case in point). I'm sure they all know the ins and outs of the
code intimately, but due to their job focus they might not have the time
to be power users of the product themselves.

It would be informative for Logos developers to be able to observe
existing power users and watch how they use the product. Set up a program where
power users could let Logos devs connect remotely to their machines and
watch them use Logos for a while, with a phone line connection
established at the same time so the user could talk the Logos dev
through what they are doing and what they'd
wish they could do that they can't or wish they could do more efficiently.

People
could volunteer, or Logos could hand-pick the people to observe, or
have other users nominate people. For example, I'd want to see Mark
Barnes and some of the other power users who use Logos heavily for their
pastoral work. People who have huge libraries, huge numbers of notes,
etc.

Find more posts tagged with

Comments

Mark Barnes

I'm sure one of the exciting things for developers is finding out people are using their product in imaginative ways that were never envisioned. I'm also sure there'd be no shortage of volunteers for the task Rosie suggests. However, despite its appeal to me, can I suggest an alternative task? Have a Logos engineer sit down (and not intervene to help) whilst a complete newbie turns Logos on for the first time, and works out what on earth they're supposed to do! This could be repeated for other groups of users (people moving from BW, Accordance or L3, people good with computers but not used to Logos, etc.)

Rosie Perera

https://community.logos.com/discussion/comment/223434#Comment_223434

However, despite its appeal to me, can I suggest an alternative task? Have a Logos engineer sit down (and not intervene to help) whilst a complete newbie turns Logos on for the first time, and works out what on earth they're supposed to do!

This was implied by my first paragraph, but I should have emphasized it more. Watching newbies struggle with your software is a real eye-opener. I got a chance to do it for a while through a one-way mirror at Microsoft's usability lab when we were developing Word, and it sent us back to the drawing board on a number of features.

Ron

Great idea Rosie...I hope it gets some traction.

Richard Wardman

https://community.logos.com/discussion/comment/223437#Comment_223437

Watching newbies struggle with your software is a real eye-opener. I got a chance to do it for a while through a one-way mirror at Microsoft's usability lab.

One word. Creepy.

MJ. Smith

https://community.logos.com/discussion/comment/223434#Comment_223434

Have a Logos engineer sit down (and not intervene to help) whilst a complete newbie turns Logos on for the first time, and works out what on earth they're supposed to do!

I think this is the most important usability test. A less expensive way of setting up the power user test would be a simple capture keystroke program installed on computers along with a commentary by the user as to when they felt they were using a "work around". I keep trying to put together instructions on how to build a Bible study unit on Logos but keep running into places that point out the weaknesses in Logos rather than it's strengths.

David Paul

LOL. I'd love to watch someone watch me use L3...for a variety of reasons. Their heads would spin off their shoulders and blow a hole through the roof.

Rosie Perera

https://community.logos.com/discussion/comment/226917#Comment_226917

A less expensive way of setting up the power user test would be a simple capture keystroke program installed on computers along with a commentary by the user as to when they felt they were using a "work around".

Uggh. The output of a simple capture keystroke program would be very difficult to wade through and make much use of or correspond it to the times when the user was making vocal commentary, even with timestamps in the output.. If it's power users we're talking about, why not have them record a screencast using Jing or some other software. That would be much easier to follow and see what is actually causing difficulties, or new ways users are wanting to use the software that were not anticipated by development.

David Ames

https://community.logos.com/discussion/comment/223434#Comment_223434

Have a Logos engineer sit down (and not intervene to help) whilst a complete newbie turns Logos on for the first time, and works out what on earth they're supposed to do!

15 seconds into the very first session: "Mr Logos engineer - Please, where is the manual?"

01 seconds after Mr logos engineer replies - "WHAT! I have to BUY the MANUAL?"

MJ. Smith

https://community.logos.com/discussion/comment/227478#Comment_227478

If it's power users we're talking about, why not have them record a screencast using Jing or some other software

Unfortunately, to the best of my knowledge there are still no great search routines for video which makes it a slower process to glean information. Some visual user testing is a highly desirable supplement but having data that can be searched and mined is a very effective starting point. Yes, there are problems one can't find this way. I've run problems that are caught only through interviews (color blindness effecting color choices). I've run into problems that are identifiable only in the actual workplace (record lock timeouts - think interrupted work of a receptionist). But my thinking is that in initiating program at Logos, it is important to be cost effective which to me translates to KISS (keep it simple, stupid).

Rosie Perera

https://community.logos.com/discussion/comment/227580#Comment_227580

Unfortunately, to the best of my knowledge there are still no great search routines for video which makes it a slower process to glean information. Some visual user testing is a highly desirable supplement but having data that can be searched and mined is a very effective starting point.

Who said it had to be either/or? Searching within videos isn't feasible, but I wasn't talking about making hours and hours of video. Just a few might be quite eye-opening, and it just takes watching them once to see that info jump out at you.

Yes, there are problems one can't find this way. I've run problems that are caught only through interviews (color blindness effecting color choices). I've run into problems that are identifiable only in the actual workplace (record lock timeouts - think interrupted work of a receptionist).

The thing about collecting reams of data is that if you don't know what you're looking for you might not find it. In other words, if you already have a preconceived idea of what you're looking for, you overlook the really obvious, which only shows up when you are casually watching somebody use your software. If you're just wading through all the data, you might not see the forest for the trees.

But my thinking is that in initiating program at Logos, it is important to be cost effective which to me translates to KISS (keep it simple, stupid).

Actually the suggestion of having users make videos with Jing showing how they are actually using the product and what frustrates them about it is essentially free. Making an instrumented version of the software costs development time. (EDIT: Note that over the course of this thread, I've vastly revised my original idea. So don't compare the cost of your suggestion with the cost of setting up an in-house usability lab.)