Set up a "Listen in on a power user" program for developers
Often software companies set up in-house usability labs to test out prototypes and observe how newbie users are struggling with their products so that they can improve the design of the features. That would be a great idea for Logos if they're not doing it already, but I digress from my main point:
Sometimes it surprises me to hear Logos devs being amazed at what
we're trying to do with their product, or why we might want
such-and-such a feature (this thread
is a case in point). I'm sure they all know the ins and outs of the
code intimately, but due to their job focus they might not have the time
to be power users of the product themselves.
It would be informative for Logos developers to be able to observe
existing power users and watch how they use the product. Set up a program where
power users could let Logos devs connect remotely to their machines and
watch them use Logos for a while, with a phone line connection
established at the same time so the user could talk the Logos dev
through what they are doing and what they'd
wish they could do that they can't or wish they could do more efficiently.
People
could volunteer, or Logos could hand-pick the people to observe, or
have other users nominate people. For example, I'd want to see Mark
Barnes and some of the other power users who use Logos heavily for their
pastoral work. People who have huge libraries, huge numbers of notes,
etc.
Comments
-
I'm sure one of the exciting things for developers is finding out people are using their product in imaginative ways that were never envisioned. I'm also sure there'd be no shortage of volunteers for the task Rosie suggests. However, despite its appeal to me, can I suggest an alternative task? Have a Logos engineer sit down (and not intervene to help) whilst a complete newbie turns Logos on for the first time, and works out what on earth they're supposed to do! This could be repeated for other groups of users (people moving from BW, Accordance or L3, people good with computers but not used to Logos, etc.)
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
However, despite its appeal to me, can I suggest an alternative task? Have a Logos engineer sit down (and not intervene to help) whilst a complete newbie turns Logos on for the first time, and works out what on earth they're supposed to do!
This was implied by my first paragraph, but I should have emphasized it more. Watching newbies struggle with your software is a real eye-opener. I got a chance to do it for a while through a one-way mirror at Microsoft's usability lab when we were developing Word, and it sent us back to the drawing board on a number of features.
0 -
Great idea Rosie...I hope it gets some traction.
0 -
Rosie Perera said:
Watching newbies struggle with your software is a real eye-opener. I got a chance to do it for a while through a one-way mirror at Microsoft's usability lab.
One word. Creepy.
0 -
Mark Barnes said:
Have a Logos engineer sit down (and not intervene to help) whilst a complete newbie turns Logos on for the first time, and works out what on earth they're supposed to do!
I think this is the most important usability test. A less expensive way of setting up the power user test would be a simple capture keystroke program installed on computers along with a commentary by the user as to when they felt they were using a "work around". I keep trying to put together instructions on how to build a Bible study unit on Logos but keep running into places that point out the weaknesses in Logos rather than it's strengths.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
LOL. I'd love to watch someone watch me use L3...for a variety of reasons. Their heads would spin off their shoulders and blow a hole through the roof.
ASUS ProArt x570s Creator, AMD R9 5950x, HyperX 64gb 3600 RAM, ASUS Strix RTX 2080 ti
"The Unbelievable Work...believe it or not." Little children...Biblical prophecy is not Christianity's friend.
0 -
MJ. Smith said:
A less expensive way of setting up the power user test would be a simple capture keystroke program installed on computers along with a commentary by the user as to when they felt they were using a "work around".
Uggh. The output of a simple capture keystroke program would be very difficult to wade through and make much use of or correspond it to the times when the user was making vocal commentary, even with timestamps in the output.. If it's power users we're talking about, why not have them record a screencast using Jing or some other software. That would be much easier to follow and see what is actually causing difficulties, or new ways users are wanting to use the software that were not anticipated by development.
0 -
Mark Barnes said:
Have a Logos engineer sit down (and not intervene to help) whilst a complete newbie turns Logos on for the first time, and works out what on earth they're supposed to do!
15 seconds into the very first session: "Mr Logos engineer - Please, where is the manual?"
01 seconds after Mr logos engineer replies - "WHAT! I have to BUY the MANUAL?"
0 -
Rosie Perera said:
If it's power users we're talking about, why not have them record a screencast using Jing or some other software
Unfortunately, to the best of my knowledge there are still no great search routines for video which makes it a slower process to glean information. Some visual user testing is a highly desirable supplement but having data that can be searched and mined is a very effective starting point. Yes, there are problems one can't find this way. I've run problems that are caught only through interviews (color blindness effecting color choices). I've run into problems that are identifiable only in the actual workplace (record lock timeouts - think interrupted work of a receptionist). But my thinking is that in initiating program at Logos, it is important to be cost effective which to me translates to KISS (keep it simple, stupid).
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
MJ. Smith said:
Unfortunately, to the best of my knowledge there are still no great search routines for video which makes it a slower process to glean information. Some visual user testing is a highly desirable supplement but having data that can be searched and mined is a very effective starting point.
Who said it had to be either/or? Searching within videos isn't feasible, but I wasn't talking about making hours and hours of video. Just a few might be quite eye-opening, and it just takes watching them once to see that info jump out at you.
MJ. Smith said:Yes, there are problems one can't find this way. I've run problems that are caught only through interviews (color blindness effecting color choices). I've run into problems that are identifiable only in the actual workplace (record lock timeouts - think interrupted work of a receptionist).
The thing about collecting reams of data is that if you don't know what you're looking for you might not find it. In other words, if you already have a preconceived idea of what you're looking for, you overlook the really obvious, which only shows up when you are casually watching somebody use your software. If you're just wading through all the data, you might not see the forest for the trees.
MJ. Smith said:But my thinking is that in initiating program at Logos, it is important to be cost effective which to me translates to KISS (keep it simple, stupid).
Actually the suggestion of having users make videos with Jing showing how they are actually using the product and what frustrates them about it is essentially free. Making an instrumented version of the software costs development time. (EDIT: Note that over the course of this thread, I've vastly revised my original idea. So don't compare the cost of your suggestion with the cost of setting up an in-house usability lab.)
0
