Open Question: Quality vs. features

Page 1 of 2 (29 items) 1 2 Next >
This post has 28 Replies | 3 Followers

Posts 2869
Don Awalt | Forum Activity | Posted: Mon, Nov 9 2015 6:12 PM

Let me be clear - I love getting new features every 6 weeks. But I am starting to think quality is suffering. I think I have posted at least 10 bug reports in the last 2 weeks, and I don't ever remember posting that many. They are all little things but annoyances that impact workflow nonetheless. It's not like I am just trying random things to see if they break, I come across them in trying to do work.

There have been several posts with a comment of support being stretched - they been struggling to respond to bug reports online. I have complained about that recently, and I have other bug reports that have still not been responded to. I think this could be a symptom of the same issue.

Is it just me, or how do you see it? Just a statistical anomaly for me, bad luck? I am wondering if some tuning of the model is in order - need for more automated regression testing on such an aggressive schedule, maybe 8 weeks instead of 6, etc.

Posts 36375
Forum MVP
MJ. Smith | Forum Activity | Replied: Mon, Nov 9 2015 6:34 PM

Interesting as it was my observation that the 6 week cycle had improved quality dramatically. I am certainly continuing to find bugs but they are generally long standing rather than directly related to the new features. I've not noticed any change in the number of long standing problems I find nor in the speed with which they get fixed - range from next release to I'm still waiting.

Unfortunately I don't have the time to put into quantifying the error report rate for time ... I suspect that Faithlife has them but doubt they would share.

EDIT: My base expectation prior to the 6 week cycle was that if I tried performing a new function, I would find 5 errors per day.

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 28019
Forum MVP
Dave Hooton | Forum Activity | Replied: Mon, Nov 9 2015 9:12 PM

Don Awalt:
need for more automated regression testing on such an aggressive schedule

Any automated regression testing, which is highly desirable, is a job for Faithlife before the Beta 1 release and during the beta cycle. Extending the cycle from 6 to 8 weeks is not likely to increase the detection of bugs & regressions unless you figure that a certain number of testing hours has to be conducted and it was not achieved. If there is a shortage of FL personnel then capturing user bug reports can fall behind and more bugs will go into the final release.

Overall, the 6 week cycle has been more successful (e.g. stable) than previous beta testing and I've seen a few beta releases devoted solely to bugs. The biggest issues have arisen from Mac beta OS releases and the recent "fix" for Windows Smart Tags in Office 2016. The longer term issues are still coming from Multiple Resources with its effect on (normal) resource panel preferences and Visual filters.

Dave
===

Windows 11 & Android 8

Posts 13428
Mark Barnes | Forum Activity | Replied: Tue, Nov 10 2015 1:42 AM

Don Awalt:
I think I have posted at least 10 bug reports in the last 2 weeks,

Your post intrigued me. I counted six bug reports in the last two weeks.

And three more in the previous week:

Don Awalt:
Is it just me, or how do you see it?

Looking at those bug reports, one is a resource issue (rather than a program bug). Two of them couldn't be reproduced. One was a PB sync error, which is probably not a bug with the app - there have been some sync issues recently. That leaves five, in three weeks.

You probably see slightly more bugs than some because you're using Verbum on a Mac. That will be the least tested combination during the beta stages. Indeed, most of the bugs you reported seem to be Mac issues. (Just for clarity, I'm not suggesting Faithlife treats Macs as second-class citizens, just that because fewer people use them they get less testing from the community during beta.)

Don Awalt:
I am wondering if some tuning of the model is in order - need for more automated regression testing on such an aggressive schedule, maybe 8 weeks instead of 6, etc.

Of the bugs you reported, I suspect most of those bugs have been round a long time - it's just that you've happened to notice them recently. They're not bugs directly related to new features, and some of them are pretty obscure (illegal characters in PB descriptions, for example).

Don Awalt:
they been struggling to respond to bug reports online.

I think that's because of where we are in the release cycle. Faithlife are most proactive about bugs early in the beta release cycle. At this stage in the release cycle (after a stable release, but before a beta release), the testers will be working on alpha builds internally, and preparing those for the beta release. They'll naturally be in the forums less. But when the first beta is released, they'll reduce internal testing, and spend more time monitoring the forums for user reports.

However, to be be fair, apart from your two most recent reports, FL have responded to nearly all your bug reports.

This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!

Posts 2869
Don Awalt | Forum Activity | Replied: Tue, Nov 10 2015 4:19 AM

OK thanks for the perspective Mark Dave and MJ. Funny the only replies so far are MVPs - feels like I walked down the wrong dark alley on this one :-)

I realized the bugs were not directly new functions, my fear is adding new functions quickly had unintended consequences with older functions. I think you are right about the Mac version Mark, it is getting much better but it is not IMHO as good a shape as the Windows version.

And Mark, most of the bug reports have been addressed, but the timeframe for getting an address is longer anecdotally than before. I am ok with it if they all get addressed eventually, just saying. Support is very busy, no harm in acknowledging that.

For my own "to be fair", I think this is going to make the ultimate 7.0 release significantly more stable/higher quality because of all the features that have been used and refined over the prior time periods since 6.0. That will be nice.

Like I started out saying in the OP - I love the new features. I hope it keeps working out.

Posts 2869
Don Awalt | Forum Activity | Replied: Tue, Nov 10 2015 4:24 AM

Mark Barnes:
However, to be be fair, apart from your two most recent reports, FL have responded to nearly all your bug reports.

Actually to be specific Mark, this is incorrect - I think my comment about responsiveness is accurate - of those 9 bugs, 2 had a Faithlife reply of a case being opened or already opened, 2 had a could not reproduce, the other 5 either did not have a response at all or had maybe 1 comment midway through the thread asking a question. 

Posts 30868
Forum MVP
JT (alabama24) | Forum Activity | Replied: Tue, Nov 10 2015 4:26 AM

Don - Your "problem" is that you are a Mac user and you just expect more. You expect things to be designed properly and to "just work." Shame on you. Stick out tongue

macOS, iOS & iPadOS | Logs |  Install

Posts 28019
Forum MVP
Dave Hooton | Forum Activity | Replied: Tue, Nov 10 2015 12:35 PM

alabama24:
Don - Your "problem" is that you are a Mac user and you just expect more.

Of course he does, but Windows sets the standard!

Dave
===

Windows 11 & Android 8

Posts 244
LogosEmployee
Ryan Gano (Faithlife) | Forum Activity | Replied: Mon, Nov 30 2015 1:20 PM

Don,

I'm sorry you feel like you've not been heard recently. Yes, we've been busy lately but that is no excuse to allow a customer to feel ignored.

You are correct that we're busy working on new features but we have about 10-20% of our team working on existing feature cleanup at any one time. Our attempt is to make a big impact in in a single existing area rather than a lot of little impacts in several areas. We'd like to put out a release and (essentially) say, "We added all of these features and we made notes better." Notes could of course be, performance, or clippings, or any other feature area of the app.

I notice that several of the posts listed in this thread have been about personal books. This area had a refresh several months ago and we have not yet gotten it back into the rotation. This may explain why you feel like it's unloved right now.

I will ensure that we take a look through your recent posts and respond where needed. If you have any specific issues you feel we're not addressing please let me know.

Ryan

Posts 819
LogosEmployee
Eli Evans (Faithlife) | Forum Activity | Replied: Tue, Dec 1 2015 1:13 PM

Email me at eli@faithlife.com any time you feel we've missed something. I'll dispatch it right away. 

Our internal metrics mostly agree with MJ that new features are being released with fewer defects than in the past. However, Don also has a point that focusing on new features tends to de-emphasize fixes to longstanding issues. It's all about the trade-offs. We're aware of that and other issues, and we'll keep fine-tuning as we go.

To follow on Ryan's comment, we've also been focusing on fixing crashes recently, which is a two-edged sword. On the one hand, a crash is maximally critical for anyone who experiences it. On the other, the fix goes completely unnoticed to anyone who doesn't.

Posts 1753
Forum MVP
Veli Voipio | Forum Activity | Replied: Wed, Dec 2 2015 6:29 AM

Eli Evans:
On the other, the fix goes completely unnoticed to anyone who doesn't.

It might be good to notice other users some way, I am not sure how. Too many new about bug fixes may cause some unreasonable concern among some users...

Another thing is the updates for the resources. For example today I received an update for "Complete Jewsih Bible" and I assume some people at Logos have done quite a lot of work, it would be nice what has been done, like does it have more links to other Logos resources or so? This kind of info does add to the feeling that FL takes care of the owners of the resources

Gold package, and original language material and ancient text material, SIL and UBS books, discourse Hebrew OT and Greek NT. PC with Windows 11

Posts 12639
Forum MVP
NB.Mick | Forum Activity | Replied: Wed, Dec 2 2015 7:36 AM

Veli Voipio:

Another thing is the updates for the resources. For example today I received an update for "Complete Jewsih Bible" and I assume some people at Logos have done quite a lot of work, it would be nice what has been done, like does it have more links to other Logos resources or so? This kind of info does add to the feeling that FL takes care of the owners of the resources

There's a monthly blog-post about the resource updates since some months, maybe through the whole of 2015. (I remember a thread asking about 'the usual monthly post' in summer, when one didn't appear 'on schedule'). 

Running Logos 9 latest (beta) version on Win 10

Posts 1753
Forum MVP
Veli Voipio | Forum Activity | Replied: Wed, Dec 2 2015 8:19 AM

NB.Mick:
There's a monthly blog-post about the resource updates

Thanks NB, I am getting closer. I found https://blog.logos.com/2015/11/resource-updates-october-2015/ , perhaps it it the latest, but it took some effort to find and is not terribly informative and the info has some lag.

What I wish that by right-clicking I can right away find some exciting info what the Logos employees have done to improve the resource. 

Gold package, and original language material and ancient text material, SIL and UBS books, discourse Hebrew OT and Greek NT. PC with Windows 11

Posts 819
LogosEmployee
Eli Evans (Faithlife) | Forum Activity | Replied: Wed, Dec 2 2015 9:26 AM

Veli Voipio:
What I wish that by right-clicking I can right away find some exciting info what the Logos employees have done to improve the resource. 

We still want to do something like resource-specific release notes in resource information, but we've had bigger fish to fry when it comes to the resource compilers.

Posts 2069
Forum MVP
Reuben Helmuth | Forum Activity | Replied: Thu, Dec 3 2015 9:55 PM

Unfortunately I have to agree with Don in that I feel like too much emphasis is given to new (though I like (many of) the new features) areas. See this thread for an example of what feels like something that is presented half baked (and gets us to say WOW at first glance), before "dropping" it for newer more exciting things.

If the current model is 10%-20% of developers deployed to existing fields, my vote would be to move that to at least 30% if not 35-40. My plea is to focus on performance until we no longer need to hang our heads when company "A" shows off. And as for new tools... it's great having tools that no other company does, but how about also making sure that the reverse isn't true (think searching on Cantillation marks, highlighting Hebrew affixes correctly, etc.)!!

Posts 36375
Forum MVP
MJ. Smith | Forum Activity | Replied: Thu, Dec 3 2015 10:34 PM

Reuben Helmuth:
My plea is to focus on performance until we no longer need to hang our heads when company "A" shows off

A recent post explained that Company "A" converts the text into a database which gives it lightning speed but difficulty in offering a broad range of resources. Company "L" treats the text as unstructured text which gives them great flexibility in adding additional resource types but at the cost of speed. That fundamental difference in approach is not overcome by throwing people at the issue. That's not to say the Logos can't excede its current speed but until there are some significant gains in Natural Language Processing, they aren't going to match company "A".

Reuben Helmuth:
my vote would be to move that to at least 30% if not 35-40.

One needs to know a fair amount about the system, inheritance structure and component divisions before you can say how many people can be on maintenance before they start stepping all over each other and slowing productivity rather than increasing it. I haven't a clue as to the appropriate balance for Logos.

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 30868
Forum MVP
JT (alabama24) | Forum Activity | Replied: Fri, Dec 4 2015 4:32 AM

MJ. Smith:
A recent post explained that Company "A" converts the text into a database which gives it lightning speed but difficulty in offering a broad range of resources.

Blog? Forum? 

macOS, iOS & iPadOS | Logs |  Install

Posts 13428
Mark Barnes | Forum Activity | Replied: Fri, Dec 4 2015 4:39 AM

alabama24:
Blog? Forum? 

https://community.logos.com/forums/p/88436/782343.aspx#782343

Eli Evans:

However, I know they have heard this before, Logos do not overlook the small stuff.  Give us the speed of Accordance in searching!  This may be a small thing but those small things do matter to us. 

It's not a small thing, actually. It's largely a function of the different architectural approaches to electronic texts I've alluded to. Logos books are essentially documents, that is, large, mostly unstructured chunks of text, with indexes for finding locations within them at the time a search is executed. Accordance and Bibleworks modules (last I checked, which was a while ago, so apologies if this isn't the case any more) are essentially databases, that is, small, highly structured chunks of data that are assembled at run time to resemble documents. We think the Logos architecture is more flexible and more closely models non-electronic books. This flexibility and higher-fidelity modeling of a print book is part of what allows us to have such an expansive and diverse library. Other architectures, on the other hand, are more readily optimized for speed.

Flexibility versus speed is one of the basic trade offs in computer science. We went one way, they went another. That said, we never stop optimizing. It matters to us, too! Smile

FWIW, the document model is the more prevalent model (see Kindle, Google) and we're confident it was the right choice.

This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!

Posts 1753
Forum MVP
Veli Voipio | Forum Activity | Replied: Fri, Dec 4 2015 5:37 AM

Some years ago I compared A and L in my home, neither one had SSD. He had a laptop and I had a some years old but good desktop. Both having about the same size of library. When comparing the search capabilities, we had the feeling that the results are about the same.

Currently I have a faster computer with an SSD and everything works fast. For those feeling that Logos is slow, Logos 6 may not be the main problem, could be the computer (or the Internet link?) 

There is some investment to a fast computer, but compare if I had to put 5000 books into my home, I should buy a larger house and that would be much more expensive. If each book is one inch wide that would require something like 40 feet wide and 10 feet high bookshelf, if I calculated correctly. Needs one more room, and how much that would cost in your area? Hmm 

Gold package, and original language material and ancient text material, SIL and UBS books, discourse Hebrew OT and Greek NT. PC with Windows 11

Posts 819
LogosEmployee
Eli Evans (Faithlife) | Forum Activity | Replied: Fri, Dec 4 2015 11:39 AM

Yep, I'm rather proud of the post Mark linked to. Go ahead and read the whole thing

Logos is architected for large, heterogenous libraries. It is a generalist, not a specialist. For any specialized task you can name, a system that is optimized for performing that task will outperform a general system that is more or less task agnostic.

Think: Cheetas versus bears. Is the task running down a gazelle, or surviving many unpredictable habitats? Think: Aquaman versus Batman. Does your go-to solution involve mind-controlled dolphins, or are you more about fighting all the crime on land, sea, and air? Is a smart phone the best camera? No, but it's decent, and you can't check email and play Angry Birds on a camera.

Or better: My favorite Olympic sport is the decathalon. They don't break records in the hundred meter dash the way that dedicated sprinters do, but a sprinter never even goes near a javelin or a long jump. (Okay, I lied. I don't like sports.)

That's just to say: We care a lot about how we perform in the game, but we have to play our game, not someone else's.

One more metaphor ...

Squeezing performance out of a mature software platform like L is a little bit like drilling for oil. First, you suck up the crude that bubbles to the surface. Easy win. Then you start drilling, deeper and deeper. Then you figure out how to drill the ocean floor. (Scary.) All along the way, you can help the drilling problem in part with infrastructure and replaced underlying architecture: By building more fuel-efficient engines and roadways and adjusting the refining process. (Tricky.)

Before you know it, you're talking about Arctic oil sands and fracking. (Scary!) Sure, there may be a lot of oil left, but you can only get it one drop at a time with tremendous effort, and at some point the process starts breaking things that can't be fixed.

When it comes to extracting speed from Logos, we're past the easy wins. We're not all the way to fracking ... yet. We will run up against some infrastructure and deep architecture issues. This is about the time we start thinking maybe we should build an electric car.

That said, it is our intention to set some specific performance goals for Logos 7. I can't tell you what fraction of our people-power we'll spend on it, because we have found that method of management isn't as effective as setting actual targets to aim for. I don't want to say that work expands to fill the time allotted (our team is all-pro), but it's entirely possible to spend a lot of hours squeezing out tiny droplets of general performance that don't amount to anything noticable.

Merry Christmas!

Page 1 of 2 (29 items) 1 2 Next > | RSS