Open Question: Quality vs. features

Let me be clear - I love getting new features every 6 weeks. But I am starting to think quality is suffering. I think I have posted at least 10 bug reports in the last 2 weeks, and I don't ever remember posting that many. They are all little things but annoyances that impact workflow nonetheless. It's not like I am just trying random things to see if they break, I come across them in trying to do work.

There have been several posts with a comment of support being stretched - they been struggling to respond to bug reports online. I have complained about that recently, and I have other bug reports that have still not been responded to. I think this could be a symptom of the same issue.

Is it just me, or how do you see it? Just a statistical anomaly for me, bad luck? I am wondering if some tuning of the model is in order - need for more automated regression testing on such an aggressive schedule, maybe 8 weeks instead of 6, etc.

Find more posts tagged with

Comments

MJ. Smith

Interesting as it was my observation that the 6 week cycle had improved quality dramatically. I am certainly continuing to find bugs but they are generally long standing rather than directly related to the new features. I've not noticed any change in the number of long standing problems I find nor in the speed with which they get fixed - range from next release to I'm still waiting.

Unfortunately I don't have the time to put into quantifying the error report rate for time ... I suspect that Faithlife has them but doubt they would share.

EDIT: My base expectation prior to the 6 week cycle was that if I tried performing a new function, I would find 5 errors per day.

Dave Hooton

need for more automated regression testing on such an aggressive schedule

Any automated regression testing, which is highly desirable, is a job for Faithlife before the Beta 1 release and during the beta cycle. Extending the cycle from 6 to 8 weeks is not likely to increase the detection of bugs & regressions unless you figure that a certain number of testing hours has to be conducted and it was not achieved. If there is a shortage of FL personnel then capturing user bug reports can fall behind and more bugs will go into the final release.

Overall, the 6 week cycle has been more successful (e.g. stable) than previous beta testing and I've seen a few beta releases devoted solely to bugs. The biggest issues have arisen from Mac beta OS releases and the recent "fix" for Windows Smart Tags in Office 2016. The longer term issues are still coming from Multiple Resources with its effect on (normal) resource panel preferences and Visual filters.

Mark Barnes

I think I have posted at least 10 bug reports in the last 2 weeks,

Your post intrigued me. I counted six bug reports in the last two weeks.

And three more in the previous week:

Is it just me, or how do you see it?

Looking at those bug reports, one is a resource issue (rather than a program bug). Two of them couldn't be reproduced. One was a PB sync error, which is probably not a bug with the app - there have been some sync issues recently. That leaves five, in three weeks.

You probably see slightly more bugs than some because you're using Verbum on a Mac. That will be the least tested combination during the beta stages. Indeed, most of the bugs you reported seem to be Mac issues. (Just for clarity, I'm not suggesting Faithlife treats Macs as second-class citizens, just that because fewer people use them they get less testing from the community during beta.)

I am wondering if some tuning of the model is in order - need for more automated regression testing on such an aggressive schedule, maybe 8 weeks instead of 6, etc.

Of the bugs you reported, I suspect most of those bugs have been round a long time - it's just that you've happened to notice them recently. They're not bugs directly related to new features, and some of them are pretty obscure (illegal characters in PB descriptions, for example).

they been struggling to respond to bug reports online.

I think that's because of where we are in the release cycle. Faithlife are most proactive about bugs early in the beta release cycle. At this stage in the release cycle (after a stable release, but before a beta release), the testers will be working on alpha builds internally, and preparing those for the beta release. They'll naturally be in the forums less. But when the first beta is released, they'll reduce internal testing, and spend more time monitoring the forums for user reports.

However, to be be fair, apart from your two most recent reports, FL have responded to nearly all your bug reports.

Don Awalt

https://community.logos.com/discussion/comment/778836#Comment_778836

OK thanks for the perspective Mark Dave and MJ. Funny the only replies so far are MVPs - feels like I walked down the wrong dark alley on this one :-)

I realized the bugs were not directly new functions, my fear is adding new functions quickly had unintended consequences with older functions. I think you are right about the Mac version Mark, it is getting much better but it is not IMHO as good a shape as the Windows version.

And Mark, most of the bug reports have been addressed, but the timeframe for getting an address is longer anecdotally than before. I am ok with it if they all get addressed eventually, just saying. Support is very busy, no harm in acknowledging that.

For my own "to be fair", I think this is going to make the ultimate 7.0 release significantly more stable/higher quality because of all the features that have been used and refined over the prior time periods since 6.0. That will be nice.

Like I started out saying in the OP - I love the new features. I hope it keeps working out.

Don Awalt

https://community.logos.com/discussion/comment/778836#Comment_778836

However, to be be fair, apart from your two most recent reports, FL have responded to nearly all your bug reports.

Actually to be specific Mark, this is incorrect - I think my comment about responsiveness is accurate - of those 9 bugs, 2 had a Faithlife reply of a case being opened or already opened, 2 had a could not reproduce, the other 5 either did not have a response at all or had maybe 1 comment midway through the thread asking a question.

JT (alabama24)

https://community.logos.com/discussion/comment/778854#Comment_778854

Don - Your "problem" is that you are a Mac user and you just expect more. You expect things to be designed properly and to "just work." Shame on you. [:P]

Dave Hooton

https://community.logos.com/discussion/comment/778860#Comment_778860

Don - Your "problem" is that you are a Mac user and you just expect more.

Of course he does, but Windows sets the standard!

Ryan Gano (Logos)

Don,

I'm sorry you feel like you've not been heard recently. Yes, we've been busy lately but that is no excuse to allow a customer to feel ignored.

You are correct that we're busy working on new features but we have about 10-20% of our team working on existing feature cleanup at any one time. Our attempt is to make a big impact in in a single existing area rather than a lot of little impacts in several areas. We'd like to put out a release and (essentially) say, "We added all of these features and we made notes better." Notes could of course be, performance, or clippings, or any other feature area of the app.

I notice that several of the posts listed in this thread have been about personal books. This area had a refresh several months ago and we have not yet gotten it back into the rotation. This may explain why you feel like it's unloved right now.

I will ensure that we take a look through your recent posts and respond where needed. If you have any specific issues you feel we're not addressing please let me know.

Ryan

Eli Evans (Logos)

https://community.logos.com/discussion/comment/785092#Comment_785092

Email me at eli@faithlife.com any time you feel we've missed something. I'll dispatch it right away.

Our internal metrics mostly agree with MJ that new features are being released with fewer defects than in the past. However, Don also has a point that focusing on new features tends to de-emphasize fixes to longstanding issues. It's all about the trade-offs. We're aware of that and other issues, and we'll keep fine-tuning as we go.

To follow on Ryan's comment, we've also been focusing on fixing crashes recently, which is a two-edged sword. On the one hand, a crash is maximally critical for anyone who experiences it. On the other, the fix goes completely unnoticed to anyone who doesn't.

Veli Voipio

https://community.logos.com/discussion/comment/785377#Comment_785377

On the other, the fix goes completely unnoticed to anyone who doesn't.

It might be good to notice other users some way, I am not sure how. Too many new about bug fixes may cause some unreasonable concern among some users...

Another thing is the updates for the resources. For example today I received an update for "Complete Jewsih Bible" and I assume some people at Logos have done quite a lot of work, it would be nice what has been done, like does it have more links to other Logos resources or so? This kind of info does add to the feeling that FL takes care of the owners of the resources

NB.Mick

https://community.logos.com/discussion/comment/785500#Comment_785500

Another thing is the updates for the resources. For example today I received an update for "Complete Jewsih Bible" and I assume some people at Logos have done quite a lot of work, it would be nice what has been done, like does it have more links to other Logos resources or so? This kind of info does add to the feeling that FL takes care of the owners of the resources

There's a monthly blog-post about the resource updates since some months, maybe through the whole of 2015. (I remember a thread asking about 'the usual monthly post' in summer, when one didn't appear 'on schedule').

Veli Voipio

https://community.logos.com/discussion/comment/785520#Comment_785520

There's a monthly blog-post about the resource updates

Thanks NB, I am getting closer. I found https://blog.logos.com/2015/11/resource-updates-october-2015/ , perhaps it it the latest, but it took some effort to find and is not terribly informative and the info has some lag.

What I wish that by right-clicking I can right away find some exciting info what the Logos employees have done to improve the resource.

Eli Evans (Logos)

https://community.logos.com/discussion/comment/785529#Comment_785529

What I wish that by right-clicking I can right away find some exciting info what the Logos employees have done to improve the resource.

We still want to do something like resource-specific release notes in resource information, but we've had bigger fish to fry when it comes to the resource compilers.

Reuben Helmuth

https://community.logos.com/discussion/comment/785547#Comment_785547

Unfortunately I have to agree with Don in that I feel like too much emphasis is given to new (though I like (many of) the new features) areas. See this thread for an example of what feels like something that is presented half baked (and gets us to say WOW at first glance), before "dropping" it for newer more exciting things.

If the current model is 10%-20% of developers deployed to existing fields, my vote would be to move that to at least 30% if not 35-40. My plea is to focus on performance until we no longer need to hang our heads when company "A" shows off. And as for new tools... it's great having tools that no other company does, but how about also making sure that the reverse isn't true (think searching on Cantillation marks, highlighting Hebrew affixes correctly, etc.)!!

MJ. Smith

https://community.logos.com/discussion/comment/785950#Comment_785950

My plea is to focus on performance until we no longer need to hang our heads when company "A" shows off

A recent post explained that Company "A" converts the text into a database which gives it lightning speed but difficulty in offering a broad range of resources. Company "L" treats the text as unstructured text which gives them great flexibility in adding additional resource types but at the cost of speed. That fundamental difference in approach is not overcome by throwing people at the issue. That's not to say the Logos can't excede its current speed but until there are some significant gains in Natural Language Processing, they aren't going to match company "A".

my vote would be to move that to at least 30% if not 35-40.

One needs to know a fair amount about the system, inheritance structure and component divisions before you can say how many people can be on maintenance before they start stepping all over each other and slowing productivity rather than increasing it. I haven't a clue as to the appropriate balance for Logos.

JT (alabama24)

https://community.logos.com/discussion/comment/785952#Comment_785952

A recent post explained that Company "A" converts the text into a database which gives it lightning speed but difficulty in offering a broad range of resources.

Blog? Forum?

Mark Barnes

https://community.logos.com/discussion/comment/786004#Comment_786004

Blog? Forum?

https://community.logos.com/forums/p/88436/782343.aspx#782343

Rev. 14:6">However, I know they have heard this before, Logos do not overlook the small stuff. Give us the speed of Accordance in searching! This may be a small thing but those small things do matter to us.

It's not a small thing, actually. It's largely a function of the different architectural approaches to electronic texts I've alluded to. Logos books are essentially documents, that is, large, mostly unstructured chunks of text, with indexes for finding locations within them at the time a search is executed. Accordance and Bibleworks modules (last I checked, which was a while ago, so apologies if this isn't the case any more) are essentially databases, that is, small, highly structured chunks of data that are assembled at run time to resemble documents. We think the Logos architecture is more flexible and more closely models non-electronic books. This flexibility and higher-fidelity modeling of a print book is part of what allows us to have such an expansive and diverse library. Other architectures, on the other hand, are more readily optimized for speed.

Flexibility versus speed is one of the basic trade offs in computer science. We went one way, they went another. That said, we never stop optimizing. It matters to us, too!

FWIW, the document model is the more prevalent model (see Kindle, Google) and we're confident it was the right choice.

Veli Voipio

https://community.logos.com/discussion/comment/786006#Comment_786006

Some years ago I compared A and L in my home, neither one had SSD. He had a laptop and I had a some years old but good desktop. Both having about the same size of library. When comparing the search capabilities, we had the feeling that the results are about the same.

Currently I have a faster computer with an SSD and everything works fast. For those feeling that Logos is slow, Logos 6 may not be the main problem, could be the computer (or the Internet link?)

There is some investment to a fast computer, but compare if I had to put 5000 books into my home, I should buy a larger house and that would be much more expensive. If each book is one inch wide that would require something like 40 feet wide and 10 feet high bookshelf, if I calculated correctly. Needs one more room, and how much that would cost in your area? [^o)]

Eli Evans (Logos)

https://community.logos.com/discussion/comment/786021#Comment_786021

Yep, I'm rather proud of the post Mark linked to. Go ahead and read the whole thing.

Logos is architected for large, heterogenous libraries. It is a generalist, not a specialist. For any specialized task you can name, a system that is optimized for performing that task will outperform a general system that is more or less task agnostic.

Think: Cheetas versus bears. Is the task running down a gazelle, or surviving many unpredictable habitats? Think: Aquaman versus Batman. Does your go-to solution involve mind-controlled dolphins, or are you more about fighting all the crime on land, sea, and air? Is a smart phone the best camera? No, but it's decent, and you can't check email and play Angry Birds on a camera.

Or better: My favorite Olympic sport is the decathalon. They don't break records in the hundred meter dash the way that dedicated sprinters do, but a sprinter never even goes near a javelin or a long jump. (Okay, I lied. I don't like sports.)

That's just to say: We care a lot about how we perform in the game, but we have to play our game, not someone else's.

One more metaphor ...

Squeezing performance out of a mature software platform like L is a little bit like drilling for oil. First, you suck up the crude that bubbles to the surface. Easy win. Then you start drilling, deeper and deeper. Then you figure out how to drill the ocean floor. (Scary.) All along the way, you can help the drilling problem in part with infrastructure and replaced underlying architecture: By building more fuel-efficient engines and roadways and adjusting the refining process. (Tricky.)

Before you know it, you're talking about Arctic oil sands and fracking. (Scary!) Sure, there may be a lot of oil left, but you can only get it one drop at a time with tremendous effort, and at some point the process starts breaking things that can't be fixed.

When it comes to extracting speed from Logos, we're past the easy wins. We're not all the way to fracking ... yet. We will run up against some infrastructure and deep architecture issues. This is about the time we start thinking maybe we should build an electric car.

That said, it is our intention to set some specific performance goals for Logos 7. I can't tell you what fraction of our people-power we'll spend on it, because we have found that method of management isn't as effective as setting actual targets to aim for. I don't want to say that work expands to fill the time allotted (our team is all-pro), but it's entirely possible to spend a lot of hours squeezing out tiny droplets of general performance that don't amount to anything noticable.

Merry Christmas!

Mark Barnes

https://community.logos.com/discussion/comment/786103#Comment_786103

Logos is architected for large, heterogenous libraries

That's the reason I'm so committed to Logos. It's underlying architecture is orders of magnitude more advanced than anything else in the market, and has been since the earliest Libronix days. Logos does things now that other applications still won't be able to do in three years time, and then only if they rewrite their apps virtually from scratch.

R. Mansfield

https://community.logos.com/discussion/comment/786006#Comment_786006

It's not a small thing, actually. It's largely a function of the different architectural approaches to electronic texts I've alluded to. Logos books are essentially documents, that is, large, mostly unstructured chunks of text, with indexes for finding locations within them at the time a search is executed. Accordance and Bibleworks modules (last I checked, which was a while ago, so apologies if this isn't the case any more) are essentially databases, that is, small, highly structured chunks of data that are assembled at run time to resemble documents. We think the Logos architecture is more flexible and more closely models non-electronic books. This flexibility and higher-fidelity modeling of a print book is part of what allows us to have such an expansive and diverse library. Other architectures, on the other hand, are more readily optimized for speed.

Flexibility versus speed is one of the basic trade offs in computer science. We went one way, they went another. That said, we never stop optimizing. It matters to us, too!

Okay, I try very hard not to come on these forums and troll. And, in fact, I said nothing when this statement was first made; I just let it go. But now that it's being repeated, at the very least, I want to say that the characterization being made above as to why Accordance is significantly faster than Logos is simply not factual--at all.

There are reasons as to why Accordance is faster than Logos. I don't feel it is in the spirit of propriety to go in to such in the Logos forums, but if anyone here wants to come to the Accordance forums and ask that question, you have my blessing.

Jay R

https://community.logos.com/discussion/comment/786178#Comment_786178

I originally started with Logos, and switched over to Accordance (in case its relevant, all on a mac). There are a few main reasons I switched, but the primary one was this.

When I opened Logos, because I wanted to check a bible verse, it would start opening, and then I would be like, "argh this is annoyingly slow" then I would start opening accordance, and it would open and I would type the bible verse in accordance, and be reading it before Logos was even open. This happened 4-5 times, and I just decided, this is stilly, and deleted Logos from the menubar.

MJ. Smith

https://community.logos.com/discussion/comment/786494#Comment_786494

If opening speed is what you value most, your choice makes perfect sense - and serves as a perfect example of why different software is the best software for a particular person doing a particular job.

Veli Voipio

https://community.logos.com/discussion/comment/786501#Comment_786501

We are very impatient when waiting the computer to do something but very patient when waiting for the parousia [:|]

BKMitchell

https://community.logos.com/discussion/comment/785950#Comment_785950

think searching on Cantillation marks, highlighting Hebrew affixes correctly, etc.)!!

Those are important issues to think about and features I wish LogosBible Software had.

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/786178#Comment_786178

But now that it's being repeated, at the very least, I want to say that the characterization being made above as to why Accordance is significantly faster than Logos is simply not factual--at all.

See update here: https://community.logos.com/forums/p/88436/786633.aspx#786633

R. Mansfield

https://community.logos.com/discussion/comment/786635#Comment_786635

Yes, I saw. I am subscribed to both threads :-)

Thanks.

Bob Pritchett

https://community.logos.com/discussion/comment/786178#Comment_786178

There are reasons as to why Accordance is faster than Logos.

Rick makes a great point, and reminds me to point out that our choice of .NET as a platform has some performance implications; we're writing code at a higher level of abstraction on a platform that isn't quite as performant as C++ or other languages. (Though our core book display engine is still in C++...)

I don't know for sure that we would make this choice again if we had a chance to change it, but we're heavily invested in .NET at this point, and it does bring some advantages. We're able to share a remarkable amount of code over multiple platforms (using Mono to run on the Mac), and we're able to code many things faster and more reliably, at a pretty reasonable price in raw performance. And .NET is actually improving, with Microsoft investing in true compiling technology and sharing their (often fast, better) implementations via open source in a way that improves Mono.

I think it's awesome that we get to work in an industry where everyone has the same interests: providing tools for more and better Bible study. We all do that in different ways, and offer different solutions, and our competition sometimes encourages each of us to 'step up our game.' In the end I think that makes you -- the users -- better off.