Mobile App Quality & QA (FAO Ali Pope)

Hi Ali

I wanted to raise a concern about the quality of the mobile product on Android. I don't wish to be too critical but it does seem that there are an awful lots of bugs creeping into the Android version that ought to be picked up during testing before release. I've retested the issues that you identified as being resolved in the mobile app, and by and large they are resolved, which is great. However I had also identified five new issues in this release, four of which are in areas which have received development attention recently. For reference they are:

Notifications inconsistency - https://community.logos.com/forums/p/212444/1238479.aspx#1238479

Search, save as passage list errors - https://community.logos.com/forums/p/212443/1238478.aspx#1238478

Logos - Add to reading list notification appears in wrong place - https://community.logos.com/forums/p/212442/1238476.aspx#1238476

Notes - Paste & Delete fails - https://community.logos.com/forums/p/212413/1238324.aspx#1238324

Notes - Paste fails - https://community.logos.com/forums/p/212414/1238326.aspx#1238326

I would hope, before release, that the app would undergo some QA including a basic functionality test, which involves checking each screen, menu and function. If this were done before a stable release such as 10.1.2 a number of the issues above would have been identified. We pay quite a lot of money for new features in Logos which also supports the ongoing development. Ultimately the Mobile edition is a great product and a blessings, however I am left with some serious concerns about the quality of QA in this area of Logos' product line. Can you offer any insight or reassurance regarding your processes and application quality?

thanks

Paul

Find more posts tagged with

Mobile App

Comments

Paul M

Also since posting this, I have reported another 10 issues on Android on mobile. Again probably 8/10 of these issues should have been identified during a pre-release functionality test.

DMB

QA including a basic functionality test, which involves checking each screen, menu and function.

I've always wondered about that at FL in general (the source of 'at most 2 users at FL' ... how else could there be such obvious issues). But to give the beta people credit, the software is pretty 'combinatorial' ... meaning in one usage combination, works fine, in another, completely bad.

But I hope you keep on the job; benefits many.

Paul M

https://community.logos.com/discussion/comment/1238963#Comment_1238963

Thanks DMB. My issue with the product quality is two fold:

1. It seems evident from the breadth of issues across app function and their simplicity to identify (just clicking a menu option in some cases), that a comprehensive, or even partly comprehensive functionality test has not happened. Or if it did, that it wasn't done properly and/or the app was release anyway.

2. These issues are present in release software not beta. One should expect issues with beta's and a higher product quality with release.

Hence my questions to FL.

thanks

Paul

Mark Barnes (Logos)

Ultimately the Mobile edition is a great product and a blessings, however I am left with some serious concerns about the quality of QA in this area of Logos' product line. Can you offer any insight or reassurance regarding your processes and application quality?

Thanks for your questions. We appreciate your attention to detail and feedback.

This period of our release cycle - immediately after a major release - is probably when users will experience most bugs. That's true of any software product made by any company.

But it's also a period when we focus most on fixing bugs, and from what you've said, it seems you're seeing plenty of bugs being fixed over the last couple of months.

So I hope you can see a renewed commitment to us fixing bugs. But I think what you're asking is whether we have an equal commitment to fixing bugs before release.

The answer to that is "yes" and "no".

It's "yes" in that we've recently begun tracking whether we spot bugs before software is released (we call them "captured" or "escaped"). We're tracking this metric precisely because we're not happy with the number of bugs that we didn't find during beta testing, and we want to do better. Our teams will be tracking the percentage of "escaped" bugs, and we'll do what we need to do to drive that figure down.

But the answer is also "no" because we will not be aiming for zero "escaped" bugs. Zero escaped bugs is appropriate at a nuclear power station or onboard the space shuttle. But in software like Logos, we have to balance users' desire for bug-free software with users' desire to use the new features that we're building. Where the new features offer significant value to our customers, we may choose to release, even if there are still a few minor bugs outstanding.

In summary:

There are too many bugs right now. There are some understandable reasons for that, but we're working hard to squish as many as we can.
We're working specifically to reduce the number of bugs we find prior after release, and making changes to our processes and testing in order to drive that number down.
We'll never reach zero bugs, and that's a good thing.

Paul M

https://community.logos.com/discussion/comment/1238989#Comment_1238989

Thanks Mark, that is helpful to know. Thank you for acknowledging that there is an issue. From my perspective this disproportionally affects the mobile Android Version, for instance I've reported I think 47 issue on this platform since release. Whilst a few bugs have been resolved, if you check my posts on this forum you'll see that I'm reporting many more bugs that are currently being resolved. I do appreciate that this is, for some businesses, the most 'buggy' period. However, this doesn't address the question of why these issues are not being caught during QA. Do the Android QA team conduct a basic functionality test prior to release? Is there a specific issue with QA on Android? If they did, conduct a basic functionality test, they would identify c. 90% of the c.47 issues I've reported to date. This makes me wonder what is happening, hence this post.

At this moment, the product quality isn't there on Android, almost every function has issues including some very serious ones with notes. Matters have got worse with the last release not better. Can you confirm what sort of testing is done before release, on Android, so we know what to expect?

MJ. Smith

https://community.logos.com/discussion/comment/1239000#Comment_1239000

From my perspective this disproportionally affects the mobile Android Version,

I suspect this reflects the external beta testers (users) being skewed towards the more popular platforms and/or choosing to concentrate their testing efforts on their primary platforms.

Mark Barnes (Logos)

https://community.logos.com/discussion/comment/1239000#Comment_1239000

Whilst a few bugs have been resolved, if you check my posts on this forum you'll see that I'm reporting many more bugs that are currently being resolved.

FWIW, this is also a metric we measure - bugs reported vs. bugs resolved. Our leadership is very focused on ensuring that measure is trending down.

Can you confirm what sort of testing is done before release, on Android, so we know what to expect?

There are multiple levels of testing/review:

Automated unit tests, whereby automated tests check to see if there have been any regressions when we commit new code.
Peer review, whereby code will be reviewed by another developer (who didn't write the code) before it's committed.
Developer-led tests, whereby the developer working on a feature will test their own code to ensure it functions as designed.
Demos or reviews, whereby the developer demos their changes, and those changes are signed off by a designer or product manager.
QA, whereby Ali or someone else will test that the new feature or bug fix does what it was supposed to.
Public or private beta tests, whereby the application is tested by users.

Is there a specific issue with QA on Android? If they did, conduct a basic functionality test, they would identify c. 90% of the c.47 issues I've reported to date.

I don't think there's a specific issue with testing on Android. But, several years ago, Android development wasn't as closely integrated as it is now, and some parts of the code are quite quirky. That can lead to some unforeseen regressions. That is, we might touch some part of the code today, and it has an unexpected impact on an entirely different part of the code on Android. That doesn't happen that often, but it probably accounts for some of the bugs you've reported.

Regarding basic functionality tests, it's easy for us to focus on the features we've added, and perhaps neglect to test related features that may be impacted by changes. I don't think that temptation is specific to Android, nor indeed specific to Faithlife. But it is to say that even the most committed QA tester will never put the app through as much testing as a regular user just using the app normally for an extended period of time.

I have passed your comments (and this thread) to others on the team. We do take criticism seriously, and we do want to improve the experience of all our Android users. Thank you for helping us to do that.

Paul M

https://community.logos.com/discussion/comment/1239035#Comment_1239035

Thanks Mark, I appreciate all your comments and am reassured. I'm glad that you have some automated reading in your commit/build/release pipeline. I'm glad that there is clearly some significant QA effort being put into the app. However regarding the question of functionality coverage, the existing QA regime isn't working wrt to Android (due to the code base reasons you suggest) and I think you should consider some changes, at least until you reach some stability. A basic run through all the main functions won't take too long (a few hours or so) and would identify 80-90% of the iasues I've reported, which 'escaped'.

Thanks again for a comprehensive answer.

Blessings Paul

Mark Barnes (Logos)

https://community.logos.com/discussion/comment/1239038#Comment_1239038

A basic run through all the main functions won't take too long (a few hours or so) and would identify 80-90% of the iasues I've reported, which 'escaped'.

Unfortunately, the guy who would normally do this is currently on paternity leave. But we'll see if we can find someone else to do it, or maybe Ali can. (Ali's following the thread, by the way. We chatted about it this morning. I'm the one responding because I had some spare capacity today.)

Fr Devin Roza

https://community.logos.com/discussion/comment/1239035#Comment_1239035

I have passed your comments (and this thread) to others on the team. We do take criticism seriously, and we do want to improve the experience of all our Android users. Thank you for helping us to do that.

I just wanted to add a +1 to these concerns from Paul. I've long used both the iOS and Android apps, and have often had the impression that, while the iOS app is very carefull tested and everything almost always works correctly (and if not, is quickly fixed), the Android app does not seem to receive the same care, and bugs and missing features are more common, and then can take years to fix... if ever.

I'm glad to hear you are trying to take some steps to improve that situation.

Paul M

Hey @Mark Barnes (Logos) would it be possible to get an update on where the business is at with respect to QA processes please (in particular with Android)? I don't regularly report the issues I see with Logos, as I have other things to do. A few weeks ago I happened to see a number of issues appearing regularly in Reading Plans, with both Android and iOS. When you look at the issues together and the ease of reproducibility (e.g. I got them every day when using a reading plan) it make me wonder if this particular feature has had any QA at all? Here is the thread for reference:

https://community.logos.com/discussion/comment/1355764#Comment_1355764

In addition to the above, many of the issues I reported around the time of this original post remain open/unresolved and there seems little systematic feedback to users who report issues on the forums. I posted on a few tagging @Jason Stone (Logos) for an update to highlight the issue. I know that providing feedback is time consuming, and as a result would be in favour of a public facing side to your bug tracking system to streamline reporting and feedback of issues. However in the absence of that, if Logos want to improve software quality, it will require effort to report, resolve and respond to users on issues reported in the forum.

So, would you mind giving an update on the status of the QA effort within the business, as we are now 2 years down the line, and it seems like there is still progress needed!

thanks

Paul

Mark Barnes (Logos)

Hi @Paul M.

When I last responded a couple of years ago, I said there was a focus on fixing bugs. That effort has now shifted to preventing bugs from escaping into the wild. That doesn't mean fixing bugs is no longer a priority. It's a recognition that prevention is better than a cure. Developers and QA are held accountable for how many bugs make it to our customers. The whole of the engineering teams pays careful attention to those numbers, right up to the CTO.

To be entirely transparent, long-standing bugs that have only been reported once or twice might not make it to the top of our list of bugs to work on because we naturally try to prioritize fixing bugs that have the biggest impact. That's doubly true if the bugs are in code that we hope to rewrite soon. For example, we've "fixed" many bugs in Factbook recently by completely rewriting those Factbook sections from scratch rather than fixing individual bugs. That said, knowing which bugs have the biggest impact on users is always helpful. What would your answer be?

Lew Worthington

@Mark Barnes (Logos) said: That said, knowing which bugs have the biggest impact on users is always helpful. What would your answer be?

Is this an open question for discussion, or one directed at @Paul M specifically? I have opinions but don't want to share them in the wrong place.

Mark Barnes (Logos)

I was directing it primarily at Paul, but I'm very glad to have other input here, too. I can't guarantee we'll fix bugs that are mentioned because sometimes it can be incredibly difficult to fix a bug. Occasionally the bug isn't even in our code. But it's always helpful to hear about the bugs that are most problematic.

Paul M

https://community.logos.com/discussion/comment/1357626#Comment_1357626

Thanks for the response Mark, from my experience of the mobile apps in recent years I would have to say that the text editor is the area of most concern. There are two reasons for this a) I've frequently had problems with the text editor, as you will see if you look back at my posts b) anything with the potential to lose user data unexpectedly is amongst the highest priority. I've felt that the mobile app (and Android in particular) has lacked proper QA, which is mirrored by the thoughts of others on the forum. My recent problems with Reading plans (https://community.logos.com/discussion/comment/1357664) made me wonder whether these problems remained, hence the reason for resurrecting this thread.