Mobile App Quality & QA (FAO Ali Pope)
Hi Ali
I wanted to raise a concern about the quality of the mobile product on Android. I don't wish to be too critical but it does seem that there are an awful lots of bugs creeping into the Android version that ought to be picked up during testing before release. I've retested the issues that you identified as being resolved in the mobile app, and by and large they are resolved, which is great. However I had also identified five new issues in this release, four of which are in areas which have received development attention recently. For reference they are:
Notifications inconsistency - https://community.logos.com/forums/p/212444/1238479.aspx#1238479
Search, save as passage list errors - https://community.logos.com/forums/p/212443/1238478.aspx#1238478
Logos - Add to reading list notification appears in wrong place - https://community.logos.com/forums/p/212442/1238476.aspx#1238476
Notes - Paste & Delete fails - https://community.logos.com/forums/p/212413/1238324.aspx#1238324
Notes - Paste fails - https://community.logos.com/forums/p/212414/1238326.aspx#1238326
I would hope, before release, that the app would undergo some QA including a basic functionality test, which involves checking each screen, menu and function. If this were done before a stable release such as 10.1.2 a number of the issues above would have been identified. We pay quite a lot of money for new features in Logos which also supports the ongoing development. Ultimately the Mobile edition is a great product and a blessings, however I am left with some serious concerns about the quality of QA in this area of Logos' product line. Can you offer any insight or reassurance regarding your processes and application quality?
thanks
Paul
Comments
-
Also since posting this, I have reported another 10 issues on Android on mobile. Again probably 8/10 of these issues should have been identified during a pre-release functionality test.
0 -
Paul M said:
QA including a basic functionality test, which involves checking each screen, menu and function.
I've always wondered about that at FL in general (the source of 'at most 2 users at FL' ... how else could there be such obvious issues). But to give the beta people credit, the software is pretty 'combinatorial' ... meaning in one usage combination, works fine, in another, completely bad.
But I hope you keep on the job; benefits many.
0 -
Thanks DMB. My issue with the product quality is two fold:
1. It seems evident from the breadth of issues across app function and their simplicity to identify (just clicking a menu option in some cases), that a comprehensive, or even partly comprehensive functionality test has not happened. Or if it did, that it wasn't done properly and/or the app was release anyway.
2. These issues are present in release software not beta. One should expect issues with beta's and a higher product quality with release.
Hence my questions to FL.
thanks
Paul0 -
Paul M said:
Ultimately the Mobile edition is a great product and a blessings, however I am left with some serious concerns about the quality of QA in this area of Logos' product line. Can you offer any insight or reassurance regarding your processes and application quality?
Thanks for your questions. We appreciate your attention to detail and feedback.
This period of our release cycle - immediately after a major release - is probably when users will experience most bugs. That's true of any software product made by any company.
But it's also a period when we focus most on fixing bugs, and from what you've said, it seems you're seeing plenty of bugs being fixed over the last couple of months.
So I hope you can see a renewed commitment to us fixing bugs. But I think what you're asking is whether we have an equal commitment to fixing bugs before release.
The answer to that is "yes" and "no".
It's "yes" in that we've recently begun tracking whether we spot bugs before software is released (we call them "captured" or "escaped"). We're tracking this metric precisely because we're not happy with the number of bugs that we didn't find during beta testing, and we want to do better. Our teams will be tracking the percentage of "escaped" bugs, and we'll do what we need to do to drive that figure down.
But the answer is also "no" because we will not be aiming for zero "escaped" bugs. Zero escaped bugs is appropriate at a nuclear power station or onboard the space shuttle. But in software like Logos, we have to balance users' desire for bug-free software with users' desire to use the new features that we're building. Where the new features offer significant value to our customers, we may choose to release, even if there are still a few minor bugs outstanding.
In summary:
- There are too many bugs right now. There are some understandable reasons for that, but we're working hard to squish as many as we can.
- We're working specifically to reduce the number of bugs we find prior after release, and making changes to our processes and testing in order to drive that number down.
- We'll never reach zero bugs, and that's a good thing.
0 -
Thanks Mark, that is helpful to know. Thank you for acknowledging that there is an issue. From my perspective this disproportionally affects the mobile Android Version, for instance I've reported I think 47 issue on this platform since release. Whilst a few bugs have been resolved, if you check my posts on this forum you'll see that I'm reporting many more bugs that are currently being resolved. I do appreciate that this is, for some businesses, the most 'buggy' period. However, this doesn't address the question of why these issues are not being caught during QA. Do the Android QA team conduct a basic functionality test prior to release? Is there a specific issue with QA on Android? If they did, conduct a basic functionality test, they would identify c. 90% of the c.47 issues I've reported to date. This makes me wonder what is happening, hence this post.
At this moment, the product quality isn't there on Android, almost every function has issues including some very serious ones with notes. Matters have got worse with the last release not better. Can you confirm what sort of testing is done before release, on Android, so we know what to expect?
0 -
Paul M said:
From my perspective this disproportionally affects the mobile Android Version,
I suspect this reflects the external beta testers (users) being skewed towards the more popular platforms and/or choosing to concentrate their testing efforts on their primary platforms.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Paul M said:
Whilst a few bugs have been resolved, if you check my posts on this forum you'll see that I'm reporting many more bugs that are currently being resolved.
FWIW, this is also a metric we measure - bugs reported vs. bugs resolved. Our leadership is very focused on ensuring that measure is trending down.
Paul M said:Can you confirm what sort of testing is done before release, on Android, so we know what to expect?
There are multiple levels of testing/review:
- Automated unit tests, whereby automated tests check to see if there have been any regressions when we commit new code.
- Peer review, whereby code will be reviewed by another developer (who didn't write the code) before it's committed.
- Developer-led tests, whereby the developer working on a feature will test their own code to ensure it functions as designed.
- Demos or reviews, whereby the developer demos their changes, and those changes are signed off by a designer or product manager.
- QA, whereby Ali or someone else will test that the new feature or bug fix does what it was supposed to.
- Public or private beta tests, whereby the application is tested by users.
Paul M said:Is there a specific issue with QA on Android? If they did, conduct a basic functionality test, they would identify c. 90% of the c.47 issues I've reported to date.
I don't think there's a specific issue with testing on Android. But, several years ago, Android development wasn't as closely integrated as it is now, and some parts of the code are quite quirky. That can lead to some unforeseen regressions. That is, we might touch some part of the code today, and it has an unexpected impact on an entirely different part of the code on Android. That doesn't happen that often, but it probably accounts for some of the bugs you've reported.
Regarding basic functionality tests, it's easy for us to focus on the features we've added, and perhaps neglect to test related features that may be impacted by changes. I don't think that temptation is specific to Android, nor indeed specific to Faithlife. But it is to say that even the most committed QA tester will never put the app through as much testing as a regular user just using the app normally for an extended period of time.
I have passed your comments (and this thread) to others on the team. We do take criticism seriously, and we do want to improve the experience of all our Android users. Thank you for helping us to do that.
0 -
Thanks Mark, I appreciate all your comments and am reassured. I'm glad that you have some automated reading in your commit/build/release pipeline. I'm glad that there is clearly some significant QA effort being put into the app. However regarding the question of functionality coverage, the existing QA regime isn't working wrt to Android (due to the code base reasons you suggest) and I think you should consider some changes, at least until you reach some stability. A basic run through all the main functions won't take too long (a few hours or so) and would identify 80-90% of the iasues I've reported, which 'escaped'.
Thanks again for a comprehensive answer.
Blessings Paul
0 -
Paul M said:
A basic run through all the main functions won't take too long (a few hours or so) and would identify 80-90% of the iasues I've reported, which 'escaped'.
Unfortunately, the guy who would normally do this is currently on paternity leave. But we'll see if we can find someone else to do it, or maybe Ali can. (Ali's following the thread, by the way. We chatted about it this morning. I'm the one responding because I had some spare capacity today.)
0 -
Mark Barnes (Faithlife) said:
I have passed your comments (and this thread) to others on the team. We do take criticism seriously, and we do want to improve the experience of all our Android users. Thank you for helping us to do that.
I just wanted to add a +1 to these concerns from Paul. I've long used both the iOS and Android apps, and have often had the impression that, while the iOS app is very carefull tested and everything almost always works correctly (and if not, is quickly fixed), the Android app does not seem to receive the same care, and bugs and missing features are more common, and then can take years to fix... if ever.
I'm glad to hear you are trying to take some steps to improve that situation.
0
