Quality and Update Cycles Beyond Logos 6

Page 1 of 7 (122 items) 1 2 3 4 5 Next > ... Last »
This post has 121 Replies | 9 Followers

Posts 1721
LogosEmployee
Bob Pritchett | Forum Activity | Posted: Tue, Jan 20 2015 4:27 PM

I appreciate all the feedback about quality of both code and data in recent forum threads. You are right, and we are working on it.

We've been through many major release cycles, and every one involves a very similar pattern:

0. Faithlife teams secretly works on new product with previously un-imagined features and data sets we invented. We quietly test ideas on each other and some customers, and come up with a plan and release date. The release date is chosen long in advance and (now that we're big and complicated) becomes set in stone, because hundreds of people, data, licenses, contracts, vacation schedules, baby births, and big-budget expenses are all coordinated against it.

The team works super-hard, has to cut some favored features and content, and sometimes has to choose to ship something incomplete for future delivery (new Atlas maps) or to cut the feature, because the date is set. The team also has to build many features in an environment with limited user-feedback. Sometimes the data set is being shaped by the act of creating it (cultural concepts) -- some of these things are so new we don't know what they'll be like and how they'll work when we're done.  

1. The surprise release on the set date.

2(a). Joy, enthusiasm, excitement over new features (25% of regular users).

2(b). We hear that "I never buy point-zero releases -- they're always buggy" and "it's too soon -- I can't afford an upgrade yet" and others are just not paying attention / are in no hurry. (75% of regular users.)

[update with fixes and improvements ships]

3. Another 25% start upgrading while the first 25% send in 'bug reports'. Many of these are good, useful, and 'real bugs'. Some are clearly our fault (bad coding, rushed release, inattentive editing). Others are good, useful, 'real bugs', and the fault of the latest Microsoft / Apple OS release. (Too often our release coincides with one of theirs. Probably because we all like the same pre-Christmas release dates.) Others are good, useful feedback that isn't really a bug, it's a misunderstanding of scope or intention. This feedback helps us improve descriptions, documentation, or even change a feature or data set.

[update with fixes and improvements ships]

4. Strategic-disagreement ensues. :-)  "Why would you release something so buggy / unfinished / poorly-document / unexplained / not-what-I-expected-at-this-moment." [tiny fix ships] "You should have waited a year till it was right." [tiny fix ships] "I don't want an upgrade every two years, I want an upgrade every three / four / ten years." [tiny fix ships]

5. Bigger update (the 'point-one' release) ships. Much happiness from many users. More users upgrade.

6. GOTO 4 (and repeat for years until next release).

<smile>

Releases follow a cycle. After a major release like Logos 6, the cycle turns from 'create new things, think bold ideas, and ship them!' to 'fix bugs, respond to complaints, address pain points, improve performance, re-tag the book that nobody paid attention to until the new bundle included it or a new feature made it more prominent', etc.

So the good news is, we're in the maintenance phase now. We've had multiple meetings and internal email threads, and performance, bug fixes, interface improvements, documentation, and resource maintenance (re-tagging with new data types, label markup, data sets, etc.) are top priorities.

Logos 6 has been out for less than 90 days, and we've had many service releases and improvements, and Logos 6.1 went into beta today. (https://wiki.logos.com/Logos_6.1_Beta_1)

Everything is getting more complicated. Sometimes a bug is a bug. Sometimes a bug is a financial or logistical constraint that forces a difficult choice.

Examples:

- We implemented a new Atlas for Logos 6. It works differently and offers different value than the previous implementation, which we largely left in place. We planned it, wrote the code, and wrote the marketing copy over many months, while the maps (250 planned) were slowly created in parallel. There was a lot of prep work on the system, the process, and the background maps that are beneath every thematic map. We couldn't make any thematic maps until the background was right, and that took a lot longer than anticipated (lots of reasons), and we realized we wouldn't have all the maps done by the release date. Question: Do we ship what we have, knowing we can ship new maps at a rate of several a week (roughly) until they're all delivered, or do we pull the feature and just not have a new Atlas feature in Logos 6? It would actually take more work to reverse the new Atlas, change the marketing, etc., but we probably would sell just as many upgrades / earn as much revenue without this one feature.... We decided to ship it and deliver more maps after release. (That's ongoing.) Is this a bug, or a feature we're delivering over time that you're glad to have when it's ready? Or should we have stopped the whole process and delayed launch by a few months to let Atlas catch up?

- LCV, Cultural Concepts (and other data sets) involve manually reading books in our system and tagging them with a new ontology of our own design. We rank books by number of users and importance as key reference works, and then we come up with a list of X books to tag in the first release, and then a budget of how much time (=money) to allocate to continuing down the priority list in future years, even though there's no (direct) new revenue to updating those old books. We have 45,000 books; LCV is on dozens, and Cultural Concepts is on about two dozen of four-dozen key books we have identified. (Though it could arguably be useful on even more.) Question: Is it a bug that Cultural Concepts aren't applied to Pliny's Letters yet? Should we have held the data set for a future release? We decided to ship with the books that were tagged, and to establish a budget for ongoing tagging that works down the priority list, and we insert books into that list based on user feedback.

- Publishers send pre-made EPUB books to us for Vyrso. We never touch a physical book, there are often no page numbers in these EPUBs, and many sell 0 or 1 units per year, on which we make as little as one dollar. We can't afford to page number these texts. Question: Should we not sell them? ('Not high enough quality!') We decided to offer them because many people want the content, and we want to let them get that one book they need in a Logos-compatible format. But we label it an 'Ebook Edition' to distinguish it from our higher-quality 'Logos Edition'.

We can't make everyone happy. Somebody wants the newest stuff now, whatever shape it's in. Somebody doesn't care about Pliny's Letters, and finds Cultural Concepts tagging useful even if only applied to the Bible. Someone else doesn't want anything until it's rock solid, bug-free, and comprehensive. Someone else doesn't want a single tag applied to the Bible or a database including Bible references unless every book in every Christian canon in all of church history is included on equal footing. :-)

Our planned solution. 

1. More communication. (It seems like that's the solution to most problems...)

A. Our Content Production team allocates 15% of their effort to maintenance on existing resources. This year it turns out it was more like 12%, with all the work on Logos 6 material. They'll get it back to 15%, and we'll go as high as 20% if it's necessary to keep you (collectively) happy. (Beyond 20%, for just revising things we already shipped and sold, feels like it might be financially difficult for us.) The team is also planning to post more often, and to be clearer about what they're doing, so all their hard word doesn't just silently download in the night.

B. Our Content Innovation department is working on better documentation of our data sets. We plan to add a Library entry for each data set, even if it's an 'invisible' data set that's exposed through features or other resource panels. This will give us a place to provide an Information Pane on the data set, and we'll explain how we created it, who created it, the process, and discuss strengths/weaknesses, as appropriate. We're looking into ways to share 'what's been tagged, and what's next' priority lists, too.

2. Shorter cycles. (This is the new part.)

We're not going to retreat to three or four year release cycles. We believe it's a false conceit that simply taking more time will fix everything. Some quality issues don't show up until thousands of people test thousands of combinations of hardware, network connection, and other software installs. Some quality issues are differences of opinion, and we need to hear the other opinions to know a change needs to be made. 

In a world where the iPhone is updated annually and that feels (to some of us) like not often enough, where Google Chrome silently updates every six weeks, and where Facebook and other web sites change behavior, layout, and functionality every single day, it's just weird for an increasingly-connected product to go three years without any significant changes.

A shorter cycle means new ideas get feedback faster. A shorter cycle means bad ideas don't waste as many resources before being abandoned. And a shorter cycle lets us serve everyone. If you don't like shorter cycles, you can just skip themMake your own 'long cycle' by opting out of the short-cycle offering for one, two, or three years. 

This is a conceptual overview, not a product announcement. 

It's too early to get bogged down with configuration and pricing details. But I hope I can help with examples, possibilities, and reassurances:

- We're considering subscription content. We license Proclaim by subscription, and offer Pro Media by subscription: each month is just gets better, as we release new content every month. Logos Bible Software could benefit from the same model, especially with our online content and 'never ending' data sets. We could make 10 more Interactives (like Psalms Explorer, Feasts and Sacrifices, etc.) and sit on them for two years until they're 'monetized' in a Logos 7 release. We could create 20 more sets of Teaching Slides and Teaching Videos (deSilva) and included them in that big release. But in reality these are built one at a time, over many months. Wouldn't some users rather subscribe and get new content every month rather than have that new content 'sit in a drawer' for two years, helping no one, waiting for the big bundle release?

- This isn't a money-grab. Don't confuse shorter-cycles, or subscription content, with a money-grab. We aren't trying to get the 'every few years' upgrade revenue from you every month. A shorter cycle will involve a smaller release and it will be priced appropriately. A subscription product goes on indefinitely and would be priced appropriately. Shorter cycles can just lead to 'smoothing' the funding, not necessarily increasing it. And this can be helpful to both the customer and the producer.

- We know some people have workflows that don't adapt to frequent change. Some users have spent a lot of time figuring out how to do their study / sermon-prep, etc. They know what books they use, in what layouts, and exactly what keys to hit and buttons to click, and they may even use cheat-sheets from MP Seminars or other guides they've studied carefully. They don't want things to move and mess them up. We understand this, and intend to support the 'don't move anything for a few years!' users.

- There are renters and there are owners. Some people stream all their music and all their movies. Some people lovingly catalog their audio CDs and buy their favorite movies and shows in boxed sets of DVDs. There are advantages and disadvantages to each model. We get it, and we want to support both models.

- Shorter cycles improve transparency. In a short-cycle model, there won't be years of hidden work. Even if you choose to only upgrade / add-to-your-library every couple years, you'll know what we're doing, what features we've been adding and improving, and what content we have been creating. You'll even be able to speak into that process and make suggestions on what to do and what not to do.

- We know that offline access (on both mobile and laptops) is important. While we will certainly create more features that need online access -- for reasons of logistics (often) and physics (sometimes) -- we know that many of you need offline use of your Bible study tools and we will keep it a top priority.

We know that all of our users are interested in a solid, fast, quality product. And we know that most of them are also interested (enough to pay -- which feeds us!) in new tools and content. It's a perpetual balancing act, and one where we're off balance to someone all the time. I believe that shorter cycles and more communication can help address both, leading to a better product that is even more in tune with what you want. (While still delivering the occasional delightful surprise you didn't know you wanted until you saw it!)

What do you think?

-- Bob

Posts 33246
Forum MVP
MJ. Smith | Forum Activity | Replied: Tue, Jan 20 2015 4:41 PM

Sorry Bob but your post doesn't give me anything to complain about. What fun is that? Seriously, the devil is in the details of documentation, subscription features and transparency. Done right it will work. Done wrong and it will just increase the ratio of whines to praise.

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 591
Rayner | Forum Activity | Replied: Tue, Jan 20 2015 4:43 PM

I like the sound of all of it (including shorter release cycles) except for the point about subscriptions.  The only thing I'd be interested in by subscription would be up to date journals, which I'd then want to purchase licences for later (once they were a few years old)...

Perhaps I've misunderstood though... When you say:

Bob Pritchett:
Wouldn't some users rather subscribe and get new content every month rather than have that new content 'sit in a drawer' for two years, helping no one, waiting for the big bundle release?
I would say that it doesn't matter to me that I receive content quickly so much as that I can keep it permanently even if I stop paying the subscription.  I would choose receiving content slowly in order to keep the content permanently over receiving content quickly, but only being able to access it temporarily, simply because I have so much content that I'm not in a huge hurry to read extra resources.  Except journals.  I would really like up to date academic journals.

Also, I think the point about focus on documentation and improving existing resources will make lots of people very happy.

Posts 11433
DMB | Forum Activity | Replied: Tue, Jan 20 2015 4:49 PM

You can certainly do whatever it is you wish to do.  Your company.  Just deliver what you charge for.  And communicate when you don't. It's not complicated.

"God will save his fallen angels and their broken wings He'll mend."

Posts 623
JAL | Forum Activity | Replied: Tue, Jan 20 2015 5:28 PM

Bob Pritchett:
1. More communication. (It seems like that's the solution to most problems...)

More complete documentation is imperative.

  • Thank you for the very open forums in which we users may bark.
  • Thank you for considering the barking and responding substantively.
  • Thank you for producing products which we find useful and can reasonably hope will improve.
  • I trust that you are in a position to weigh what is written in these forums against fact. Thank you for the transparency of your responses.

Bob Pritchett:
2. Shorter cycles. (This is the new part.)

As described here: https://community.logos.com/forums/p/98979/685108.aspx#685108

This seems hopeful.

Bob Pritchett:
We know that all of our users are interested in a solid, fast, quality product. And we know that most of them are also interested (enough to pay -- which feeds us!) in new tools and content. It's a perpetual balancing act...

I've added emphasis to the above quote.

I'm not sure I understand why it's a balancing act to add innovation without breaking the core functions.

"The Christian mind is the prerequisite of Christian thinking. And Christian thinking is the prerequisite of Christian action." - Harry Blamires, 1963

Posts 13419
Mark Barnes | Forum Activity | Replied: Tue, Jan 20 2015 5:36 PM

Bob Pritchett:
A. Our Content Production team allocates 15% of their effort to maintenance on existing resources… The team is also planning to post more often, and to be clearer about what they're doing, so all their hard word doesn't just silently download in the night.

I'd love to see a monthly blog post which said, "Here are the resources we've updated this month", with a brief notes on what's new. That would really help to reinforce some of the extra value we get with Logos resources.

Bob Pritchett:
We're not going to retreat to three or four year release cycles.

I'm glad to hear this (and I'm not surprised). As you say, it's the way the world is going (or has gone).

Bob Pritchett:
Wouldn't some users rather subscribe and get new content every month rather than have that new content 'sit in a drawer' for two years, helping no one, waiting for the big bundle release?

If you're considering a subscription offering and shorter release cycles, please consider implementing three different models:

  1. Rental — pay $50/month for your base package. You can use the resources so long as you keep paying.
  2. Lease — pay $200/month for the first 12 months, then $50/month thereafter. After the first year you get to keep everything you have even if you stop paying. It also rewards Logos for producing new content (because if you don't, we'll stop paying).
  3. Pay-As-You-Go — Pay $2,000 upfront and keep everything. Pay smaller dyamically-priced upgrade fees for new content, whenever you want.

I'd love the lease model. That way, I get the benefit of fixed pricing, but I also keep what I've paid for.

Posts 1327
Sean | Forum Activity | Replied: Tue, Jan 20 2015 5:54 PM

Bob Pritchett:

What do you think?

-- Bob

Regarding subscriptions: when I'm doing an in-depth project, I'd be interested in being able to pay to open up by categories a wide range of resources that I don't own, such as all journals in Logos, all (non-PD) Bible commentaries, etc., for a month or three, then unsubscribing and closing them up when I'm done. That would be about the extent of my interest; I would have no interest in subscribing to get in-house produced content (sorry, but that's the way it is).

Posts 1109
EastTN | Forum Activity | Replied: Tue, Jan 20 2015 5:56 PM

Rayner:

I like the sound of all of it (including shorter release cycles) except for the point about subscriptions.  The only thing I'd be interested in by subscription would be up to date journals...

... I would choose receiving content slowly in order to keep the content permanently over receiving content quickly, but only being able to access it temporarily, simply because I have so much content that I'm not in a huge hurry to read extra resources.  Except journals.  I would really like up to date academic journals.

I agree. The only reasons I see wanting a subscription would be to:

1) Have an equivalent within Logos to EBSCO for theological journals;

2) Access expensive specialist resources for a limited time for a paper I'm working on.

Posts 9619
Forum MVP
Bruce Dunning | Forum Activity | Replied: Tue, Jan 20 2015 7:09 PM

Bob Pritchett:
You are right, and we are working on it.

I always appreciate your openness and your continually effort to improve. Thank you!

Bob Pritchett:
Everything is getting more complicated.

That seems to be the norm these days for many things. It is easy for someone on the "outside" of any issue offer criticism but not really fully understand the extent of the issue. Thank you for reminding us of this.

Bob Pritchett:

Our planned solution. 

1. More communication. (It seems like that's the solution to most problems...)

I agree that communication is at the root of most problems. Thanks for your commitment to improve this.

Bob Pritchett:
2. Shorter cycles...(This is the new part.)

Personally, I like your reasoning for shorter cycles and would prefer this and I like the idea that those who do not like this can opt out.

Bob Pritchett:
We're considering subscription content.

Personally I don't like subscriptions but I know others do and understand the reasoning for this.

Bob Pritchett:
There are renters and there are owners. Some people stream all their music and all their movies. Some people lovingly catalog their audio CDs and buy their favorite movies and shows in boxed sets of DVDs. There are advantages and disadvantages to each model. We get it, and we want to support both models.

I'm glad you recognize this. I'm one of those who prefers to own than rent. Supporting both models is a great way to go.

Bob Pritchett:
we know that many of you need offline use of your Bible study tools and we will keep it a top priority.

Great to hear that this will be a priority moving forward.

Thanks Bob for listening to our feedback. Yes

Using adventure and community to challenge young people to continually say "yes" to God

Posts 4138
abondservant | Forum Activity | Replied: Tue, Jan 20 2015 7:48 PM

Bob - I like what I hear. Especially about not forcing us to rent, and the faster product cycle.

I like new features Yes.

Blessings brother.

ALSO, if you need another Alpha or Private Beta Tester ;) ;) ;)

L2 lvl4 (...) WORDsearch, L9

Posts 826
JRS | Forum Activity | Replied: Tue, Jan 20 2015 8:23 PM

At the risk of being branded a contrarian ...

Perhaps bending over backwards to try and develop an ever increasingly complex product via public transparency and consensus is counterproductive.  Rather than keeping a wet finger in the forum air and trying to maintain a constant dialog with users, maybe Logos needs to 'Nike-up' and just do it without involving the users so much. I know the concept of constant dialog sounds like a great idea (and it probably was a necessity in the early years when the company and the product were smaller and more fragile), nevertheless, it becomes unwieldy much as pure democracy does with a growing population.  You and your managers undoubtedly know the product, the business, the competition, the variables, the limitations, the market, and the possibilities far better and more intimately than any of us could possibly know them, and as a result, you and they are much more qualified  to productively fix, refine, and develop the product than trying to crowd-source and dialog everything.  Sure, user input will always be of value, and similarly, an occasional note from on high as to which way the product is headed is always appreciated, but, perhaps far less of both is needed than you perceive. 

For example, if Logos thinks that a some sort of a subscription model is the way to go, then don't ask, go for it.  BTW, the marketplace will let you know quickly, ruthlessly, and more accurately than a few forum voices if you have a winner, Windows 8, or something that just needs more tweaking.

For example, if Logos thinks a regular patch Tuesday, once per month, like clockwork, instead of having crisis-like, internal fire drills trying to fix this or that whine from the forums and then issuing hurried fixes, then don't ask, go for it. 

For example, if Logos thinks one or two regularly scheduled reveals per year of major new versions instead of publishing a continual string of public betas, then don't ask, go for it. 

Bottom line ... More structure, less Kum-By-Ya.

Just my two cents Money 

JRS has left the building.

Posts 5318
Dan Francis | Forum Activity | Replied: Tue, Jan 20 2015 8:26 PM

abondservant:
ALSO, if you need another beta tester ;) ;) ;)

set update channel to beta Stick out tongue

Posts 4138
abondservant | Forum Activity | Replied: Tue, Jan 20 2015 8:29 PM

JRS:

For example, if Logos thinks that a some sort of a subscription model is the way to go, then don't ask, go for it.  BTW, the marketplace will let you know quickly, ruthlessly, and more accurately than a few forum voices if you have a winner, Windows 8, or something that just needs more tweaking.



That made me laugh. Love the synonymizing of windows 8 and failure :)

L2 lvl4 (...) WORDsearch, L9

Posts 4138
abondservant | Forum Activity | Replied: Tue, Jan 20 2015 8:29 PM

Dan Francis:

abondservant:
ALSO, if you need another Alpha tester ;) ;) ;)

set update channel to beta Stick out tongue



Stick out tongue Better? :)

L2 lvl4 (...) WORDsearch, L9

Posts 11433
DMB | Forum Activity | Replied: Tue, Jan 20 2015 8:54 PM

Yep. Too much kumbaya'ing.  That Logos dating game thing would really have hit the spot.  My spouse was even curious.  Customer input.  Nah.

"God will save his fallen angels and their broken wings He'll mend."

Posts 1506
Wild Eagle | Forum Activity | Replied: Tue, Jan 20 2015 9:05 PM

Eventhough I am enjoying with Logos bells and whistles, I would prefer quality features over the quantity. To my fear, it seems to me that Logos core value is quantity over the quality. It would be better to have less things but rock solid than many with flaws

"No man is greater than his prayer life. The pastor who is not praying is playing; the people who are not praying are straying." Leonard Ravenhill 

Posts 623
JAL | Forum Activity | Replied: Tue, Jan 20 2015 9:33 PM

JRS:
...maybe Logos needs to 'Nike-up' and just do it without involving the users so much. 

Denise:
That Logos dating game thing would really have hit the spot.

"The Christian mind is the prerequisite of Christian thinking. And Christian thinking is the prerequisite of Christian action." - Harry Blamires, 1963

Posts 1359
Edwin Bowden | Forum Activity | Replied: Tue, Jan 20 2015 10:03 PM

Bob,

I always appreciate it when you draw back the curtain and let us see what is going on behind the scenes at Logos. Unfortunately, it usually happens when someone has gone on a rant based on a off-base assumption about the sinister plans or intentions of Logos.

I have had the joy over the last couple of months in helping and guiding a graduate student buy his first Logos software pkg. One of the key points that I made to him was that you can trust Logos. After spending thousands on 2 Platinum base pkgs, he is very happy and excited.

I also post daily on a FB page devoted to helping Logos Users in my denomination. I post free and great values that I find as well as tips and topics.

Logos users are a diverse group in their interests and where they are in their life. Be sure that you remember the variety of situations that Logos users are in. A variety of solutions (like Mark Barnes outlined) may be necessary. I am now retired, which means I have limited finances. I have been teaching Sunday School for over 45 yrs. Logos has been the best thing that ever happened to my class. I don't know if I will still be teaching when my final volumes of EEC are released, but I bought it anyway because of my confidence in Logos.  

Lack of communication has always been one of the weaknesses of Logos. Communication related to the release of L6 was a major improvement over recent releases. I encourage as much communication and transparency as possible.

Shorter cycles seems to be a good idea. As major new parts of the program become available, users should be able to access them in some way. The new features of L6 intrigued me, but I was not able to add a huge amount of resources to my library. I am in the "buy only if I can't live without it" stage. L5 Platinum plus lots of extras (4300 total) makes a good library for me. L6 Gold was the right upgrade move for me.

Posts 4
David Potete | Forum Activity | Replied: Wed, Jan 21 2015 4:11 AM

I love Logos.  But if you go to only subscription, then I'm done.  Maybe I'm old school, but I want to keep what I buy.

Posts 948
Everett Headley | Forum Activity | Replied: Wed, Jan 21 2015 7:53 AM

In looking at your plan above, I am trying to apply it to the progression of the Journal saga.

It would all make sense, in that I see what you are trying do bringing them in house, increasing their quality and offering them in a greater selection.

But, in the meantime, why stop another project for 2 years that was fulfilling this need (to whatever quality/capacity it was) before Logos' Journals are completed? It would seem that your financials would be increased by the revenue generated; your customers would be satisfied with what they already had; and when you rolled out your final project it would polished.

Page 1 of 7 (122 items) 1 2 3 4 5 Next > ... Last » | RSS