Faithlife: Business Continuity And Disaster Recovery Plans

We were given some insight into the mirror site Logos was building when they had a major failure just weeks before its completion. I personally would rather have the Logos employees maintaining, re-evaluating and planning the next steps for the mirror site rather than reporting to me the details of their plan ... and that it with many years of experience giving the operations side of the house my systems requirements for offsite recovery and every 6 months proving I could be up and running offsite within 24 hours - a requirement of our contract.

Dan Francis

Bob has stated in the past he has no plans on abandoning the desktop model where All things feasible are done locally. Visual copy, atlas and I believe some NOW features are so content intensive (it has been estimated that the completed atlas will take up well over 3 terabytes of space) as to never be feasible offline. DR plans are in place with servers in multiple states). We had the rather nasty experience where something like a failure happened while other equipment was being upgraded... There was panic confusion and it took a few days to get everything back online, but FL stated they learned a lot from that. As for a BCP, I very much want a good interactive version of the Book of Common Prayer in Logos too. [:D] But jokes aside Bob has stated stuff regarding that as well we know his brother is heavily involved and when he talked of this he spoke of being in good health and not expecting anything, but things were in place.

There are people with better memories than me, that would be able to offer more details, but I do not think from everything that has been shared in the past and experiences with an equipment disaster of recent memory that everything is well in hand. I don't feel like I have to have all the details to have confidence that Faithlife has decent plans for a number of foreseeable events. I personally am of "Starfeet" protocol where I would love to see a third server system in a third state maybe more eastward. To paraphrase Cheif Obrian, It may not be likely a primary system and its backup goes down at the same time but hate to be without a secondary back up. But what is enough is something only FL can decide and for all I know maybe they have a third in place now. This is a fairly standard practice since the banking industry tends to try to ensure that anyone of the three major centres can effectively handle all the worlds transactions if the other two are down. ( that factoid was from the mid 90s, and banking standards maybe even more robust now a days).

There is nothing eternal beyond God and Love, however like I said I have enough confidence in FL to have no serious concerns.

Dan

I was recently wondering the same. I have full trust that FL will be around for some while, but would feel much safer having a full library backup from which to restore Logos desktop and mobile without cloud (which would also reduce FL server load and improve re-installation for users with bad internet connection or limited bandwidth).

I'll create a user voice request shortly.

https://community.logos.com/discussion/comment/784481#Comment_784481

Jan, I would not support your request for a very specific reason. Yes, as a rule of thumb I don't believe one user should knock down another's idea but I'm making an exception. I run with a mirror drive and an automatic backup of my entire system (I happen to use Carbonite but there are several good products out there). I would NOT want my Logos backups segregated from the rest of my system. It would only increase the complexity of recovery and increase the probability of a non-recoverable error. I do, however, believe that everyone should have a working recovery plan should their system go down - one that works for equipment failure or equipment loss (2 very different scenarios).

By the way I am not suggesting the plans be published, but be sure that they are there, tested periodically, and users are assured of them.

https://community.logos.com/discussion/comment/784484#Comment_784484

Another area to look into is risk management and controls especially that Faithlife is a family business and are recently diversifying greatly.

https://community.logos.com/discussion/comment/784483#Comment_784483

Sure I have my own recovery plan as well. However, when I buy new equipment, I install everything from scratch, and don't restore the previous system (that might have leftovers from Windows 7, that again has leftovers from Windows XP, that again has leftovers from Windows 98).

Logos did sell backup DVDs in the past for good reasons. These disks allowed you to install the full Logos with all resources without cloud. Obviously, the amount of data wouldn't fit on a single disk these days. But the idea behind it is still relevant.

https://community.logos.com/discussion/comment/784487#Comment_784487

If the Logos Cloud was down, I could reload from my mirror disk otherwise I have options. my mirror disk simply replaces the Logos disks and has the added benefit of always being up to date. Very little of my software has disks - only small companies that have distribution systems that don't allow for downloading of old versions if that is what I own. Usually I do archive the install executable.

https://community.logos.com/discussion/comment/784489#Comment_784489

Think 10 or even 20 years in the future. If FL goes bust this year, will Lovos still run from the mirror drive on Windows 14 or Windows 18? Or will it require to be installed from scratch on such a system?

No one knows today. And that's exactly my point for an installable backup.

https://community.logos.com/discussion/comment/784491#Comment_784491

One doesn't run from the mirror drive - one reinstalls from it. I suspect that my chances of being able to find settings for reinstalling would be better than from old media that may no longer be readable. Think of my sad lost of Houdini when floppies were no longer supported. [:'(]

But I have no objections for you asking for it and if I were to support it, I would argue from the perspective of the individual in relatively isolated circumstances not disaster recovery. I'm just saying don't expect me to want a disk or to personally want Faithlife spending resources on producing them. But I'm used to not getting my priorities which are skewed towards the liturgical, faith formation and hymnody.

abondservant

https://community.logos.com/discussion/comment/784493#Comment_784493

An interesting product would be a Logos Branded thumbdrive containing every resource you have a liscense for, L6 (or whatever the current version is), and what not. A personal recovery solution should you be in a place where you can't, or its impractical to download 40gb of data (or however much).

I'd buy one. I suspect it could be mostly done with a script (eg - check for available liscenses, if available copy this file to g:\logos\wherever\) and I'd frankly like to have a logos branded thumb drive anyway.

This probably warrants a post in the suggestions forum.

https://community.logos.com/discussion/comment/784553#Comment_784553

A thumbdrive would be good, or even a micro SD. Most users' libraries should fit on those.

Problem with disks is, they would in most cases be outdated the day they arrive in the mail, so building thosewould be a waste of time indeed.

I was more thinking about an in-program option to save an installable backup to pen drive or external disk. That'd be useful to install Logos on two or three machines as well.

Do you remember the license backup in Logos 3? That was a useful tool. If O remember right, the license file and the backup disk were sufficient to get Libronix back running.

We must also not forget mobile backup.

I think user voice is the place for this, not the suggestions forum.

Doc B

https://community.logos.com/discussion/comment/784480#Comment_784480

Bob has stated in the past he has no plans on abandoning the desktop model

The desktop model has already been abandoned, practically speaking.

Try running Logos offline for a few hours to see what I mean. Yes, it'll still run, and yes, most of the features are available.

Most.

Under a 'desktop model' scenario, they'd all work, indefinitely. Now try syncing your devices with your desktop, with a cable instead of the cloud.

But that's probably too much to ask, and a bit unfair. What would be fair, IMO, is some kind of switch available on the local install to run Logos offline long-term, disabling online-only functions so their absence wouldn't interfere with the operation of the rest of the program, and enabling local backup and cable-sync with mobile devices and on intranets. In other words, it could be enabled to run completely cloud-free for as long as necessary.

Bellingham is less than 150 miles from the Cascadia Subduction zone. This alone is a huge threat to the viability of the company.

http://discovermagazine.com/2012/extreme-earth/01-big-one-earthquake-could-devastate-pacific-northwest

An event of this magnitude would take out the power grid on the entire west coast. I hope their secondary system is located at least 400 miles east of the coast. Somewhere near Kansas City or San Antonio would be even better.

https://community.logos.com/discussion/comment/784587#Comment_784587

Perhaps Faithlife can do 'Scenario Planning' and publish best practice guidelines to users to handle different scenarios

Dan Francis

An event of this magnitude would take out the power grid on the entire west coast. I hope their secondary system is located at least 400 miles east of the coast. Somewhere near Kansas City or San Antonio would be even better.

I know that sync devices with a cable is not supported nor is there a way to easily manually transfer notes and highlights from say your desktop machine to your laptop machine. When I said not abandon the desktop model I meant having a desktop app that will run independently of an internet connection. Right now to the best of my knowledge if I disconnected my computer from the internet and never connected again Verbum would never sync, but most all functions, except the 6.0 Atlas and Visual copy (and of course access to scans of books archived at FL) would be available to me. Bob had stated words to the effect that there was no plans to make any drastic changes in 7, he made no guarantees for 8 or even 11, because he has no crystal ball (this is extremely loose paraphrasing). As for Cascade that is a concern but I believe the secondary servers are in Tempe Arizona (I might be wrong but i believe that is what was said). I fully agree with you some place further inland would be good or even someplace in inland Canada...

-Dan

Bob Pritchett

We're continually working to improve our infrastructure.

We maintain data centers on campus (with things spread around our five buildings in Bellingham), and at a data center along the waterfront (where we're on the fiber backbone, and to which we're connected by fiber), and in a remote data center south of Seattle (about 100 miles away).

We have an office in Tempe, Arizona with more infrastructure, too.

Like every connected company there are risks -- a regional power outage that exceeded a week, or a major earthquake or tsunami, could take even the generator-backed data centers down -- but I think our worst-case scenario is some offline time. Essential data, code, resources, user data, etc. are all backed up, at various levels of safety and accessibility.

This is a cost/benefit trade-off. We aren't spending the huge sums required to ensure we'll rarely go down more than 1 second; we try to be set up so that it's unlikely we'd be off for more than a few hours. In a really bad circumstance we could be down 24 hours, but almost any likely scenario could be fixed/addressed in that time.

It's not impossible something could take us offline for days, but that would probably be on the order of a major, regional catastrophe -- the 'once in a century' earthquake or something; the kind of incident that would prevent UPS / FedEx from delivering for days, and where the people who could get us back online were necessarily more focused on food and water. In other words, we're redundant and backed up, but we aren't maintaining multi-million dollar redundant systems in abandoned missile silos with their own water and air supplies.

Presumably if we're down again for more than 24 hours it's something big enough that you won't be worried about it, because you'll be out loading trucks with supplies to ship to us. Or else we'll have made a big mistake... but we're doing what we can to make that increasingly less likely. Though not impossible, and more likely than the earthquake. :-)

We have more than 400 employees right now, and lots of really smart people on the team. So you're also pretty well insulated against the problem of losing one key person. In fact, if something happened to me, operations might even get more efficient. :-)

-- Bob

https://community.logos.com/discussion/comment/784617#Comment_784617

Thanks for the re-assurance!

Mark Barnes

https://community.logos.com/discussion/comment/784617#Comment_784617

These posts also give some information about Faithlife's infrastructure:

This post gives the story on a major outage at the beginning of year. Although lots of things went wrong (some outside Faithlife's control), the postmortem will probably reassure you:

https://ops.faithlife.com/?p=6

The outage occurred before the Seattle datacentre was deployed, so things are more secure now than they were then.