Faithlife: Business Continuity And Disaster Recovery Plans
Logos Bible Software has moved from a Desktop Installation to a hybrid model - Desktop, Mobile and Cloud. Much of the functionality and long term usability of the Software and Libraries now dependant on Faithlife and their Cloud Infrastructure continuing to operate in spite of adverse scenarios that may occur - both natural and man made. With users having made significant investments, we trust that Faithlife has well defined Business Continuity Plans (BCP) And Disaster Recovery (DR) plans. We trust that Faithlife is acting responsibly in this area. For example, would like to see someone other than Bob reply re-assuringly to this post with the same rigour and detail as Bob 
Comments
-
We were given some insight into the mirror site Logos was building when they had a major failure just weeks before its completion. I personally would rather have the Logos employees maintaining, re-evaluating and planning the next steps for the mirror site rather than reporting to me the details of their plan ... and that it with many years of experience giving the operations side of the house my systems requirements for offsite recovery and every 6 months proving I could be up and running offsite within 24 hours - a requirement of our contract.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Bob has stated in the past he has no plans on abandoning the desktop model where All things feasible are done locally. Visual copy, atlas and I believe some NOW features are so content intensive (it has been estimated that the completed atlas will take up well over 3 terabytes of space) as to never be feasible offline. DR plans are in place with servers in multiple states). We had the rather nasty experience where something like a failure happened while other equipment was being upgraded... There was panic confusion and it took a few days to get everything back online, but FL stated they learned a lot from that. As for a BCP, I very much want a good interactive version of the Book of Common Prayer in Logos too. [:D] But jokes aside Bob has stated stuff regarding that as well we know his brother is heavily involved and when he talked of this he spoke of being in good health and not expecting anything, but things were in place.
There are people with better memories than me, that would be able to offer more details, but I do not think from everything that has been shared in the past and experiences with an equipment disaster of recent memory that everything is well in hand. I don't feel like I have to have all the details to have confidence that Faithlife has decent plans for a number of foreseeable events. I personally am of "Starfeet" protocol where I would love to see a third server system in a third state maybe more eastward. To paraphrase Cheif Obrian, It may not be likely a primary system and its backup goes down at the same time but hate to be without a secondary back up. But what is enough is something only FL can decide and for all I know maybe they have a third in place now. This is a fairly standard practice since the banking industry tends to try to ensure that anyone of the three major centres can effectively handle all the worlds transactions if the other two are down. ( that factoid was from the mid 90s, and banking standards maybe even more robust now a days).
There is nothing eternal beyond God and Love, however like I said I have enough confidence in FL to have no serious concerns.
Dan
0 -
I was recently wondering the same. I have full trust that FL will be around for some while, but would feel much safer having a full library backup from which to restore Logos desktop and mobile without cloud (which would also reduce FL server load and improve re-installation for users with bad internet connection or limited bandwidth).
I'll create a user voice request shortly.
0 -
Jan, I would not support your request for a very specific reason. Yes, as a rule of thumb I don't believe one user should knock down another's idea but I'm making an exception. I run with a mirror drive and an automatic backup of my entire system (I happen to use Carbonite but there are several good products out there). I would NOT want my Logos backups segregated from the rest of my system. It would only increase the complexity of recovery and increase the probability of a non-recoverable error. I do, however, believe that everyone should have a working recovery plan should their system go down - one that works for equipment failure or equipment loss (2 very different scenarios).
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
By the way I am not suggesting the plans be published, but be sure that they are there, tested periodically, and users are assured of them.
0 -
Another area to look into is risk management and controls especially that Faithlife is a family business and are recently diversifying greatly.
0 -
Sure I have my own recovery plan as well. However, when I buy new equipment, I install everything from scratch, and don't restore the previous system (that might have leftovers from Windows 7, that again has leftovers from Windows XP, that again has leftovers from Windows 98).
Logos did sell backup DVDs in the past for good reasons. These disks allowed you to install the full Logos with all resources without cloud. Obviously, the amount of data wouldn't fit on a single disk these days. But the idea behind it is still relevant.
0 -
If the Logos Cloud was down, I could reload from my mirror disk otherwise I have options. my mirror disk simply replaces the Logos disks and has the added benefit of always being up to date. Very little of my software has disks - only small companies that have distribution systems that don't allow for downloading of old versions if that is what I own. Usually I do archive the install executable.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Think 10 or even 20 years in the future. If FL goes bust this year, will Lovos still run from the mirror drive on Windows 14 or Windows 18? Or will it require to be installed from scratch on such a system?
No one knows today. And that's exactly my point for an installable backup.
0 -
One doesn't run from the mirror drive - one reinstalls from it. I suspect that my chances of being able to find settings for reinstalling would be better than from old media that may no longer be readable. Think of my sad lost of Houdini when floppies were no longer supported. [:'(]
But I have no objections for you asking for it and if I were to support it, I would argue from the perspective of the individual in relatively isolated circumstances not disaster recovery. I'm just saying don't expect me to want a disk or to personally want Faithlife spending resources on producing them. But I'm used to not getting my priorities which are skewed towards the liturgical, faith formation and hymnody.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
An interesting product would be a Logos Branded thumbdrive containing every resource you have a liscense for, L6 (or whatever the current version is), and what not. A personal recovery solution should you be in a place where you can't, or its impractical to download 40gb of data (or however much).
I'd buy one. I suspect it could be mostly done with a script (eg - check for available liscenses, if available copy this file to g:\logos\wherever\) and I'd frankly like to have a logos branded thumb drive anyway.
This probably warrants a post in the suggestions forum.L2 lvl4 (...) WORDsearch, all the way through L10,
0 -
A thumbdrive would be good, or even a micro SD. Most users' libraries should fit on those.
Problem with disks is, they would in most cases be outdated the day they arrive in the mail, so building thosewould be a waste of time indeed.
I was more thinking about an in-program option to save an installable backup to pen drive or external disk. That'd be useful to install Logos on two or three machines as well.
Do you remember the license backup in Logos 3? That was a useful tool. If O remember right, the license file and the backup disk were sufficient to get Libronix back running.
We must also not forget mobile backup.
I think user voice is the place for this, not the suggestions forum.
0 -
Dan Francis said:
Bob has stated in the past he has no plans on abandoning the desktop model
The desktop model has already been abandoned, practically speaking.
Try running Logos offline for a few hours to see what I mean. Yes, it'll still run, and yes, most of the features are available.
Most.
Under a 'desktop model' scenario, they'd all work, indefinitely. Now try syncing your devices with your desktop, with a cable instead of the cloud.
But that's probably too much to ask, and a bit unfair. What would be fair, IMO, is some kind of switch available on the local install to run Logos offline long-term, disabling online-only functions so their absence wouldn't interfere with the operation of the rest of the program, and enabling local backup and cable-sync with mobile devices and on intranets. In other words, it could be enabled to run completely cloud-free for as long as necessary.
Bellingham is less than 150 miles from the Cascadia Subduction zone. This alone is a huge threat to the viability of the company.
An event of this magnitude would take out the power grid on the entire west coast. I hope their secondary system is located at least 400 miles east of the coast. Somewhere near Kansas City or San Antonio would be even better.
Eating a steady diet of government cheese, and living in a van down by the river.
0 -
Perhaps Faithlife can do 'Scenario Planning' and publish best practice guidelines to users to handle different scenarios
0 -
Doc B said:
An event of this magnitude would take out the power grid on the entire west coast. I hope their secondary system is located at least 400 miles east of the coast. Somewhere near Kansas City or San Antonio would be even better.
I know that sync devices with a cable is not supported nor is there a way to easily manually transfer notes and highlights from say your desktop machine to your laptop machine. When I said not abandon the desktop model I meant having a desktop app that will run independently of an internet connection. Right now to the best of my knowledge if I disconnected my computer from the internet and never connected again Verbum would never sync, but most all functions, except the 6.0 Atlas and Visual copy (and of course access to scans of books archived at FL) would be available to me. Bob had stated words to the effect that there was no plans to make any drastic changes in 7, he made no guarantees for 8 or even 11, because he has no crystal ball (this is extremely loose paraphrasing). As for Cascade that is a concern but I believe the secondary servers are in Tempe Arizona (I might be wrong but i believe that is what was said). I fully agree with you some place further inland would be good or even someplace in inland Canada...
-Dan
0 -
We're continually working to improve our infrastructure.
We maintain data centers on campus (with things spread around our five buildings in Bellingham), and at a data center along the waterfront (where we're on the fiber backbone, and to which we're connected by fiber), and in a remote data center south of Seattle (about 100 miles away).
We have an office in Tempe, Arizona with more infrastructure, too.
Like every connected company there are risks -- a regional power outage that exceeded a week, or a major earthquake or tsunami, could take even the generator-backed data centers down -- but I think our worst-case scenario is some offline time. Essential data, code, resources, user data, etc. are all backed up, at various levels of safety and accessibility.
This is a cost/benefit trade-off. We aren't spending the huge sums required to ensure we'll rarely go down more than 1 second; we try to be set up so that it's unlikely we'd be off for more than a few hours. In a really bad circumstance we could be down 24 hours, but almost any likely scenario could be fixed/addressed in that time.
It's not impossible something could take us offline for days, but that would probably be on the order of a major, regional catastrophe -- the 'once in a century' earthquake or something; the kind of incident that would prevent UPS / FedEx from delivering for days, and where the people who could get us back online were necessarily more focused on food and water. In other words, we're redundant and backed up, but we aren't maintaining multi-million dollar redundant systems in abandoned missile silos with their own water and air supplies.
Presumably if we're down again for more than 24 hours it's something big enough that you won't be worried about it, because you'll be out loading trucks with supplies to ship to us. Or else we'll have made a big mistake... but we're doing what we can to make that increasingly less likely. Though not impossible, and more likely than the earthquake. :-)
We have more than 400 employees right now, and lots of really smart people on the team. So you're also pretty well insulated against the problem of losing one key person. In fact, if something happened to me, operations might even get more efficient. :-)
-- Bob
0 -
Thanks for the re-assurance!
0 -
These posts also give some information about Faithlife's infrastructure:
This post gives the story on a major outage at the beginning of year. Although lots of things went wrong (some outside Faithlife's control), the postmortem will probably reassure you:
The outage occurred before the Seattle datacentre was deployed, so things are more secure now than they were then.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
I've written something on user voice now:
Please vote if you support the idea!
Jan
0 -
Jan Krohn said:
I've written something on user voice now:
Please vote if you support the idea!
Jan
I added my vote because I have always felt that relying too heavily on the Cloud for backup/reinstallation is worrisome. IMO, a local reinstallation option is always the best.
Instead of Artificial Intelligence, I prefer to continue to rely on Divine Intelligence instructing my Natural Dullness (Ps 32:8, John 16:13a)
0
