Logos needs a disaster Recovey Plan

Page 2 of 2 (38 items) < Previous 1 2
This post has 37 Replies | 1 Follower

Posts 6
Bill Smith | Forum Activity | Replied: Tue, Nov 25 2014 9:08 AM

Thank you Bob. I am my own worst enemy on my assumptions especially when I get worked up. I mean "no" disrespect. 

Actully we are middle size with less than 10tb of data. We are in the recovering from our investment in infrastructure (3 secure redundant data centers). We made the budgeting decision last year from marketing and development to backend support. We took a heavy toll on funds but are in a good place to move forward with database cleanup (we were at 100tb before optimization). This allowed us to backup fast and effective. Keep up the good work. Bill

Posts 5285
Dan Francis | Forum Activity | Replied: Tue, Nov 25 2014 11:53 AM

Thank you Bob for weighing in once again. Best laid plans of mice and men go astray. I was glad to hear of your plans from your other thread and it is ashamed that had this outage happened in a few weeks your backup centre would have handled it, but it is also good that the backup centre had much of the equipment you needed to get things back up and running. I try to look at this outage as a blessing. It showed Faithlife where significant changes need to be made and for proclaim users perhaps showed them leaving things to the last minute is never a good idea. I am thankful that 90%+ of Logos application functionality works very well without Logos servers. I can live without the following...(https://community.logos.com/forums/p/93180/647603.aspx)

Todd White (Faithlife):

This is a great discussion and a great reminder that not everyone has internet access all the time. We try to strike a balance with features that require internet and those that don't and in some cases the feature will not work offline initially but has the capability of working offline.

Here are some of my thoughts on the features listed by Mark. 

Atlas - The new maps are quickly approaching terabytes of data. It would not be practical (or desired) to download that content to every users machine. However, we do cache the metadata for every map the user owns and the recently viewed zoom levels (we call them tiles) to the users computer. If you go offline passage guide and other features will tell you that you have a map, but you may not be able to load the map if you haven't viewed it recently. We even evaluated a few options of pre-downloading some of the maps so users would have a good out of the box offline experience, but decided to wait and see if that will be needed based on user feedback.

Community Tags - The nature of this data is online and would also be very large to download. We also had the difficulty deciding if we downloaded it how often would people want it downloaded. Some people may not like this feature and would we need to add a way for those people to turn it off completely? We landed on leaving this one as online only.

Visual Copy - This feature takes advantage of online templates that don't have to be stored on the application as well as the technology of Proclaim to render slides. We didn't want to completely integrate Proclaim into the application and didn't want to require users to have Proclaim installed to use it. As we gauge the popularity of this feature all of the decisions may be revisited.

Ask the Author - The nature of this feature is online only. We could possibly queue up your question offline, but the complexity was enough that we didn't even consider offline support.

Groups - Very similar to Ask the Author.

Wikipedia - After you view a Wikipedia article the article should be viewable offline.

Books.logos.com search - You can download and integrate books into Library search for offline use.

Online Manuscripts - This isn't our content so it will only work online.

Logos Media Library - There are a large number of images provided by this library. Storing all of the images, videos, handouts, etc. on your computer in the possibility you might use them someday would be a tough sell.

Online media - This is media on external sites so it makes sense to keep online only.

Home Page Blogs - Home page works offline, but you need to be online to get the blogs. I think this is the right balance.

Overall I think we made reasonable decisions when striking a balance for each feature online and offline; however, we may have missed the mark somewhere. Your feedback on the features that we missed the mark on is invaluable, so please keep the feedback coming.

Faithlife has a plan, Verbum/Logos desktop apps function with close to full power with no internet connection. Bob has apologized and explained things fully again and again. And I will say yet again, this was an inconvenience not a disaster.

-Dan

Posts 790
James Hiddle | Forum Activity | Replied: Tue, Nov 25 2014 12:25 PM

Posts 623
JAL | Forum Activity | Replied: Tue, Nov 25 2014 1:06 PM

Dan Francis:
Best laid plans of mice and men go astray.

A well-turned phrase ... Find more here: https://www.logos.com/product/9902/the-harvard-classics-vol-6-the-poems-and-songs-of-robert-burns

"The Christian mind is the prerequisite of Christian thinking. And Christian thinking is the prerequisite of Christian action." - Harry Blamires, 1963

Posts 28926
Forum MVP
MJ. Smith | Forum Activity | Replied: Tue, Nov 25 2014 1:31 PM

Denise - you missed the obvious lahar danger which, given that is comes from different mountains, is unlikely to be simultaneous.

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 10871
Denise | Forum Activity | Replied: Tue, Nov 25 2014 2:59 PM

True.  But I was trying to be brazenly subtle, in noting the primary disaster threat did not seem to be natural (and rarely is).  And the recovery plan seemed to be maximized, by minimizing the distance between the problem and the solution.  I'd assume re-locating Seatle computer parts stores to Bellingham would be a big plus.  Or visa versa.

"I didn't know God made honky tonk angels."

Posts 167
David Kirk Davis | Forum Activity | Replied: Tue, Nov 25 2014 4:13 PM

Silicon Valley and Seattle are both on major fault lines. A lot more tech companies than Faithlife are in a world of hurt if there ever is a major earthquake. Thankfull, we haven't had a major earthquake since 2001 here in Seattle. 

Posts 28926
Forum MVP
MJ. Smith | Forum Activity | Replied: Tue, Nov 25 2014 4:18 PM

Denise, you're perceptive as usual. They said nothing about sending people north to Vancouver, which I think is closer. Did they let a little thing like a national border stand in the way of shortening the down time? I'm sure they have people with experience smuggling er ah I mean giving ambiguous answers to border patrol to get through faster.

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 2433
mab | Forum Activity | Replied: Tue, Nov 25 2014 6:13 PM

I think the real problem of a disaster recovery plan is that it still relies on certain assumptions which are not considered. The company I worked for had a rule that the heads of the company would not travel together in case of a major accident. Now if they were in a room together, rather happened often, and the ceiling collapsed, the rule's precaution was of no use.

If your computer's CPU goes up in smoke on Friday, what would you do? Everyone needs to plan ahead. (Hint: work ahead of schedule)

The mind of man is the mill of God, not to grind chaff, but wheat. Thomas Manton | Study hard, for the well is deep, and our brains are shallow. Richard Baxter

Posts 18867
Rosie Perera | Forum Activity | Replied: Tue, Nov 25 2014 6:47 PM

Dan Francis:
And I will say yet again, this was an inconvenience not a disaster.

It was "an inconvenience not a disaster" to most users, but if you multiply the inconvenience by the number of users affected, and then that by how long the outage lasted (there are still things that aren't working yet four days after the problem began), then from the company's perspective, it is a disaster. You can bet they are not going to want to repeat this episode ever again. It has been very costly to them: having to buy emergency replacement equipment and fly team members around or send them driving down I-5 to go pick up these replacement parts by hand, having people working round the clock to fix everything, getting a new support contract on short notice, etc. And then they're also going to have to take the time to rethink their backup and restore strategy, probably buy more equipment and hire more people to prevent this from happening again. Yes, it was a disaster. A big learning experience, but an expensive event which I hope they never have to live through again. In company parlance, this is called putting in place a "disaster recovery plan" because these sorts of events are called "disasters" in corporate lingo.

Posts 5285
Dan Francis | Forum Activity | Replied: Tue, Nov 25 2014 7:22 PM

Rosie Perera:
A big learning experience, but an expensive event which I hope they never have to live through again. In company parlance, this is called putting in place a "disaster recovery plan" because these sorts of events are called "disasters" in corporate lingo.

Yes and I am sure Faithlife is well underway in trying to ensure this never happens again. and for FL it was a bad thing... I am sorry for everyone who was inconvenienced. But I still feel all users have tended to over react. A worse case scenario seems to be a Worship service was disrupted. This is unfortunate but I personally do not think anyone should consider that horrible. Technology can help us worship but should never be needed to worship. I never meant to say it was not a big deal for Faithlife, just that for us affected it was not as bad a scenario as people made it sound. This thread in particular seemed pointless (I am hoping the person starting this simply missed Bob's explanation in which he said they had already started prep for such a disaster when murphy's law caught up before they were done). I do not feel I am owed anything for this disaster, perhaps proclaim users are owed something for their inconvenience but that is debatable of course. The Faithlife employees are doing everything possible. I want software that works 100% too, but i fail to see how our complaints help any. After all it is not about me or you or church a or b, it's about Christ, and I do not see how this can help. In some ways much of it feels like james and john wanting to call down hell fire. It is not Jesus' way.

-Dan 

Posts 2574
Lee | Forum Activity | Replied: Tue, Nov 25 2014 7:38 PM

With respect, Dan, you're the one missing the point of the original post. It's from a Security Officer, wanting to offer some advice to Logos. Bob P. has humbly accepted it. You're just another user, and you're missing the entire corporate context that they are dialoging about, with nothing to add other than that you suffered nothing, feel nothing and it's not a disaster (missing the "disaster" point entirely).

Dan Francis:

In some ways much of it feels like james and john wanting to call down hell fire. It is not Jesus' way.

-Dan 

I would love for you to quote people and offer relevant, wise rebuke to the people who said too much or out of turn, rather than impose a lofty Jesus-would-not-approve-it kind of blanket statement. I think, in fact, that there is a lot of misinterpretation going on in your statement.

Posts 18867
Rosie Perera | Forum Activity | Replied: Tue, Nov 25 2014 7:40 PM

Dan Francis:
This thread in particular seemed pointless (I am hoping the person starting this simply missed Bob's explanation in which he said they had already started prep for such a disaster when murphy's law caught up before they were done).

The person who started this thread seemed to be trying to give his expert advice (one could almost read it in a tone of "I feel for you guys, I'm sure it must be tough; here's what I've learned in my experience in the business, hope it helps for next time..."). He seems to have the kind of experience that would have helped out in this situation if Faithlife had had someone of his background on hand instead of having to learn the ropes as they went along. Bill probably had not seen Bob's explanation that there was already a plan in the works.

Yes, there were other threads which implied Faithlife owed users something, and I agree with you about those threads. But this thread in particular was not one such. It was just simply "Logos needs a disaster recovery plan" (which Bob and his teammates would not dispute), with an elaboration of what he meant by that subject line. It was unfortunate that he included one sentence in there which he was later called out on and admitted he had mischosen his words. But the rest of it was purely professional and very wise, regardless of whether Bob and his company could have said "yup, we already know that." He was saying it in a tone of charity towards the company, and he reiterated later that he loves Logos. He really was just trying to be helpful.

Posts 5285
Dan Francis | Forum Activity | Replied: Tue, Nov 25 2014 7:54 PM

Too Lee and Rosie... I was not meaning to suggest Bill was being unreasonable. I admit I was lumping his thread in to many others. When I first read this thread I thought it was being more critical than helpful (we have seen a lot of that in the past few days). My James john statement was aimed more at a comment on a different thread. At times I will admit my train of thought can get very tangental. Bill likely was meaning to be helpful and he was not the one in any respect who claimed damnation for the Logos staff. At one point someone complained people were being big babies in not so many words. I hate how this outage has caused many people to lash out. My apologies once again. I should know better than post with a migraine. This is not the first time I have put my foot in my mouth because of that.

-Dan

Posts 334
Paul Strickert | Forum Activity | Replied: Tue, Nov 25 2014 8:00 PM

Bill Smith:

It happens to the best of us. As a cyber security officer at a major Sass in California I would like to point out a couple things. First you are not the only company with a major malfunction in the past month. You have now joined the club. So here are some points to reflect on.  

1. To have a weak engine (data center and its servers) under the hood of a Porsche does not do the customer any good. This is the biggest head scratcher is why you did not spend the money on server infrastructure like you do on you image. (Nice buildings in Bellingham--I visited your town last year and it was very nice)

2. Get a sound disaster  recovery plan in place. You do not have one! There are plenty of templates out there. Heck there are companies who specializes in this. If you have the cash I would drop your ego and invest in one. 3.  Put you data centers geographically apart with redundant realtime backup. Not west coast. Central US would be best. And a data center that is not going anywhere. The big ones are swallowing up the smaller ones.  5.  Do not put your servers in a data centers with low security (some will say we have great security  but do not) . 6. Do not try and coverup your mistakes. Human answers within at lest one hour of an outage will do. Be honest and to the point. Presidents have gotten in trouble with the type of padding lawyer answers you gave and look where it got them.  7. Be honest be fast. A good liaison point person to your customers does not half to be a IT staff. Just someone who can get a quick update form your lead IT.  8. I hope you are secure now. The biggest mistake is recovering after a disaster and shortcutting security.   Hang in. I use your products at our church ( 10 thousand people) and it does great for our volume. Bill

Lee:

Please, nothing in what “Bill Smith" wrote is off base. His techincal points are very pertinent and he is commenting about prospective action. He was not malicious, snarky or anything.

I read it a bit differently, I guess (esp. the parts I bolded).  Not that it matters what I think, but I'd probably call those statements at least somewhat condescending.

Posts 2574
Lee | Forum Activity | Replied: Tue, Nov 25 2014 8:07 PM

Bob P.'s response was measured and wise, because this guy knows what he's talking about.

I've lived long enough to realize that the best advice is often not couched in soft language. Advice given, advice taken. So it really doesn't matter what you (or I or anyone) would think.

Posts 18867
Rosie Perera | Forum Activity | Replied: Tue, Nov 25 2014 8:07 PM

Dan Francis:
I should know better than post with a migraine. This is not the first time I have put my foot in my mouth because of that.

No worries. To quote Bill in the words with which he started this thread, "it happens to the best of us." I hope and pray that you recover from your migraine soon.

Yes, indeed, people have been lashing out in unhelpful ways, and that is not very nice. I hope all of that hasn't contributed to your headache.

Peace to you, and get some rest.

Posts 790
James Hiddle | Forum Activity | Replied: Tue, Nov 25 2014 8:17 PM

Dan Francis:

Too Lee and Rosie... I was not meaning to suggest Bill was being unreasonable. I admit I was lumping his thread in to many others. When I first read this thread I thought it was being more critical than helpful (we have seen a lot of that in the past few days). My James john statement was aimed more at a comment on a different thread. At times I will admit my train of thought can get very tangental. Bill likely was meaning to be helpful and he was not the one in any respect who claimed damnation for the Logos staff. At one point someone complained people were being big babies in not so many words. I hate how this outage has caused many people to lash out. My apologies once again. I should know better than post with a migraine. This is not the first time I have put my foot in my mouth because of that.

-Dan

Don't feel bad. I suffered with Foot In Mouth Syndrome quite a bit since I've been here. It happens to the best of us(now where did I hear that from).

Also prayed for you to get better soon. God Bless you always and like it was said get some rest.

Page 2 of 2 (38 items) < Previous 1 2 | RSS