Data Center Storage Failure, 11/21
A storage system that backs many core websites and services started failing at approximately 4:30 PM PST on Friday. The components that failed (hard drives & a RAID controller battery) have been replaced. However Ceph, the distributed object storage technology we use to run databases, application servers, and file storage requires a re-balance to heal. This healing process takes many hours, potentially up to 20, to distribute ~190 TB of data across 5 storage nodes (computers). The re-balance started around 1 pm PST, and once the re-balance completes, all systems will be back to normal.
We’re currently migrating core services off the failed storage system to expedite system availability. Commerce websites including Logos.com, Verbum.com, Vyrso.com, Noet.com, etc. should migrate within the next hour or two.
We’re taking the same steps with Proclaim to get it functioning in time for Sunday morning service.
Ceph, and other central storage technologies are common place in data center deployments. They are essentially many-eggs-in-one-basket architectures that are vulnerable to large scale failures. To reduce the impact of hardware failures, we’re researching decentralized storage options that minimize the dependence of core systems on any one component.
We’re also building a redundant data center deployment in south Seattle. This will enable rapid fail over when a system wide event occurs. Unfortunately, that second data center is several weeks/months from completion.
We’re very sorry for the inconvenience caused by this event, and appreciate your continued patience and support as we work to resolution. Know that we appreciate your business and are making every effort to improve system availability.
Comments
-
Jim Straatman said:
We’re also building a redundant data center deployment in south Seattle. This will enable rapid fail over when a system wide event occurs. Unfortunately, that second data center is several weeks/months from completion.
This was going to be my question.
I know building a redundant datacenter can be very costly, but considering the growth and dependency, it may be time to do this.
0 -
Jim Straatman said:
A storage system that backs many core websites and services started failing at approximately 4:30 PM PST on Friday. The components that failed (hard drives & a RAID controller battery) have been replaced. However Ceph, the distributed object storage technology we use to run databases, application servers, and file storage requires a re-balance to heal. This healing process takes many hours, potentially up to 20, to distribute ~190 TB of data across 5 storage nodes (computers). The re-balance started around 1 pm PST, and once the re-balance completes, all systems will be back to normal.
We’re currently migrating core services off the failed storage system to expedite system availability. Commerce websites including Logos.com, Verbum.com, Vyrso.com, Noet.com, etc. should migrate within the next hour or two.
We’re taking the same steps with Proclaim to get it functioning in time for Sunday morning service.
Ceph, and other central storage technologies are common place in data center deployments. They are essentially many-eggs-in-one-basket architectures that are vulnerable to large scale failures. To reduce the impact of hardware failures, we’re researching decentralized storage options that minimize the dependence of core systems on any one component.
We’re also building a redundant data center deployment in south Seattle. This will enable rapid fail over when a system wide event occurs. Unfortunately, that second data center is several weeks/months from completion.
We’re very sorry for the inconvenience caused by this event, and appreciate your continued patience and support as we work to resolution. Know that we appreciate your business and are making every effort to improve system availability.
Ok that went way over my head. Too technical for my taste but from what I read is that logos is down and will be up when you fix the problem.
Which would be nice!
0 -
Jim Straatman said:
We’re taking the same steps with Proclaim to get it functioning in time for Sunday morning service.
It is Sunday morning in Kenya. I don't appreciate being a second class citizen to my American Coleagues
Jim Straatman said:Know that we appreciate your business
Your actions do not say so. The last time that there was an issue and the forums were down, I complained that you did not communicate the issue via social media or respond to my enquiries via the same. Your handling of this issue shows that my complaint was not taken seriously and that you have no protocols in place for communicating with your customers during such an event as this. Don't remind me that you responded via twitter because it was not to my tweet but to the latest ones and after many hours had passed.
Recently I purchased L6 and at no point was I given any clue that there was an issue with Yosemite.
I do not feel valued as a customer and talk is cheap. However, since you did nothing to change last time, I guess that nothing will change this time.
0 -
What caused the failure to begin with?
0 -
Sean,
My apologies for my comments regarding Sunday morning and our timezone difference.
You are right, we should communicate better when things like this happen. It can be difficult to broadcast information in all the appropriate channels, especially when our attention is focused on resolving the problem. We're currently building a status dashboard to expose key performance metrics and information when an event like this occurs. Something similar to what you might find at Amazon Web Services or GitHub.
0 -
James Hiddle said:
What caused the failure to begin with?
The hard drives (and a battery) failed just as they do on your home computer or laptop - wear, heat, manufacturing flaw ... all the things you think of at home.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Jim Straatman said:
My apologies for my comments regarding Sunday morning and our timezone difference.
Thanks. I appreciate that you are taking steps to improve and a status page would go a long way towards reassuring me in the future. I assume that it would be insulated somehow from everything else. One frustration for me was that the forums went down leaving me completely in the dark.
0 -
Yes, Sean, we'll host it somewhere completely separate from our production infrastructure.
0 -
God bless you Jim and the IT team at Logos. I know these events can be physically and emotionally exhausting. Hang in there!
0 -
All Proclaim databases have migrated to a more stable deployment and are back online. We're currently adding additional web servers to accommodate peak load.
0 -
John McGovern said:
God bless you Jim and the IT team at Logos. I know these events can be physically and emotionally exhausting. Hang in there!
Amen.
0 -
MJ. Smith said:
... all the things you think of at home.
Yes, you put things in a better perspective. Thanks!
I think the thing that makes us easily frustrated/angry is that we can't control these things anymore. We are so depedent on others that when something doesn't work as we want to.
Just came back from Tanzania for a workvisit. Everytime I am in Africa I realise that our abilities to control our life are much more limited then we think. Much more than just webservers. Turning back to stone tablets won't do the trick. We (or maybe it is just me) take much more for granted than we realize.
Suppose the power fails today? Suppose we can't shower us in clean drinkwater anymore (or drink it)? What to do if candles are finished? What to do when the gasbottle is empty and the shops don't have none (until nobody knows)? What to do when there is no good wokring public service for rubbish (like printer toners, toxic fluorescent bulbs, polypropyleen bags, shaving foam cans)? Just throw it on the street? Just burn it? Just bury it?
What to do if the bread that is sold is just another form of powder? What to do if the bus to the next city three hours down the road doesn't show up? What to do if it shows up but breaks down after one and a half hour?
I find it scary that I can't do much myself. I am scared that I depend so much on others. Frustrated, since so many times 'the others don't do as I like'. Scared, since people would think that also about me.
But it is also a miracle that many times it works! A miracle and a blessing: we don't need to do everything ourselves. We have each other. So let us enjoy and celebrate that.
This is the WORLD the Lord has made. We will rejoice and be glad in it.
0 -
John McGovern said:
God bless you Jim and the IT team at Logos. I know these events can be physically and emotionally exhausting. Hang in there!
Amen
0 -
Community APIs that support Faithlife.com & groups functionality within various applications has been restored. We're currently working on alternative infrastructure for the commerce engine to get Logos.com, Verbum.com, Noet.com, & Vyrso.com online.
The storage rebalance is about 50% complete. We'll continue to migrate core services off the failed storage system while it's recovering.
0 -
Jim Straatman said:
We’re also building a redundant data center deployment in south Seattle. This will enable rapid fail over when a system wide event occurs. Unfortunately, that second data center is several weeks/months from completion.
Jim, now is the time for some serious reflection at Faith Life. As someone alluded to elsewhere, your systems are expected to be up 24/7. Faith Life is now a global service and anything less than 24/7 is out of the question.
In your situation I wouldn't be able to sleep at night knowing that you have a single data center in Seattle. As far as I'm concerned, a second data center in Seattle is not a solution. Does the earth not shake in Seattle? How about a Tsunami? You need to be looking at east or midwest.
Personally, I work in a small company where it is my responsibility to keep a website and company infrastructure up 24/7. Our customers may only be numbered in the thousands but they are are all over the globe. I have found, as well as it being my own expectation, that people who have been sold a product or service expect it to be available to them when they want it and when they need it. The significance of any outage goes way beyond the financial. It affects our reputation and how the customer sees our dependability.
I know that you guys work hard and I know how overwhelming the technology can be. I just wanted to voice my concern.
0 -
I am thankful that I am able to use Logos 6 on my Mac and that resources that were downloaded prior to the current problems appear to be accessible and useable. Once the server is back to full capacity, I wonder if you might also address a question I have about the Logos mobile app (or refer it to the appropriate IT person). I had downloaded a number of resources to both my iPad and iPhone for offline use and although the apps now show me as signed in to my account at logos, none of the resources appear in my library on either device. I know that there are a lot of people who rely on the software more than I do, but I'm guessing that others have had similar problems with their mobile Logos. Thanks so much for your hard work. Best wishes for continued progress.
0 -
Jim Straatman said:
A storage system that backs many core websites and services started failing at approximately 4:30 PM PST on Friday.
Logos.com and Vyrso.com are still down at 5:22 CST on Sunday, Nov 23rd
0 -
Think of all the free books we will get when it comes back up!
0 -
Ralph Tyner said:
I am thankful that I am able to use Logos 6 on my Mac and that resources that were downloaded prior to the current problems appear to be accessible and useable. Once the server is back to full capacity, I wonder if you might also address a question I have about the Logos mobile app (or refer it to the appropriate IT person). I had downloaded a number of resources to both my iPad and iPhone for offline use and although the apps now show me as signed in to my account at logos, none of the resources appear in my library on either device. I know that there are a lot of people who rely on the software more than I do, but I'm guessing that others have had similar problems with their mobile Logos. Thanks so much for your hard work. Best wishes for continued progress.
This was my experience exactly. Resources I download for offline use should be available when I want them. A dependency on network availability and availability of your servers (as certainly appears to be the case) should not exist for this use case. Kindle & iBooks work fine without connectivity. I simply love Noet but my confidence has been badly shaken. Thanks, and good luck with the continuing mop up.
Fred
0 -
I can only imagine the frustration of Faithlife's technical team. Hang in there. My prayers are with you guys.
Using adventure and community to challenge young people to continually say "yes" to God
0 -
"It is Sunday morning in Kenya. I don't appreciate being a second class citizen to my American Coleagues"
Yeah, Sunday in Australia comes a long time before Sunday in Logosland. But it's like Charlie Brown said: "You don't have to worry about the world coming to an end today. It's already tomorrow in Australia."
Gold: Logos 8 | Bronze: Logos 9, Baptist, Reformed, Ordinariate | Starter: Messianic Jewish, Eastern Rite | Academic: Premium
0 -
Since this thread seems to be the primary communication thread during the Data Center Storage Failure situation is it possible that this can be temporarily made a sticky post on the top of the page?
Using adventure and community to challenge young people to continually say "yes" to God
0 -
Can we get something more helpful than the sad sheep page as a status page at logos.com. something that actually says what the problem is, the steps to be completed before it is back, and time estimates? At the moment it is not describing the real problem and the sheep gives the impression it is not a serious issue.
I don't know how many users actually read the forum but if they don't, all they can see is something very unhelpful.
0 -
0
-
I hope they get it all back up ASAP. There is no telling how much money will be lost over the course of the weekend due to the outage. I feel like this has been a bad couple or three days for Logos (including the potential company growth solution that sparked some pretty heated debate and comments). I think this will certainly make it more likely that this will never happen again. Good luck to those working on it!
Disclaimer: I hate using messaging, texting, and email for real communication. If anything that I type to you seems like anything other than humble and respectful, then I have not done a good job typing my thoughts.
0 -
With my own many years of IT background, I appreciate and understand the problems your team is facing. Thank you for burning the midnight oil to resolve the matter.
I did not even notice the problem -- my mobile apps continued to work without any apparent issues -- until I attempted to load up the Logos software on my laptop late last evening. Access, it seems, is prevented in situations such as this because the software often checks for and attempts to download any and all available updates before presenting the user with a usable study environment. If there is no connectivity or the download fails, the software does not open -- even after many minutes to hours of waiting. It may benefit from a code modification which will allow the software to run regardless internet connectivity. I am rarely "offline", and understood the software was supposed to maintain an offline copy of my library. To find differently adds a new wrinkle to my ministry plan using Logos. My intent was/is to utilize Logos in a rather rural setting and in many cases will not have Internet readily available. I would appreciate feedback regarding this matter and whether or not a background updating code change might be in the project works.
0 -
Paul Coleman said:
If there is no connectivity or the download fails, the software does not open -- even after many minutes to hours of waiting.
My understanding is that Logos should still work (even when offline).
I used Logos (on Windows 7) many times over the past few days. The only thing I noticed that was different was a small yellow warning sign in the upper left corner of the app which stated "An error occurred during synchronization, click to try again."
What O/S are you using? And which version of Logos?
0 -
Paul Coleman said:
I am rarely "offline", and understood the software was supposed to maintain an offline copy of my library.
Logos does work offline. On the sign-in, if online isn't available, you should be able to sign in without using the internet.
There could be a problem when the Logos servers are available intermittently so the software "thinks" it can login but can't. In that case hold the control key down (Windows) as Logos loads and choose "work offline." That should put you back up and running.
Help links: WIKI; Logos 6 FAQ. (Phil. 2:14, NIV)
0 -
Dit it ocur to you that you are on a different timeline?
0 -
Moving updates to this thread which does not have replies enabled, so status is easier to discover.
0 -
I don’t depend on any Faithlife products so haven’t had to make the kinds of evaluations that I would if I did. Churches, ministers of the word, working scholars and, students who do depend on Faithlife products are within their right to expect the highest levels of reliability. I hope reflection, and scrutiny of this incident, will lead to a broader understanding of the risks and a well-informed confidence that Faithlife is responsibly acting to mitigate them.
MJ. Smith said:... all the things you think of at home.
Faithlife has abstracted the library software in such a way that it is difficult if not impossible to assume individual responsibility for the integrity of an installed ‘library system’. I have assented to the fact that we will, probably, never again have the type of flexibility and security that the Libronix\LDLS provided with regard to installation, system level configuration, and local backup.
I have made a very substantial commitment and believe now that the ongoing value of this to me depends upon the ongoing success of Faithlife. My library of over 8000, including Perseus and about 100 other free, resources is a joy to use and I will certainly add to it. I have concerns though, which have tempered my enthusiasm. I am all in so far but am also prepared to cut my losses if needed.
Thank you Faithlife for consistently good products and great, almost two decades to me, customer service.
Thanks to all the individuals who regularly participate in the forums. MJ Smith you are constantly a voice for my interests and concerns. Denise I always read your posts with glee and you nudge my thinking in meaningful ways.
John McGovern said:God bless you Jim and the IT team at Logos. I know these events can be physically and emotionally exhausting. Hang in there!
Patrick Rietveld said:But it is also a miracle that many times it works! A miracle and a blessing: we don't need to do everything ourselves. We have each other. So let us enjoy and celebrate that.
Jim Straatman said:Moving updates to this thread which does not have replies enabled, so status is easier to discover.
"The Christian mind is the prerequisite of Christian thinking. And Christian thinking is the prerequisite of Christian action." - Harry Blamires, 1963
0 -
Patrick Rietveld said:MJ. Smith said:
... all the things you think of at home.
Yes, you put things in a better perspective. Thanks!
I think the thing that makes us easily frustrated/angry is that we can't control these things anymore. We are so depedent on others that when something doesn't work as we want to.
Just came back from Tanzania for a workvisit. Everytime I am in Africa I realise that our abilities to control our life are much more limited then we think. Much more than just webservers. Turning back to stone tablets won't do the trick. We (or maybe it is just me) take much more for granted than we realize.
Suppose the power fails today? Suppose we can't shower us in clean drinkwater anymore (or drink it)? What to do if candles are finished? What to do when the gasbottle is empty and the shops don't have none (until nobody knows)? What to do when there is no good wokring public service for rubbish (like printer toners, toxic fluorescent bulbs, polypropyleen bags, shaving foam cans)? Just throw it on the street? Just burn it? Just bury it?
What to do if the bread that is sold is just another form of powder? What to do if the bus to the next city three hours down the road doesn't show up? What to do if it shows up but breaks down after one and a half hour?
I find it scary that I can't do much myself. I am scared that I depend so much on others. Frustrated, since so many times 'the others don't do as I like'. Scared, since people would think that also about me.
But it is also a miracle that many times it works! A miracle and a blessing: we don't need to do everything ourselves. We have each other. So let us enjoy and celebrate that.
This is the WORLD the Lord has made. We will rejoice and be glad in it.
That's when you move to Alaska and live off the land [:)]
0 -
MJ. Smith said:James Hiddle said:
What caused the failure to begin with?
The hard drives (and a battery) failed just as they do on your home computer or laptop - wear, heat, manufacturing flaw ... all the things you think of at home.
Are you sure it wasn't a ton of Faithline employees overloading the servers while playing Angry Birds online [:D]
0 -
Jim Straatman said:
Moving updates to this thread which does not have replies enabled, so status is easier to discover.
Thank you, Mr. Straatman, for your devotedness to seeing that the Faithlife/Logos server infrastructure is fully restored. I sure do appreciate this as well as the communicated updates. [:)][:)]
Praying for you all.
Blessings.
<!-- -->
0 -
Jim
Prayers are being offered for you and your team - for wisdom and endurance during this time. Logos is an amazing tool -- it is because it is so valuable to most of us that when it is down it feels like my left arm has been amputated. The work all of you do at Logos really is a blessing to the Church -- to pastors and people everywhere. So thank you.
A question -- could you set it up to move the templates that drive Visual Copy from the server to the local computers? If that were so I would have been able to manage one of my projects while the server is down. I suppose the same is true on some of the other datasets in Logos 6. Having them reside on your server makes the use of L6 subject to your server being up -- and users being connected to the net. That seems a bit shaky to me.
I am glad to see that you are working on an offsite mirror for the Logos system. Expensive I'm sure -- but pretty important given how we have come to rely on Logos. And lets face it -- this software is not inexpensive. I just dropped $1500 on an upgrade from L5Gold to L6Platinum. This is serious money and we need a reliable platform.
OK.... I feel better. Having shared my angst -- let me repeat my assurance of prayer.... and my gratitude to you and your team who are no doubt putting in a very long weekend!
Vicars
0 -
Vicars,
Thank you for your prayers. They are very helpful and appreciated as we push through this event.
With respect to local visual copy templates, I'll defer that decision to our products team. I'm primarily responsible for growing the web infrastructure, so my focus will be continuing to improve the performance and availability of data center functions. I can see value in your suggestions not only during an outage, but also for someone working offline. When we have an opportunity to reflect on the event, we'll take your suggestion into consideration.
Again, thank you for your prayers and support.
Jim
0 -
Thank you, Daniel. We really appreciate your support.
0 -
Thanks for all the hard work from you and your team to get everything up and running again: Prayers and support as always...... [:)]
0 -
Thank you, David.
0 -
Joseph Turner said:
including the potential company growth solution that sparked some pretty heated debate and comments
Joseph, can you elaborate on this or give me the threads on this. I have no idea what you're talking about here.
Thank you for your time/help.
Myke H.
Myke Harbuck
Lead Pastor, www.ByronCity.Church
Adjunct Professor, Georgia Military College0 -
Myke, just look for the Forum thread that has the most views of any you can probably remember. It's quite recent.
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
It has also been my experience that some bibles, that I had already downloaded to my phone, are now inaccessible. Quite frustrating.
0 -
The store front seems to be back up!
0 -
I am going to say it in anticipation, by faith, because its going to be any moment now:
Hey! I'm syncing! I'm syncing!
0 -
Yasmin Stephen said:
The store front seems to be back up!
I can access it as well. Thanks to all the hard working staff [Y]
0 -
Yasmin Stephen said:
The store front seems to be back up!
Yup, Vyrso.com and Verbum.com and Noet.com too.
0 -
If Jesus had been an IT professional, how would the wording be in Luke 6:48–49?
It is interesting that Jesus did not say that build a flimsy house and pray that it holds together.
Gold package, and original language material and ancient text material, SIL and UBS books, discourse Hebrew OT and Greek NT. PC with Windows 11
0 -
I say we stop asking what would Jesus do if he were a customer in this situation, what would Jesus do if he were a tech, WWJD, WWJD...
It trivializes our Lord by assuming the answer, for who knows whether Jesus would not have done something less expected, e.g. stop using said software, quit the job, lay hands on the computer, take time-out for prayer, etc.
Instead, I suggest that client and company alike be civil and professional in dealing with this extended downtime.
0 -
I have been inconvenienced more than damaged by this outage. I still had Logos on my Mac and on my Android phone. Where I lost out was with my iPad and iPod on which I had downloaded thousands of books. [As bi-vocational I actually do more of my preparation work on my iPad than on any other device, using it "on the go" and whenever opportunity presents itself]. Immediately before the outage I had a technical issue on the iPad which required reinstalling the app. I did that and was in the process of downloading my resources when the whole thing went awry. My immediate assumption was that something had gone wrong in the download process so I uninstalled and reinstalled the app again. I was totally surprised to discover my library had shrunk to only a handful of books. It wasn't until I tried the iPod and saw that it too had only 32 resources out of more than 3,000 that I realised the problem was with Logos and not with me.
I have two concerns from a user perspective - [I can have no concept of the nightmare you have endured / are enduring from a service provider perspective] - the first is communication. I spent hours trying to figure out what had gone wrong in my process of installing and downloading. I can't remember how many times I tried to reinstall on the iPad before it dawned on me to check the iPod. Admittedly, this time I was in the unusual situation of following your support team member's advice to reinstall the app to resolve a particular problem when the crash happened. If it were a normal usage situation I might not have lost so much time before realising the problem was not with me or my device. So the communication of an issue would be helpful in future if it is possible to do so in this type of event.
The second concern has also been raised by others and stems from an assumption under which I operated, and apparently others do too. That assumption is that downloaded resources are available to us offline irrespective of what technical problems Logos is experiencing. I have a second phone that I use exclusively for my secular employment. It is an early Android model and I have Logos downloaded on it from "Market," which will give you some idea of when I downloaded it. It doesn't have Google Play. I never go online now with this phone and my downloaded resources are available to me all the time. I expected the same would be true of resources downloaded on my iPad and iPod.
In the light of this incident I think we [more precisely, I] need an understanding of what "downloaded" actually means in terms of accessibility, and in what circumstances we are likely to lose access in the future. I am back and forward to the Philippines from Ireland and consequently, Logos has been such a great solution for me. I've been giving away my physical library bit by bit as I have replaced the physical books with Logos editions. Being able to bring a library in a phone or other hand held device makes ministry preparation so much easier when one ministers in locations 6,000 miles apart. My fears now are that divesting myself of the physical books appears to me - "non-techie" that I am - in the light of this incident, as possibly an expensive mistake or partial mistake [depending on the resources given away]. I won't make that a definitive judgement until I actually understand why my downloaded resources were not available to me, and what if any solutions there may be for the future. The answer to this question will determine if I need to bring a greater balance into my book buying between physical and electronic copies. By which I don't mean a 50/50 type balance, but only that I need to ensure that I have an adequate physical back-up of essential exegetical tools for emergency needs.
I will continue to be a huge fan of Logos and the genius and creativity that have given us an incredible tool without which my bi-vocational life would not be viable. What I need is understanding to inform future directions.
Blessings.
0 -
ManilaDave said:
I have two concerns from a user perspective - [I can have no concept of the nightmare you have endured / are enduring from a service provider perspective] - the first is communication. I spent hours trying to figure out what had gone wrong in my process of installing and downloading. I can't remember how many times I tried to reinstall on the iPad before it dawned on me to check the iPod. Admittedly, this time I was in the unusual situation of following your support team member's advice to reinstall the app to resolve a particular problem when the crash happened. If it were a normal usage situation I might not have lost so much time before realising the problem was not with me or my device. So the communication of an issue would be helpful in future if it is possible to do so in this type of event.
The second concern has also been raised by others and stems from an assumption under which I operated, and apparently others do too. That assumption is that downloaded resources are available to us offline irrespective of what technical problems Logos is experiencing. I have a second phone that I use exclusively for my secular employment. It is an early Android model and I have Logos downloaded on it from "Market," which will give you some idea of when I downloaded it. It doesn't have Google Play. I never go online now with this phone and my downloaded resources are available to me all the time. I expected the same would be true of resources downloaded on my iPad and iPod.
Dave, thanks for sharing your thoughts on this situation. While I wouldn't say I was even inconvenienced by the outage (the Logos app gets little use from me on weekends and I was able to continue as normal on my laptop), I do share these exact same two concerns you enumerated above. I was able to browse the forums intermittently so I knew early on something was wrong on the Logos end. However, I kept checking Facebook and Twitter and nothing was announced there for the longest while, though other things were being posted (regarding free books; verse of the day etc). While I know these things may be scheduled and doesn't necessarily mean someone is actually sitting at a desk posting them the moment they appear, to most people it just looks like there's a problem and no one is addressing it, though, from the updates given here, the problems were being addressed early on. The first order of the day when something goes wrong that affects customers is to inform as many of them as possible, as early as possible. And with social media, it's easy peasy.
(Also, the maintenance page for the website. Please, Logos, if (God forbid!) you ever have another serious outage like this, please do not use the defluffed sheep page again. It seemed vaguely insulting for a problem of that nature, and also many persons I'm sure thought that this was a scheduled maintenance, which in turn most likely caused greater irritation - why would you schedule maintenance over the weekend and not make a prior announcement.)
Your second point about the downloaded resources is troubling to me as well. While the disappearance of resources caused me no inconvenience, if it wasn't the weekend it would have. Downloaded resources should be off-limits and not messed with.
When the Logos team is able to move past the immediacy of the situation, I look forward to hearing what they have to say about all this.
0
