Proclaim Services Outage Sunday, September 20

From about 7:40 to 8:10 PST, one of the databases that supports the Proclaim Remote became unavailable. During this time, the remote would not have been able to change slides in Proclaim. All services are back online now. I am sorry for any inconvenience that this has caused. Thank you for your patience.

Find more posts tagged with

Comments

lanny smith

Can you elaborate on "became unavailable" please? This is not a trivial matter. We rely on the remote app to a large extent and the outage was a real problem for us. Is this likely to repeat? Details would be helpful.

Stephen Massey

https://community.logos.com/discussion/comment/764562#Comment_764562

We are still investigating the issue. What we do know is that the database that handles slide change operations started sending back very large responses. This caused access times to become slow to the point that the web nodes could no longer contact the database. The root cause remains unknown but we will be working on it as our top priority this week. At the same time, we will be looking into a plan to identify the effects of the issue and quickly recover from it if it does happen again.

I will post to this thread when we find and fix the root issue. I will also announce what our mitigation strategy is if a similar issue happens in the future.

Again I am deeply sorry for the problems that this caused you and everyone else.

Stephen Massey

https://community.logos.com/discussion/comment/764605#Comment_764605

We've identified and fixed the bug that caused a web service to try to obtain large amounts of data from the database. Certain data was not always cleaned up properly and was causing the service to obtain ever larger data sets. In addition to making sure that all data is cleaned up, we've updated the service to request data in a way that it will not be affected by a similar issue in the future.