Bug: Missling Highlights

I recommend using the following command in the command box: update note attachments

This requires that you are running the application online. It will update all of your notes that are attached to an older version of a resource. This could take some time to complete, but you will be shown a progress indicator notification at the top of the main window.

Once this completes, the notes should be able to find their correct location in the updated resource(s).

John Marino said:

Andrew,

Admittedly I'm a little nervous about this, having lost the highlights to this specific resource in the first place. I have thousands of notes/highlights along with thousands of hyperlinks stored in my notes documents. Losing them would be catastrophic for me. Are you able to do two things for me first? (1) Confirm for me that this in no way puts my other notes documents or highlights at risk of being lost and (2) tell me how I can (for Mac) perform a backup of all my notes documents and highlights that I can store locally on my machine.

Unfortunately, there's not a practical way to back up your notes/highlights. However, every single change that is made to your notes/highlights that is sent to our server is stored forever. We don't throw away past revisions. That means that if something catastrophic happens, we have the capability to manually roll the state of your notes back to a previous point in time. That means there is no risk of loss.

If anything looks wrong as a result of this command, please let us know right away. Maintaining the integrity of your data is an extremely high priority for us.

John Marino said:

Andrew,

I ran the command and it appears to have restored all my highlights. I'm grateful for your help and grateful that they've been restored. This entire event has reminded me how dependent I am upon the reliability of the Logos servers, and I'm walking away from this conversation feeling a bit uneasy about how vulnerable my data now appears to be.

I'd certainly feel a lot more comfortable if I had the opportunity to talk through this situation with one of your software engineers. I'd like to understand what is causing this to happen, and feel better about the security of my data. You said there was no practical way to backup data, but then said that it could be done. I'd like to explore the options here.

Could you have one of your engineers give me a call? My phone number is on my Logos account. Perhaps he/she could email me first to setup a time.

I trust that this won't be a problem. In every other area possible, I'm an extremely satisfied Logos user. I use it daily, and wouldn't trade it for anything. Grateful for all you guys do.

Let me try to clarify some things for you that might help ease your mind.

Each one of your notes is stored with a little piece of information that points to the exact location in the resource to which they are attached. This information looks something like the ID of the resource, the version of the resource when the note was created, the identifier for an article (a section in the resource), an offset into that article, and a bit of text providing the context of that location. As long as the resource has not been updated, we can find the exact position using the article ID and the offset. If the resource has been updated, then we make use of the context to try and find the correct location in the article in the new version.

For the vast majority of the changes that are made to resources, this process works. However, it has the constraint that text can't move from one article to another or else the correct location will not be found. Normally, when we update a resource, we don't allow text to move between articles so as to prevent this problem. However, there were a number of e-books (which are automatically generated resources) that were missing a table-of-contents. We wanted to update these resources to include that, but we missed the fact that this would make article changes, which would result in breaking the location of the notes. Ben is investigating a solution to this problem.

Nothing about your created data is missing or changed, the application just can't find the location in the current version of the resource where you originally created the note. Happily, using all the past revisions of our resources, we can store a bunch of information on our servers that allows us to exactly determine from one revision to the next where any given text in a resource moves. The command I gave you takes advantage of this information to update the positions of any notes that were created on an old version of a resource. The updated locations are then synced up to the server.

This command is almost never needed because we make a point to avoid this kind of breaking change. We're actually in the process of shipping this information with new versions of resources so that locations in the resource can be resolved accurately in the application without the need to connect with our servers. This should entirely eliminate this problem.

With regards to backing up your data, what I was trying to say about lacking a practical backup mechanism is that while you can backup the local files on your machine, there's no way for you to restore that state of your data if you have subsequently made any changes to this data. That said, in the event that Faithlife servers were no longer available an existing installation (or a backed-up installation) would continue to operate normally in offline mode. Take a look at "Is there a faster way to install Logos on another computer?" on the wiki for details about what needs to be copied in order to quickly get a new installation up and running. This will help you know what would need to be backed up.

I hope that helps!

PetahChristian said:

So the patch information would be downloaded alongside the resource, temporarily used, then deleted? (Just looking for clarification that disk space usage won't be increasing.)

The patch information will be built into the resource file. I don't currently have information about the impact on file size.

PetahChristian said:

Sorry for assuming that no one had time to already address this.

No problem. The issue is one that's been around for a while, and some of the pieces of the solution have been in place for some time. It hasn't been a high priority to complete the implementation because it almost never causes a problem. Happily, changes required for the upcoming new notes system prompted finishing things up.

Dave Hooton said:

The (older) backup data would be overwritten by the newer data from the server and you would still have the same issue! But if you start Logos in Offline mode you would continue to use the backup data. If you then manually fix the data or make other changes, that (newer) data would not be overwritten when you start Logos normally (online).

This is correct, provided that you make some kind of change to each piece of data that you don't want overwritten by the server. In the case of having had sweeping changes to all (or most) of your notes, this would mean making some kind of edit each note. I would call that "not practical". Alternatively, if you don't know what may have changed, then you'd be pretty much out of luck.