Bug: Missling Highlights

All of my highlights are missing in the book "Hide or Seek." When I go to my note file, the note says, "unavailable resource" even though I can open the book. Highlights are missing in all three devices: MacBook Pro, Dell PC, iPad. "Notes and Highlights" visual filter is turned on.

Find more posts tagged with

Comments

NB.Mick

All of my highlights are missing in the book "Hide or Seek." When I go to my note file, the note says, "unavailable resource" even though I can open the book. Highlights are missing in all three devices: MacBook Pro, Dell PC, iPad. "Notes and Highlights" visual filter is turned on.

Hm. Your description sounds as if you were looking at two editions of a book (one you open, the other you had made your highlights in and then lost access to). Another alternative is that the recent update of the book (on Friday) disrupted the highlights for some reason. Maybe Faithlife can speak to that.

Can we just compare the support info for a second?

LLS:9781939946645
2018-02-16T15:52:02Z
9781939946645.logos4

Edit: and what does it show, if you double click the blue "unavailable resource" text - it should open a "you don't have a licence" tab with a resource identifier.

marino389@gmail.com

https://community.logos.com/discussion/comment/955097#Comment_955097

Here's the identifier info...

LLS:9781939946645
2018-02-16T15:52:02Z
9781939946645.logos4

Interestingly enough, when I click on the link in the notes document ("unavailable resource") it does open up the correct book.

I think you're correct about the update. I saw the update yesterday, and then today had this problem.

NB.Mick

https://community.logos.com/discussion/comment/955113#Comment_955113

I think you're correct about the update. I saw the update yesterday, and then today had this problem.

This seems to be the reason - FL updated a number of ebook resources recently, and another user found highlights missing as well: https://community.logos.com/forums/t/159417.aspx

marino389@gmail.com

https://community.logos.com/discussion/comment/955151#Comment_955151

That's a big discouragement. I hope that the highlights are still there and can be restored.

Kenute P. Curry

https://community.logos.com/discussion/comment/955152#Comment_955152

Well I can't restore mine, so now I have to go and highlight this book all over again.....

JT (alabama24)

https://community.logos.com/discussion/comment/955177#Comment_955177

Well I can't restore mine, so now I have to go and highlight this book all over again.....

Unless you need the highlights ASAP, I would wait for a response from FL.

Andrew Batishko

I recommend using the following command in the command box: update note attachments

This requires that you are running the application online. It will update all of your notes that are attached to an older version of a resource. This could take some time to complete, but you will be shown a progress indicator notification at the top of the main window.

Once this completes, the notes should be able to find their correct location in the updated resource(s).

marino389@gmail.com

https://community.logos.com/discussion/comment/955433#Comment_955433

Andrew,

Admittedly I'm a little nervous about this, having lost the highlights to this specific resource in the first place. I have thousands of notes/highlights along with thousands of hyperlinks stored in my notes documents. Losing them would be catastrophic for me. Are you able to do two things for me first? (1) Confirm for me that this in no way puts my other notes documents or highlights at risk of being lost and (2) tell me how I can (for Mac) perform a backup of all my notes documents and highlights that I can store locally on my machine.

Andrew Batishko

https://community.logos.com/discussion/comment/955441#Comment_955441

Andrew,

Admittedly I'm a little nervous about this, having lost the highlights to this specific resource in the first place. I have thousands of notes/highlights along with thousands of hyperlinks stored in my notes documents. Losing them would be catastrophic for me. Are you able to do two things for me first? (1) Confirm for me that this in no way puts my other notes documents or highlights at risk of being lost and (2) tell me how I can (for Mac) perform a backup of all my notes documents and highlights that I can store locally on my machine.

Unfortunately, there's not a practical way to back up your notes/highlights. However, every single change that is made to your notes/highlights that is sent to our server is stored forever. We don't throw away past revisions. That means that if something catastrophic happens, we have the capability to manually roll the state of your notes back to a previous point in time. That means there is no risk of loss.

If anything looks wrong as a result of this command, please let us know right away. Maintaining the integrity of your data is an extremely high priority for us.

Andrew Batishko

https://community.logos.com/discussion/comment/955441#Comment_955441

Also note success using this command on the other thread: https://community.logos.com/forums/p/159417/955448.aspx#955448

marino389@gmail.com

https://community.logos.com/discussion/comment/955451#Comment_955451

How about for PC? I have an older PC that I use mainly for backup, but it does have Logos installed on it. Is there a practical way to export/backup notes documents for PC?

Andrew Batishko

https://community.logos.com/discussion/comment/955453#Comment_955453

How about for PC? I have an older PC that I use mainly for backup, but it does have Logos installed on it. Is there a practical way to export/backup notes documents for PC?

There is no practical way to backup the data that is synced to the server on any machine. You could back up the files where the notes are stored locally, but after restoring them, as soon as you ran the application online, the server would send all of the recent changes down to your machine, where they would overwrite the restored notes (because it knows that the changes on the server are more recent than the changes on your machine).

marino389@gmail.com

https://community.logos.com/discussion/comment/955455#Comment_955455

Andrew,

I ran the command and it appears to have restored all my highlights. I'm grateful for your help and grateful that they've been restored. This entire event has reminded me how dependent I am upon the reliability of the Logos servers, and I'm walking away from this conversation feeling a bit uneasy about how vulnerable my data now appears to be.

I'd certainly feel a lot more comfortable if I had the opportunity to talk through this situation with one of your software engineers. I'd like to understand what is causing this to happen, and feel better about the security of my data. You said there was no practical way to backup data, but then said that it could be done. I'd like to explore the options here.

Could you have one of your engineers give me a call? My phone number is on my Logos account. Perhaps he/she could email me first to setup a time.

I trust that this won't be a problem. In every other area possible, I'm an extremely satisfied Logos user. I use it daily, and wouldn't trade it for anything. Grateful for all you guys do.

Dave Hooton

https://community.logos.com/discussion/comment/955457#Comment_955457

I'd like to understand what is causing this to happen, and feel better about the security of my data.

It would be good to know why this happened as we have been assured in the past that resource updates won't affect our notes. Does it affect only Highlighting? Will this "Update Note.." command have to be applied whenever our "notes" go missing?

PetahChristian

https://community.logos.com/discussion/comment/955466#Comment_955466

I'd like to understand what is causing this to happen, and feel better about the security of my data.

It would be good to know why this happened as we have been assured in the past that resource updates won't affect our notes. Does it affect only Highlighting? Will this "Update Note.." command have to be applied whenever our "notes" go missing?

It would be nice if this process was automated, so whenever this "loss of highlights" would have happened (for whatever reason), the system would detect and transparently apply the correction.

I think FL has a lot of brilliant people working there, but not enough man-hours to accomplish everything they want to do.

JT (alabama24)

https://community.logos.com/discussion/comment/955466#Comment_955466

It would be good to know why this happened as we have been assured in the past that resource updates won't affect our notes.

We haven't been given a definitive answer, but it seems to be related to the re-processing of Vyrso books. I know that this can happen with personal books when they are updated (primarily formatting related) and I think it is the same with Vyrso resources.

Andrew Batishko

https://community.logos.com/discussion/comment/955457#Comment_955457

Andrew,

I ran the command and it appears to have restored all my highlights. I'm grateful for your help and grateful that they've been restored. This entire event has reminded me how dependent I am upon the reliability of the Logos servers, and I'm walking away from this conversation feeling a bit uneasy about how vulnerable my data now appears to be.

I'd certainly feel a lot more comfortable if I had the opportunity to talk through this situation with one of your software engineers. I'd like to understand what is causing this to happen, and feel better about the security of my data. You said there was no practical way to backup data, but then said that it could be done. I'd like to explore the options here.

Could you have one of your engineers give me a call? My phone number is on my Logos account. Perhaps he/she could email me first to setup a time.

I trust that this won't be a problem. In every other area possible, I'm an extremely satisfied Logos user. I use it daily, and wouldn't trade it for anything. Grateful for all you guys do.

Let me try to clarify some things for you that might help ease your mind.

Each one of your notes is stored with a little piece of information that points to the exact location in the resource to which they are attached. This information looks something like the ID of the resource, the version of the resource when the note was created, the identifier for an article (a section in the resource), an offset into that article, and a bit of text providing the context of that location. As long as the resource has not been updated, we can find the exact position using the article ID and the offset. If the resource has been updated, then we make use of the context to try and find the correct location in the article in the new version.

For the vast majority of the changes that are made to resources, this process works. However, it has the constraint that text can't move from one article to another or else the correct location will not be found. Normally, when we update a resource, we don't allow text to move between articles so as to prevent this problem. However, there were a number of e-books (which are automatically generated resources) that were missing a table-of-contents. We wanted to update these resources to include that, but we missed the fact that this would make article changes, which would result in breaking the location of the notes. Ben is investigating a solution to this problem.

Nothing about your created data is missing or changed, the application just can't find the location in the current version of the resource where you originally created the note. Happily, using all the past revisions of our resources, we can store a bunch of information on our servers that allows us to exactly determine from one revision to the next where any given text in a resource moves. The command I gave you takes advantage of this information to update the positions of any notes that were created on an old version of a resource. The updated locations are then synced up to the server.

This command is almost never needed because we make a point to avoid this kind of breaking change. We're actually in the process of shipping this information with new versions of resources so that locations in the resource can be resolved accurately in the application without the need to connect with our servers. This should entirely eliminate this problem.

With regards to backing up your data, what I was trying to say about lacking a practical backup mechanism is that while you can backup the local files on your machine, there's no way for you to restore that state of your data if you have subsequently made any changes to this data. That said, in the event that Faithlife servers were no longer available an existing installation (or a backed-up installation) would continue to operate normally in offline mode. Take a look at "Is there a faster way to install Logos on another computer?" on the wiki for details about what needs to be copied in order to quickly get a new installation up and running. This will help you know what would need to be backed up.

I hope that helps!

PetahChristian

https://community.logos.com/discussion/comment/955489#Comment_955489

Thanks for sharing details that help preserve links. It's actually reassuring that so much thought has gone into the process to keep everything working, resource update after update.

We're actually in the process of shipping this information with new versions of resources so that locations in the resource can be resolved accurately in the application without the need to connect with our servers. This should entirely eliminate this problem.

Thanks for the work in progress to automate this process!

So the patch information would be downloaded alongside the resource, temporarily used, then deleted? (Just looking for clarification that disk space usage won't be increasing.)

I think FL has a lot of brilliant people working there, but not enough man-hours to accomplish everything they want to do.

Sorry for assuming that no one had time to already address this.

Andrew Batishko

https://community.logos.com/discussion/comment/955512#Comment_955512

So the patch information would be downloaded alongside the resource, temporarily used, then deleted? (Just looking for clarification that disk space usage won't be increasing.)

The patch information will be built into the resource file. I don't currently have information about the impact on file size.

Sorry for assuming that no one had time to already address this.

No problem. The issue is one that's been around for a while, and some of the pieces of the solution have been in place for some time. It hasn't been a high priority to complete the implementation because it almost never causes a problem. Happily, changes required for the upcoming new notes system prompted finishing things up.

Dave Hooton

https://community.logos.com/discussion/comment/955489#Comment_955489

With regards to backing up your data, what I was trying to say about lacking a practical backup mechanism is that while you can backup the local files on your machine, there's no way for you to restore that state of your data if you have subsequently made any changes to this data.

The (older) backup data would be overwritten by the newer data from the server and you would still have the same issue! But if you start Logos in Offline mode you would continue to use the backup data. If you then manually fix the data or make other changes, that (newer) data would not be overwritten when you start Logos normally (online).

Andrew Batishko

https://community.logos.com/discussion/comment/955521#Comment_955521

The (older) backup data would be overwritten by the newer data from the server and you would still have the same issue! But if you start Logos in Offline mode you would continue to use the backup data. If you then manually fix the data or make other changes, that (newer) data would not be overwritten when you start Logos normally (online).

This is correct, provided that you make some kind of change to each piece of data that you don't want overwritten by the server. In the case of having had sweeping changes to all (or most) of your notes, this would mean making some kind of edit each note. I would call that "not practical". Alternatively, if you don't know what may have changed, then you'd be pretty much out of luck.

PetahChristian

https://community.logos.com/discussion/comment/955515#Comment_955515

The patch information will be built into the resource file. I don't currently have information about the impact on file size.

Does that mean that every resource in our library would need to be downloaded, and reindexed, even if we've never highlighted the majority of them?

Can there be some additional benefit (beyond the anticipated new notes system) to any update of an entire library, such as FL also adding missing hyperlinks to resources which became available after the original had been converted?

MJ. Smith

https://community.logos.com/discussion/comment/955523#Comment_955523

Does that mean that every resource in our library

I believe that he is talking only of resources in which a rebuild by Faithlife caused a piece of text to move from one article to another.

Andrew Batishko

https://community.logos.com/discussion/comment/955523#Comment_955523

The patch information will be built into the resource file. I don't currently have information about the impact on file size.

Does that mean that every resource in our library would need to be downloaded, and reindexed, even if we've never highlighted the majority of them?

Can there be some additional benefit (beyond the anticipated new notes system) to any update of an entire library, such as FL also adding missing hyperlinks to resources which became available after the original had been converted?

I don't know what the plan is for rolling this out. It's possible that no plan has been decided on yet. One possibility would be similar to when sentence boundary data was added to resources. The most popular resources were updated, while the remainder were automatically updated the next time the resource was updated for any other reason.

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/955523#Comment_955523

The patch information will be built into the resource file. I don't currently have information about the impact on file size.

Preliminary investigation has shown that this may lead to an average 0.64% increase in file size. This estimate was created by analysing some of the “worst-case” resources (i.e., ones that we know have had many versions, lots of textual changes, or complex articles) so it's probably on the high side. (I suspect it's likely that there will be tens of thousands of resources that increase in size by less than 0.1%.)

Does that mean that every resource in our library would need to be downloaded, and reindexed, even if we've never highlighted the majority of them?

No, this information is only necessary if a resource is rebuilt with textual changes, so (in general) it only needs to go in future versions of the resources. We don't have to rebuild all existing resources “just because”.

However, if we have information that leads us to believe that many users have lost highlights in a particular resource, we might choose to proactively rebuild it; the updated version would include the information that can automatically restore the highlights. Let me give an example. Let's say that many users highlighted v1 of a particular Faithlife Ebooks resource. It was then automatically rebuilt to fix the TOC (v2) but the change was significant enough that the application couldn't display highlights that were taken on v1 (Andrew described this situation earlier). We might choose to rebuild and ship v3, which would be exactly the same as v2 except that it would include “position upgrade tables” for v1→v3 and v2→v3. This would allow the application to display highlights from all prior versions of the resource. Because the text of the resource itself hasn't changed, our existing "resource patching" technology should make this a very small download.

Can there be some additional benefit (beyond the anticipated new notes system) to any update of an entire library, such as FL also adding missing hyperlinks to resources which became available after the original had been converted?

If we have to do a mass rebuild of resources, we do try to see how much work we can do at once, to minimise the total number of updates and downloads.

PetahChristian

https://community.logos.com/discussion/comment/955532#Comment_955532

No, this information is only necessary if a resource is rebuilt with textual changes, so (in general) it only needs to go in future versions of the resources. We don't have to rebuild all existing resources “just because”.

Thanks for confirming what MJ had shared.

If we have to do a mass rebuild of resources, we do try to see how much work we can do at once, to minimise the total number of updates and downloads.

Thanks very much. I am grateful not only for all the improvements, but that FL also cares to minimize any impact.

scooter

https://community.logos.com/discussion/comment/955532#Comment_955532

However, if we have information that leads us to believe that many users have lost highlights in a particular resource, we might choose to proactively rebuild it; the updated version would include the information that can automatically restore the highlights.

I highlight ALL books I read. The following is in Logos. I finished reading Schreiner's ''New Testament Theology'' 2011 May. The only highlights in the book are ones I created in 2018; all 2011's are gone.

Hours of highlighting.. Can FL restore them?

PetahChristian

https://community.logos.com/discussion/comment/955539#Comment_955539

scooter, did you try the command posted earlier in the thread?

JT (alabama24)

https://community.logos.com/discussion/comment/955539#Comment_955539

However, if we have information that leads us to believe that many users have lost highlights in a particular resource, we might choose to proactively rebuild it; the updated version would include the information that can automatically restore the highlights.

I highlight ALL books I read. The following is in Logos. I finished reading Schreiner's ''New Testament Theology'' 2011 May. The only highlights in the book are ones I created in 2018; all 2011's are gone.

Hours of highlighting.. Can FL restore them?

2011 is a long time ago. I say that because the system for highlights changed at some point. Do you know which document you used, or was it before documents were being used? If The latter, the notes would have automatically been placed into a pallet specific note document. Are you familiar with how that works? Have you made sure the note document is still present? Turned on in the visual filter menu?