Bug: very very very strange results with sync.
Because of the performance issues, I keep the internet setting turned off. I will turn it on after I have purchased a new book.
To get my notes from my laptop to my desktop, I use L3 for all of my notes. I then use a program that I use to sync all of my documents to move my L3 notes from one computer to the other. When I want my updated notes in L4, I delete the notes in L4 and then re-import them into L4.
Because I wanted to sync some of my other settings in L4, I deleted all of my notes on both computers. I then imported my notes into L4 on my laptop. I then turn on the internet setting to sync my info. After a day and a half or so, L4 on my laptop stopped syncing.
I then turned the internet setting off on my laptop and turned the setting on for the desktop. After running close to a week, the desktop was still syncing (as indicated by the icon), but nothing was actually syncing (that I can tell). Here are some screen prints.
Laptop #1:
Desktop 1a and 1b:
Not only did my notes lose their names, I do not have access to all of my notes, my visual filters, and sentence diagrams.
Note on my notes: The db files are close (not exact) to the same size.
Here are my highlights:
laptop #2:
Desktop #2:
I am note sure what happened here, and why I have "dup." palettes and I am missing others.
My layouts also did not sync.
laptop #3:
Desktop #3:
A lot of data can be transfered in a weeks time. Not only would I think everything would be completed by now, I am surprised that my notes have lost their names.
FYI... I did not have logging enabled when I did this, but I will do this all over again with logging when the next beta comes out.
Comments
-
Scary
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
ouch
0 -
If you have one machine that still has a correct .db file, make a copy of that now, for safety. If you delete notes on L4, and then sync, it will naturally delete them on the other machines.
I think if I was you I would close Logos on all machines, and delete all the notes/highlighting/etc. .db files from the machines that are causing problems, and just leave them on one machine. Then i'd make sure that one machine was up-to-date with its syncing. Then, one at a time, I'd restart Logos, and let it sync (download) the notes data etc. Once that had finished, I'd do it again on the next machine, and so on.
In the future, I would never delete note files unless I wanted those deletions synced.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
If you have one machine that still has a correct .db file, make a copy of that now, for safety. If you delete notes on L4, and then sync, it will naturally delete them on the other machines.
I think if I was you I would close Logos on all machines, and delete all the notes/highlighting/etc. .db files from the machines that are causing problems, and just leave them on one machine. Then i'd make sure that one machine was up-to-date with its syncing. Then, one at a time, I'd restart Logos, and let it sync (download) the notes data etc. Once that had finished, I'd do it again on the next machine, and so on.
In the future, I would never delete note files unless I wanted those deletions synced.
Like I said in my original post, I am going to see if I can recreate this issue with logging enabled. This will give the developers something more to look at.
What I was trying to do is to move my highlighting and filters from my laptop to my desktop. If I get the same problem next time, I will simply delete everything off my desktop along with deleting my notes and sentence diagrams from laptop. Sync the two computers so that I will have my highlighting and filters on my desktop. Then I will Import everything again on both computers.
0 -
Can we assume that you're shutting down Logos on your machines so that only one instance of Logos (with your user ID) is running at any given time?
Help links: WIKI; Logos 6 FAQ. (Phil. 2:14, NIV)
0 -
Richard DeRuiter said:
Can we assume that you're shutting down Logos on your machines so that only one instance of Logos (with your user ID) is running at any given time?
Yes, I only had one computer calling home at a time. I first did my laptop. Once it finished calling home, I let my desktop call home. The problem is that the desktop never stopped talking.
0 -
I am in the process of recreating this issue, and I think the log file from my laptop will help explain what is going on with L4.
This is what I did, I deleted all of my personal (notes, layouts, highlighting, etc...) off of my desktop and I deleted my notes and sentence diagrams from my laptop.
Here are some screen prints of my laptop:
File menu:
My custom highlighting:
Then I imported my notes and sentence diagrams from L3:
I then turn sync on my laptop, and it has been running for close to four days now, and it is still syncing.
Here is a screen shot that has the log file and the notes db file being displayed. Noticed that the notes.db file has not changed in over 24 hours while the notes-journal file is always being updated. When you look at the log file, it states that it is downloading notes the entire time. NOTE: the notes.db file did doubled in size while L4 has been syncing. So at one time, notes were being added to the notes.db file.
Here is the log file:1830.Logos4.zip
0 -
Here's what I think is happening. The log file shows that the server is trying to sync notes back from 23rd February - presumably a day when you deleted or uploaded thousands of notes. It's likely that the server is trying to propagate those deletions to this machine - even though the notes no longer exist. That's probably why the notes file isn't growing.
It's worth noting you've got half a gigabytes in your notes database. When you remember that sync traffic adds significant overhead, there is a huge amount to upload/download if you change all your data.
To be sure, you'd need to watch traffic between your installation and Logos' servers. If you want to do this you can install Fiddler2. You'll need to enable https decryption.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
Here's what I think is happening. The log file shows that the server is trying to sync notes back from 23rd February - presumably a day when you deleted or uploaded thousands of notes. It's likely that the server is trying to propagate those deletions to this machine - even though the notes no longer exist. That's probably why the notes file isn't growing.
It's worth noting you've got half a gigabytes in your notes database. When you remember that sync traffic adds significant overhead, there is a huge amount to upload/download if you change all your data.
To be sure, you'd need to watch traffic between your installation and Logos' servers. If you want to do this you can install Fiddler2. You'll need to enable https decryption.
I was thinking the same, and I have two questions. #1 - Why is L4 is downloading deleted notes? True, we need a way to restore deleted notes, but we do not need to keep deleted notes forever.
#2 - why did the notes.db file stop getting updated? I have been downloading notes for more than 24 hours without the notes.db file being updated.
0 -
tom collinge said:
I was thinking the same, and I have two questions. #1 - Why is L4 is downloading deleted notes? True, we need a way to restore deleted notes, but we do not need to keep deleted notes forever.
#2 - why did the notes.db file stop getting updated? I have been downloading notes for more than 24 hours without the notes.db file being updated.
We don't know for certain that it is downloading notes. It may just be downloading the command to delete the note.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:tom collinge said:
I was thinking the same, and I have two questions. #1 - Why is L4 is downloading deleted notes? True, we need a way to restore deleted notes, but we do not need to keep deleted notes forever.
#2 - why did the notes.db file stop getting updated? I have been downloading notes for more than 24 hours without the notes.db file being updated.
We don't know for certain that it is downloading notes. It may just be downloading the command to delete the note.
Then why did my note file almost double in size?
0 -
tom collinge said:
Then why did my note file almost double in size?
I think I might know why. Reading through your steps, it seems that there may be a step missing: "After removing notes, restart L4 and let it finish syncing on both machines." This step will remove the old notes from the server and, I think, significantly speed things up.
Another thing I'm seeing in your log file are a lot of errors with RichText links ("Error RichTextUriLink Uri is not valid:"), this also seems to be slowing down the syncing process. If you could identify those errors in your Libronix notes and remove them, that could also speed things up considerably.
I'm wondering if changing strategies here might be worth the effort. I notice that many annotations are not being updated, but the sync process checks each one anyway. What would happen if you only deleted and imported those notes that were changed from your last session? I don't know of an L4 command to import a specific note, but if you just moved the notes you didn't change out of the target folder (temporarily, of course) and issued your update command, the process could be speeded up quite a bit.
On the other hand, as Mark says, the amount of content here may be the problem, and it will continue to be a problem regardless of the steps we take. I just don't have the necessary inside information on how syncing works in L4 to know. (Have you considered purchasing the PBB when it comes out?)
Help links: WIKI; Logos 6 FAQ. (Phil. 2:14, NIV)
0 -
Richard DeRuiter said:tom collinge said:
Then why did my note file almost double in size?
I think I might know why. Reading through your steps, it seems that there may be a step missing: "After removing notes, restart L4 and let it finish syncing on both machines." This step will remove the old notes from the server and, I think, significantly speed things up.
Another thing I'm seeing in your log file are a lot of errors with RichText links ("Error RichTextUriLink Uri is not valid:"), this also seems to be slowing down the syncing process. If you could identify those errors in your Libronix notes and remove them, that could also speed things up considerably.
I'm wondering if changing strategies here might be worth the effort. I notice that many annotations are not being updated, but the sync process checks each one anyway. What would happen if you only deleted and imported those notes that were changed from your last session? I don't know of an L4 command to import a specific note, but if you just moved the notes you didn't change out of the target folder (temporarily, of course) and issued your update command, the process could be speeded up quite a bit.
On the other hand, as Mark says, the amount of content here may be the problem, and it will continue to be a problem regardless of the steps we take. I just don't have the necessary inside information on how syncing works in L4 to know. (Have you considered purchasing the PBB when it comes out?)
Hello Richard, and thanks for suggestions.
I import my notes from L3 into L4 about once a month. While I would have a good idea of the notes that I have added over the past month, I simply cannot remember every single note that is a new note. There could be a way to separate my new notes from my old notes, but this will also increase my time.
I too noticed the many errors concerning uri, but these errors are coming from just a few "deleted" notes from L4 server. I tried to figure out which notes are causing this problem by trying to match the uuid with the value in my notes.db file, but I cannot find a match. Still, these errors do not add a lot of time to the sync process overall. What is making sync run for such a long time is sync. It is taking anywhere from five to ten seconds to check and update each note file. At most, it should only take a second to download/upload the record, compare the date/time stamps, and to update the record.
I do not believe the amount of content is the issue. I believe the amount of content is showing the problems of an inefficient sync process.
Have I thought about PBB, yes I have. I would say about a third to maybe half of my notes would be good in a PBB. Still, I am not taking the time to convert my notes that I have created over the years into a PBB. I have my work flow for my sermon/worship prep, and it works for me.
FYI... The only reason why I am syncing my notes is so that I can sync my other documents. I already have a very efficient system to sync my notes (and it uses a different syncing program). It takes about two to three hours to have all of my notes updated on all of my computers. (FYI... My laptop is still trying to sync my laptop notes to L4 servers, and it is approaching six days now that L4 on my laptop has been trying to accomplish this task.
0 -
tom collinge said:
I import my notes from L3 into L4 about once a month. While I would have a good idea of the notes that I have added over the past month, I simply cannot remember every single note that is a new note. There could be a way to separate my new notes from my old notes, but this will also increase my time.
Since the Libronix notes are stored in a unique folder, you could simply sort them in Windows Explorer by "Date modified" and figure out pretty quickly which ones were changed since your last update. You could at least eliminate those notes that would make good PBB's, since they likely never change, and (my guess) are likely among the largest files.
tom collinge said:I do not believe the amount of content is the issue. I believe the amount of content is showing the problems of an inefficient sync process.
Which means that for now the amount of content is a problem. Reducing the amount that must sync is likely to have positive effects on that initial sync after updating your notes from Libronix, even (especially?) if the syncing process is inefficient. I believe that sync only deals with notes it detects have been changed since the last sync. Since you're changing everything, everything is being checked. This is very time consuming, as you noted. (The turn-around time (pause) between the batch checks (of up to 100 at a time), is likely a server issue, rather than programming, but the result is the same regardless.)
BTW, did you consider my suggestion of syncing both computers after deleting the notes and before re-importing them?
Help links: WIKI; Logos 6 FAQ. (Phil. 2:14, NIV)
0 -
As mentioned in previous posts, Logos is currently working on improvements to the Notes sync process.
0 -
While the designers and programmers are working on it, please add local backup and restore of user-data. I still can't believe that this is not a priority.
0 -
JimT said:
While the designers and programmers are working on it, please add local backup and restore of user-data. I still can't believe that this is not a priority.
Bob Pritchett just addressed this in a post on another thread today:
Bob Pritchett said:Mark A. Smith said:The ability to do local backup and restore has also been asked for a number of times.
I don't want to rehash the conversations (they're still online in old
forum posts), but I can say that while this isn't off the table, it's
also not being worked on right now.0 -
Thanks Rosie. Sadly, a disapointing answer from Bob on a number of points, including still no protection for user-data, and no way to be sure if the Apple / Android portion was a tease, joking, of he just does not yet know where Logos stands re Apple and license terms.
0 -
Melissa Snyder said:
As mentioned in previous posts, Logos is currently working on improvements to the Notes sync process.
I have had to reboot my laptop twice now. After nine days, my laptop is till syncing. My notes.db file has increased in its size again, but I still do not know what notes are downloading.
Here is the log file between the two reboots: 3806.Logos4_pt2.zip
0 -
Logos crashed while it was syncing, and here is the latest log and crash reports: 3225.Logos Log Files.zip
0 -
L4 crashed again while syncing, but no crash log was produced this time (Windows closed L4 because L4 stop responding). Here is the latest log file.0172.Logos4.zip
0 -
tom collinge said:
L4 crashed again while syncing, but no crash log was produced this time (Windows closed L4 because L4 stop responding). Here is the latest log file.0172.Logos4.zip
Same thing happened again, so here are my latest logs2604.Logos Log Files.zip
FYI... over two and a half weeks of syncing note files, and the end is still not in sight.
0 -
tom collinge said:tom collinge said:
L4 crashed again while syncing, but no crash log was produced this time (Windows closed L4 because L4 stop responding). Here is the latest log file.0172.Logos4.zip
Same thing happened again, so here are my latest logs2604.Logos Log Files.zip
FYI... over two and a half weeks of syncing note files, and the end is still not in sight.
L4 crashed again while syncing. This time it did produced a crash report about being out of memory.
I did noticed that L4 is now downloading notes that it uploaded to L4's server when I started this process. I do not know why it is downloading notes that it "just" uploaded because I have not changed any of my notes. If I did change my notes, I would expect that L4 would upload them to the L4 servers instead of downloading them.
0 -
tom collinge said:
L4 crashed again while syncing. This time it did produced a crash report about being out of memory.
I did noticed that L4 is now downloading notes that it uploaded to L4's server when I started this process. I do not know why it is downloading notes that it "just" uploaded because I have not changed any of my notes. If I did change my notes, I would expect that L4 would upload them to the L4 servers instead of downloading them.
Well, I thought the process was coming close to the end. Now L4 is downloaded deleted notes that it has already downloaded in this process. Here is the current crash (out of memory again) report and log files
0 -
tom collinge said:
Well, I thought the process was coming close to the end. Now L4 is downloaded deleted notes that it has already downloaded in this process. Here is the current crash (out of memory again) report and log files
FYI...L4 is now downloading the same deleted notes for the third time. I sure hope that the new sync system is more efficient.
IMHO... downloading deleted notes is odd to me. Downloading the same deleted notes three times = bad programing.
0 -
tom collinge said:
FYI...L4 is now downloading the same deleted notes for the third time. I sure hope that the new sync system is more efficient.
IMHO... downloading deleted notes is odd to me. Downloading the same deleted notes three times = bad programing.
After 28 days and five crashes (InvalidOperationException, 2 OutOfMemoryExceptions, and windows closed L4 twice because L4 stop responding), L4 has finally stopped syncing.
Here is the last log of this sync process: 7127.Logos4.zip
I am going to update L4 to the latest beta, and do this process all over again with my laptop to see how the new sync process works.
0 -
tom collinge said:
I am going to update L4 to the latest beta, and do this process all over again with my laptop to see how the new sync process works.
You're a brave man, Tom.
Dave
===Windows 11 & Android 13
0 -
tom collinge said:
I am going to update L4 to the latest beta, and do this process all over again with my laptop to see how the new sync process works.
Because L4 crashed (PBB issue) here is my first log. L4 was still in the process of importing my notes.
0 -
tom collinge said:
Because L4 crashed (PBB issue) here is my first log. L4 was still in the process of importing my notes.
L4 crashed again for being out of memory.
I had to divide the log into two files so that it will upload.0121.Logos4_#2b.zip
0 -
Tom, it feels like you're speaking into an echo chamber here. I wonder if anyone from Logos is still following this thread? I hope so. You're going to lots of trouble to continue documenting this problem which they already said they're working on (hopefully) fixing. I hope it does fix it for you. You've been incredibly patient and longsuffering (with emphasis on the suffering)
0 -
Rosie Perera said:
Tom, it feels like you're speaking into an echo chamber here. I wonder if anyone from Logos is still following this thread? I hope so. You're going to lots of trouble to continue documenting this problem which they already said they're working on (hopefully) fixing. I hope it does fix it for you. You've been incredibly patient and longsuffering (with emphasis on the suffering)
Thanks Rosie,
Because of the amount of notes that I have and the people who design L4 probably think people would not have the notes that I have, I thought I would provide as much data as I can to Logos.
Because I still have logging enabled, I am still importing my notes. I cannot wait until this has finished so that I can test the new sync process.
FYI... I know at least two Logos employees know about this thread. If they are still reading it, I do not know. Still, this is not going to stop me from providing them with my data.
0 -
tom collinge said:
L4 crashed again for being out of memory.
I had to divide the log into two files so that it will upload.0121.Logos4_#2b.zip
L4 crashed for the second time already. Again, it it was for being out of memory.
Here is the log file, and again I had to divide the file into smaller files to upload it to the server.
0 -
tom collinge said:
L4 crashed for the second time already. Again, it it was for being out of memory.
Here is the log file, and again I had to divide the file into smaller files to upload it to the server.
Because I have received two out of memory crashes in two days, I turn off the setting to use the internet. I did this so I could import all of my notes.
I will turn the setting back on to see how the new sync works.
0 -
tom collinge said:
Because I have received two out of memory crashes in two days, I turn off the setting to use the internet. I did this so I could import all of my notes.
I will turn the setting back on to see how the new sync works.
Once I turn the use internet setting to no, I was able to import my notes. Everything now seems to be working.
0 -
tom collinge said:
Once I turn the use internet setting to no, I was able to import my notes. Everything now seems to be working.
I'm glad. I was starting to feel sorry for you.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
MJ. Smith said:
I'm glad. I was starting to feel sorry for you.
Thanks.
This does tell me that the new sync does not work with the import all command. (That is the new sync does not work with the Import All command when you have 38,445 notes that needs 288MB on the hard drive.)
0 -
tom collinge said:
This does tell me that the new sync does not work with the import all command. (That is the new sync does not work with the Import All command when you have 38,445 notes that needs 288MB on the hard drive.)
The only new sync implemented as of Beta 6 is for Copy Bible Verses. New sync for Notes is still coming.
0 -
Melissa Snyder said:
The only new sync implemented as of Beta 6 is for Copy Bible Verses. New sync for Notes is still coming.
Thanks for the update Melissa.
I was going to say that I have not seen any changes so far in the process. I will stop what I am doing until the new sync for notes comes out.
0 -
Melissa Snyder said:
The only new sync implemented as of Beta 6 is for Copy Bible Verses. New sync for Notes is still coming.
Hi Melissa,
I am impressed on how you have improved the speed of syncing notes. With the new beta, it took L4 under 24 hours to sync everything. Now I am going to see how much sync impacts the running of L4.
I did notice that clippings still take a very long time to upload. Is the change that was made to notes going to be made to clippings?
0 -
tom collinge said:
I did notice that clippings still take a very long time to upload. Is the change that was made to notes going to be made to clippings?
Check to see if this is faster in Beta 9.
0 -
0
-
I have not had the time to checked clippings, but I have finally got around to sync my desktop to my laptop.
For some reason, I now have an additional 253 note files.
Here is a screen print of my laptop:
Here is a screen print on how my desktop started:
Here is a screen print of my current desktop:
I tried to upload my log file, but the zip file is too large. Because I need to leave my house, I will upload the log file when I get back home tonight.
0 -
Here is the log file for my desktop syncing:
Now on my laptop, I am getting these unnamed notes. I am assuming this is because my desktop L4 uploaded these old notes right after L4 downloaded them. Why hasn't my laptop downloaded these before now?
0
