Thanks for "delete unlicensed resources"

Page 1 of 2 (30 items) 1 2 Next >
This post has 29 Replies | 0 Followers

Posts 27232
Forum MVP
Dave Hooton | Forum Activity | Posted: Tue, Jun 9 2015 2:19 AM

The new "delete unlicensed resources” command has removed 17 files and recovered 1 GB of space!

Dave
===

Windows 10 & Android 8

Posts 1991
mike | Forum Activity | Replied: Tue, Jun 9 2015 4:08 AM

17 files for 1gb.. that's big. what were they?

Posts 3771
Francis | Forum Activity | Replied: Tue, Jun 9 2015 4:27 AM

Yes

Posts 27232
Forum MVP
Dave Hooton | Forum Activity | Replied: Tue, Jun 9 2015 5:51 AM

mike:

17 files for 1gb.. that's big. what were they?

I don't know, they got deletedBig Smile - from past beta trials

Dave
===

Windows 10 & Android 8

Posts 12149
Forum MVP
NB.Mick | Forum Activity | Replied: Tue, Jun 9 2015 7:44 AM

Okay Dave, Big Smile I see this as an invitation to switch roles. You play the NewbieMick card, getting all excited over a new Logos feature. I put on the veteran cap of someone who has seen all the ways Logos has messed up or not developed to the full extent the many good-intended features, helped others get around the most in-obvious pitfalls and looks up from down under with a wrinkled brow and holds us all (and especially Faithlife) to a high standard of quality. So I try to post in a Dave-like spirit:  

I don't get the excitement. In a move unusual but understandable given the hoopla over the Logos Cloud download, Logos is touting this beta feature everywhere outside of beta forum and the release notes only say "this exists" and link to the promise of this feature in a Cloud-forum thread.

No one has discussed how this thing intends to work. No one has discussed the utter lack of documentation. No one has discussed a testing strategy or posted about a test conducted with the beta (and just that someone has a number of files less and a couple of MB more on his HD is no test that qualifies the Australian norm). 

The minimum documentation we would need is at least a sentence like:

This command will put into the OS trash all files with filetypes *.logos4, *.lbx............. from the folder(s) xyz\ResourceManager\Resources and ..... that are not part of currently licenced library resources, as well as the entries for those resources from the library database; empty the trash and re-index to reclaim space on your HD.

or

based on an initial run of update resources this command will update the library database and eliminate all files from all folders existing under the folder  xyz\ResourceManager that are not listed in the database table as belonging to the licenced resources and datasets. 

Beta tests could then make sure

  • yes, those files are removed (one could place some from the FTP-server into the folder to make sure or create mock files - does this make a difference, ie. are these really Logos files or just named as such?)
  • more importantly: no legitimately licenced files are removed, and no other files are removed (one could have ABC.logos4 - licenced - and ABC.logos4.bak - a legitimate copy of the licenced file and ABC-myveryimportantbackup.logos4 - probably this will be removed - and thisisimportanttoremember.docx for some reason in the folder/folder structure affected by the command)
  • does removal mean they are gone from the HD, or are they in the hidden OS trash folder to get back in case one thinks again over this?     
  • ....

All this in the beta - with the potential to refine any issues, perhaps even set up a wiki page which explains the whereabouts and possibly some caveats - before people get frustrated with Logos Cloud, switch to beta, issue the command and due to some glitch are missing their only working Logos installation plus everything they saved in a Logos folder (e.g. as a backup for just this case).

okay, I hope this does it in time for second breakfast at Bellingham, now switching back:

Nice feature! Great to see it coming prior to end of June! 

Wink

Running Logos 9 latest (beta) version on Win 10

Posts 2816
LogosEmployee
Andrew Batishko | Forum Activity | Replied: Tue, Jun 9 2015 8:32 AM

Dave Hooton:

mike:

17 files for 1gb.. that's big. what were they?

I don't know, they got deletedBig Smile - from past beta trials

The files that get deleted are listed in the log file. Search for "Deleted unlicensed file:"

Andrew Batishko | Faithlife software developer

Posts 2816
LogosEmployee
Andrew Batishko | Forum Activity | Replied: Tue, Jun 9 2015 8:37 AM

NB.Mick:

The minimum documentation we would need is at least a sentence like:

This command permanently deletes all files in xyz\ResourceManager\Resources\ that can not be opened as valid resources. This will include resources for which you no longer have a valid license. It will not include duplicate copies of valid resource files.

Andrew Batishko | Faithlife software developer

Posts 12149
Forum MVP
NB.Mick | Forum Activity | Replied: Tue, Jun 9 2015 9:08 AM

Thanks, Andrew!

Running Logos 9 latest (beta) version on Win 10

Posts 27232
Forum MVP
Dave Hooton | Forum Activity | Replied: Wed, Jun 10 2015 12:26 AM

NB.Mick:

Dave Hooton:
I don't know, they got deletedBig Smile - from past beta trials.

Okay Dave, Big Smile I see this as an invitation to switch roles.

...

...

Beta tests could then make sure...

My reply to Mike was Big Smilecasual. I knew what resource files got deleted (via a third party utility), and that they weren't licensed resources because my Library resource count was unchanged and I didn't get a subsequent download! But I didn't know exactly where they came from until I tried the command on another installation, which showed me that the majority came from a cancelled "Cloud" download before I opted out of the trial/early release; hence the rather large "couple of MB" saved on the HD (nothing in trash).

Dave
===

Windows 10 & Android 8

Posts 12149
Forum MVP
NB.Mick | Forum Activity | Replied: Tue, Jun 16 2015 2:14 PM

Andrew Batishko (Faithlife):
This command permanently deletes all files in xyz\ResourceManager\Resources\ that can not be opened as valid resources. This will include resources for which you no longer have a valid license. It will not include duplicate copies of valid resource files.

Okay, I tested that today. It deleted 18 files from ResourceManager\Resources and it (unexpectedly) reduced my library count by one which is not part of the cloud try-out. I can only guess which book, since I did not keep track with a bibliography or tool (my risk, I know).

The 18 files were  

  • 4 from https://www.logos.com/product/36318/select-works-of-raymond-e-brown - the Cloud "adjustment"
  • 6 from a former private Beta, including three authored by D.L. Moody, a Pilgrim's Progress edition, and two Lexham resources 
  • 2 (HebrewPronunciationsButh.lbsprn and AramaicPronunciationsButh.lbsprn) from https://www.logos.com/product/5960/hebrew-audio-pronunciations - never knew I had them, maybe from that beta, too, but: L6 gives the "you don't have a licence" message tab, whereas with others it gives the "This resource is no longer in the library" message tab that even has a working info button.
  • 1 Spanish-language people diagram resource (recall that from an earlier beta, they now go with UI language, I think)
  • 1 RCSOVB.lix - the Welcome to the RC Sproul Library file. This one is maybe my loss in the library, since it has communitytags ("sproul") and says I read 2% of it. It may be that there's an error in that the deleter doesn't verify this filetype. Update resources didn't bring this or any other resource back.
  • 2 files I put into the folder on purpose, one a Word document with .docx, one a txt with fake .logos4 type
  • 2 files that were instances of Youcat.logos4 - interestingly it deleted one with appended .bak ending without ado, whereas it compared two instances of Youcat that ended with .logos4 and kept the copy, deleted the original - but the copy works!  

So it seems it really deletes all files that either don't end with a valid ending from a short list or (if they have such an ending) can't be opened for technical / current license reasons. Regarding duplicate copies of valid files: it keeps only the most recent. So it's not advisable to store in this folder any backup copies of licensed files. Hopefully nobody used this folder for PB sources!

Besides the library count reduction it seems everything is fine. We should create a wiki page for this command which gives the 'documentation' and caveats for users wanting to try that.  

Running Logos 9 latest (beta) version on Win 10

Posts 27232
Forum MVP
Dave Hooton | Forum Activity | Replied: Tue, Jun 16 2015 2:41 PM

NB.Mick:
Hopefully nobody used this folder for PB sources!

Hopefully, nobody was silly enough to do that; especially as it is a hidden folder.

NB.Mick:
We should create a wiki page for this command

It's already in https://wiki.logos.com/Logos_6_Commands

Dave
===

Windows 10 & Android 8

Posts 2816
LogosEmployee
Andrew Batishko | Forum Activity | Replied: Tue, Jun 16 2015 4:09 PM

Thanks for the report, NB.Mick. I'm doing some investigation.

NB.Mick:

RCSOVB.lix - the Welcome to the RC Sproul Library file. This one is maybe my loss in the library, since it has communitytags ("sproul") and says I read 2% of it. It may be that there's an error in that the deleter doesn't verify this filetype. Update resources didn't bring this or any other resource back.

Does this file show up in your library? If so, can you open it? Do you have the file RCSOVB.lsf in your Resources folder?

Andrew Batishko | Faithlife software developer

Posts 2816
LogosEmployee
Andrew Batishko | Forum Activity | Replied: Tue, Jun 16 2015 5:04 PM

NB.Mick:

RCSOVB.lix - the Welcome to the RC Sproul Library file. This one is maybe my loss in the library, since it has communitytags ("sproul") and says I read 2% of it. It may be that there's an error in that the deleter doesn't verify this filetype. Update resources didn't bring this or any other resource back.

Also, do you happen to still have your log file from the session when the files were deleted?

Andrew Batishko | Faithlife software developer

Posts 12149
Forum MVP
NB.Mick | Forum Activity | Replied: Wed, Jun 17 2015 4:45 AM

Andrew Batishko (Faithlife):

NB.Mick:

RCSOVB.lix - the Welcome to the RC Sproul Library file. This one is maybe my loss in the library, since it has communitytags ("sproul") and says I read 2% of it. It may be that there's an error in that the deleter doesn't verify this filetype. Update resources didn't bring this or any other resource back. But maybe that's just because this file is not on your ftp server (no rcso* files at all - and no youcat either! Glad my testing didn't kill that)

Does this file show up in your library? If so, can you open it? Do you have the file RCSOVB.lsf in your Resources folder?

If "this file" is RCSOVB.lix: it can't show up, since L6 deleted it.

I do have RCSOVB.lsf in my resources folder - which is strange since it can't be opened as a valid resource so probably the code should have deleted both or neither. 

When I use command logosres:rcsovb.lsf (orange) or logosres:rcsovb.lix (red) I will be told that I have no licence - which we all know is the wrong error message since it just means "can't open this for any reason (file corrupt, file missing, licence issue ...)" but with logosres:rcsovb (green) I get the "This resource is no longer in your library" tab with working info-button, community tag and reading progress.

EDIT:

For curiosity, I copied the RSCOVB.lix file from an older version of the resources folder (I hadn't done a full backup of the 35 GB directly before - which I might have should) and after a Logos restart, it appears back in my library  

 

Running Logos 9 latest (beta) version on Win 10

Posts 2816
LogosEmployee
Andrew Batishko | Forum Activity | Replied: Wed, Jun 17 2015 6:17 AM

Your description both matches what I would expect to see, and puzzles me.

RSCOVB is a very old resource that requires two different files in order to open (.lsf and .lix). When the .lix file was deleted, it no longer allowed the resource to be open despite the fact that the .lsf was still there.

It's no surprise that the logosres:rcsovb.lsf and logosres:rcsovb.lix commands indicate no license, because the command requires a resource ID not a filename.

I had actually tested the delete command with the old resource files before release, and tested it again after your post yesterday (with a different resource), and in both cases the .lix file was not deleted.

I haven't been able to get my hands on a copy of this resource yet. But I'll be doing some more investigation today. Would you mind emailing the following files to logosbugs@logos.com?

  • Data\<random>\ResourceManager\logos\Discovered.db
  • Data\<random>\ResourceManager\logos\Identified.db

Andrew Batishko | Faithlife software developer

Posts 12149
Forum MVP
NB.Mick | Forum Activity | Replied: Wed, Jun 17 2015 6:59 AM

Andrew Batishko (Faithlife):
do you happen to still have your log file from the session when the files were deleted?

Unfortunately, I think I don't. I copied some interesting lines and commented them to reconcile the file count, but I fear I did not keep all of it. See attached what I have:

2043.new 2.txt 

Since this is mainly for the RCSOVB.lix file and whether it signs off on the RCSOVB.lsf: since I again have both files, I ran the command again (twice: found it did not delete the resource while it was opened). Logs: 5732.Logos Logs for delete run.zip

At line 42993 in the Logos log, RCSOVB.LSF is identified, whereas RCSOVB.lix is again deleted (in line 45119, plus following deletion of a really unlicenced resource ZNIBD I put in from the ftp server and the logs always claim deleting a subfolder Resources\ScannedResources even if it wasn't there). The library resource count decreased by one and the Sproul resource is no longer there.

Hope this helps fixing this!

Running Logos 9 latest (beta) version on Win 10

Posts 12149
Forum MVP
NB.Mick | Forum Activity | Replied: Wed, Jun 17 2015 7:10 AM

Andrew Batishko (Faithlife):

I'll be doing some more investigation today. Would you mind emailing the following files to logosbugs@logos.com?

  • Data\<random>\ResourceManager\logos\Discovered.db
  • Data\<random>\ResourceManager\logos\Identified.db

done, and I put the lix file into it, too, for your testing. 

Running Logos 9 latest (beta) version on Win 10

Posts 2816
LogosEmployee
Andrew Batishko | Forum Activity | Replied: Wed, Jun 17 2015 9:59 AM

NB.Mick:

Hope this helps fixing this!

OK, rename RCSOVB.lix to RCSOVB.LIX (note the case difference). Make sure that the case is also correct on RCSOVB.LSF. Then start the application and try the command again. The file should no longer be deleted.

I'm figuring out the correct fix for this problem. Thank you so much for your help in tracking it down.

Andrew Batishko | Faithlife software developer

Posts 12149
Forum MVP
NB.Mick | Forum Activity | Replied: Wed, Jun 17 2015 11:17 AM

Andrew Batishko (Faithlife):

NB.Mick:

Hope this helps fixing this!

OK, rename RCSOVB.lix to RCSOVB.LIX (note the case difference). Make sure that the case is also correct on RCSOVB.LSF. Then start the application and try the command again.

I did (noting in passing that currently the LSF file is the odd one out: all other file extensions in the Resources folder are lowercase). This time it didn't delete any files from the folder or resources from the library - which is nice: I had renamed one .logos4 into .LOGOS4 and it stayed there - the log shows that it identifies and discovers only the LSF resource, but the LIX stays in place and the resource is properly working in L6. Do you care for the log? 

NB: The log always claims it deleted the xxxxx\ResourceManager\ScannedResources folder in this cleanup process (which is a little bit incorrect if there was no such folder in the firstplace). I created one, put two files into it and ran the command again - the log again only says it deleted the folder, neither remarking its prior existence nor its content. This may be due to the way you delete (e.g. the program doesn't care for the folder's content and doesn't evaluate a return code from the activity on the HD). The deletion in fact occurs, which at least should be part of the documentation. 

Running Logos 9 latest (beta) version on Win 10

Posts 2816
LogosEmployee
Andrew Batishko | Forum Activity | Replied: Wed, Jun 17 2015 11:33 AM

NB.Mick:

Andrew Batishko (Faithlife):

OK, rename RCSOVB.lix to RCSOVB.LIX (note the case difference). Make sure that the case is also correct on RCSOVB.LSF. Then start the application and try the command again.

I did (noting in passing that currently the LSF file is the odd one out: all other file extensions in the Resources folder are lowercase). This time it didn't delete any files from the folder or resources from the library - which is nice: I had renamed one .logos4 into .LOGOS4 and it stayed there - the log shows that it identifies and discovers only the LSF resource, but the LIX stays in place and the resource is properly working in L6. Do you care for the log? 

No need for the log. This is working how I would expect. The LSF/LIX resources are kind of different, and the LIX file in particular has special handling that doesn't handle case differences well. I would not expect to see any similar problems with other resource files, with the caveat that renaming files in your Resources folder (including case changes) is definitely unsupported and could result in unexpected behavior.

I've just finished writing a fix that will prevent the LIX file from being deleted, even if the extension doesn't match the case of the LSF file, so everything should be good.

NB.Mick:

NB: The log always claims it deleted the xxxxx\ResourceManager\ScannedResources folder in this cleanup process (which is a little bit incorrect if there was no such folder in the firstplace). I created one, put two files into it and ran the command again - the log again only says it deleted the folder, neither remarking its prior existence nor its content. This may be due to the way you delete (e.g. the program doesn't care for the folder's content and doesn't evaluate a return code from the activity on the HD). The deletion in fact occurs, which at least should be part of the documentation.

The directory does actually exist for a short period of time before it's deleted. This is a normal part of the discovery process, and not related to the use of this command (beyond the fact that the use of the command triggers discovery, similar to the use of the scan command).

Thanks again for the help.

Andrew Batishko | Faithlife software developer

Page 1 of 2 (30 items) 1 2 Next > | RSS