[feature request] scriptable API to interface with Logos
Comments
-
Thanks for the heads up. I knew about these. Seems like their new {} syntax fixed it. The issue is the original syntax is too general, and allows partial match, i.e. they are matching patterns all the time. But with the new syntax, it only looks for exact match, therefore it is much faster (no pattern expansion), and depending on how they implement it, they can even hash it to do a reverse look up to save time, and this is what I guess they have done.
(It's somewhat obvious that these were never anticipated when you look at the single-line editing UI where you enter these rules; how are you even supposed to edit them easily?)
It's actually quite simple, edit them in a proper text editor with syntax highlighting. Choose a language (for syntax highlight) that's closer to their syntax, something like Python. In software like Textmate, one can even define a new language (syntax highlight rule). And if I were to develop such a complex match, I would have put it in multiline, kind of C like line break convention for easy editing and version control and then have another script to generate the 1-liner.
By the way, it is easier for cross-referencing if the original link is provided together with the quote: https://community.logos.com/forums/p/174759/1010186.aspx#1010186 (Edit: use the "pasting button" on the left of "Bradley Grainger (Faithlife) | Forum Activity | Replied: Tue Nov 6, 2018 4:21 PM".)
0 -
Also, as far as I understand, the abundance of custom collections is just slowing down the indexing time, which I don't often care about (I remember in the old days Logos indexing can be quite annoying, but that forces one to develop a habit to, say, running "renew resources" on Saturday night and let it run over night.)
0 -
Kolen Cheung said:
By the way, have you tried to parse the sitemap and obtain all products from there? I think you said there's some reason you can't get the Faithlife ebooks unless you know its existence. Did you try parsing Faithlife's sitemap to get them? I only parsed logos.com's sitemap so don't know what's there yet.
The Faithlife eBooks site doesn't contain any resource IDs, so scraping the site doesn't help.
Kolen Cheung said:I really would like to write to it eventually, e.g. for creating collection. I created ~100 collections last weekend and it is very tedious to click click click. But I think I've had enough warning to take the computer offline first and check if things are working before allowing it to sync.
Why? You should be using dynamic collections for the most part. How can it be quicker to write a script to create collections, rather than write a rule?
Kolen Cheung said:Another thing I want to do is to merge all highlighting scattering into 3 notebooks into one (somehow Logos has different kinds of highlights, some called emphasis marking, and applying them would fall into different kinds of notebooks automatically...) I think currently there's no GUI way to do this (like cmd+A and drag won't work.)
If you set your highlighting up properly, this won't happen in the future — and I doubt it will take very long to fix the problem for the current highlights.
Kolen Cheung said:But I think I've had enough warning to take the computer offline first and check if things are working before allowing it to sync.
Writing the collection rules to the database isn't necessarily the problem. That's fairly easy. But you've then got to be sure you've set the sync flags correctly — and there's no way of testing that until you go online and try to sync.
For what it's worth, my own opinion — based on all your questions over the last two weeks — is that you don't yet know enough Logos' web-services or database structure to go anywhere near writing to your databases. I continue to strongly advise that you do not do it.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
Also, as far as I understand, the abundance of custom collections is just slowing down the indexing time, which I don't often care about (I remember in the old days Logos indexing can be quite annoying, but that forces one to develop a habit to, say, running "renew resources" on Saturday night and let it run over night.)
This is something else that you've entirely misunderstood. Custom collections have zero impact on indexing time.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
Why? You should be using dynamic collections for the most part. How can it be quicker to write a script to create collections, rather than write a rule?
If you look at Logos' community collection such as those Logos 5 collections, they are "manual collections". I was thinking about doing something like that, again, like Scholar (LE). Also some other series that Logos doesn't regard as series. Those are tedious to be done manually.
Mark Barnes said:If you set your highlighting up properly, this won't happen in the future
Then I guess I still don't understand how to set it up correctly. I've been using it for a decade but I just use it. IMO this is a bit over-engineering here, I just want to highlight and they turn it into a document/notebook... Certainly others has better use of highlight but for me it is just when I revisit I know where my eyes should start to stare at...
Mark Barnes said:and I doubt it will take very long to fix the problem for the current highlights.
Long enough for me to give up (and I give up easily since I don't like to routinely clicking things because I know there has to be better way to do it. It might be some macOS bug too. I know in Library cmd+A doesn't work as expected. And I tried in the new notebook cmd+A don't work. May be it works on Windows then I guess it's worth to boot into Windows to do this.
Mark Barnes said:For what it's worth, my own opinion — based on all your questions over the last two weeks — is that you don't yet know enough Logos' web-services or database structure to go anywhere near writing to your databases. I continue to strongly advise that you do not do it.
May be that's when someone should start to gets their hands dirty [;)] I've been trying to be an "obedient" Logos users for a decade but now I decided I should start getting my hands dirty.
And then there's really not much things I can mess up in my Logos "cloud", I don't like to put stuffs in the cloud in a non-transparent manner. I always want to be in control, want everything reproducible. If the system do not support a way to completely throw out everything and start over, the system has a problem. And right now frankly Logos has. In this regard it is the worst software I've ever used. (And the 2nd worst is Apple's iCloud. It isn't the worst because one can largely not rely on iCloud in most cases, and for those one can't escape from, it behaves quite well.) "Just works" doesn't really work, as I often told my wife, "When it works, it's magic. When it doesn't work, it's magic!"
0 -
Mark Barnes said:
This is something else that you've entirely misunderstood. Custom collections have zero impact on indexing time.
Actually I got that impression from https://community.logos.com/forums/p/166404/972021.aspx#972021,
Mark Barnes said:Large collections will cause slow-downs in two main places:
- Startup
- Any time anything changes in your library (new resources, tags/ratings added, community tags downloaded, metadata updated, etc.)
At this time all collections are calculated, regardless of whether they're in use.
This calculation step ain't indexing? I thought you were referring that to indexing. Because say if one is changing a rating, the program is saying "indexing... library result may not be accurate..." something like that.
0 -
Kolen Cheung said:
This calculation step ain't indexing? I thought you were referring that to indexing. Because say if one is changing a rating, the program is saying "indexing... library result may not be accurate..." something like that.
No it isn't. And the program says: "Updating library catalog", not "indexing".
Have joy in the Lord!
0 -
Kolen Cheung said:
If you look at Logos' community collection such as those Logos 5 collections, they are "manual collections". I was thinking about doing something like that, again, like Scholar (LE). Also some other series that Logos doesn't regard as series. Those are tedious to be done manually.
If you want a collection like that, you're correct that it will have to be done manually. But you won't be able to automate it, even if you could write to the database — because you won't be able to 100% accurately match resource titles on a wiki page or spreadsheet to the resource titles in your library. They're often not identical.
Kolen Cheung said:Then I guess I still don't understand how to set it up correctly. I've been using it for a decade but I just use it. IMO this is a bit over-engineering here, I just want to highlight and they turn it into a document/notebook... Certainly others has better use of highlight but for me it is just when I revisit I know where my eyes should start to stare at...
In the Highlighting tool hover over the name of the palette (e.g. "Emphasis Markup"). Click the down arrow. You'll then be able to select the notebook where highlights from that palette are stored in the future.
Kolen Cheung said:Long enough for me to give up (and I give up easily since I don't like to routinely clicking things because I know there has to be better way to do it. It might be some macOS bug too. I know in Library cmd+A doesn't work as expected. And I tried in the new notebook cmd+A don't work. May be it works on Windows then I guess it's worth to boot into Windows to do this.
I know cmd+A doesn't work. But (at least on Windows), SHIFT+click does. Click on the first note in the notebook. SHIFT+click on the last. Drag to the new notebook. Delete the now empty notebook. Four clicks.
Kolen Cheung said:May be that's when someone should start to gets their hands dirty
I've been trying to be an "obedient" Logos users for a decade but now I decided I should start getting my hands dirty.I enjoy poking around behind the scenes. I'm not trying to stop you from doing that. I am suggesting very strongly that you are 100% confident you've understood the format and how everything works before you write to the database. I've spent years poking around to extract resource metadata and use it in various projects. I think I have a good understand of everything from verse mapping to hidden library fields to the web services. Despite that, I've never written to the databases.
So if you get in a mess, you'll be completely on your own.
Kolen Cheung said:I always want to be in control, want everything reproducible.
That was fine in 1990. This is 2019. The vast majority of software and services doesn't work like that any more.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
This calculation step ain't indexing? I thought you were referring that to indexing. Because say if one is changing a rating, the program is saying "indexing... library result may not be accurate..." something like that.
As Mick says, when you change a rating you get an entirely different message ("Updating library catalog…"). That's not indexing. Indexing occurs when you add or update resources. Updating the library catalog occurs when you add resources or update metadata.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
because you won't be able to 100% accurately match resource titles on a wiki page or spreadsheet to the resource titles in your library. They're often not identical.
Text manipulation is often messy, a pity. But one can try to canonicalize both side and compare if they are the same. That’s the data cleaning part. Moreover, while being 100% accurate is nice, if spending 10% of time getting one 90% there then I’m all for it. I don’t even know how to manually add hundreds of resources by hand. But again I hope Logos will do it, as they have said. But the logos community collections last updated years ago, and years bejng Logos users taught me unfortunately never trust their promises. Good intension, it there are always obstacles that make things dally and delay. Bob once promised me PBB reverse interlinear soon almost a decade ago, and it hasnt arrived yet. I hope that this feature came soon since the brother I know spending his whole life working on a Chinese reverse interlinear and a fortune on an in house bible software is, you know, getting very old. He actualy once produced a reverse interlinear Chinese recovery version in NT but due to a history related to the people involved in recovery version, it never s got published. Anyway.
Mark Barnes said:But (at least on Windows), SHIFT+click does.
Unfortunately I tried that on Mac and it doesn't work. Probably should give Windows a try. I wish Logos develop multi platform apps like Microsoft did to its office. Logos is just not behaving like a Mac app. (I was tempted to use the other Mac native Bible software. But I first drawn to Logos by a unique resource published by Lexham. So I can't live without Logos. Imagine I can decrypt Logos format and convert them to the other format such that resources and the softwares are decoupled. Among all people one would guess Christians should have more incentive to have a common standard. I guess the unique complexity involved in Biblical related resources are too much and yet their world are too small for industrial standard to win.
Mark Barnes said:That was fine in 1990. This is 2019. The vast majority of software and services doesn't work like that any more.
Most I used does. Among those I used Logos is uniquely behind in this aspect as if they don't even considered this.
And from my personal experience, it's getting easier and easier to do this. Partly because I learnt more, mostly because the world is so big and new tools emerges that makes this possible and easier.
May be one day I'll be fully using open source softwares and OS. But I'm greedy and I want the best and some of the best is still proprietary GUI like Logos, Lightroom, etc. (not to mention in the case of Logos it is the resources + feature/data sets that set it apart, not the software, making an open source alternative almost absence.) That's partly why I started this thread because I worry for a future that the "philosophical distance" between me and Logos are getting farther and farther away. E.g. Applescript support is also important but from Logos' not Apple native commitment seems to be low priority. I just can't imagine how I can still tolerate those clicking a decade from now, which could've been scripted. Like PBB, I have a very small collection of PBB and managing that already is a headache. It is relatively easy to batch obtain a lot of public domain resources to be added by PBB, but the bottleneck is the clicking of thousands times. May be that API already exist but it isn't clear what function does or doesn't.
0 -
NB.Mick said:Kolen Cheung said:
This calculation step ain't indexing? I thought you were referring that to indexing. Because say if one is changing a rating, the program is saying "indexing... library result may not be accurate..." something like that.
No it isn't. And the program says: "Updating library catalog", not "indexing".
Right, I just recalled it from memory so I wasn't using the exact wording.
However, it must be doing more than just updating library catalog, because if so it is just one single atomic transaction which should be very fast. What I imagine it is doing is that it is like adding the record and then perform an optimization to that database. Lightroom does something like that. Lightroom's catalog is an SQLite database, and there's an "optimize catalog" button allowing the user to optimize and cleanup the catalog on demand, but not on real time. So that a mere change of a tag/rating/metadata won't have that much a penalty.
So if what I said is true, I suggest Logos to separate these 2 steps. e.g. try manually tagging and rating the new base package one has bought, the "updating library catalog" is very annoying. The user is almost like racing with that when rating resources one by one.
So indexing / updating library catalog or else, the reason they take so long to finish is to "pre-cache" something so that in the future a certain operation is very fast, like O(1) kind of fast. So if this is indeed what's happening behind the scene when one has a ton of collections, I don't care. Because the slowness only shows up in the preparatory step, not in real time when one need to interact with resources. If not true, then probably they should tell us what's the expectation here.
0 -
Yes, the contents of the resource files are encrypted. This is a DRM issue.
One more question: is .lbspbb essentially identical to .logos4 in terms of spec and DRM? I peek into one of them and it sort of looks like DRM'd. If not I'd like to be able to have the spec of lbspbb to generate it directly without going through Logos' PBB compiler. And if yes, why DRM is applied to PBB?
0 -
Kolen Cheung said:
One more question: is .lbspbb essentially identical to .logos4 in terms of spec and DRM? I peek into one of them and it sort of looks like DRM'd. If not I'd like to be able to have the spec of lbspbb to generate it directly without going through Logos' PBB compiler.
Logos PBB compiler includes metadata that is needed for .lbspbb to be usable, which is synced. Hence puzzled by desire to generate lbspbb without using PBB compiler ?
Keep Smiling [:)]
0 -
Kolen Cheung said:
One more question: is .lbspbb essentially identical to .logos4 in terms of spec and DRM? I peek into one of them and it sort of looks like DRM'd. If not I'd like to be able to have the spec of lbspbb to generate it directly without going through Logos' PBB compiler.
Logos PBB compiler includes metadata that is needed for .lbspbb to be usable, which is synced. Hence puzzled by desire to generate lbspbb without using PBB compiler ?
Keep Smiling
- Get rid of MS Word. Since the beginning they introduce PBB (post Logos 4), I thought it was a big mistake to make it depends on MS Word.
- A “principle” reason is that makes it not free. Imagine someone can’t afford to buy any Logos resources for various reasons (which is not uncommon), they can get free Logos engine, they can get some free Logos resources (recall when they first introduced PBB there were very few free Logos resources), then the next logical step is PBB for classic, public domain resources. But without a commercial MS Office, one can’t produce that.
- MS Word’s spec is in principle open, but in reality a very complicated thing that requires a lot of reverse engineering.
- A free option to generate docx exists, but the problem from its spec making generating docx a hit and miss thing. Sometimes the generated docx is corrupted, and occasionally opening in MS Word and “save as” can fix that, which again requires the commercial application.
- Authoring a book in MS Office is almost a joke. MS Word is just not for book publishing (The state of the art is LaTeX, and other solutions exists, and yes a lot of publishers accept Word but they are doing their conversions and editing). Regardless the origin of PBB (public domain books, “self-publishing” kind of long form notes, etc.), being a PBB almost guarantee it is very long. Producing long, structured documents is very hard in MS Office (I mean making sure the whole document falls in a uniform typesetting standard is hard. It is easy to apply a slightly different choice of, font, font size, indentation, section, etc. somewhere in the document and hard to spot.)
- Do something PBB compiler can’t do. If indeed lbspbb really support everything logos4 does, then although we aren’t suppose to author logos4 documents (for the prevention of piracy), we can author lbspbb directly for features already supported in the spec but not by the PBB compiler. One such example is (reverse) interlinear than Bob has promised but still didn’t deliver (I think they changed their mind) after almost a decade.
Note that I have my own solution to (1) so the main focus is (2), which focus on getting features not supported by the PBB compiler.
1 - Get rid of MS Word. Since the beginning they introduce PBB (post Logos 4), I thought it was a big mistake to make it depends on MS Word.
-
Kolen Cheung said:
Get rid of MS Word.
LibreOffice is a free alternative that can create docx file(s) for Personal Book(s).
Years ago remember learning a MS Word trick to save a document as RTF followed by opening RTF for save as docx.
Kolen Cheung said:One such example is (reverse) interlinear ...
Reverse Interlinear resource additions are *.lbsrvi
Keep Smiling [:)]
0 -
" rel="nofollow">Keep Smiling 4 Jesus
said:LibreOffice is a free alternative that can create docx file(s) for Personal Book(s).Yes, if you read a few rows after that,
Kolen Cheung said:A free option to generate docx exists, but the problem from its spec making generating docx a hit and miss thing. Sometimes the generated docx is corrupted, and occasionally opening in MS Word and “save as” can fix that, which again requires the commercial application.
I should’ve said “free options” as there are many. But none of them is perfect. Speaking of Libre Office, it has many problems. e.g. HTML to docx conversion can sometimes be extremely slow which takes hours, overnight conversion. While its cli program,
sofficeis useful for scripting (and hence automation and batch processing), it cannot be parallelized (this can be gotten around by setting it up as a daemon which makes parallel execution very complicated.)Reverse Interlinear resource additions are *.lbsrvi
I think the main point is to open the spec of native Logos formats so that personal book is not limited to what the PBB compiler supports.
Yes, there is DRM concern, but it is irrelevant. Just like e.g. epub, the spec is open, and DRM is optional. i.e. Logos can release the spec of all their formats, while on top of it keep using their DRM for official, commercial products.
Even MS office releases the spec of their XML formats. So why not for Logos?
As an example, this is the steps involved in PBB output from BibleMultiConverter:
- detect versification in BibleMultiConverter (no output yet)
- export to “Logos HTML” in BibleMultiConverter, which is in HTML format, but written in a way to includes Logos PBB syntax
- HTML to docx conversion, e.g. using soffice from Libre Office
- docx to docx in 2 passes by BibleMultiConverter for syntax fixes
- docx to PBB using Logos' PBB compiler
All these conversion steps (ignoring the versification detection) can be done in 1 go if a spec in the final format is given, say the current
.lbspbb.So the issue isn’t related to the file extension (file extension is an arbitrary convention that didn’t change its content, its spec is the one that’s relevant. In principle given the some file extension, or even no extension at all, as long as e.g. the mime type is provided it’ll be fine), but the spec to generates these files.
0 -
Mark Barnes said:
For what it's worth the code is on GitHub. The code wasn't written for the public, and I'm not really willing to answer lots of questions about it, but you can start with the update_metadata function.
But this service is probably more useful than my code. Lots of the functions require authorisation (i.e. your Logos cookie), which you can't generate through the site. But GetMetadata works unauthenticated. Just add the ResourceIDs in the request XML.
Thanks Mark again. I just started to use this and it is actually quite useful. The only bump I got is I spent almost an hour just to figure out resourceIds is something like 'LLS:EEC15EZR' not '24998' (which I assumed from the REST API...)
0 -
The numbers are product IDs, not resource IDs. Products are what is sold (e.g. “Logos 8 Gold”). Products often (but not always) contain multiple resources. You only need product IDs if you’re interacting with the store.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Thanks. I figured that out.
0 -
Kolen Cheung said:
Thanks Mark again. I just started to use this and it is actually quite useful. The only bump I got is I spent almost an hour just to figure out resourceIds is something like 'LLS:EEC15EZR' not '24998' (which I assumed from the REST API...)
We appreciate your enthusiasm for our products and your desire to programmatically explore the Faithlife platform. We have public facing developer documentation for a couple of our web APIs at https://developer.faithlife.com/, but these APIs were specifically designed with public consumption in mind and have built in rate limiting. Unfortunately, some of our older APIs have not yet been retrofitted with rate limiting and there is potential for the additional load to impact other users of our services.
We don’t object to you calling external APIs called by the desktop application, but we do ask that you throttle your calls to our APIs to a couple hundred requests per minute. We’ll plan to add rate limiting in the near future.
Thanks for your understanding!
0 -
Nate Merritt (Faithlife) said:Kolen Cheung said:
Thanks Mark again. I just started to use this and it is actually quite useful. The only bump I got is I spent almost an hour just to figure out resourceIds is something like 'LLS:EEC15EZR' not '24998' (which I assumed from the REST API...)
We appreciate your enthusiasm for our products and your desire to programmatically explore the Faithlife platform. We have public facing developer documentation for a couple of our web APIs at https://developer.faithlife.com/, but these APIs were specifically designed with public consumption in mind and have built in rate limiting. Unfortunately, some of our older APIs have not yet been retrofitted with rate limiting and there is potential for the additional load to impact other users of our services.
We don’t object to you calling external APIs called by the desktop application, but we do ask that you throttle your calls to our APIs to a couple hundred requests per minute. We’ll plan to add rate limiting in the near future.
Thanks for your understanding!
Thanks for letting me know. I did try to find the limit and see there's 300 requests per minute from the Products API but not mentioned in ResourceMetadata API. Will it be ok if I limit it at the same rate (300 requests per minute)?
To prevent sending a ton of requests, I actually was trying to cache the metadata locally first yesterday. But the SOAP interface causes me some problem.
By the way, to others in this thread, I think I've found a way to "write" without writing to the database. Essentially the problem is how to "create" something from the data gathered back into Logos. I think I've found a way to do that without actually writing to the database, which is something every response to that is negative. I'll test this idea once I have cached all the metadata.
0 -
Kolen Cheung said:
Thanks for letting me know. I did try to find the limit and see there's 300 requests per minute from the Products API but not mentioned in ResourceMetadata API. Will it be ok if I limit it at the same rate (300 requests per minute)?
We'll probably set a rate limit much lower than 300 requests per minute in the near future for the ResourceMetadata API and require authentication for the GetMetadata route.
0 -
Nate Merritt (Faithlife) said:
We'll probably set a rate limit much lower than 300 requests per minute in the near future for the ResourceMetadata API and require authentication for the GetMetadata route.
I see that it is now requiring authentication. Could you tell us what's the rate limit?
Thanks.
0 -
Kolen Cheung said:
I see that it is now requiring authentication. Could you tell us what's the rate limit?
The rate limit is currently 4000 requests per hour for the GetMetadata route.
0 -
Did anyone ever get anywhere with this? Looking to do something similar.. Logos is far too slow when it comes to string searches and other types of queries for biblical research - I can do something a fair bit faster but need raw data to work with.
0 -
Nate Merritt (Faithlife) said:Kolen Cheung said:
I see that it is now requiring authentication. Could you tell us what's the rate limit?
The rate limit is currently 4000 requests per hour for the GetMetadata route.
Has anything changed on the SOAP interface? e.g. starting with product ID of 165462, with resource ID of {A54626B9-439E-42F2-BFAD-3AB0443E8AFF}, it used to give me its metadata such as the AbbreviatedTitles, etc. and now it returns nothing. Are other people experiencing the same issue or is it just me? If it has been changed, how might I query a collection such as the "Logos 8 Academic Professional" with all its children recursively?
Thanks.
0
