[feature request] scriptable API to interface with Logos
The feature request is simple to state: provide a programable/scriptable interface to Logos Desktop. Example programming language are Python, Lua.
Has this ever been suggested before? And if so what’s the reception on this and Logos/Faithlife’s view on the matter?
One can formulate a formal proposal but we’re not yet there. Just name a few advantages and reasons:
- some routine stuffs can be done more effectively with a program. e.g.
- Let’s say from Wayback Machine I find the old listing of Logos 4 (LE) and want to create a custom collection on it, it is very tedious to drag and drop hundreds of resources to create a collection, but with scripting it is much faster and scalable (to any given list once data cleaned up.)
- grab definitions from https://wiki.logos.com/Canonical_Commentary_Collections and loop through them to create 100s of collections.
- write a script to get ratings from <bestcommentaries.com>, then add them as 5 star scale ratings in logos, updated routinely.
- listing out some information that is not shown through the software, e.g. in that list subjects such as O.T.Leviticus, O.T.Pentateuch exists but clicking on info panel doesn’t show this.
- to cook up some functionality not yet exist in Logos. e.g. there aren’t a tool to merge a few notebooks into one. But with a proper API one can again create a notebook consists of contents of the others easy.
- enabling data analysis using datasets available in Logos. e.g.
- I used to do stuffs like this manually a decade ago: counting no. of verses per book/testament/bible and doing statistics on it (e.g. total no. of original words, average no. of words per chapter, longest book by chapter/verse/words/etc.) These can be done easily, and repeated (say with a new resource or new update) if such API exists.
- e.g. one can analyze the graph of cross references between the resources and use that partly to determine a popularity of a resource and rates accordingly. (this depends on the no. of resources one has, so one could even think that’s a web API that Logos' server answering queries of stuffs like this, having access to the metadata of the whole Logos library. And this paid by hours of access like AWS.)
- This can enable some other form of research not often done yet. e.g. my advisor in the old days (more than a decade ago when terminology like “big data” doesn’t exist yet) had performed a research to study complexity between languages given Bible translated in those languages. With Logos' metadata one can ask similar but more advanced, correct questions: e.g. using the reverse interlinear datasets, one can construct matrices of “semantic clusters” between the original language and the translated language, and from this one can infer the complexity of the resulted translation, comparing to the original language. This is interesting to know, and can be used as some sort of index for Bible translations in terms of formal vs. dynamic correspondence.
- allowing one to combine tools from programming and Logos' dataset to perform unique kind of research. e.g. using natural language processing from a library together with Logos' datasets/internal resources.
- plugins can be written by the community to enhance Logos in various ways. e.g. one can imagine the automation on creations of PBBs, even not from docx by chaining tools such as pandoc.
- allow one to create resources that aren’t supported by PBB. Examples such as a interlinear / reverse interlinear Bible. I know a very old brother who has a database of that (in Chinese), which has been completed years ago. He insisted it should be given away for free. He hired a programmer to create a Bible software to show that interlinear tool (somewhat similar to Interlinear Scripture Analyzer in what it does.) But the programmer was too expensive, and left, and he’s left with a software only works in Windows XP and no documentation so no one to follow up to improve upon. I think he has talked with Bob years ago (who told him it took at least a million for create such software, sort of discouraging him to do so.) I asked Bob a decade ago if PBB would support the creation of interlinear / release interlinear and he answered yes but that feature never materialized yet. Even if that feature in PBB exist, it would be very tedious to convert a database into the Word format PBB required. It would be much easier if Logos instead open up the internals of generating a Logos resource through an API. This will democratize the creation of resources and Bible software (since now one don’t have to create their own in order to give away for free. Instead direct people to download free Logos engine, provide the resource for free as a file.) One can even create this resource and donate to Logos to give away for free (because without a way to create this on our own, Logos won’t invest his staffs' time to create it and give away.)
Out of the many reasons that one might want to reject this idea, I’ll only start to reply to this most important one I can think of: this would be extra work for Logos/Faithlife to provide such a feature for such a niche audience. The investment is not worth it. Now the counterpoint to this:
- Logos' employees has to have ways to touch the internals for various reasons. Such API, which might already exist in various forms for their internal use to debug stuffs, can be refactored to allow part of them to be public facing. Such release is only going to make their employees' experience so much better (because for something to be ready to the general public, it has to pass some sort of user-friendliness test). Such improvements would increase the productivities of their employees, and lower the cost to train new ones. (Ok, this is just my opportunistic guess.)
- capability to program was for the few elites but becomes part of the standard curriculum. i.e. the new generations (including mine to some extents) will all know programming as part of their common knowledge, just as Math, etc. For those it is incredibly time wasting to manually clicking dozens times when they know repetitive tasks should always be scripted and automated. i.e. this feature which might seem to be for niche can becomes the mainstream for the next generations. And Logos can again lead in innovations in this aspect. (More example: when I was undergraduate more than a decade ago, one can get a degree in Physics and Mathematics without programming knowledge at all. For now, not that it’s impossible, but it becomes very common for an undergraduate at least know Python to perform some simulations. Some University even choose to teach Introductory Physics with programming embedded in their course such that they can visualize the simulation of their theoretical equations and know how it translate into complex dynamics.)
- allow rapid prototyping some ideas. e.g. those wheels in Bible study, can be coded up easily given the access of APIs. Again this can benefits Logos internally, and also potentially some creative ideas from the community can becomes a feature in next version of Logos.
- not uncommon for softwares to provide a scriptable interface. e.g. macros in MS Office. Lua and/or Javascript scripting support in Adobe Lightroom, Photoshop, etc.
Regarding the programming language of choice, I would suggest providing a Python binding, allowing people to install it in their own environments such that they can use them with other libraries such as nltk, pandas, numpy, seaborn, etc.
Note that Python can be used as an embedded interpreter within Logos too, this is more self-contained for simple scripting but is inferior because one can’t use other than standard library. Also Lua is also a common choice as an embedded scripting language but I think it is too lightweight for doing anything really powerful together with Logos.
Comments
-
In my library, I still have Libronix Corporation. Libronix DLS Object Model Reference. Libronix Corporation, 2002. [8-|]
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Kolen Cheung said:
The feature request is simple to state: provide a programable/scriptable interface to Logos Desktop.
Isaiah 65:24.
https://wiki.logos.com/How_to_Use_the_COM_API (although it's anticipated uses are very different to the ones you suggest).
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Wow. It would be interesting if the forum can parse the verse like PBB does.
And honestly I haven't heard about that at all. Why wouldn't Logos be more vocal about that?
Anyway, that one obviously has a couple of problems, Windows only (or not?), and last committed on March 2011. The choice of C# is problematic too (I guess Logos itself is written in C# too?) And it also stated that it is read only (many of the features suggested above requires the ability to write/create.)
But I'm glad it existed, it means that Logos does not reject that idea. It's just that it wasn't sccessful and may have been designed for people creating third party applications for Logos rather than as something the end user will use directly.
It also means that it is very easy to create a Pythonic, object oriented binding to that, using C FFI. Please consider this, Logos, revive this project and breath new life into it.
0 -
Over the years there have been tools built on it. But over time they become outdated or their creator dies/loses interest which stops maintenance.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
See https://community.logos.com/forums/p/179259/1035772.aspx#1035772 for example on how an API like this could be used.
Note that if more data is given e.g. through cloud (i.e. where I don’t need to own all resources in order to obtain data on all Logos resources), many interesting things can be analyzed:
- focusing on different denominations of Logos packages, one can ask what that distribution would looks like. e.g. if there’s any bias in some denomination base package that favors “good” or “bad” resources.
- focusing on how different people rated a certain resources, using the information e.g. from their denomination specified in their Faithlife profile, one can infers if a particular resources is biased towards being like by a particular group or else. (This one would touches a much larger database though.)
- one can even try to compare this across different base packages “color” to see which one is more optimal.
By the way, on the topic of making buying decision through these data, I really hope that there's something like
from logos.cart import dynamicPricing # assembling a list of resources objects resources = ... dynamicPricing(resources)
It is related to some crazy experience I got when upgrading to Logos 8. I’ve found a very interesting combination, after contacting CS to buy Unfiltered with Platinum with dynamic pricing, which is cheaper than Platinum. But the process to this discovery is time consuming. Having something scriptable to explore multitude of options would be much easier. Another topic for another day.
1 -
Kolen Cheung said:
Note that if more data is given e.g. through cloud (i.e. where I don’t need to own all resources in order to obtain data on all Logos resources), many interesting things can be analyzed:
I can't see Faithlife being able to afford the investment required to give us access to its big data — especially when there's nothing in it for them, or for most users.
There are actually a large number of web services that provide lots of useful information, especially with resource metadata. Some of what you're asking in earlier posts can be done already. None of it is documented though, and none of it is 'public'. (It's publicly accessible, but it's not be designed for public use.) If you want to play around with these services, just watch the traffic between Logos and its servers. The mobile app has some additional services, and the web app some others.
There are no apis or services for the store, however (other than converting a resource ID into a store URL). To be honest, the store is creaking a lot (hence ebooks.faithlife.com), so no APIs are going to be added at least until the whole store's been thoroughly overhauled.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
Wow. It would be interesting if the forum can parse the verse like PBB does.
It's just RefTagger.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:Kolen Cheung said:
Note that if more data is given e.g. through cloud (i.e. where I don’t need to own all resources in order to obtain data on all Logos resources), many interesting things can be analyzed:
I can't see Faithlife being able to afford the investment required to give us access to its big data — especially when there's nothing in it for them, or for most users.
I assume we're talking their site. Or maybe the library.
But Logos is truly unique in allowing access to raw data. When I first saw Libby (kind of a brouchure), I squinted at the example images ... especially the hebrew. At the time, there was plenty of public domain greek. So I ordered Libby ... CDs. Loaded it, to see if I could get the hebrew. Whoa! After that, I copied and copied, to feed my neural and genetic engines. Personal use, I always followed. Follow the rules.
When Logos4 downloaded, that's what I checked first. Export the data? Yep! Oh my. Molassas but speeding bullet downloaded.
And that tiny piece of value was everything to me. They had to have allowed research on purpose. Couldn't be accidental. Had to be Bob.
To my knowledge, no one allows bulk access like Logos does. Accordance and Bibleworks, piecemeal. Some of the mobiles did, by accident, but public domain.
Sure, a high-speed script maybe. But no user would ever be happy ... differing needs.
I'll never forget that Libby brouchure. It was heaven. A lady in a sewing room could calculate redaction layers from 3,000 years ago.
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
Kolen Cheung said:
And honestly I haven't heard about that at all. Why wouldn't Logos be more vocal about that?
Having to support power users as they do complex things is not something they are eager to do as they don't have the staff for it. I wonder if they originally created this just for their own use and then left it exposed to users just because, figuring very few would discover it and use it but those who did might be happy it was there.
0 -
Rosie Perera said:
Having to support power users as they do complex things is not something they are eager to do as they don't have the staff for it.
Read the 2nd bullet clusters in the OP for the debunk of this.
In short, many company open source their softwares (sometimes not all of their softwares but a particular stack, e.g. Twitter Bootstrap) for the reason that the more user uses it, the more problems they discovers (some are bugs, some are usability issue.) And in the end they get the benefits from it.
What I tried to argue is if Logos provides a Pythonic, Object Oriented interface, that can benefits its own employees for faster prototyping, etc. Having an API well documented itself can benefits the employees somehow.
Also as they did in 2011, releasing it in somewhere like GitHub convey the proper message that this is not for the average users but programmers. RTFM (F stands for Funny, Functional) is the standard in any such kinds of open source projects (be it only the API binding is open sourced) so supporting them shouldn't be that difficult one would have imagine. (If people ask stupid question over there, I'd politely say that project is not for them and they should start by learning the Logos GUI in this forum first.)
So this lack-of-staff-to-support is a myth. One should try to think how to organize it in a way that can directly benefits Logos (and I argued by benefiting its own staffs for ease of training, rapid prototyping, increased productivity.) Once that formulated it becomes something worth invested in (because it is not consuming but providing productivity.)
Identifying "power user" as a minority group is some sort of myth too. Because the prior of what Logos currently is largely defines these users as minority; also, as I've argued in the OP, these minority group is growing and eventually every educated person say from high school has this as part of their common knowledge. We're growing into the future where people expects automations and batch processing and increasingly going to ask the question: why am I clicking this and that routinely but not script it into automated processes?
0 -
0
-
Mark Barnes said:
I can't see Faithlife being able to afford the investment required to give us access to its big data
Limits can be applied. Many such web APIs are like this. You can't do without else you're under attack. (e.g. NYTimes API, Twitter API, GitHub API.)
0 -
Mark Barnes said:
There are actually a large number of web services that provide lots of useful information, especially with resource metadata.
Note that this doesn't cover many of the examples in the OP though.
Arguably the most valuable datasets to explore is the one we pay for, since Logos 5 days. These manually tagged metadata is very valuable (in terms of accuracy) to be explored, vs. e.g. natural language processing which can be wrong (not saying human can't be wrong though, but like the degree of accuracy is different, and also manual tagging can be fixed in time.)
Just as simple as the "semantic" mixing matrix I mentioned above cannot be done without touching Logos' reverse interlinear datasets. At least not in large scale, reproducible manner. In the past, I've even manually created these matrices for the word I'm studying, and quickly found to time consuming to not carry out that frequently. Logos word study wheels appears later in a certain Logos version does show part of this matrix (either a row or a column of information, but never the whole matrix.)
Note that one can resort to export Logos resources and perform data cleaning, but why make it so hard and error prone?
0 -
Kolen Cheung said:
RefTagger is added to the forum (it's javascript), and RefTagger auto tags them.
Kolen Cheung said:Mark Barnes said:I can't see Faithlife being able to afford the investment required to give us access to its big data
Limits can be applied. Many such web APIs are like this. You can't do without else you're under attack. (e.g. NYTimes API, Twitter API, GitHub API.)
The major cost isn't server time, it's development time.
Kolen Cheung said:Mark Barnes said:There are actually a large number of web services that provide lots of useful information, especially with resource metadata.
Note that this doesn't cover many of the examples in the OP though.
I know. But it's better than nothing.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
Has this ever been suggested before? And if so what’s the reception on this and Logos/Faithlife’s view on the matter?
Yes, it has been, and we built a simple API: https://wiki.logos.com/Logos_4_COM_API
No one (to a close approximation) used it.
We periodically consider adding more API support or embedding a scripting language or similar other approaches. It's not on the roadmap for 2019.
Kolen Cheung said:Out of the many reasons that one might want to reject this idea, I’ll only start to reply to this most important one I can think of: this would be extra work for Logos/Faithlife to provide such a feature for such a niche audience. The investment is not worth it.
You have hit the proverbial nail on the head. Your counterpoints don't (IMHO) sufficiently address the large (and ongoing) cost of building, documenting, supporting, and maintaining a public API.
Kolen Cheung said:Anyway, that one obviously has a couple of problems, Windows only (or not?), and last committed on March 2011. The choice of C# is problematic too (I guess Logos itself is written in C# too?) And it also stated that it is read only (many of the features suggested above requires the ability to write/create.)
It's a COM API; the choice of programming language for the example code is irrelevant. I believe there are people consuming it via Python. (But yes, it is Windows-only; see comments about "cost" above when proposing a cross-platform solution.)
0 -
"No one (to a close approximation) used it."
I did! So references in my Bibleworks notes could jump into Logos
"The whole modern world has divided itself into Conservatives and Progressives. The business of Progressives is to go on making mistakes. The business of Conservatives is to prevent mistakes from being corrected."- G.K. Chesterton
0 -
Mark Barnes said:Kolen Cheung said:Mark Barnes said:
I can't see Faithlife being able to afford the investment required to give us access to its big data
Limits can be applied. Many such web APIs are like this. You can't do without else you're under attack. (e.g. NYTimes API, Twitter API, GitHub API.)
The major cost isn't server time, it's development time.
That is arguable. As long as servers need to communicate there exists some form of API, whether you choose to expose it or not (or make it exposable.)
Just to list one example of things I'd like to query Logos' server, I'd like to be able to send a request to obtain all subjects on all resources Logos has, for example. (This request again can be limited like how other Web API is used.) Now this is not something serving logos.com would have needed and already there, but just an example of things I'd want to get access to.
0 -
Kolen Cheung said:
I'd like to be able to send a request to obtain all subjects on all resources Logos has, for example.
An undocumented SOAP interface allows you to obtain the metadata of resources where you know the resourceID. Getting resourceIDs is more tricky, although your library is a good start. If you're willing to dig around and write some code, you can find out a lot. Here's my list of subjects: https://www.logosresourcesguide.com/subject.php
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Oh, great! Thanks for pointing out this tool.
Performing a `head` on a resource will show the resourceID. By the way, do you know how to decode the binary logos4 format?
0 -
Actually did you release your source code of library on getting these metadata? Else are there any pointers to where I start? I want to be able to slice at the data itself.
Also a bunch of related questions:
- did you know how, once a list of resource is obtained, to create a collection in Logos (i.e. programmatically but not manually drag and dropping, once per resource)
- some how obtain a collection of old Logos packages (such as "Scholar (LE)")? Is that information somewhere?
0 -
Kolen Cheung said:
Actually did you release your source code of library on getting these metadata
It's not at all supported, but others have written tools that just directly read from the LibraryCatalog SQLite database that contains all of your resource metadata. That seems far simpler than trying to access the service APIs.
https://community.logos.com/forums/t/85971.aspx
Note before you go trying to ask the author questions, he passed a few years ago.
Kolen Cheung said:did you know how, once a list of resource is obtained, to create a collection in Logos (i.e. programmatically
Hacking in changes to synced data (data created in the application that is kept in sync between installations by storing it on the server) is something we strongly recommend against. It's a potential way to corrupt or otherwise lose your synced user data.
Andrew Batishko | Logos software developer
0 -
Kolen Cheung said:
do you know how to decode the binary logos4 format?
Among other things, the license agreement says that we may not reverse engineer, disassemble, decompile or make any attempt to discover the source code of the Software, ... or ... "unlock" or circumvent the digital copyright protection of the Content.
I don't know if talking about their file format is a grey area or not, but we should respect the license agreement.
Thanks to FL for including Carta and a Hebrew audio bible in Logos 9!
0 -
directly read from the LibraryCatalog SQLite database
Did you refer to this one?
~/Library/Application Support/Logos4/Data/gf1dasn3.0cf/LibraryCatalog/catalog.dbWhat about the db that Mark Barnes website ask us to upload?
~/Library/Application Support/Logos4/Data/gf1dasn3.0cf/ResourceManager/ResourceManager.dbI’ll peek into it probably at the weekend. Starting from the db first seems easier than scrapping the whole Logos db that Mark has done. (Can Mark share that by the way? Either the software of obtaining this db preferably or just the db you’ve already collected.)
Hacking in changes to synced data (data created in the application that is kept in sync between installations by storing it on the server) is something we strongly recommend against. It's a potential way to corrupt or otherwise lose your synced user data.
That’s a pity. You know, no offense, but I completely don’t trust Logos' cloud (or any cloud trying to be “just work”.) Simply put, that took away control from the user, so that experiments like this becomes very dangerous. A few days ago I accidentally removed 350 ratings from my library and that’s scary because there’s no way to recover that (there’s no universal undo like e.g. Lightroom does.) In the end I force quit the Logos (which still seems to late) and I jump right to my other computer and disconnect its internet immediately in order to retain the old, not-yet-synced library and then doing some manual migration of ratings to tags just so that after it syncs I can recover the ratings from these tags.
On this matter, that’s why I didn’t create any document (except 1 for historical reason (which is on church history), and some other testing documents, also other stuffs are also classified as documents in Logos which I did have, like visual filters, highlights, reading plans etc.) in Logos, despite some of them being quite attractive. The good old PBB is better (although more time consuming to tag) because the user is in control.
More on PBB, it would have been better if it was designed using a plaintext syntax such as markdown (and create Logos specific extension to it just as PBB did using docx), Word is just too heavy weight for that and bad for VCS.
PetahChristian said:Among other things, the license agreement says that we may not reverse engineer, disassemble, decompile or make any attempt to discover the source code of the Software, ... or ... "unlock" or circumvent the digital copyright protection of the Content.
I don't know if talking about their file format is a grey area or not, but we should respect the license agreement.
I'm not talking about the source code of the software, but the spec of the digital resource itself. Just like docx, odt, xml, markdown, etc. they all have a spec (and most of the time a reference implementation of reading it, just as Word, etc.) And we're not talking about DRM here, because AFAIK it isn't encrypted. (Just as if you buy a song from iTunes, it isn't encrypted either. Some vendors would like to put DRM to protect them. Luckily for us just as Apple, Logos trust us enough to not encrypt them.) Decoding it is just for doing something Logos doesn't provide yet. e.g. as simple as word count seems to be missing? Also the "semantic mixing matrix" I mentioned above definitely is not there but doable once the resource itself can be read. Not that it would interests anyone in this forum but e.g. treating Bible as a network and doing topological analysis on it, etc. would be fun. (Not sure if anyone has done this before yet though. Because the world is so big...)
0 -
I think I mentioned somewhere already, another major use of a spec of the logos4 format is not reading but writing. Imagine one to write a reverse interlinear from a certain database (that I know a brother spend his whole life working on and want to give away for free) which is impossible from PBB.
Also, I personally want to have Complete Biblical Library which is only available from a Logos competitor. I want to be able to buy it from that competitor, unpack it, and repack it in Logos. But actually the blocking step here is first I'd have to know I can decode (or even decrypt if they have DRM) it before I buy it...
0 -
Kolen Cheung said:
And we're not talking about DRM here, because AFAIK it isn't encrypted
Yes, the contents of the resource files are encrypted. This is a DRM issue.
Andrew Batishko | Logos software developer
0 -
Oh, I’m doomed. May be only the old Libronix files are unencrypted.
DRM removal is much more tricky (but legal) and I can’t possibly be getting help here... Hm...
ok, back to the API thing.
By the way, even if licensed resources are encrypted, a spec can still be provided for user to generate unencrypted resource (like the reverse interlinear I talked about.) Analogy will be DRM ePub and DRM free epub.
0 -
Kolen Cheung said:
Actually did you release your source code of library on getting these metadata? Else are there any pointers to where I start? I want to be able to slice at the data itself.
For what it's worth the code is on GitHub. The code wasn't written for the public, and I'm not really willing to answer lots of questions about it, but you can start with the update_metadata function.
But this service is probably more useful than my code. Lots of the functions require authorisation (i.e. your Logos cookie), which you can't generate through the site. But GetMetadata works unauthenticated. Just add the ResourceIDs in the request XML.
Kolen Cheung said:- did you know how, once a list of resource is obtained, to create a collection in Logos (i.e. programmatically but not manually drag and dropping, once per resource)
This is only possible by writing to the databases Logos uses. For all the reasons Andrew gave, I strongly advise against this. Personally, I only ever read from the databases. I never write to them.
Kolen Cheung said:- some how obtain a collection of old Logos packages (such as "Scholar (LE)")? Is that information somewhere?
Once the products are taken off the website, the information is lost. Wiki editors have tried to preserve much of it.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
Did you refer to this one?
~/Library/Application Support/Logos4/Data/gf1dasn3.0cf/LibraryCatalog/catalog.dbWhat about the db that Mark Barnes website ask us to upload?
~/Library/Application Support/Logos4/Data/gf1dasn3.0cf/ResourceManager/ResourceManager.dbCatalog.db contains all the metadata in your library. It can get very big. ResourceManager.db is much smaller, but gives all your ResourceIDs (among other things). My website doesn't need catalog.db because I can use those resourceIDs to query the Metadata service.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
For what it's worth the code is on GitHub. The code wasn't written for the public, and I'm not really willing to answer lots of questions about it, but you can start with the update_metadata function.
I don't know that language so it won't be useful for me. But thanks for open sourcing it.
Mark Barnes said:But this service is probably more useful than my code. Lots of the functions require authorisation (i.e. your Logos cookie), which you can't generate through the site. But GetMetadata works unauthenticated. Just add the ResourceIDs in the request XML.
Hm... it blocks me since I'm using content blocker, and "reload without content blockers" doesn't work. Chrome works fine though.
I think I'll start from the local SQL database first.
By the way, have you tried to parse the sitemap and obtain all products from there? I think you said there's some reason you can't get the Faithlife ebooks unless you know its existence. Did you try parsing Faithlife's sitemap to get them? I only parsed logos.com's sitemap so don't know what's there yet.
Mark Barnes said:This is only possible by writing to the databases Logos uses. For all the reasons Andrew gave, I strongly advise against this. Personally, I only ever read from the databases. I never write to them.
I really would like to write to it eventually, e.g. for creating collection. I created ~100 collections last weekend and it is very tedious to click click click. But I think I've had enough warning to take the computer offline first and check if things are working before allowing it to sync.
Another thing I want to do is to merge all highlighting scattering into 3 notebooks into one (somehow Logos has different kinds of highlights, some called emphasis marking, and applying them would fall into different kinds of notebooks automatically...) I think currently there's no GUI way to do this (like cmd+A and drag won't work.)
Mark Barnes said:Once the products are taken off the website, the information is lost. Wiki editors have tried to preserve much of it.
Too bad... People from Logos seems to have said they might eventually release them as community collections (where they have a bunch Logos 5 collections already.) I also found my old collections pages via archive.org (Wayback Machine.) If Logos doesn't provide it, in principle I just need to cleanup the HTML table and match them with resources I own and create a collection for that. But of course it'd be better if they provide it officially.
0 -
Kolen Cheung said:
. I created ~100 collections last weekend and it is very tedious to click click click.
Much to my disappointment, I ended up deleting many of my collections as they degraded performance ... I don't know to what extent the problem has been resolved although it has been improved.
Rick Ausdahl said:What good is it if it has such a negative affect on performance?
Most collections are fine, and very little is gained by taking the time to prune them back.
We originally envisioned that users might create a few dozen simple automatic collections, such as lang:Greek type:Bible or type:Commentary title:Genesis.
However, we found out that users had much grander visions and started created collections like these ones: https://community.logos.com/forums/p/88829/978914.aspx#978914
(It's somewhat obvious that these were never anticipated when you look at the single-line editing UI where you enter these rules; how are you even supposed to edit them easily?)
Some of these rules take multiple seconds to evaluate, and some users have dozens or hundreds of these rules. This is what can slow down the system.
As we became aware of this problem, we started investigating ways to mitigate it. In some cases, such as "* -mytag:*", we were able to ship an update to Logos that made those queries evaluate hundreds of times faster. But for the Author/Denomination/Tradition/Commentary collections, we were unable to do this; the next best approach we were able to develop was new syntax that was both more precise and much more efficient. You can see some examples here: https://community.logos.com/forums/p/88829/624737.aspx#624737
Mark rewrote his collections to use the new syntax, which addresses most of the performance problems. You shouldn't see any issues from having a dozen or so collections in that format.
But if you've got hundreds of "denomination" collections using the old syntax, especially if you use them very infrequently, you may see substantial performance improvements in Logos by deleting them from your application.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Thanks for the heads up. I knew about these. Seems like their new {} syntax fixed it. The issue is the original syntax is too general, and allows partial match, i.e. they are matching patterns all the time. But with the new syntax, it only looks for exact match, therefore it is much faster (no pattern expansion), and depending on how they implement it, they can even hash it to do a reverse look up to save time, and this is what I guess they have done.
(It's somewhat obvious that these were never anticipated when you look at the single-line editing UI where you enter these rules; how are you even supposed to edit them easily?)
It's actually quite simple, edit them in a proper text editor with syntax highlighting. Choose a language (for syntax highlight) that's closer to their syntax, something like Python. In software like Textmate, one can even define a new language (syntax highlight rule). And if I were to develop such a complex match, I would have put it in multiline, kind of C like line break convention for easy editing and version control and then have another script to generate the 1-liner.
By the way, it is easier for cross-referencing if the original link is provided together with the quote: https://community.logos.com/forums/p/174759/1010186.aspx#1010186 (Edit: use the "pasting button" on the left of "Bradley Grainger (Faithlife) | Forum Activity | Replied: Tue Nov 6, 2018 4:21 PM".)
0 -
Also, as far as I understand, the abundance of custom collections is just slowing down the indexing time, which I don't often care about (I remember in the old days Logos indexing can be quite annoying, but that forces one to develop a habit to, say, running "renew resources" on Saturday night and let it run over night.)
0 -
Kolen Cheung said:
By the way, have you tried to parse the sitemap and obtain all products from there? I think you said there's some reason you can't get the Faithlife ebooks unless you know its existence. Did you try parsing Faithlife's sitemap to get them? I only parsed logos.com's sitemap so don't know what's there yet.
The Faithlife eBooks site doesn't contain any resource IDs, so scraping the site doesn't help.
Kolen Cheung said:I really would like to write to it eventually, e.g. for creating collection. I created ~100 collections last weekend and it is very tedious to click click click. But I think I've had enough warning to take the computer offline first and check if things are working before allowing it to sync.
Why? You should be using dynamic collections for the most part. How can it be quicker to write a script to create collections, rather than write a rule?
Kolen Cheung said:Another thing I want to do is to merge all highlighting scattering into 3 notebooks into one (somehow Logos has different kinds of highlights, some called emphasis marking, and applying them would fall into different kinds of notebooks automatically...) I think currently there's no GUI way to do this (like cmd+A and drag won't work.)
If you set your highlighting up properly, this won't happen in the future — and I doubt it will take very long to fix the problem for the current highlights.
Kolen Cheung said:But I think I've had enough warning to take the computer offline first and check if things are working before allowing it to sync.
Writing the collection rules to the database isn't necessarily the problem. That's fairly easy. But you've then got to be sure you've set the sync flags correctly — and there's no way of testing that until you go online and try to sync.
For what it's worth, my own opinion — based on all your questions over the last two weeks — is that you don't yet know enough Logos' web-services or database structure to go anywhere near writing to your databases. I continue to strongly advise that you do not do it.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
Also, as far as I understand, the abundance of custom collections is just slowing down the indexing time, which I don't often care about (I remember in the old days Logos indexing can be quite annoying, but that forces one to develop a habit to, say, running "renew resources" on Saturday night and let it run over night.)
This is something else that you've entirely misunderstood. Custom collections have zero impact on indexing time.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
Why? You should be using dynamic collections for the most part. How can it be quicker to write a script to create collections, rather than write a rule?
If you look at Logos' community collection such as those Logos 5 collections, they are "manual collections". I was thinking about doing something like that, again, like Scholar (LE). Also some other series that Logos doesn't regard as series. Those are tedious to be done manually.
Mark Barnes said:If you set your highlighting up properly, this won't happen in the future
Then I guess I still don't understand how to set it up correctly. I've been using it for a decade but I just use it. IMO this is a bit over-engineering here, I just want to highlight and they turn it into a document/notebook... Certainly others has better use of highlight but for me it is just when I revisit I know where my eyes should start to stare at...
Mark Barnes said:and I doubt it will take very long to fix the problem for the current highlights.
Long enough for me to give up (and I give up easily since I don't like to routinely clicking things because I know there has to be better way to do it. It might be some macOS bug too. I know in Library cmd+A doesn't work as expected. And I tried in the new notebook cmd+A don't work. May be it works on Windows then I guess it's worth to boot into Windows to do this.
Mark Barnes said:For what it's worth, my own opinion — based on all your questions over the last two weeks — is that you don't yet know enough Logos' web-services or database structure to go anywhere near writing to your databases. I continue to strongly advise that you do not do it.
May be that's when someone should start to gets their hands dirty [;)] I've been trying to be an "obedient" Logos users for a decade but now I decided I should start getting my hands dirty.
And then there's really not much things I can mess up in my Logos "cloud", I don't like to put stuffs in the cloud in a non-transparent manner. I always want to be in control, want everything reproducible. If the system do not support a way to completely throw out everything and start over, the system has a problem. And right now frankly Logos has. In this regard it is the worst software I've ever used. (And the 2nd worst is Apple's iCloud. It isn't the worst because one can largely not rely on iCloud in most cases, and for those one can't escape from, it behaves quite well.) "Just works" doesn't really work, as I often told my wife, "When it works, it's magic. When it doesn't work, it's magic!"
0 -
Mark Barnes said:
This is something else that you've entirely misunderstood. Custom collections have zero impact on indexing time.
Actually I got that impression from https://community.logos.com/forums/p/166404/972021.aspx#972021,
Mark Barnes said:Large collections will cause slow-downs in two main places:
- Startup
- Any time anything changes in your library (new resources, tags/ratings added, community tags downloaded, metadata updated, etc.)
At this time all collections are calculated, regardless of whether they're in use.
This calculation step ain't indexing? I thought you were referring that to indexing. Because say if one is changing a rating, the program is saying "indexing... library result may not be accurate..." something like that.
0 -
Kolen Cheung said:
This calculation step ain't indexing? I thought you were referring that to indexing. Because say if one is changing a rating, the program is saying "indexing... library result may not be accurate..." something like that.
No it isn't. And the program says: "Updating library catalog", not "indexing".
Have joy in the Lord!
0 -
Kolen Cheung said:
If you look at Logos' community collection such as those Logos 5 collections, they are "manual collections". I was thinking about doing something like that, again, like Scholar (LE). Also some other series that Logos doesn't regard as series. Those are tedious to be done manually.
If you want a collection like that, you're correct that it will have to be done manually. But you won't be able to automate it, even if you could write to the database — because you won't be able to 100% accurately match resource titles on a wiki page or spreadsheet to the resource titles in your library. They're often not identical.
Kolen Cheung said:Then I guess I still don't understand how to set it up correctly. I've been using it for a decade but I just use it. IMO this is a bit over-engineering here, I just want to highlight and they turn it into a document/notebook... Certainly others has better use of highlight but for me it is just when I revisit I know where my eyes should start to stare at...
In the Highlighting tool hover over the name of the palette (e.g. "Emphasis Markup"). Click the down arrow. You'll then be able to select the notebook where highlights from that palette are stored in the future.
Kolen Cheung said:Long enough for me to give up (and I give up easily since I don't like to routinely clicking things because I know there has to be better way to do it. It might be some macOS bug too. I know in Library cmd+A doesn't work as expected. And I tried in the new notebook cmd+A don't work. May be it works on Windows then I guess it's worth to boot into Windows to do this.
I know cmd+A doesn't work. But (at least on Windows), SHIFT+click does. Click on the first note in the notebook. SHIFT+click on the last. Drag to the new notebook. Delete the now empty notebook. Four clicks.
Kolen Cheung said:May be that's when someone should start to gets their hands dirty
I've been trying to be an "obedient" Logos users for a decade but now I decided I should start getting my hands dirty.I enjoy poking around behind the scenes. I'm not trying to stop you from doing that. I am suggesting very strongly that you are 100% confident you've understood the format and how everything works before you write to the database. I've spent years poking around to extract resource metadata and use it in various projects. I think I have a good understand of everything from verse mapping to hidden library fields to the web services. Despite that, I've never written to the databases.
So if you get in a mess, you'll be completely on your own.
Kolen Cheung said:I always want to be in control, want everything reproducible.
That was fine in 1990. This is 2019. The vast majority of software and services doesn't work like that any more.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
This calculation step ain't indexing? I thought you were referring that to indexing. Because say if one is changing a rating, the program is saying "indexing... library result may not be accurate..." something like that.
As Mick says, when you change a rating you get an entirely different message ("Updating library catalog…"). That's not indexing. Indexing occurs when you add or update resources. Updating the library catalog occurs when you add resources or update metadata.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
because you won't be able to 100% accurately match resource titles on a wiki page or spreadsheet to the resource titles in your library. They're often not identical.
Text manipulation is often messy, a pity. But one can try to canonicalize both side and compare if they are the same. That’s the data cleaning part. Moreover, while being 100% accurate is nice, if spending 10% of time getting one 90% there then I’m all for it. I don’t even know how to manually add hundreds of resources by hand. But again I hope Logos will do it, as they have said. But the logos community collections last updated years ago, and years bejng Logos users taught me unfortunately never trust their promises. Good intension, it there are always obstacles that make things dally and delay. Bob once promised me PBB reverse interlinear soon almost a decade ago, and it hasnt arrived yet. I hope that this feature came soon since the brother I know spending his whole life working on a Chinese reverse interlinear and a fortune on an in house bible software is, you know, getting very old. He actualy once produced a reverse interlinear Chinese recovery version in NT but due to a history related to the people involved in recovery version, it never s got published. Anyway.
Mark Barnes said:But (at least on Windows), SHIFT+click does.
Unfortunately I tried that on Mac and it doesn't work. Probably should give Windows a try. I wish Logos develop multi platform apps like Microsoft did to its office. Logos is just not behaving like a Mac app. (I was tempted to use the other Mac native Bible software. But I first drawn to Logos by a unique resource published by Lexham. So I can't live without Logos. Imagine I can decrypt Logos format and convert them to the other format such that resources and the softwares are decoupled. Among all people one would guess Christians should have more incentive to have a common standard. I guess the unique complexity involved in Biblical related resources are too much and yet their world are too small for industrial standard to win.
Mark Barnes said:That was fine in 1990. This is 2019. The vast majority of software and services doesn't work like that any more.
Most I used does. Among those I used Logos is uniquely behind in this aspect as if they don't even considered this.
And from my personal experience, it's getting easier and easier to do this. Partly because I learnt more, mostly because the world is so big and new tools emerges that makes this possible and easier.
May be one day I'll be fully using open source softwares and OS. But I'm greedy and I want the best and some of the best is still proprietary GUI like Logos, Lightroom, etc. (not to mention in the case of Logos it is the resources + feature/data sets that set it apart, not the software, making an open source alternative almost absence.) That's partly why I started this thread because I worry for a future that the "philosophical distance" between me and Logos are getting farther and farther away. E.g. Applescript support is also important but from Logos' not Apple native commitment seems to be low priority. I just can't imagine how I can still tolerate those clicking a decade from now, which could've been scripted. Like PBB, I have a very small collection of PBB and managing that already is a headache. It is relatively easy to batch obtain a lot of public domain resources to be added by PBB, but the bottleneck is the clicking of thousands times. May be that API already exist but it isn't clear what function does or doesn't.
0 -
NB.Mick said:Kolen Cheung said:
This calculation step ain't indexing? I thought you were referring that to indexing. Because say if one is changing a rating, the program is saying "indexing... library result may not be accurate..." something like that.
No it isn't. And the program says: "Updating library catalog", not "indexing".
Right, I just recalled it from memory so I wasn't using the exact wording.
However, it must be doing more than just updating library catalog, because if so it is just one single atomic transaction which should be very fast. What I imagine it is doing is that it is like adding the record and then perform an optimization to that database. Lightroom does something like that. Lightroom's catalog is an SQLite database, and there's an "optimize catalog" button allowing the user to optimize and cleanup the catalog on demand, but not on real time. So that a mere change of a tag/rating/metadata won't have that much a penalty.
So if what I said is true, I suggest Logos to separate these 2 steps. e.g. try manually tagging and rating the new base package one has bought, the "updating library catalog" is very annoying. The user is almost like racing with that when rating resources one by one.
So indexing / updating library catalog or else, the reason they take so long to finish is to "pre-cache" something so that in the future a certain operation is very fast, like O(1) kind of fast. So if this is indeed what's happening behind the scene when one has a ton of collections, I don't care. Because the slowness only shows up in the preparatory step, not in real time when one need to interact with resources. If not true, then probably they should tell us what's the expectation here.
0 -
Yes, the contents of the resource files are encrypted. This is a DRM issue.
One more question: is .lbspbb essentially identical to .logos4 in terms of spec and DRM? I peek into one of them and it sort of looks like DRM'd. If not I'd like to be able to have the spec of lbspbb to generate it directly without going through Logos' PBB compiler. And if yes, why DRM is applied to PBB?
0 -
Kolen Cheung said:
One more question: is .lbspbb essentially identical to .logos4 in terms of spec and DRM? I peek into one of them and it sort of looks like DRM'd. If not I'd like to be able to have the spec of lbspbb to generate it directly without going through Logos' PBB compiler.
Logos PBB compiler includes metadata that is needed for .lbspbb to be usable, which is synced. Hence puzzled by desire to generate lbspbb without using PBB compiler ?
Keep Smiling [:)]
0 -
Kolen Cheung said:
One more question: is .lbspbb essentially identical to .logos4 in terms of spec and DRM? I peek into one of them and it sort of looks like DRM'd. If not I'd like to be able to have the spec of lbspbb to generate it directly without going through Logos' PBB compiler.
Logos PBB compiler includes metadata that is needed for .lbspbb to be usable, which is synced. Hence puzzled by desire to generate lbspbb without using PBB compiler ?
Keep Smiling
- Get rid of MS Word. Since the beginning they introduce PBB (post Logos 4), I thought it was a big mistake to make it depends on MS Word.
- A “principle” reason is that makes it not free. Imagine someone can’t afford to buy any Logos resources for various reasons (which is not uncommon), they can get free Logos engine, they can get some free Logos resources (recall when they first introduced PBB there were very few free Logos resources), then the next logical step is PBB for classic, public domain resources. But without a commercial MS Office, one can’t produce that.
- MS Word’s spec is in principle open, but in reality a very complicated thing that requires a lot of reverse engineering.
- A free option to generate docx exists, but the problem from its spec making generating docx a hit and miss thing. Sometimes the generated docx is corrupted, and occasionally opening in MS Word and “save as” can fix that, which again requires the commercial application.
- Authoring a book in MS Office is almost a joke. MS Word is just not for book publishing (The state of the art is LaTeX, and other solutions exists, and yes a lot of publishers accept Word but they are doing their conversions and editing). Regardless the origin of PBB (public domain books, “self-publishing” kind of long form notes, etc.), being a PBB almost guarantee it is very long. Producing long, structured documents is very hard in MS Office (I mean making sure the whole document falls in a uniform typesetting standard is hard. It is easy to apply a slightly different choice of, font, font size, indentation, section, etc. somewhere in the document and hard to spot.)
- Do something PBB compiler can’t do. If indeed lbspbb really support everything logos4 does, then although we aren’t suppose to author logos4 documents (for the prevention of piracy), we can author lbspbb directly for features already supported in the spec but not by the PBB compiler. One such example is (reverse) interlinear than Bob has promised but still didn’t deliver (I think they changed their mind) after almost a decade.
Note that I have my own solution to (1) so the main focus is (2), which focus on getting features not supported by the PBB compiler.
1 - Get rid of MS Word. Since the beginning they introduce PBB (post Logos 4), I thought it was a big mistake to make it depends on MS Word.
-
Kolen Cheung said:
Get rid of MS Word.
LibreOffice is a free alternative that can create docx file(s) for Personal Book(s).
Years ago remember learning a MS Word trick to save a document as RTF followed by opening RTF for save as docx.
Kolen Cheung said:One such example is (reverse) interlinear ...
Reverse Interlinear resource additions are *.lbsrvi
Keep Smiling [:)]
0 -
" rel="nofollow">Keep Smiling 4 Jesus
said:LibreOffice is a free alternative that can create docx file(s) for Personal Book(s).Yes, if you read a few rows after that,
Kolen Cheung said:A free option to generate docx exists, but the problem from its spec making generating docx a hit and miss thing. Sometimes the generated docx is corrupted, and occasionally opening in MS Word and “save as” can fix that, which again requires the commercial application.
I should’ve said “free options” as there are many. But none of them is perfect. Speaking of Libre Office, it has many problems. e.g. HTML to docx conversion can sometimes be extremely slow which takes hours, overnight conversion. While its cli program,
sofficeis useful for scripting (and hence automation and batch processing), it cannot be parallelized (this can be gotten around by setting it up as a daemon which makes parallel execution very complicated.)Reverse Interlinear resource additions are *.lbsrvi
I think the main point is to open the spec of native Logos formats so that personal book is not limited to what the PBB compiler supports.
Yes, there is DRM concern, but it is irrelevant. Just like e.g. epub, the spec is open, and DRM is optional. i.e. Logos can release the spec of all their formats, while on top of it keep using their DRM for official, commercial products.
Even MS office releases the spec of their XML formats. So why not for Logos?
As an example, this is the steps involved in PBB output from BibleMultiConverter:
- detect versification in BibleMultiConverter (no output yet)
- export to “Logos HTML” in BibleMultiConverter, which is in HTML format, but written in a way to includes Logos PBB syntax
- HTML to docx conversion, e.g. using soffice from Libre Office
- docx to docx in 2 passes by BibleMultiConverter for syntax fixes
- docx to PBB using Logos' PBB compiler
All these conversion steps (ignoring the versification detection) can be done in 1 go if a spec in the final format is given, say the current
.lbspbb.So the issue isn’t related to the file extension (file extension is an arbitrary convention that didn’t change its content, its spec is the one that’s relevant. In principle given the some file extension, or even no extension at all, as long as e.g. the mime type is provided it’ll be fine), but the spec to generates these files.
0 -
Mark Barnes said:
For what it's worth the code is on GitHub. The code wasn't written for the public, and I'm not really willing to answer lots of questions about it, but you can start with the update_metadata function.
But this service is probably more useful than my code. Lots of the functions require authorisation (i.e. your Logos cookie), which you can't generate through the site. But GetMetadata works unauthenticated. Just add the ResourceIDs in the request XML.
Thanks Mark again. I just started to use this and it is actually quite useful. The only bump I got is I spent almost an hour just to figure out resourceIds is something like 'LLS:EEC15EZR' not '24998' (which I assumed from the REST API...)
0 -
The numbers are product IDs, not resource IDs. Products are what is sold (e.g. “Logos 8 Gold”). Products often (but not always) contain multiple resources. You only need product IDs if you’re interacting with the store.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Thanks. I figured that out.
0 -
Kolen Cheung said:
Thanks Mark again. I just started to use this and it is actually quite useful. The only bump I got is I spent almost an hour just to figure out resourceIds is something like 'LLS:EEC15EZR' not '24998' (which I assumed from the REST API...)
We appreciate your enthusiasm for our products and your desire to programmatically explore the Faithlife platform. We have public facing developer documentation for a couple of our web APIs at https://developer.faithlife.com/, but these APIs were specifically designed with public consumption in mind and have built in rate limiting. Unfortunately, some of our older APIs have not yet been retrofitted with rate limiting and there is potential for the additional load to impact other users of our services.
We don’t object to you calling external APIs called by the desktop application, but we do ask that you throttle your calls to our APIs to a couple hundred requests per minute. We’ll plan to add rate limiting in the near future.
Thanks for your understanding!
0
