[feature request] scriptable API to interface with Logos
The feature request is simple to state: provide a programable/scriptable interface to Logos Desktop. Example programming language are Python, Lua.
Has this ever been suggested before? And if so what’s the reception on this and Logos/Faithlife’s view on the matter?
One can formulate a formal proposal but we’re not yet there. Just name a few advantages and reasons:
- some routine stuffs can be done more effectively with a program. e.g.
- Let’s say from Wayback Machine I find the old listing of Logos 4 (LE) and want to create a custom collection on it, it is very tedious to drag and drop hundreds of resources to create a collection, but with scripting it is much faster and scalable (to any given list once data cleaned up.)
- grab definitions from https://wiki.logos.com/Canonical_Commentary_Collections and loop through them to create 100s of collections.
- write a script to get ratings from <bestcommentaries.com>, then add them as 5 star scale ratings in logos, updated routinely.
- listing out some information that is not shown through the software, e.g. in that list subjects such as O.T.Leviticus, O.T.Pentateuch exists but clicking on info panel doesn’t show this.
- to cook up some functionality not yet exist in Logos. e.g. there aren’t a tool to merge a few notebooks into one. But with a proper API one can again create a notebook consists of contents of the others easy.
- enabling data analysis using datasets available in Logos. e.g.
- I used to do stuffs like this manually a decade ago: counting no. of verses per book/testament/bible and doing statistics on it (e.g. total no. of original words, average no. of words per chapter, longest book by chapter/verse/words/etc.) These can be done easily, and repeated (say with a new resource or new update) if such API exists.
- e.g. one can analyze the graph of cross references between the resources and use that partly to determine a popularity of a resource and rates accordingly. (this depends on the no. of resources one has, so one could even think that’s a web API that Logos' server answering queries of stuffs like this, having access to the metadata of the whole Logos library. And this paid by hours of access like AWS.)
- This can enable some other form of research not often done yet. e.g. my advisor in the old days (more than a decade ago when terminology like “big data” doesn’t exist yet) had performed a research to study complexity between languages given Bible translated in those languages. With Logos' metadata one can ask similar but more advanced, correct questions: e.g. using the reverse interlinear datasets, one can construct matrices of “semantic clusters” between the original language and the translated language, and from this one can infer the complexity of the resulted translation, comparing to the original language. This is interesting to know, and can be used as some sort of index for Bible translations in terms of formal vs. dynamic correspondence.
- allowing one to combine tools from programming and Logos' dataset to perform unique kind of research. e.g. using natural language processing from a library together with Logos' datasets/internal resources.
- plugins can be written by the community to enhance Logos in various ways. e.g. one can imagine the automation on creations of PBBs, even not from docx by chaining tools such as pandoc.
- allow one to create resources that aren’t supported by PBB. Examples such as a interlinear / reverse interlinear Bible. I know a very old brother who has a database of that (in Chinese), which has been completed years ago. He insisted it should be given away for free. He hired a programmer to create a Bible software to show that interlinear tool (somewhat similar to Interlinear Scripture Analyzer in what it does.) But the programmer was too expensive, and left, and he’s left with a software only works in Windows XP and no documentation so no one to follow up to improve upon. I think he has talked with Bob years ago (who told him it took at least a million for create such software, sort of discouraging him to do so.) I asked Bob a decade ago if PBB would support the creation of interlinear / release interlinear and he answered yes but that feature never materialized yet. Even if that feature in PBB exist, it would be very tedious to convert a database into the Word format PBB required. It would be much easier if Logos instead open up the internals of generating a Logos resource through an API. This will democratize the creation of resources and Bible software (since now one don’t have to create their own in order to give away for free. Instead direct people to download free Logos engine, provide the resource for free as a file.) One can even create this resource and donate to Logos to give away for free (because without a way to create this on our own, Logos won’t invest his staffs' time to create it and give away.)
Out of the many reasons that one might want to reject this idea, I’ll only start to reply to this most important one I can think of: this would be extra work for Logos/Faithlife to provide such a feature for such a niche audience. The investment is not worth it. Now the counterpoint to this:
- Logos' employees has to have ways to touch the internals for various reasons. Such API, which might already exist in various forms for their internal use to debug stuffs, can be refactored to allow part of them to be public facing. Such release is only going to make their employees' experience so much better (because for something to be ready to the general public, it has to pass some sort of user-friendliness test). Such improvements would increase the productivities of their employees, and lower the cost to train new ones. (Ok, this is just my opportunistic guess.)
- capability to program was for the few elites but becomes part of the standard curriculum. i.e. the new generations (including mine to some extents) will all know programming as part of their common knowledge, just as Math, etc. For those it is incredibly time wasting to manually clicking dozens times when they know repetitive tasks should always be scripted and automated. i.e. this feature which might seem to be for niche can becomes the mainstream for the next generations. And Logos can again lead in innovations in this aspect. (More example: when I was undergraduate more than a decade ago, one can get a degree in Physics and Mathematics without programming knowledge at all. For now, not that it’s impossible, but it becomes very common for an undergraduate at least know Python to perform some simulations. Some University even choose to teach Introductory Physics with programming embedded in their course such that they can visualize the simulation of their theoretical equations and know how it translate into complex dynamics.)
- allow rapid prototyping some ideas. e.g. those wheels in Bible study, can be coded up easily given the access of APIs. Again this can benefits Logos internally, and also potentially some creative ideas from the community can becomes a feature in next version of Logos.
- not uncommon for softwares to provide a scriptable interface. e.g. macros in MS Office. Lua and/or Javascript scripting support in Adobe Lightroom, Photoshop, etc.
Regarding the programming language of choice, I would suggest providing a Python binding, allowing people to install it in their own environments such that they can use them with other libraries such as nltk, pandas, numpy, seaborn, etc.
Note that Python can be used as an embedded interpreter within Logos too, this is more self-contained for simple scripting but is inferior because one can’t use other than standard library. Also Lua is also a common choice as an embedded scripting language but I think it is too lightweight for doing anything really powerful together with Logos.
Comments
-
In my library, I still have Libronix Corporation. Libronix DLS Object Model Reference. Libronix Corporation, 2002. [8-|]
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Kolen Cheung said:
The feature request is simple to state: provide a programable/scriptable interface to Logos Desktop.
Isaiah 65:24.
https://wiki.logos.com/How_to_Use_the_COM_API (although it's anticipated uses are very different to the ones you suggest).
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Wow. It would be interesting if the forum can parse the verse like PBB does.
And honestly I haven't heard about that at all. Why wouldn't Logos be more vocal about that?
Anyway, that one obviously has a couple of problems, Windows only (or not?), and last committed on March 2011. The choice of C# is problematic too (I guess Logos itself is written in C# too?) And it also stated that it is read only (many of the features suggested above requires the ability to write/create.)
But I'm glad it existed, it means that Logos does not reject that idea. It's just that it wasn't sccessful and may have been designed for people creating third party applications for Logos rather than as something the end user will use directly.
It also means that it is very easy to create a Pythonic, object oriented binding to that, using C FFI. Please consider this, Logos, revive this project and breath new life into it.
0 -
Over the years there have been tools built on it. But over time they become outdated or their creator dies/loses interest which stops maintenance.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
See https://community.logos.com/forums/p/179259/1035772.aspx#1035772 for example on how an API like this could be used.
Note that if more data is given e.g. through cloud (i.e. where I don’t need to own all resources in order to obtain data on all Logos resources), many interesting things can be analyzed:
- focusing on different denominations of Logos packages, one can ask what that distribution would looks like. e.g. if there’s any bias in some denomination base package that favors “good” or “bad” resources.
- focusing on how different people rated a certain resources, using the information e.g. from their denomination specified in their Faithlife profile, one can infers if a particular resources is biased towards being like by a particular group or else. (This one would touches a much larger database though.)
- one can even try to compare this across different base packages “color” to see which one is more optimal.
By the way, on the topic of making buying decision through these data, I really hope that there's something like
from logos.cart import dynamicPricing # assembling a list of resources objects resources = ... dynamicPricing(resources)
It is related to some crazy experience I got when upgrading to Logos 8. I’ve found a very interesting combination, after contacting CS to buy Unfiltered with Platinum with dynamic pricing, which is cheaper than Platinum. But the process to this discovery is time consuming. Having something scriptable to explore multitude of options would be much easier. Another topic for another day.
1 -
Kolen Cheung said:
Note that if more data is given e.g. through cloud (i.e. where I don’t need to own all resources in order to obtain data on all Logos resources), many interesting things can be analyzed:
I can't see Faithlife being able to afford the investment required to give us access to its big data — especially when there's nothing in it for them, or for most users.
There are actually a large number of web services that provide lots of useful information, especially with resource metadata. Some of what you're asking in earlier posts can be done already. None of it is documented though, and none of it is 'public'. (It's publicly accessible, but it's not be designed for public use.) If you want to play around with these services, just watch the traffic between Logos and its servers. The mobile app has some additional services, and the web app some others.
There are no apis or services for the store, however (other than converting a resource ID into a store URL). To be honest, the store is creaking a lot (hence ebooks.faithlife.com), so no APIs are going to be added at least until the whole store's been thoroughly overhauled.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
Wow. It would be interesting if the forum can parse the verse like PBB does.
It's just RefTagger.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:Kolen Cheung said:
Note that if more data is given e.g. through cloud (i.e. where I don’t need to own all resources in order to obtain data on all Logos resources), many interesting things can be analyzed:
I can't see Faithlife being able to afford the investment required to give us access to its big data — especially when there's nothing in it for them, or for most users.
I assume we're talking their site. Or maybe the library.
But Logos is truly unique in allowing access to raw data. When I first saw Libby (kind of a brouchure), I squinted at the example images ... especially the hebrew. At the time, there was plenty of public domain greek. So I ordered Libby ... CDs. Loaded it, to see if I could get the hebrew. Whoa! After that, I copied and copied, to feed my neural and genetic engines. Personal use, I always followed. Follow the rules.
When Logos4 downloaded, that's what I checked first. Export the data? Yep! Oh my. Molassas but speeding bullet downloaded.
And that tiny piece of value was everything to me. They had to have allowed research on purpose. Couldn't be accidental. Had to be Bob.
To my knowledge, no one allows bulk access like Logos does. Accordance and Bibleworks, piecemeal. Some of the mobiles did, by accident, but public domain.
Sure, a high-speed script maybe. But no user would ever be happy ... differing needs.
I'll never forget that Libby brouchure. It was heaven. A lady in a sewing room could calculate redaction layers from 3,000 years ago.
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
Kolen Cheung said:
And honestly I haven't heard about that at all. Why wouldn't Logos be more vocal about that?
Having to support power users as they do complex things is not something they are eager to do as they don't have the staff for it. I wonder if they originally created this just for their own use and then left it exposed to users just because, figuring very few would discover it and use it but those who did might be happy it was there.
0 -
Rosie Perera said:
Having to support power users as they do complex things is not something they are eager to do as they don't have the staff for it.
Read the 2nd bullet clusters in the OP for the debunk of this.
In short, many company open source their softwares (sometimes not all of their softwares but a particular stack, e.g. Twitter Bootstrap) for the reason that the more user uses it, the more problems they discovers (some are bugs, some are usability issue.) And in the end they get the benefits from it.
What I tried to argue is if Logos provides a Pythonic, Object Oriented interface, that can benefits its own employees for faster prototyping, etc. Having an API well documented itself can benefits the employees somehow.
Also as they did in 2011, releasing it in somewhere like GitHub convey the proper message that this is not for the average users but programmers. RTFM (F stands for Funny, Functional) is the standard in any such kinds of open source projects (be it only the API binding is open sourced) so supporting them shouldn't be that difficult one would have imagine. (If people ask stupid question over there, I'd politely say that project is not for them and they should start by learning the Logos GUI in this forum first.)
So this lack-of-staff-to-support is a myth. One should try to think how to organize it in a way that can directly benefits Logos (and I argued by benefiting its own staffs for ease of training, rapid prototyping, increased productivity.) Once that formulated it becomes something worth invested in (because it is not consuming but providing productivity.)
Identifying "power user" as a minority group is some sort of myth too. Because the prior of what Logos currently is largely defines these users as minority; also, as I've argued in the OP, these minority group is growing and eventually every educated person say from high school has this as part of their common knowledge. We're growing into the future where people expects automations and batch processing and increasingly going to ask the question: why am I clicking this and that routinely but not script it into automated processes?
0 -
0
-
Mark Barnes said:
I can't see Faithlife being able to afford the investment required to give us access to its big data
Limits can be applied. Many such web APIs are like this. You can't do without else you're under attack. (e.g. NYTimes API, Twitter API, GitHub API.)
0 -
Mark Barnes said:
There are actually a large number of web services that provide lots of useful information, especially with resource metadata.
Note that this doesn't cover many of the examples in the OP though.
Arguably the most valuable datasets to explore is the one we pay for, since Logos 5 days. These manually tagged metadata is very valuable (in terms of accuracy) to be explored, vs. e.g. natural language processing which can be wrong (not saying human can't be wrong though, but like the degree of accuracy is different, and also manual tagging can be fixed in time.)
Just as simple as the "semantic" mixing matrix I mentioned above cannot be done without touching Logos' reverse interlinear datasets. At least not in large scale, reproducible manner. In the past, I've even manually created these matrices for the word I'm studying, and quickly found to time consuming to not carry out that frequently. Logos word study wheels appears later in a certain Logos version does show part of this matrix (either a row or a column of information, but never the whole matrix.)
Note that one can resort to export Logos resources and perform data cleaning, but why make it so hard and error prone?
0 -
Kolen Cheung said:
RefTagger is added to the forum (it's javascript), and RefTagger auto tags them.
Kolen Cheung said:Mark Barnes said:I can't see Faithlife being able to afford the investment required to give us access to its big data
Limits can be applied. Many such web APIs are like this. You can't do without else you're under attack. (e.g. NYTimes API, Twitter API, GitHub API.)
The major cost isn't server time, it's development time.
Kolen Cheung said:Mark Barnes said:There are actually a large number of web services that provide lots of useful information, especially with resource metadata.
Note that this doesn't cover many of the examples in the OP though.
I know. But it's better than nothing.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
Has this ever been suggested before? And if so what’s the reception on this and Logos/Faithlife’s view on the matter?
Yes, it has been, and we built a simple API: https://wiki.logos.com/Logos_4_COM_API
No one (to a close approximation) used it.
We periodically consider adding more API support or embedding a scripting language or similar other approaches. It's not on the roadmap for 2019.
Kolen Cheung said:Out of the many reasons that one might want to reject this idea, I’ll only start to reply to this most important one I can think of: this would be extra work for Logos/Faithlife to provide such a feature for such a niche audience. The investment is not worth it.
You have hit the proverbial nail on the head. Your counterpoints don't (IMHO) sufficiently address the large (and ongoing) cost of building, documenting, supporting, and maintaining a public API.
Kolen Cheung said:Anyway, that one obviously has a couple of problems, Windows only (or not?), and last committed on March 2011. The choice of C# is problematic too (I guess Logos itself is written in C# too?) And it also stated that it is read only (many of the features suggested above requires the ability to write/create.)
It's a COM API; the choice of programming language for the example code is irrelevant. I believe there are people consuming it via Python. (But yes, it is Windows-only; see comments about "cost" above when proposing a cross-platform solution.)
0 -
"No one (to a close approximation) used it."
I did! So references in my Bibleworks notes could jump into Logos
"The whole modern world has divided itself into Conservatives and Progressives. The business of Progressives is to go on making mistakes. The business of Conservatives is to prevent mistakes from being corrected."- G.K. Chesterton
0 -
Mark Barnes said:Kolen Cheung said:Mark Barnes said:
I can't see Faithlife being able to afford the investment required to give us access to its big data
Limits can be applied. Many such web APIs are like this. You can't do without else you're under attack. (e.g. NYTimes API, Twitter API, GitHub API.)
The major cost isn't server time, it's development time.
That is arguable. As long as servers need to communicate there exists some form of API, whether you choose to expose it or not (or make it exposable.)
Just to list one example of things I'd like to query Logos' server, I'd like to be able to send a request to obtain all subjects on all resources Logos has, for example. (This request again can be limited like how other Web API is used.) Now this is not something serving logos.com would have needed and already there, but just an example of things I'd want to get access to.
0 -
Kolen Cheung said:
I'd like to be able to send a request to obtain all subjects on all resources Logos has, for example.
An undocumented SOAP interface allows you to obtain the metadata of resources where you know the resourceID. Getting resourceIDs is more tricky, although your library is a good start. If you're willing to dig around and write some code, you can find out a lot. Here's my list of subjects: https://www.logosresourcesguide.com/subject.php
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Oh, great! Thanks for pointing out this tool.
Performing a `head` on a resource will show the resourceID. By the way, do you know how to decode the binary logos4 format?
0 -
Actually did you release your source code of library on getting these metadata? Else are there any pointers to where I start? I want to be able to slice at the data itself.
Also a bunch of related questions:
- did you know how, once a list of resource is obtained, to create a collection in Logos (i.e. programmatically but not manually drag and dropping, once per resource)
- some how obtain a collection of old Logos packages (such as "Scholar (LE)")? Is that information somewhere?
0 -
Kolen Cheung said:
Actually did you release your source code of library on getting these metadata
It's not at all supported, but others have written tools that just directly read from the LibraryCatalog SQLite database that contains all of your resource metadata. That seems far simpler than trying to access the service APIs.
https://community.logos.com/forums/t/85971.aspx
Note before you go trying to ask the author questions, he passed a few years ago.
Kolen Cheung said:did you know how, once a list of resource is obtained, to create a collection in Logos (i.e. programmatically
Hacking in changes to synced data (data created in the application that is kept in sync between installations by storing it on the server) is something we strongly recommend against. It's a potential way to corrupt or otherwise lose your synced user data.
Andrew Batishko | Logos software developer
0 -
Kolen Cheung said:
do you know how to decode the binary logos4 format?
Among other things, the license agreement says that we may not reverse engineer, disassemble, decompile or make any attempt to discover the source code of the Software, ... or ... "unlock" or circumvent the digital copyright protection of the Content.
I don't know if talking about their file format is a grey area or not, but we should respect the license agreement.
Thanks to FL for including Carta and a Hebrew audio bible in Logos 9!
0 -
directly read from the LibraryCatalog SQLite database
Did you refer to this one?
~/Library/Application Support/Logos4/Data/gf1dasn3.0cf/LibraryCatalog/catalog.dbWhat about the db that Mark Barnes website ask us to upload?
~/Library/Application Support/Logos4/Data/gf1dasn3.0cf/ResourceManager/ResourceManager.dbI’ll peek into it probably at the weekend. Starting from the db first seems easier than scrapping the whole Logos db that Mark has done. (Can Mark share that by the way? Either the software of obtaining this db preferably or just the db you’ve already collected.)
Hacking in changes to synced data (data created in the application that is kept in sync between installations by storing it on the server) is something we strongly recommend against. It's a potential way to corrupt or otherwise lose your synced user data.
That’s a pity. You know, no offense, but I completely don’t trust Logos' cloud (or any cloud trying to be “just work”.) Simply put, that took away control from the user, so that experiments like this becomes very dangerous. A few days ago I accidentally removed 350 ratings from my library and that’s scary because there’s no way to recover that (there’s no universal undo like e.g. Lightroom does.) In the end I force quit the Logos (which still seems to late) and I jump right to my other computer and disconnect its internet immediately in order to retain the old, not-yet-synced library and then doing some manual migration of ratings to tags just so that after it syncs I can recover the ratings from these tags.
On this matter, that’s why I didn’t create any document (except 1 for historical reason (which is on church history), and some other testing documents, also other stuffs are also classified as documents in Logos which I did have, like visual filters, highlights, reading plans etc.) in Logos, despite some of them being quite attractive. The good old PBB is better (although more time consuming to tag) because the user is in control.
More on PBB, it would have been better if it was designed using a plaintext syntax such as markdown (and create Logos specific extension to it just as PBB did using docx), Word is just too heavy weight for that and bad for VCS.
PetahChristian said:Among other things, the license agreement says that we may not reverse engineer, disassemble, decompile or make any attempt to discover the source code of the Software, ... or ... "unlock" or circumvent the digital copyright protection of the Content.
I don't know if talking about their file format is a grey area or not, but we should respect the license agreement.
I'm not talking about the source code of the software, but the spec of the digital resource itself. Just like docx, odt, xml, markdown, etc. they all have a spec (and most of the time a reference implementation of reading it, just as Word, etc.) And we're not talking about DRM here, because AFAIK it isn't encrypted. (Just as if you buy a song from iTunes, it isn't encrypted either. Some vendors would like to put DRM to protect them. Luckily for us just as Apple, Logos trust us enough to not encrypt them.) Decoding it is just for doing something Logos doesn't provide yet. e.g. as simple as word count seems to be missing? Also the "semantic mixing matrix" I mentioned above definitely is not there but doable once the resource itself can be read. Not that it would interests anyone in this forum but e.g. treating Bible as a network and doing topological analysis on it, etc. would be fun. (Not sure if anyone has done this before yet though. Because the world is so big...)
0 -
I think I mentioned somewhere already, another major use of a spec of the logos4 format is not reading but writing. Imagine one to write a reverse interlinear from a certain database (that I know a brother spend his whole life working on and want to give away for free) which is impossible from PBB.
Also, I personally want to have Complete Biblical Library which is only available from a Logos competitor. I want to be able to buy it from that competitor, unpack it, and repack it in Logos. But actually the blocking step here is first I'd have to know I can decode (or even decrypt if they have DRM) it before I buy it...
0 -
Kolen Cheung said:
And we're not talking about DRM here, because AFAIK it isn't encrypted
Yes, the contents of the resource files are encrypted. This is a DRM issue.
Andrew Batishko | Logos software developer
0 -
Oh, I’m doomed. May be only the old Libronix files are unencrypted.
DRM removal is much more tricky (but legal) and I can’t possibly be getting help here... Hm...
ok, back to the API thing.
By the way, even if licensed resources are encrypted, a spec can still be provided for user to generate unencrypted resource (like the reverse interlinear I talked about.) Analogy will be DRM ePub and DRM free epub.
0 -
Kolen Cheung said:
Actually did you release your source code of library on getting these metadata? Else are there any pointers to where I start? I want to be able to slice at the data itself.
For what it's worth the code is on GitHub. The code wasn't written for the public, and I'm not really willing to answer lots of questions about it, but you can start with the update_metadata function.
But this service is probably more useful than my code. Lots of the functions require authorisation (i.e. your Logos cookie), which you can't generate through the site. But GetMetadata works unauthenticated. Just add the ResourceIDs in the request XML.
Kolen Cheung said:- did you know how, once a list of resource is obtained, to create a collection in Logos (i.e. programmatically but not manually drag and dropping, once per resource)
This is only possible by writing to the databases Logos uses. For all the reasons Andrew gave, I strongly advise against this. Personally, I only ever read from the databases. I never write to them.
Kolen Cheung said:- some how obtain a collection of old Logos packages (such as "Scholar (LE)")? Is that information somewhere?
Once the products are taken off the website, the information is lost. Wiki editors have tried to preserve much of it.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Kolen Cheung said:
Did you refer to this one?
~/Library/Application Support/Logos4/Data/gf1dasn3.0cf/LibraryCatalog/catalog.dbWhat about the db that Mark Barnes website ask us to upload?
~/Library/Application Support/Logos4/Data/gf1dasn3.0cf/ResourceManager/ResourceManager.dbCatalog.db contains all the metadata in your library. It can get very big. ResourceManager.db is much smaller, but gives all your ResourceIDs (among other things). My website doesn't need catalog.db because I can use those resourceIDs to query the Metadata service.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
For what it's worth the code is on GitHub. The code wasn't written for the public, and I'm not really willing to answer lots of questions about it, but you can start with the update_metadata function.
I don't know that language so it won't be useful for me. But thanks for open sourcing it.
Mark Barnes said:But this service is probably more useful than my code. Lots of the functions require authorisation (i.e. your Logos cookie), which you can't generate through the site. But GetMetadata works unauthenticated. Just add the ResourceIDs in the request XML.
Hm... it blocks me since I'm using content blocker, and "reload without content blockers" doesn't work. Chrome works fine though.
I think I'll start from the local SQL database first.
By the way, have you tried to parse the sitemap and obtain all products from there? I think you said there's some reason you can't get the Faithlife ebooks unless you know its existence. Did you try parsing Faithlife's sitemap to get them? I only parsed logos.com's sitemap so don't know what's there yet.
Mark Barnes said:This is only possible by writing to the databases Logos uses. For all the reasons Andrew gave, I strongly advise against this. Personally, I only ever read from the databases. I never write to them.
I really would like to write to it eventually, e.g. for creating collection. I created ~100 collections last weekend and it is very tedious to click click click. But I think I've had enough warning to take the computer offline first and check if things are working before allowing it to sync.
Another thing I want to do is to merge all highlighting scattering into 3 notebooks into one (somehow Logos has different kinds of highlights, some called emphasis marking, and applying them would fall into different kinds of notebooks automatically...) I think currently there's no GUI way to do this (like cmd+A and drag won't work.)
Mark Barnes said:Once the products are taken off the website, the information is lost. Wiki editors have tried to preserve much of it.
Too bad... People from Logos seems to have said they might eventually release them as community collections (where they have a bunch Logos 5 collections already.) I also found my old collections pages via archive.org (Wayback Machine.) If Logos doesn't provide it, in principle I just need to cleanup the HTML table and match them with resources I own and create a collection for that. But of course it'd be better if they provide it officially.
0 -
Kolen Cheung said:
. I created ~100 collections last weekend and it is very tedious to click click click.
Much to my disappointment, I ended up deleting many of my collections as they degraded performance ... I don't know to what extent the problem has been resolved although it has been improved.
Rick Ausdahl said:What good is it if it has such a negative affect on performance?
Most collections are fine, and very little is gained by taking the time to prune them back.
We originally envisioned that users might create a few dozen simple automatic collections, such as lang:Greek type:Bible or type:Commentary title:Genesis.
However, we found out that users had much grander visions and started created collections like these ones: https://community.logos.com/forums/p/88829/978914.aspx#978914
(It's somewhat obvious that these were never anticipated when you look at the single-line editing UI where you enter these rules; how are you even supposed to edit them easily?)
Some of these rules take multiple seconds to evaluate, and some users have dozens or hundreds of these rules. This is what can slow down the system.
As we became aware of this problem, we started investigating ways to mitigate it. In some cases, such as "* -mytag:*", we were able to ship an update to Logos that made those queries evaluate hundreds of times faster. But for the Author/Denomination/Tradition/Commentary collections, we were unable to do this; the next best approach we were able to develop was new syntax that was both more precise and much more efficient. You can see some examples here: https://community.logos.com/forums/p/88829/624737.aspx#624737
Mark rewrote his collections to use the new syntax, which addresses most of the performance problems. You shouldn't see any issues from having a dozen or so collections in that format.
But if you've got hundreds of "denomination" collections using the old syntax, especially if you use them very infrequently, you may see substantial performance improvements in Logos by deleting them from your application.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0
