[feature request] scriptable API to interface with Logos

Page 1 of 3 (55 items) 1 2 3 Next >
This post has 54 Replies | 4 Followers

Posts 884
Kolen Cheung | Forum Activity | Posted: Sat, Feb 23 2019 9:46 PM

The feature request is simple to state: provide a programable/scriptable interface to Logos Desktop. Example programming language are Python, Lua.

Has this ever been suggested before? And if so what’s the reception on this and Logos/Faithlife’s view on the matter?

One can formulate a formal proposal but we’re not yet there. Just name a few advantages and reasons:

  • some routine stuffs can be done more effectively with a program. e.g.
    • Let’s say from Wayback Machine I find the old listing of Logos 4 (LE) and want to create a custom collection on it, it is very tedious to drag and drop hundreds of resources to create a collection, but with scripting it is much faster and scalable (to any given list once data cleaned up.)
    • grab definitions from https://wiki.logos.com/Canonical_Commentary_Collections and loop through them to create 100s of collections.
    • write a script to get ratings from <bestcommentaries.com>, then add them as 5 star scale ratings in logos, updated routinely.
  • listing out some information that is not shown through the software, e.g. in that list subjects such as O.T.Leviticus, O.T.Pentateuch exists but clicking on info panel doesn’t show this.
  • to cook up some functionality not yet exist in Logos. e.g. there aren’t a tool to merge a few notebooks into one. But with a proper API one can again create a notebook consists of contents of the others easy.
  • enabling data analysis using datasets available in Logos. e.g.
    • I used to do stuffs like this manually a decade ago: counting no. of verses per book/testament/bible and doing statistics on it (e.g. total no. of original words, average no. of words per chapter, longest book by chapter/verse/words/etc.) These can be done easily, and repeated (say with a new resource or new update) if such API exists.
    • e.g. one can analyze the graph of cross references between the resources and use that partly to determine a popularity of a resource and rates accordingly. (this depends on the no. of resources one has, so one could even think that’s a web API that Logos' server answering queries of stuffs like this, having access to the metadata of the whole Logos library. And this paid by hours of access like AWS.)
    • This can enable some other form of research not often done yet. e.g. my advisor in the old days (more than a decade ago when terminology like “big data” doesn’t exist yet) had performed a research to study complexity between languages given Bible translated in those languages. With Logos' metadata one can ask similar but more advanced, correct questions: e.g. using the reverse interlinear datasets, one can construct matrices of “semantic clusters” between the original language and the translated language, and from this one can infer the complexity of the resulted translation, comparing to the original language. This is interesting to know, and can be used as some sort of index for Bible translations in terms of formal vs. dynamic correspondence.
  • allowing one to combine tools from programming and Logos' dataset to perform unique kind of research. e.g. using natural language processing from a library together with Logos' datasets/internal resources.
  • plugins can be written by the community to enhance Logos in various ways. e.g. one can imagine the automation on creations of PBBs, even not from docx by chaining tools such as pandoc.
  • allow one to create resources that aren’t supported by PBB. Examples such as a interlinear / reverse interlinear Bible. I know a very old brother who has a database of that (in Chinese), which has been completed years ago. He insisted it should be given away for free. He hired a programmer to create a Bible software to show that interlinear tool (somewhat similar to Interlinear Scripture Analyzer in what it does.) But the programmer was too expensive, and left, and he’s left with a software only works in Windows XP and no documentation so no one to follow up to improve upon. I think he has talked with Bob years ago (who told him it took at least a million for create such software, sort of discouraging him to do so.) I asked Bob a decade ago if PBB would support the creation of interlinear / release interlinear and he answered yes but that feature never materialized yet. Even if that feature in PBB exist, it would be very tedious to convert a database into the Word format PBB required. It would be much easier if Logos instead open up the internals of generating a Logos resource through an API. This will democratize the creation of resources and Bible software (since now one don’t have to create their own in order to give away for free. Instead direct people to download free Logos engine, provide the resource for free as a file.) One can even create this resource and donate to Logos to give away for free (because without a way to create this on our own, Logos won’t invest his staffs' time to create it and give away.)

Out of the many reasons that one might want to reject this idea, I’ll only start to reply to this most important one I can think of: this would be extra work for Logos/Faithlife to provide such a feature for such a niche audience. The investment is not worth it. Now the counterpoint to this:

  • Logos' employees has to have ways to touch the internals for various reasons. Such API, which might already exist in various forms for their internal use to debug stuffs, can be refactored to allow part of them to be public facing. Such release is only going to make their employees' experience so much better (because for something to be ready to the general public, it has to pass some sort of user-friendliness test). Such improvements would increase the productivities of their employees, and lower the cost to train new ones. (Ok, this is just my opportunistic guess.)
  • capability to program was for the few elites but becomes part of the standard curriculum. i.e. the new generations (including mine to some extents) will all know programming as part of their common knowledge, just as Math, etc. For those it is incredibly time wasting to manually clicking dozens times when they know repetitive tasks should always be scripted and automated. i.e. this feature which might seem to be for niche can becomes the mainstream for the next generations. And Logos can again lead in innovations in this aspect. (More example: when I was undergraduate more than a decade ago, one can get a degree in Physics and Mathematics without programming knowledge at all. For now, not that it’s impossible, but it becomes very common for an undergraduate at least know Python to perform some simulations. Some University even choose to teach Introductory Physics with programming embedded in their course such that they can visualize the simulation of their theoretical equations and know how it translate into complex dynamics.)
  • allow rapid prototyping some ideas. e.g. those wheels in Bible study, can be coded up easily given the access of APIs. Again this can benefits Logos internally, and also potentially some creative ideas from the community can becomes a feature in next version of Logos.
  • not uncommon for softwares to provide a scriptable interface. e.g. macros in MS Office. Lua and/or Javascript scripting support in Adobe Lightroom, Photoshop, etc.

Regarding the programming language of choice, I would suggest providing a Python binding, allowing people to install it in their own environments such that they can use them with other libraries such as nltk, pandas, numpy, seaborn, etc.

Note that Python can be used as an embedded interpreter within Logos too, this is more self-contained for simple scripting but is inferior because one can’t use other than standard library. Also Lua is also a common choice as an embedded scripting language but I think it is too lightweight for doing anything really powerful together with Logos.

Posts 26477
Forum MVP
MJ. Smith | Forum Activity | Replied: Sun, Feb 24 2019 12:06 AM

In my library, I still have Libronix Corporation. Libronix DLS Object Model Reference. Libronix Corporation, 2002. Geeked

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 13359
Forum MVP
Mark Barnes | Forum Activity | Replied: Sun, Feb 24 2019 1:49 AM

Kolen Cheung:
The feature request is simple to state: provide a programable/scriptable interface to Logos Desktop.

Isaiah 65:24.

https://wiki.logos.com/How_to_Use_the_COM_API (although it's anticipated uses are very different to the ones you suggest).

Posts 884
Kolen Cheung | Forum Activity | Replied: Sun, Feb 24 2019 2:05 AM

Wow. It would be interesting if the forum can parse the verse like PBB does.

And honestly I haven't heard about that at all. Why wouldn't Logos be more vocal about that?

Anyway, that one obviously has a couple of problems, Windows only (or not?), and last committed on March 2011. The choice of C# is problematic too (I guess Logos itself is written in C# too?) And it also stated that it is read only (many of the features suggested above requires the ability to write/create.)

But I'm glad it existed, it means that Logos does not reject that idea. It's just that it wasn't sccessful and may have been designed for people creating third party applications for Logos rather than as something the end user will use directly.

It also means that it is very easy to create a Pythonic, object oriented binding to that, using C FFI. Please consider this, Logos, revive this project and breath new life into it.

Posts 26477
Forum MVP
MJ. Smith | Forum Activity | Replied: Sun, Feb 24 2019 2:37 AM

Over the years there have been tools built on it. But over time they become outdated or their creator dies/loses interest which stops maintenance.

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 884
Kolen Cheung | Forum Activity | Replied: Sun, Feb 24 2019 5:16 AM

See https://community.logos.com/forums/p/179259/1035772.aspx#1035772 for example on how an API like this could be used.

Note that if more data is given e.g. through cloud (i.e. where I don’t need to own all resources in order to obtain data on all Logos resources), many interesting things can be analyzed:

  • focusing on different denominations of Logos packages, one can ask what that distribution would looks like. e.g. if there’s any bias in some denomination base package that favors “good” or “bad” resources.
  • focusing on how different people rated a certain resources, using the information e.g. from their denomination specified in their Faithlife profile, one can infers if a particular resources is biased towards being like by a particular group or else. (This one would touches a much larger database though.)
  • one can even try to compare this across different base packages “color” to see which one is more optimal.

By the way, on the topic of making buying decision through these data, I really hope that there's something like

from logos.cart import dynamicPricing

# assembling a list of resources objects
resources = ...
dynamicPricing(resources)

It is related to some crazy experience I got when upgrading to Logos 8. I’ve found a very interesting combination, after contacting CS to buy Unfiltered with Platinum with dynamic pricing, which is cheaper than Platinum. But the process to this discovery is time consuming. Having something scriptable to explore multitude of options would be much easier. Another topic for another day.

Posts 13359
Forum MVP
Mark Barnes | Forum Activity | Replied: Sun, Feb 24 2019 7:50 AM

Kolen Cheung:
Note that if more data is given e.g. through cloud (i.e. where I don’t need to own all resources in order to obtain data on all Logos resources), many interesting things can be analyzed:

I can't see Faithlife being able to afford the investment required to give us access to its big data — especially when there's nothing in it for them, or for most users.

There are actually a large number of web services that provide lots of useful information, especially with resource metadata. Some of what you're asking in earlier posts can be done already. None of it is documented though, and none of it is 'public'. (It's publicly accessible, but it's not be designed for public use.) If you want to play around with these services, just watch the traffic between Logos and its servers. The mobile app has some additional services, and the web app some others.

There are no apis or services for the store, however (other than converting a resource ID into a store URL). To be honest, the store is creaking a lot (hence ebooks.faithlife.com), so no APIs are going to be added at least until the whole store's been thoroughly overhauled.

Posts 13359
Forum MVP
Mark Barnes | Forum Activity | Replied: Sun, Feb 24 2019 7:51 AM

Kolen Cheung:
Wow. It would be interesting if the forum can parse the verse like PBB does.

It's just RefTagger.

Posts 10177
Denise | Forum Activity | Replied: Sun, Feb 24 2019 8:31 AM

Mark Barnes:

Kolen Cheung:
Note that if more data is given e.g. through cloud (i.e. where I don’t need to own all resources in order to obtain data on all Logos resources), many interesting things can be analyzed:

I can't see Faithlife being able to afford the investment required to give us access to its big data — especially when there's nothing in it for them, or for most users.

I assume we're talking their site. Or maybe the library.

But Logos is truly unique in allowing access to raw data. When I first saw Libby (kind of a brouchure), I squinted at the example images ... especially the hebrew. At the time, there was plenty of public domain greek. So I ordered Libby ... CDs. Loaded it, to see if I could get the hebrew. Whoa! After that, I copied and copied, to feed my neural and genetic engines. Personal use, I always followed. Follow the rules.

When Logos4 downloaded, that's what I checked first. Export the data? Yep! Oh my. Molassas but speeding bullet downloaded.

And that tiny piece of value was everything to me. They had to have allowed research on purpose. Couldn't be accidental. Had to be Bob.

To my knowledge, no one allows bulk access like Logos does. Accordance and Bibleworks, piecemeal. Some of the mobiles did, by accident, but public domain.

Sure, a high-speed script maybe. But no user would ever be happy ... differing needs.

I'll never forget that Libby brouchure. It was heaven. A lady in a sewing room could calculate redaction layers from 3,000 years ago.


Posts 18668
Rosie Perera | Forum Activity | Replied: Sun, Feb 24 2019 9:29 AM

Kolen Cheung:
And honestly I haven't heard about that at all. Why wouldn't Logos be more vocal about that?

Having to support power users as they do complex things is not something they are eager to do as they don't have the staff for it. I wonder if they originally created this just for their own use and then left it exposed to users just because, figuring very few would discover it and use it but those who did might be happy it was there.

Posts 884
Kolen Cheung | Forum Activity | Replied: Sun, Feb 24 2019 2:46 PM

Rosie Perera:
Having to support power users as they do complex things is not something they are eager to do as they don't have the staff for it.

Read the 2nd bullet clusters in the OP for the debunk of this.

In short, many company open source their softwares (sometimes not all of their softwares but a particular stack, e.g. Twitter Bootstrap) for the reason that the more user uses it, the more problems they discovers (some are bugs, some are usability issue.) And in the end they get the benefits from it.

What I tried to argue is if Logos provides a Pythonic, Object Oriented interface, that can benefits its own employees for faster prototyping, etc. Having an API well documented itself can benefits the employees somehow.

Also as they did in 2011, releasing it in somewhere like GitHub convey the proper message that this is not for the average users but programmers. RTFM (F stands for Funny, Functional) is the standard in any such kinds of open source projects (be it only the API binding is open sourced) so supporting them shouldn't be that difficult one would have imagine. (If people ask stupid question over there, I'd politely say that project is not for them and they should start by learning the Logos GUI in this forum first.)

So this lack-of-staff-to-support is a myth. One should try to think how to organize it in a way that can directly benefits Logos (and I argued by benefiting its own staffs for ease of training, rapid prototyping, increased productivity.) Once that formulated it becomes something worth invested in (because it is not consuming but providing productivity.)

Identifying "power user" as a minority group is some sort of myth too. Because the prior of what Logos currently is largely defines these users as minority; also, as I've argued in the OP, these minority group is growing and eventually every educated person say from high school has this as part of their common knowledge. We're growing into the future where people expects automations and batch processing and increasingly going to ask the question: why am I clicking this and that routinely but not script it into automated processes?

Posts 884
Kolen Cheung | Forum Activity | Replied: Sun, Feb 24 2019 3:01 PM

Mark Barnes:
It's just RefTagger.

But Logos forum hasn't auto tag them, isn't it?

Posts 884
Kolen Cheung | Forum Activity | Replied: Sun, Feb 24 2019 3:03 PM

Mark Barnes:
I can't see Faithlife being able to afford the investment required to give us access to its big data

Limits can be applied. Many such web APIs are like this. You can't do without else you're under attack. (e.g. NYTimes API, Twitter API, GitHub API.)

Posts 884
Kolen Cheung | Forum Activity | Replied: Sun, Feb 24 2019 6:03 PM

Mark Barnes:
There are actually a large number of web services that provide lots of useful information, especially with resource metadata.

Note that this doesn't cover many of the examples in the OP though.

Arguably the most valuable datasets to explore is the one we pay for, since Logos 5 days. These manually tagged metadata is very valuable (in terms of accuracy) to be explored, vs. e.g. natural language processing which can be wrong (not saying human can't be wrong though, but like the degree of accuracy is different, and also manual tagging can be fixed in time.)

Just as simple as the "semantic" mixing matrix I mentioned above cannot be done without touching Logos' reverse interlinear datasets. At least not in large scale, reproducible manner. In the past, I've even manually created these matrices for the word I'm studying, and quickly found to time consuming to not carry out that frequently. Logos word study wheels appears later in a certain Logos version does show part of this matrix (either a row or a column of information, but never the whole matrix.)

Note that one can resort to export Logos resources and perform data cleaning, but why make it so hard and error prone?

Posts 13359
Forum MVP
Mark Barnes | Forum Activity | Replied: Mon, Feb 25 2019 3:44 AM

Kolen Cheung:

Mark Barnes:
It's just RefTagger.

But Logos forum hasn't auto tag them, isn't it?

RefTagger is added to the forum (it's javascript), and RefTagger auto tags them.

Kolen Cheung:

Mark Barnes:
I can't see Faithlife being able to afford the investment required to give us access to its big data

Limits can be applied. Many such web APIs are like this. You can't do without else you're under attack. (e.g. NYTimes API, Twitter API, GitHub API.)

The major cost isn't server time, it's development time.

Kolen Cheung:

Mark Barnes:
There are actually a large number of web services that provide lots of useful information, especially with resource metadata.

Note that this doesn't cover many of the examples in the OP though.

I know. But it's better than nothing.

Posts 8006
LogosEmployee
Bradley Grainger (Faithlife) | Forum Activity | Replied: Mon, Feb 25 2019 11:38 AM

Kolen Cheung:
Has this ever been suggested before? And if so what’s the reception on this and Logos/Faithlife’s view on the matter?

Yes, it has been, and we built a simple API: https://wiki.logos.com/Logos_4_COM_API

No one (to a close approximation) used it.

We periodically consider adding more API support or embedding a scripting language or similar other approaches. It's not on the roadmap for 2019.

Kolen Cheung:
Out of the many reasons that one might want to reject this idea, I’ll only start to reply to this most important one I can think of: this would be extra work for Logos/Faithlife to provide such a feature for such a niche audience. The investment is not worth it.

You have hit the proverbial nail on the head. Your counterpoints don't (IMHO) sufficiently address the large (and ongoing) cost of building, documenting, supporting, and maintaining a public API.

Kolen Cheung:
Anyway, that one obviously has a couple of problems, Windows only (or not?), and last committed on March 2011. The choice of C# is problematic too (I guess Logos itself is written in C# too?) And it also stated that it is read only (many of the features suggested above requires the ability to write/create.)

It's a COM API; the choice of programming language for the example code is irrelevant. I believe there are people consuming it via Python. (But yes, it is Windows-only; see comments about "cost" above when proposing a cross-platform solution.)

Posts 1363
Ben | Forum Activity | Replied: Mon, Feb 25 2019 11:58 AM

"No one (to a close approximation) used it."

I did! So references in my Bibleworks notes could jump into Logos :)

"The whole modern world has divided itself into Conservatives and Progressives. The business of Progressives is to go on making mistakes. The business of Conservatives is to prevent mistakes from being corrected."- G.K. Chesterton

Posts 884
Kolen Cheung | Forum Activity | Replied: Tue, Feb 26 2019 2:34 AM

Mark Barnes:

Kolen Cheung:

Mark Barnes:
I can't see Faithlife being able to afford the investment required to give us access to its big data

Limits can be applied. Many such web APIs are like this. You can't do without else you're under attack. (e.g. NYTimes API, Twitter API, GitHub API.)

The major cost isn't server time, it's development time.

That is arguable. As long as servers need to communicate there exists some form of API, whether you choose to expose it or not (or make it exposable.)

Just to list one example of things I'd like to query Logos' server, I'd like to be able to send a request to obtain all subjects on all resources Logos has, for example. (This request again can be limited like how other Web API is used.) Now this is not something serving logos.com would have needed and already there, but just an example of things I'd want to get access to.

Posts 13359
Forum MVP
Mark Barnes | Forum Activity | Replied: Tue, Feb 26 2019 4:15 AM

Kolen Cheung:
I'd like to be able to send a request to obtain all subjects on all resources Logos has, for example.

An undocumented SOAP interface allows you to obtain the metadata of resources where you know the resourceID. Getting resourceIDs is more tricky, although your library is a good start. If you're willing to dig around and write some code, you can find out a lot. Here's my list of subjects: https://www.logosresourcesguide.com/subject.php 

Posts 884
Kolen Cheung | Forum Activity | Replied: Tue, Feb 26 2019 7:14 PM

Oh, great! Thanks for pointing out this tool.

Performing a `head` on a resource will show the resourceID. By the way, do you know how to decode the binary logos4 format?

Page 1 of 3 (55 items) 1 2 3 Next > | RSS