[feature request] scriptable API to interface with Logos

Page 3 of 3 (55 items) < Previous 1 2 3
This post has 54 Replies | 4 Followers

Posts 885
Kolen Cheung | Forum Activity | Replied: Thu, Feb 28 2019 7:57 AM

Mark Barnes:
because you won't be able to 100% accurately match resource titles on a wiki page or spreadsheet to the resource titles in your library. They're often not identical.

Text manipulation is often messy, a pity. But one can try to canonicalize both side and compare if they are the same. That’s the data cleaning part. Moreover, while being 100% accurate is nice, if spending 10% of time getting one 90% there then I’m all for it. I don’t even know how to manually add hundreds of resources by hand. But again I hope Logos will do it, as they have said. But the logos community collections last updated years ago, and years bejng Logos users taught me unfortunately never trust their promises. Good intension, it there are always obstacles that make things dally and delay. Bob once promised me PBB reverse interlinear soon almost a decade ago, and it hasnt arrived yet. I hope that this feature came soon since the brother I know spending his whole life working on a Chinese reverse interlinear and a fortune on an in house bible software is, you know, getting very old. He actualy once produced a reverse interlinear Chinese recovery version in NT but due to a history related to the people involved in recovery version, it never s got published. Anyway.

Mark Barnes:
But (at least on Windows), SHIFT+click does.

Unfortunately I tried that on Mac and it doesn't work. Probably should give Windows a try. I wish Logos develop multi platform apps like Microsoft did to its office. Logos is just not behaving like a Mac app. (I was tempted to use the other Mac native Bible software. But I first drawn to Logos by a unique resource published by Lexham. So I can't live without Logos. Imagine I can decrypt Logos format and convert them to the other format such that resources and the softwares are decoupled. Among all people one would guess Christians should have more incentive to have a common standard. I guess the unique complexity involved in Biblical related resources are too much and yet their world are too small for industrial standard to win.

Mark Barnes:
That was fine in 1990. This is 2019. The vast majority of software and services doesn't work like that any more.

Most I used does. Among those I used Logos is uniquely behind in this aspect as if they don't even considered this.

And from my personal experience, it's getting easier and easier to do this. Partly because I learnt more, mostly because the world is so big and new tools emerges that makes this possible and easier.

May be one day I'll be fully using open source softwares and OS. But I'm greedy and I want the best and some of the best is still proprietary GUI like Logos, Lightroom, etc. (not to mention in the case of Logos it is the resources + feature/data sets that set it apart, not the software, making an open source alternative almost absence.) That's partly why I started this thread because I worry for a future that the "philosophical distance" between me and Logos are getting farther and farther away. E.g. Applescript support is also important but from Logos' not Apple native commitment seems to be low priority. I just can't imagine how I can still tolerate those clicking a decade from now, which could've been scripted. Like PBB, I have a very small collection of PBB and managing that already is a headache. It is relatively easy to batch obtain a lot of public domain resources to be added by PBB, but the bottleneck is the clicking of thousands times. May be that API already exist but it isn't clear what function does or doesn't.

Posts 885
Kolen Cheung | Forum Activity | Replied: Fri, Mar 1 2019 9:27 PM

NB.Mick:

Kolen Cheung:
This calculation step ain't indexing? I thought you were referring that to indexing. Because say if one is changing a rating, the program is saying "indexing... library result may not be accurate..." something like that.

No it isn't. And the program says: "Updating library catalog", not "indexing".

Right, I just recalled it from memory so I wasn't using the exact wording.

However, it must be doing more than just updating library catalog, because if so it is just one single atomic transaction which should be very fast. What I imagine it is doing is that it is like adding the record and then perform an optimization to that database. Lightroom does something like that. Lightroom's catalog is an SQLite database, and there's an "optimize catalog" button allowing the user to optimize and cleanup the catalog on demand, but not on real time. So that a mere change of a tag/rating/metadata won't have that much a penalty.

So if what I said is true, I suggest Logos to separate these 2 steps. e.g. try manually tagging and rating the new base package one has bought, the "updating library catalog" is very annoying. The user is almost like racing with that when rating resources one by one.

So indexing / updating library catalog or else, the reason they take so long to finish is to "pre-cache" something so that in the future a certain operation is very fast, like O(1) kind of fast. So if this is indeed what's happening behind the scene when one has a ton of collections, I don't care. Because the slowness only shows up in the preparatory step, not in real time when one need to interact with resources. If not true, then probably they should tell us what's the expectation here.

Posts 885
Kolen Cheung | Forum Activity | Replied: Sat, Mar 2 2019 1:15 AM

Andrew Batishko (Faithlife):
Yes, the contents of the resource files are encrypted. This is a DRM issue.

One more question: is .lbspbb essentially identical to .logos4 in terms of spec and DRM? I peek into one of them and it sort of looks like DRM'd. If not I'd like to be able to have the spec of lbspbb to generate it directly without going through Logos' PBB compiler. And if yes, why DRM is applied to PBB?

Posts 17990
Forum MVP
Keep Smiling 4 Jesus :) | Forum Activity | Replied: Sat, Mar 2 2019 12:34 PM

Kolen Cheung:
One more question: is .lbspbb essentially identical to .logos4 in terms of spec and DRM? I peek into one of them and it sort of looks like DRM'd. If not I'd like to be able to have the spec of lbspbb to generate it directly without going through Logos' PBB compiler.

Logos PBB compiler includes metadata that is needed for .lbspbb to be usable, which is synced. Hence puzzled by desire to generate lbspbb without using PBB compiler ?

Keep Smiling Smile

Posts 885
Kolen Cheung | Forum Activity | Replied: Sat, Mar 2 2019 1:34 PM

Keep Smiling 4 Jesus :):

Kolen Cheung:
One more question: is .lbspbb essentially identical to .logos4 in terms of spec and DRM? I peek into one of them and it sort of looks like DRM'd. If not I'd like to be able to have the spec of lbspbb to generate it directly without going through Logos' PBB compiler.

Logos PBB compiler includes metadata that is needed for .lbspbb to be usable, which is synced. Hence puzzled by desire to generate lbspbb without using PBB compiler ?

Keep Smiling Smile

  1. Get rid of MS Word. Since the beginning they introduce PBB (post Logos 4), I thought it was a big mistake to make it depends on MS Word.
    1. A “principle” reason is that makes it not free. Imagine someone can’t afford to buy any Logos resources for various reasons (which is not uncommon), they can get free Logos engine, they can get some free Logos resources (recall when they first introduced PBB there were very few free Logos resources), then the next logical step is PBB for classic, public domain resources. But without a commercial MS Office, one can’t produce that.
    2. MS Word’s spec is in principle open, but in reality a very complicated thing that requires a lot of reverse engineering.
    3. A free option to generate docx exists, but the problem from its spec making generating docx a hit and miss thing. Sometimes the generated docx is corrupted, and occasionally opening in MS Word and “save as” can fix that, which again requires the commercial application.
    4. Authoring a book in MS Office is almost a joke. MS Word is just not for book publishing (The state of the art is LaTeX, and other solutions exists, and yes a lot of publishers accept Word but they are doing their conversions and editing). Regardless the origin of PBB (public domain books, “self-publishing” kind of long form notes, etc.), being a PBB almost guarantee it is very long. Producing long, structured documents is very hard in MS Office (I mean making sure the whole document falls in a uniform typesetting standard is hard. It is easy to apply a slightly different choice of, font, font size, indentation, section, etc. somewhere in the document and hard to spot.)
  2. Do something PBB compiler can’t do. If indeed lbspbb really support everything logos4 does, then although we aren’t suppose to author logos4 documents (for the prevention of piracy), we can author lbspbb directly for features already supported in the spec but not by the PBB compiler. One such example is (reverse) interlinear than Bob has promised but still didn’t deliver (I think they changed their mind) after almost a decade.

Note that I have my own solution to (1) so the main focus is (2), which focus on getting features not supported by the PBB compiler.

Posts 17990
Forum MVP
Keep Smiling 4 Jesus :) | Forum Activity | Replied: Sat, Mar 2 2019 1:52 PM

Kolen Cheung:
Get rid of MS Word.

LibreOffice is a free alternative that can create docx file(s) for Personal Book(s).

Years ago remember learning a MS Word trick to save a document as RTF followed by opening RTF for save as docx.

Kolen Cheung:
One such example is (reverse) interlinear ...

Reverse Interlinear resource additions are *.lbsrvi

Keep Smiling Smile

Posts 885
Kolen Cheung | Forum Activity | Replied: Mon, Mar 4 2019 11:58 PM

Keep Smiling 4 Jesus :):
LibreOffice is a free alternative that can create docx file(s) for Personal Book(s).

Yes, if you read a few rows after that,

Kolen Cheung:
A free option to generate docx exists, but the problem from its spec making generating docx a hit and miss thing. Sometimes the generated docx is corrupted, and occasionally opening in MS Word and “save as” can fix that, which again requires the commercial application.

I should’ve said “free options” as there are many. But none of them is perfect. Speaking of Libre Office, it has many problems. e.g. HTML to docx conversion can sometimes be extremely slow which takes hours, overnight conversion. While its cli program, soffice is useful for scripting (and hence automation and batch processing), it cannot be parallelized (this can be gotten around by setting it up as a daemon which makes parallel execution very complicated.)

Keep Smiling 4 Jesus :):
Reverse Interlinear resource additions are *.lbsrvi

I think the main point is to open the spec of native Logos formats so that personal book is not limited to what the PBB compiler supports.

Yes, there is DRM concern, but it is irrelevant. Just like e.g. epub, the spec is open, and DRM is optional. i.e. Logos can release the spec of all their formats, while on top of it keep using their DRM for official, commercial products.

Even MS office releases the spec of their XML formats. So why not for Logos?

As an example, this is the steps involved in PBB output from BibleMultiConverter:

  1. detect versification in BibleMultiConverter (no output yet)
  2. export to “Logos HTML” in BibleMultiConverter, which is in HTML format, but written in a way to includes Logos PBB syntax
  3. HTML to docx conversion, e.g. using soffice from Libre Office
  4. docx to docx in 2 passes by BibleMultiConverter for syntax fixes
  5. docx to PBB using Logos' PBB compiler

All these conversion steps (ignoring the versification detection) can be done in 1 go if a spec in the final format is given, say the current .lbspbb.

So the issue isn’t related to the file extension (file extension is an arbitrary convention that didn’t change its content, its spec is the one that’s relevant. In principle given the some file extension, or even no extension at all, as long as e.g. the mime type is provided it’ll be fine), but the spec to generates these files.

Posts 885
Kolen Cheung | Forum Activity | Replied: Tue, Mar 19 2019 1:46 AM

Mark Barnes:

For what it's worth the code is on GitHub. The code wasn't written for the public, and I'm not really willing to answer lots of questions about it, but you can start with the update_metadata function.

But this service is probably more useful than my code. Lots of the functions require authorisation (i.e. your Logos cookie), which you can't generate through the site. But GetMetadata works unauthenticated. Just add the ResourceIDs in the request XML.

Thanks Mark again. I just started to use this and it is actually quite useful. The only bump I got is I spent almost an hour just to figure out resourceIds is something like 'LLS:EEC15EZR' not '24998' (which I assumed from the REST API...)


Posts 13368
Forum MVP
Mark Barnes | Forum Activity | Replied: Tue, Mar 19 2019 6:30 AM

The numbers are product IDs, not resource IDs. Products are what is sold (e.g. “Logos 8 Gold”). Products often (but not always) contain multiple resources. You only need product IDs if you’re interacting with the store.

Posts 885
Kolen Cheung | Forum Activity | Replied: Tue, Mar 19 2019 1:09 PM

Thanks. I figured that out.

Posts 13
LogosEmployee
Nate Merritt (Faithlife) | Forum Activity | Replied: Mon, Mar 25 2019 6:01 PM

Kolen Cheung:

Thanks Mark again. I just started to use this and it is actually quite useful. The only bump I got is I spent almost an hour just to figure out resourceIds is something like 'LLS:EEC15EZR' not '24998' (which I assumed from the REST API...)


We appreciate your enthusiasm for our products and your desire to programmatically explore the Faithlife platform.  We have public facing developer documentation for a couple of our web APIs at https://developer.faithlife.com/, but these APIs were specifically designed with public consumption in mind and have built in rate limiting.  Unfortunately, some of our older APIs have not yet been retrofitted with rate limiting and there is potential for the additional load to impact other users of our services.

We don’t object to you calling external APIs called by the desktop application, but we do ask that you throttle your calls to our APIs to a couple hundred requests per minute.  We’ll plan to add rate limiting in the near future.

Thanks for your understanding!

Posts 885
Kolen Cheung | Forum Activity | Replied: Tue, Mar 26 2019 3:08 PM

Nate Merritt (Faithlife):

Kolen Cheung:

Thanks Mark again. I just started to use this and it is actually quite useful. The only bump I got is I spent almost an hour just to figure out resourceIds is something like 'LLS:EEC15EZR' not '24998' (which I assumed from the REST API...)


We appreciate your enthusiasm for our products and your desire to programmatically explore the Faithlife platform.  We have public facing developer documentation for a couple of our web APIs at https://developer.faithlife.com/, but these APIs were specifically designed with public consumption in mind and have built in rate limiting.  Unfortunately, some of our older APIs have not yet been retrofitted with rate limiting and there is potential for the additional load to impact other users of our services.

We don’t object to you calling external APIs called by the desktop application, but we do ask that you throttle your calls to our APIs to a couple hundred requests per minute.  We’ll plan to add rate limiting in the near future.

Thanks for your understanding!

Thanks for letting me know. I did try to find the limit and see there's 300 requests per minute from the Products API but not mentioned in ResourceMetadata API. Will it be ok if I limit it at the same rate (300 requests per minute)?

To prevent sending a ton of requests, I actually was trying to cache the metadata locally first yesterday. But the SOAP interface causes me some problem.

By the way, to others in this thread, I think I've found a way to "write" without writing to the database. Essentially the problem is how to "create" something from the data gathered back into Logos. I think I've found a way to do that without actually writing to the database, which is something every response to that is negative. I'll test this idea once I have cached all the metadata.

Posts 13
LogosEmployee
Nate Merritt (Faithlife) | Forum Activity | Replied: Tue, Mar 26 2019 5:54 PM

Kolen Cheung:

Thanks for letting me know. I did try to find the limit and see there's 300 requests per minute from the Products API but not mentioned in ResourceMetadata API. Will it be ok if I limit it at the same rate (300 requests per minute)?

We'll probably set a rate limit much lower than 300 requests per minute in the near future for the ResourceMetadata API and require authentication for the GetMetadata route.

Posts 885
Kolen Cheung | Forum Activity | Replied: Fri, Apr 5 2019 2:05 PM

Nate Merritt (Faithlife):
We'll probably set a rate limit much lower than 300 requests per minute in the near future for the ResourceMetadata API and require authentication for the GetMetadata route.

I see that it is now requiring authentication. Could you tell us what's the rate limit?

Thanks.

Posts 13
LogosEmployee
Nate Merritt (Faithlife) | Forum Activity | Replied: Mon, Apr 8 2019 11:41 AM

Kolen Cheung:

I see that it is now requiring authentication. Could you tell us what's the rate limit?

The rate limit is currently 4000 requests per hour for the GetMetadata route.

Page 3 of 3 (55 items) < Previous 1 2 3 | RSS