Continuous indexing issues

This is meant to be a general comment about indexing in L4. Since the inception of L4, there have been multiple indexing issues and I am wondering if there is beta testing going on to improve the indexing method. Some of the things noticed since November:

1. Long indexing taking place even after a 1.5 mb update

2. Indexing crashes

3. Indexing says it takes 8 hours but crashes before it finished and must start again from the beginning

4. Indexing hangs up or goes no further at times (even though it is not done)

5. Indexing finishes but in search function, there is something about a need to merge indexes which also takes a long time

6. Indexing renders some computers inoperable until the indexing is done or paused. In other words, on some computers, the indexing holds your work hostage until it is done. You can't do anything (or much of anything) until the indexing is finished.

7. I spoke this past weekend to the computer department at a Bible college and they told me that indexing is a continuous problem and they did not even want to go into details on how much difficulty L4 has caused the college.

8. Finally get through indexing and a short time later there is another update (whether big or small) and the indexing is another 8 hour or 4 hour or 16 hour or 3 hour job...and it does not appear that the amount of indexing time has any correlation to the size of the update.

As mentioned at the beginning, this is meant to be a general post about indexing. I am sure someone might write and suggest they cannot help without seeing a log uploaded. But that is another thing. L4 is supposed to be a huge advance from L3. L3 we were told was for techies and L4 is more user friendly. The indexing is one reason why I cannot agree that this is reality. Uploading logs and going through technical steps to find bad sectors etc on a computer or something like that is a techie's dream. But it is a nightmare for non-techies who just want the program to operate as it is supposed to operate.

I am wondering if Logos acknowledges issues with the indexing and if there is work being done to improve the process...or is it a situation where the customer's computer is at fault and the program is actually clean, quick and efficient.

This post is a critic. But it is not meant to be a whine. I think it should be a genuine concern. Any comments?

Find more posts tagged with

Comments

Mark Smith

Mark,

Good summary. And timely.

It seems indexer problems have been multiplying like proverbial rabbits lately. Almost any time you visit these forums on the first page of the latest topics you'll find someone is dealing with an indexer problem.

The trouble with indexer problems is they don't stop. Installation problems can sometimes be tricky to solve, but once they are done, you are usually fine. Not so with indexer problems. They can spring up at any time, and with the frequency of resource updates they are triggered almost at will.

did not even want to go into details on how much difficulty L4 has caused the college.

That is ominous. Esp. since Logos seems to be putting so much emphasis on marketing to colleges and seminaries.

I am wondering if Logos acknowledges issues with the indexing and if there is work being done to improve the process...or is it a situation where the customer's computer is at fault and the program is actually clean, quick and efficient.

Regardless of who's at fault, it ends up being Logos' problem to solve, so it behooves them to put a lot of effort into lowering the number of incidents, whether that is through making the indexer more fault tolerant, giving more helpful information when a problem comes up (maybe even an automated problem solver), pre-checking resources for potential hang-ups, or whatever is needed.

I hate to say so but I think what we see on the forums may be the tip of the iceberg with Logos' support problems with this new program.

Alfred Choi

Poor Service and Bad Experience

I have had such a bad experience with the indexing. Last Monday, it started indexing. The number of hours kept going up - to "1000 hours remaining". That was when I decided to terminate it on Friday. I could not use the computer from Monday to Friday, had to disconnect the Internet so that nothing comes into the system while it was indexing. Today, it started indexing again and it is now at "25 hours remaining." I am sure it will go up to "xxxx hours remaining" again. If I were to leave the computer on to let the program index, 1000 hours is equivalent to 6 weeks! Would you be able NOT to use your computer for 6 weeks while a program indexes?

Calling Logos did not help. Somebody called me back 5 hours later and expected me to be sitting at my computer so that he could help me. Of course, I was not: he could not help me.

logos is a great program. But I am not sure if I would recommend it to somebody else. I find the service at logos very poor. If they were a commercial organization, they would be out of business by now. Wait time for technical support averaged 30 minutes for me. You will NOT think of waiting 30 minutes for someone to come on the line to help you if you were calling your local utitlies or phone company.

Mark Smith

https://community.logos.com/discussion/comment/92018#Comment_92018

Alfred,

Sorry for your trouble. You've found this forum. Instead of us high-jacking this thread I suggest posting a new forum thread here on the Logos 4 forum with a description of your trouble, along with your computer specs, and what you have tried. I think you'll get some help.

Mark

https://community.logos.com/discussion/comment/92018#Comment_92018

Alfred

I think more people than are willing to post have similar problems to what you are experiencing. And I this point I also encourage you to post this again as a new thread. There are a few people on the forum who are technical enough to help you through the issues and perhaps resolve it. And they have been great in helping. I would like to keep this thread to be a general thread, not one which gets into a specific case.

The issue on this thread is whether we can get any confirmation from logos users that there are lots of problems with the indexing and then whether we can get confirmation from Logos that they acknowledge the issue and are working, brainstorming ways to resolve them.

Juanita

https://community.logos.com/discussion/comment/92017#Comment_92017

They can spring up at any time, and with the frequency of resource updates they are triggered almost at will.

Regardless of who's at fault, it ends up being Logos' problem to solve, so it behooves them to put a lot of effort into lowering the number of incidents, whether that is through making the indexer more fault tolerant, giving more helpful information when a problem comes up (maybe even an automated problem solver), pre-checking resources for potential hang-ups, or whatever is needed.

I hate to say so but I think what we see on the forums may be the tip of the iceberg with Logos' support problems with this new program.

Hi Mark,

Thanks for your candor. All I know to do to protect my system so that I can still use Logos is to refrain from downloading updates. And use L3 until these kinds of issues which seriously hinder using L4 are resolved. If it takes months, I can ride it out. Thankfully, I have had no problems that Mark S. listed (other than not using the computer for hours while it indexes).

John Hapgood

https://community.logos.com/discussion/comment/92032#Comment_92032

I don't seem to have any trouble using my computer while indexing is going on. It may be a little slower but hardly noticable.

I'm using a Home built XP SP3 machime with an AMD 64x2 6400+ 2.41GHz with 2GB RAM

John Hapgood

https://community.logos.com/discussion/comment/92039#Comment_92039

I'm using a Home built XP SP3 machime with an AMD 64x2 6400+ 2.41GHz with 2GB RAM

Correction - That would be a 4600+ . . . not a 6400+ processor

Mark Smith

https://community.logos.com/discussion/comment/92027#Comment_92027

The issue on this thread is whether we can get any confirmation from logos users that there are lots of problems with the indexing and then whether we can get confirmation from Logos that they acknowledge the issue and are working, brainstorming ways to resolve them.

[Y]

Alfred Choi

https://community.logos.com/discussion/comment/92044#Comment_92044

Hi guys,

I have moved my post to a new one. Sorry for disrupting the thread.

Mark

https://community.logos.com/discussion/comment/92064#Comment_92064

Hi guys,

I have moved my post to a new one. Sorry for disrupting the thread.

Thanks Alfred. Don't get discouraged by some of the responses you may get on the forum. As I have said, there are some people on the forum who will work with you through your issue.

Mark

https://community.logos.com/discussion/comment/92083#Comment_92083

1. Can we say that if there is an issue with indexing, that one solution always can be to erase the files in the index folder and let indexing start again?

2. Would it be possible to have an automatic pause in indexing in case of computer crash so that indexing does not always have to start from the beginning?

3. Would it be possible to have another indicator in addition to time to let us know how much indexing has already been done? (such as percentage?)

4. Could we have the ability to manually turn indexing on when we restart L4 instead of it being automatic?

5. What would happen if we need to index and we do not index? What services in L4 would be disrupted? Would it only be the search function?

6. Can someone explain why many hours of indexing needs to happen when only 1.5 mb (for example) of updates download?

7. Can updates be scheduled to always occur at the same time each month so we know when to expect them and can prepare to index overnight?

MJ. Smith

https://community.logos.com/discussion/comment/92088#Comment_92088

There are a few of these I can answer:

2. Would it be possible to have an automatic pause in indexing in case of computer crash so that indexing does not always have to start from the beginning?

Logos has indicated that the additional processing to preserve position would add significant time to an already time consuming process.

4. Could we have the ability to manually turn indexing on when we restart L4 instead of it being automatic?

We do have the ability to pause the indexing which achieves much the same result.

5. What would happen if we need to index and we do not index? What services in L4 would be disrupted? Would it only be the search function?

Primarily the search but I suspect that since L4 links are still being added that there are some links within the new/revised texts that could be affected.

6. Can someone explain why many hours of indexing needs to happen when only 1.5 mb (for example) of updates download?

There are at least three factors that affect the length of time: your machine, the density of indexed tags within the text, whether the temporary (new) index reaches its maximum and is merged into the main index.

7. Can updates be scheduled to always occur at the same time each month so we know when to expect them and can prepare to index overnight?

Possible yes, but would the users who have little problem with the indexing be willing to wait for the new/modified resources?

Kevin Becker

https://community.logos.com/discussion/comment/92088#Comment_92088

I'll comment on a couple of your items

1) Yes, deleting the index folder is one strategy. Usually "rebuild index" works but sometimes deleting the index files seems to be the only way for Logos to properly build an index.

2) This would be nice, but it sounds like it might be a major technical feat to code this.

3) I've asked for something like this before ( http://community.logos.com/forums/t/5070.aspx : second suggestion) I think this idea would give a lot more accuracy of the indexer's time estimates.

7) I really like this idea, something like Microsoft's update Tuesday..

Rosie Perera

https://community.logos.com/discussion/comment/92100#Comment_92100

Why is it that tens of thousands of machines have to all run through virtually the same chore, most of which could have been precomputed in advance on Logos's servers? I've asked for this from the beginning, but can't base packages and new resources be shipped with the indexes already built for them included so that we ALL don't have to wait hours for the indexing to take place on our own machines? For brand new users buying a base package, there would be virtually nothing left to do on their machines during the indexing phase, and for existing users buying new products, or receiving downloads of updated resources, all we'd have to do would be to merge indexes. Only in the case of the index becoming corrupted for some reason (which should never happen in solid software) would we ever have to rebuild the whole index from scratch.

Michael Birney

https://community.logos.com/discussion/comment/92102#Comment_92102

but can't base packages and new resources be shipped with the indexes already built for them included so that we ALL don't have to wait hours for the indexing to take place on our own machines? For brand new users buying a base package, there would be virtually nothing left to do on their machines during the indexing phase, and for existing users buying new products, or receiving downloads of updated resources, all we'd have to do would be to merge indexes.

Rosie, that makes a lot of sense to me. All that would be left if to merge indexes after an update or download. Of course this means bigger downloads, but it makes a lot of sense.

Mark

https://community.logos.com/discussion/comment/92099#Comment_92099

There are a few of these I can answer:

Thank you MJ for taking time to answer. I want to follow up on your answers:

on point #2: Is it possible that the extra time that would be added to have an automatic pause of indexing after a crash might actually save time? If I have an 8 hour index and after 6 hours there is a crash (this type of scenario happens often to me) and I have to start again, I would argue that the automatic pause might save time.

on point #4 you missed my point. Because of so much trouble with the indexing, I often restart L4 and dont want the indexing to start. I prefer to wait until the evening. So the ability to manually turn indexing on after startup would be great.

on point#5 you were a big help. Thanks

on point #6 I find it hard to understand how 1.5 mb of anything could case another 8 or 11 hours of indexing.

on point #7 I think Logos can figure out whether the few who have no problems with indexing outweigh the many who do (or perhaps they know that the ones with problems already are in the minority?)

Mark

https://community.logos.com/discussion/comment/92100#Comment_92100

Yes, deleting the index folder is one strategy.

Kevin, when you say deleting the index folder, do you mean EVERYTHING in the folder?

MJ. Smith

https://community.logos.com/discussion/comment/92186#Comment_92186

on point #2: Is it possible that the extra time that would be added to have an automatic pause of indexing after a crash might actually save time? If I have an 8 hour index and after 6 hours there is a crash (this type of scenario happens often to me) and I have to start again, I would argue that the automatic pause might save time.

Perhaps. I simply don't have the statistical data on the setups Logos encounters to be able to even guesstimate the trade-offs.

on point #4 you missed my point. Because of so much trouble with the indexing, I often restart L4 and dont want the indexing to start. I prefer to wait until the evening. So the ability to manually turn indexing on after startup would be great.

If even paused indexing creates problems, you are correct.

on point #6 I find it hard to understand how 1.5 mb of anything could case another 8 or 11 hours of indexing.

I had a single note file take almost 7 hours to index i.e. nearly as long as my initial installation. I attributed this to the high density of biblical references.

on point #7 I think Logos can figure out whether the few who have no problems with indexing outweigh the many who do (or perhaps they know that the ones with problems already are in the minority?)

There is no incentive for those of us who have not had problems to post in the forums. I suspect that Logos has a fairly firm grasp of the number of L4 programs in use and the number that have experienced indexing problems. But remember, I'm one of those half-glass full kind of little old ladies.[:D]

Dave Hooton

As mentioned at the beginning, this is meant to be a general post about indexing. I am sure someone might write and suggest they cannot help without seeing a log uploaded. But that is another thing. L4 is supposed to be a huge advance from L3. L3 we were told was for techies and L4 is more user friendly. The indexing is one reason why I cannot agree that this is reality. Uploading logs and going through technical steps to find bad sectors etc on a computer or something like that is a techie's dream. But it is a nightmare for non-techies who just want the program to operate as it is supposed to operate.

I think this is a huge learning curve for Logos as well as the experienced users that are able to provide advice. We could simply advise to reinstall and it would fix 99% of the issues but it would not assist with finding the root cause and truly resolving the problem. The error messages are too cryptic in most cases and that is why we need to see logs. They provide information to the developers and allow us to decide if reindexing via "rebuild index" or deleting the index folder will work , or whether a reinstall is needed (or another remedy).

In some cases the advice to test memory or hard drive has been fruitful but we are slowly learning when the fault lies elsewhere.

In general when a user performs a merge or the Indexer decides to merge indexes as a result of an update there will be problems! I would recommend that the advice to merge in the "New Resources" section be permanently removed (and for searches by Book that section does not need to be present!).

Mark

https://community.logos.com/discussion/comment/92191#Comment_92191

Thanks again, MJ for your comments. I found them helpful

Mark

https://community.logos.com/discussion/comment/92196#Comment_92196

Thanks Dave for weighing in

Kevin Becker

https://community.logos.com/discussion/comment/92189#Comment_92189

Yes, deleting the index folder is one strategy.

Kevin, when you say deleting the index folder, do you mean EVERYTHING in the folder?

Yes, sorry I was imprecise. Although I imagine if the folder was deleted Logos would probably recreate it, I'm not about to try though.

Dave Hooton

https://community.logos.com/discussion/comment/92088#Comment_92088

Some additional comments.

1. Can we say that if there is an issue with indexing, that one solution always can be to erase the files in the index folder and let indexing start again?

From a user perspective at the present time ... Yes!

EDIT: but many users prefer to have a second opinion.

3. Would it be possible to have another indicator in addition to time to let us know how much indexing has already been done? (such as percentage?)

I've never liked the time method and I have learnt** to do something else and/or let the computer index overnight. I would scrap it and just work with percentage of resources.

** the estimated time is unreliable and takes too many compute cycles.

4. Could we have the ability to manually turn indexing on when we restart L4 instead of it being automatic?

5. What would happen if we need to index and we do not index? What
services in L4 would be disrupted? Would it only be the search
function?

Manual control of indexing after a download is not a good idea as that will lead to errors both subtle and/or catastrophic. It would be feasible if the files are not moved to the resources folder, so you are working with consistent (old) resources and index. But that isn't necessary when you can set Automatically Download Updates to No and be in full control of the process ie. you refuse the download until you are ready.

6. Can someone explain why many hours of indexing needs to happen when only 1.5 mb (for example) of updates download?

Firstly (ignoring the 1.5 MB size) a single reverse interlinear can take considerably longer than several (at least 20) simple, non-interlinear resources. But the download of even a single small resource can trigger :-

a merge of indexes,
a supplemental index build, or
a full reindex

The Indexer decides what is best, but it is not based on time! The worst case is when a reindex is necessary. A supplemental build could take minutes or hours depending on its contents. And a merge can take hours (if it succeeds!).

7. Can updates be scheduled to always occur at the same time each month so we know when to expect them and can prepare to index overnight?

It would be sensible of Logos to not schedule updates knowing that there are likely to be problems with indexing eg. wait until 4.0b is released. Even if there is a periodic schedule just remember that overnight in the US is morning elsewhere[:)] If the time is an issue then set Automatically Download Updates to No as mentioned above.

Mike Childs

https://community.logos.com/discussion/comment/92017#Comment_92017

No doubt this is a big, big problem. I had no real indexing problems until two or three days ago. Now it never stops - not even if I leave my computer on all night.

It seems that I may have to opt to have logos not use the internet at all for a month or so.

This definately should be a high priority to fix.

Mark

https://community.logos.com/discussion/comment/92234#Comment_92234

Very helpful information Dave. Thanks.

May I suggest that someone compile some of these answers into the FAQ or wiki page or somewhere? I have no doubt this indexing issue is frustrating to many and some answers like these can be helpful.

If this is added, I would also add that if someone is looking for a fix, the steps be included on how to upload a log.

David Wilson

https://community.logos.com/discussion/comment/92253#Comment_92253

Based on experience with my systems:

If you have a regularly defragmented hard drive, plenty of free hard drive space, and plenty of free RAM memory, then indexing seems to go well and time taken seems reasonably well related to processor speed (comparing how long it takes on a well equipped desktop versus a well equipped laptop).

If you have an older system with not much memory such that you can barely launch and run Logos 4 without using your hard drive space to simulate extra memory (how much RAM is not enough depends on your operating system, how much software you try to run at the same time as Logos 4 (browsers, word processors, email & messenger, anti-virus etc plus how many programs try to load into memory at startup [icons cluttering the toolbar at the bottom right of our screen]), THEN indexing can start to take seemingly infinite amounts of time (which is what it was like for me before I got a new system). As indexing time heads towards infinity, the system will tend to heat up; even one small glitch with the hard drive can corrupt a file or two and then you may need to identify (per log files) and delete the corrupt file AND probably need to re-index everything from scratch.

Its getting to the point that top end Logos packages should perhaps come with a new computer (which probably costs a lot less than the software !!).

Need to start lobying for Portfolio to come with a top end iPad included (so that you have enough memory to run not only Portfolio but still be able to add other collections not in any current base packages... )

That said, re-indexing 4000 plus resources with anything less than a high end Intel i-7 with 8GB ram is an overnight job for those who have a good night's rest....