BUG: Serious issues with Swedish letters

My searches constantly get messed up because Logos can't handle the Swedish letters å, ä and ö.

Example 1: I search for 'fader' (father); I also get results for 'fäder' (fathers). That's a lot of results I don't want.

Example 2: I start typing a word on 'ä' and keep a close eye on the dropdown to find out exactly how many letters will give me all the words that I want, but none that I don't want. It's impossible: 80% of the words shown start with 'a', not 'ä'.

Example 3: I search for 'höra'; I also get results for 'hora'. Unfortunately 'höra' means '[to] hear', while 'hora' means '[to] whore' or '[a ] whore'. Not quite the same thing...

Å, ä and ö are letters, not a's and o's with accents. Letters with a completely different place in the alphabet (last: ...xyzåäö). Letters far more different in pronunciation from a and o than omikron is from omega, or epsilon from eta. Mixing a and å is like calling Dan Don, or D-i-c-k Duck.

Unlike on certain American websites, writing åäö with my Swedish keyboard works perfectly fine in Logos, so the basics are right. And since these Bibles are sold from Sweden, I presume they are typed in with Swedish keyboards, so there shouldn't be any problem there either. You just need to tell Logos to separate one from the other!

To complicate things further, though, it turns out that Match All Word Forms, which is supposed to give more results, instead gives fewer results when used on words including åäö. Then, suddenly, the program does seem to know how to separate the two, and gets rid of the wrong one. You can read here how completely confusing that made things for me: http://community.logos.com/forums/t/26173.aspx. And, unfortunately, the dropdown still mixes them up, even when Match All Word Forms is chosen.

The dropdown, btw, often seems pretty useless for Swedish words: I can write out a whole word and still not see it in the dropdown, though I can see far more unusual words there. And for Swedish searches not including åäö, I seem to be always getting the exact same number of hits, whether I have Match All Word Forms chosen or not. Are these two issues with Logos, or some kind of tagging that's missing in the Swedish resources? In other words, whom shall I put pressure on: you or SN Ljud & Data?

(Yet another 'Swedish' issue has been reported here: http://community.logos.com/forums/p/26183/193074.aspx#193074.)

(The Forum Censor is really funny: it seems 'whore' is OK, though the the perfectly normal name D-i-c-k isn't... Smile )

Find more posts tagged with

Comments

Bradley Grainger (Logos)

Å, ä and ö are letters, not a's and o's with accents. Letters with a completely different place in the alphabet (last: ...xyzåäö).

Are you sure the last letters in the alphabet aren't actually æ, ø, and å? [;)]

The Logos 4 search engine was designed to recognise regional differences regarding what constitutes a letter, what is an ignorable accent, etc. (and to not be completely US-centric). We know what the right rules should be, but it looks like we aren't applying them at query time for Swedish, which is a bug that we'll investigate.

fgh

https://community.logos.com/discussion/comment/193147#Comment_193147

Are you sure the last letters in the alphabet aren't actually æ, ø, and å?

You beat me with that one! [:)] I didn't realize Danish and Norwegian puts å last (but at least I had the sense to look it up before I answered...).

Thanks!

Do you also have an answer to the question towards the end ("Are these two...")?

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/193150#Comment_193150

Do you also have an answer to the question towards the end ("Are these two...")?

I don't have an immediate answer. It's most likely a code problem; I quickly checked a few Swedish resources I have here and they all seem to be correctly tagging Swedish text as Swedish.

fgh

https://community.logos.com/discussion/comment/193332#Comment_193332

I don't have an immediate answer.

OK, I'll be waiting for you to get back. [;)]

I was also wondering: is there some kind of learning curve going on here in Logos? I'm asking because the first time I searched for 'offer' (sacrifice, as a noun), it didn't show up in the dropdown at all, though far more unusual compound nouns starting with 'offer-' did; the second time it did show up, but far down the list; and now that I try a third time it has moved to number three. Is this deliberate, or just accidental? Also, if I've selected a Swedish Bible, is it supposed to show only Swedish words, or any words happening to start with those letters?

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/193349#Comment_193349

I was also wondering: is there some kind of learning curve going on here in Logos? I'm asking because the first time I searched for 'offer' (sacrifice, as a noun), it didn't show up in the dropdown at all, though far more unusual compound nouns starting with 'offer-' did; the second time it did show up, but far down the list; and now that I try a third time it has moved to number three. Is this deliberate, or just accidental? Also, if I've selected a Swedish Bible, is it supposed to show only Swedish words, or any words happening to start with those letters?

The suggestions that are shown shouldn't depend on past history; it should come back with the same suggestions each time.

The suggestions are not limited by the resource or collection being searched; they are always taken from across the entire library. (In a "Bible" search, the suggestions do only come from Bibles, but always come from all Bibles.)

fgh

https://community.logos.com/discussion/comment/193415#Comment_193415

The suggestions that are shown shouldn't depend on past history; it should come back with the same suggestions each time.

Well, it doesn't.

The suggestions are not limited by the resource or collection being searched

Bad news for us with minority languages.

Bradley Grainger (Logos)

OK, I have identified part of the problem: the right search settings are used if your UI Language is set to Swedish (on Windows). Searching for "väder" finds only "väder", searching for "vader" finds nothing, and searching for "våder" (with "Matcha alla ordformer" turned ON) finds "våder", "våd", "våderna", "våden", etc. I don't speak Swedish, but I'm going to assume those are different forms of the same basic word, possibly including some extra matches due to overstemming (which is also a problem we have in English from using an algorithm to find related words, rather than using a dictionary).

Of course the Mac program doesn't currently have a Swedish UI option, so you're locked into the US English behaviour you don't want.

The complicating factor is that US English keyboards (which is what the overwhelming majority of our users have) don't have keys for ä, ö, å, ø, etc., and I would guess that many English speakers don't realise that those characters are actually different letters in other alphabets. We just want to be able to type in "Soren Kierkegaard" and find results. [:)]

If we make the searching more strict, we can end up excluding results that English speakers would expect and want to find, making the program less usable. (That's one reason why the current logic is opt-in based on the UI language.) Obviously there are problems with that approach that make it much less usable for non-English speakers. We'll be discussing our options and will try to find a compromise that will make everyone happy.

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/193482#Comment_193482

Actually... there might be a workaround available right now...

Do you get the results that you would expect if you search for the following queries?

swedish:hora

swedish:höra

(I know it's a pain to have to type the "swedish:" prefix, but that clue lets the system know you're searching for Swedish words, forcing it to the right behaviour.)

fgh

https://community.logos.com/discussion/comment/193483#Comment_193483

I would guess that many English speakers don't realise that those characters are actually different letters in other alphabets. We just want to be able to type in "Soren Kierkegaard" and find results. Smile

Why would they when you are taught from infancy that the whole world should adapt to you, and you adapt to no one -- not even to show the most basic courtesy of getting someone's name right!? Just about the only thing more rude than to abuse someone's name, is to not even try to get it right.

Actually... there might be a workaround available right now...

Well, that at least gives me a few more options, but it doesn't work anywhere near like you intend it to:

• "Swedish:syn" or "swedish:synas" both give lots of forms of the noun 'syn' (vision), the adjective 'synlig' (visible), and the passive verb 'synas' (be visible). That is far too much, isn't it? But it's also far too little: it misses verb forms like 'syntes' and 'synts'.

• "Swedish:ko" (cow) only returns 'ko', but misses the plural 'kor', and the definite forms 'kon' and 'korna' (we don't have any 'the'). "Swedish:kor", on the other hand, returns nor only 'kor', but also 'kors' (cross), a form of the verb 'kora' (select as winner, king, or similar), the names Kora, Kore and Kores, and the word 'korer', which seems to be some kind of measure. But not 'ko', 'kon' or 'korna'. "Swedish:korna" gives 'korna', but also 'korn' (barley, corn), and, of course, again not what it should give.

• 'Offren' (the sacrifices) comes up under the verb 'offra' ([to] sacrifice), not under the noun 'offer' ([a ] sacrifice, sacrifices). 'Offret' (the sacrifice) doesn't come up under either.

• "Swedish:komma" ([to] come) finds 'komma', 'kommer' (come[s ], is coming), and 'kommen' ([is] come). "Swedish:kom" (came, come!) only finds 'kom'. "Swedish:kommit" ([have] come) only finds 'kommit'.

• There are serious issues with irregular verbs -- even in English! The program realizes that 'walk' and 'walked' are forms of the same word, but not that 'go' and 'gone' are, much less 'go' and 'went'. Not to talk about 'am', 'is', 'are', 'was', 'were', 'has been', 'will be'... Considering that, it's hardly surprising that it can't handle 'gå', 'gick', 'gått', or 'är', 'var', 'varit', 'bli','blev', 'blivit'. This is pretty serious, especially in English! How is anyone to know they have to make a special search for 'gone'?? Untrustworthy results are worse than no results at all, especially if you're writing an academic paper or a scholarly book!

You might want to try these examples on Windows to see what results they give there, because if "swedish:" is supposed to work exactly like the Swedish interface, you definitely have some algorithms to fix! (You might have problems reproducing all of them exactly, since I've worked, sometimes in one Bible book only, sometimes in one whole Bible (the 1982 one), and sometimes in all Swedish Bibles simultaneously, depending on how common or unusual a word I was searching for.)

And, btw, 'väder' ought to turn up 'vädret' and 'vädrets'; and 'vader' ought to turn up 'vad', 'vaden' and 'vaderna' -- but only those 'vad' and 'vaden' that refer to the calf of a leg, not those that refer to wagers or fords, and definitely not all the 'vad's that translate 'what' (try making an algorithm for that one! [;)] ) Unfortunately, a search for 'vaden' does turn up all the what-'vad's -- about 10.000 of them in a full Swedish Bible collection -- and also forms of the verb 'vada' (wade). Furthermore, 'våder' correctly finds 'våd', 'våden', våder' and 'våderna', but also 'våda', which has nothing to do with the others. 'Höra' finds a number of verb forms and some related nouns, but misses several past tense and passive forms. 'Hora' finds all verb forms I can think of, two nouns, a place name, a mountain, and a nationality which I consider a typo (it's there in print as well, but totally inconsistent with the spelling in other places in the same Bible).

So, no, I don't exactly get the results I would expect...

(And you really need to do something about the Analysis view: it can't handle searches with 0 results: it just keeps spinning!)

Mark Barnes

https://community.logos.com/discussion/comment/193679#Comment_193679

I would guess that many English speakers don't realise that those characters are actually different letters in other alphabets. We just want to be able to type in "Soren Kierkegaard" and find results.

You've missed the point. Bradley was saying that when you have the English UI selected, then the UI should behave in a way that's suitable for English speakers. When you have the Swedish UI selected, then it behaves in a way appropriate for Swedish speakers. So with the English UI, Soren and Søren are treated similarly, whilst in the Swedish UI they're not. It's called transliteration, it's incredibly common, and it's nothing to get upset about. No-one is asking anyone else to adapt.

• "Swedish:syn" or "swedish:synas" both give lots of forms of the noun 'syn' (vision), the adjective 'synlig' (visible), and the passive verb 'synas' (be visible). That is far too much, isn't it? But it's also far too little: it misses verb forms like 'syntes' and 'synts'.

Search all word forms was never intended to be a perfect solution - in any language. That just isn't possible without a massive manually-created dictionary, and even then there'd be arguments. If you want precision, you'll need to turn it off, as the wiki advises.

fgh

https://community.logos.com/discussion/comment/193684#Comment_193684

I didn't miss the point. You did. Sören might possibly be called a transliteration of Søren, Soren is not. It's simply a misspelling. Ø isn't pronounced anywhere near o. It's pronounced similar to i in girl or u in further. Would you consider Milk Burnes a transliteration of your name? You can't just take a letter in someone's name and change it to whatever you feel like, and then call it 'transliteration'! It is no harder for you to learn to spell Danish, French or German names correctly than it is for us. It's just a question of whether you respect people enough to bother.

And, yes, we are 'asked' (read: not asked/forced) to adapt every single day. There is a Swedish county with the perfectly decent name Hörby. Because Americans can't think outside the American box unless they're forced to, that poor county has to use the web address horby.se. 'Horby' means something like 'the whoring village'... Besides, my point was far far broader than just language. You simply haven't got a clue how much we have to adapt to make any interaction at all possible. Precisely because you don't ask, you just take for granted.

Ebbe Andréasson

My searches constantly get messed up because Logos can't handle the Swedish letters å, ä and ö.

Example 1: I search for 'fader' (father); I also get results for 'fäder' (fathers). That's a lot of results I don't want.

Example 2: I start typing a word on 'ä' and keep a close eye on the dropdown to find out exactly how many letters will give me all the words that I want, but none that I don't want. It's impossible: 80% of the words shown start with 'a', not 'ä'.

Example 3: I search for 'höra'; I also get results for 'hora'. Unfortunately 'höra' means '[to] hear', while 'hora' means '[to] whore' or '[a ] whore'. Not quite the same thing...

This might be a Logs 4 for Mac issue. I haven't this issues when running searches in Logos for Windows (4.2 B7). Searching for "fader" only finds "fader" and so on... I can't remember but I think I have your problem during the hidden beta testing a year ago but they where solved. I don't think this has to do with the tagging of swedish resources.

Cameron Watters (Faithlife)

https://community.logos.com/discussion/comment/193711#Comment_193711

Because Americans can't think outside the American box unless they're forced to, that poor county has to use the web address horby.se.

Actually, both horby.se and hörby.se work just fine for me. Appears they've had the latter registered since 2003.

Mark Barnes

https://community.logos.com/discussion/comment/193711#Comment_193711

For what it's worth, my wife's name is Shâron. She does not answer to
Sharon. So I understand your point. But we're talking here about how
words should be written in languages that have a different alphabet to the language of the original word.

Would you consider Milk Burnes a transliteration of your name?

That would depend entirely on the language in which it was written. In certain parts of the English-speaking world my name is pronounced entirely different to the way I myself would pronounce it. Indeed in city 50 miles from where I'm typing this it would be pronounced entirely differently again. I live with it. And when I've travelled to countries that have a different alphabet than my own, they spell it differently too. I don't make a fuss about that, either.

And, yes, we are 'asked' (read: not asked/forced)
to adapt every single day.

My comments were regarding Logos. I am not trying to get into a
political debate. And this has certainly nothing at all to do with
whether I respect you or the language(s) you speak.

Precisely because you don't ask, you just take for granted.

And the 'you' in that sentence is who?

fgh

https://community.logos.com/discussion/comment/193725#Comment_193725

You're right, Cameron. I abbreviated the argument a bit to save space.[;)] It works now (on some web sites), but for a long time it didn't. I'm pretty sure it didn't work in 2003, though they may have been able to register the address somewhere to keep it reserved. (I bet they were one of the first to register an address with å,ä or ö!) And it still works on so few sites that we tend to automatically write a or o rather than having to rewrite the URL twice when it doesn't work. I just tried försäkringskassan.se (which handles pensions, sick pay and the like), and it still didn't work. You have to use forsakringskassan.se.

Cameron Watters (Faithlife)

https://community.logos.com/discussion/comment/193732#Comment_193732

It works now (on some web sites), but for a long time it didn't. I'm pretty sure it didn't work in 2003, though they may have been able to register the address somewhere to keep it reserved.

To be fair, support for unicode 2nd-level domains isn't even universal today, though I suspect the problem is more painful in countries that don't have alphabets that share most of their letters with ASCII (e.g. India, China, Russia, Arabic-language nations, etc).

And further, to be at least a bit fairer to ignorant Americans (like me), I don't think the limitation of the DNS system is as much a function of American myopia as it is a result of the DNS designers not anticipating in 1983 the demands that would be placed on the system 15 years later. They were trying to improve on the state-of-the-art of the day, which was hand-built HOSTS.TXT files on each machine. At that time, much of the internet-powered technological realities of 2010 were the stuff of science fiction.

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/193679#Comment_193679

• "Swedish:syn" or "swedish:synas" both give lots of forms of the noun 'syn' (vision), the adjective 'synlig' (visible), and the passive verb 'synas' (be visible). That is far too much, isn't it? But it's also far too little: it misses verb forms like 'syntes' and 'synts'.

The stemmers are algorithmic, not dictionary-based, so they produce a number of false positives and negatives (and are affected greatly by how many irregular forms the language has). The option's provided as a convenience for those who find it useful, and based on IR research (though perhaps not done on a Swedish corpus with native Swedish speakers) showing that in general, imperfect stemming improves recall and precision. We realise that it's not always the right thing to do, so it is optional.

Mark Barnes

https://community.logos.com/discussion/comment/193748#Comment_193748

What languages have stemming algorithms? Another user can't get stemming working in German. Is this supported yet?

Bob Pritchett

https://community.logos.com/discussion/comment/193711#Comment_193711

You can't just take a letter in someone's name and change it to whatever you feel like, and then call it 'transliteration'! It is no harder for you to learn to spell Danish, French or German names correctly than it is for us. It's just a question of whether you respect people enough to bother.

Is this level of respect required just among those of us from Western European cultures? Or must I stop writing "Beijing" and "Shanghai", and use the correct ideographic forms? :-)

I hope you'll forgive forgive my making a joke; I just want to make the point that it's impossible to conform to every culture's requirements. We're aware that American culture has a big, and seemingly unsympathetic, footprint around the world. And so we try hard to make Logos as "universal" as possible. (We first supported indexing the Swedish alphabet correctly -- before Unicode -- in 1992.) But we're bound by our own knowledge -- which it's unrealistic to expand to accommodate every culture in the world -- and by economic realities. The bigger markets get bigger attention. I know this often, correctly, gets blamed on an American ignorance/arrogance, but I think in many ways it just comes down to size/influence. In the Soviet era, Cyrillic script and the Russian language were pushed on lots of countries; in the Chinese sphere there's similar culture export/indifference.

(While we don't have it perfect, we're at least trying to get Swedish (and all European languages) right; I confess that our layout engine, global as it is, doesn't support vertical-script Mongolian yet. It's just too small an audience.)