Bug 7029 - Zebra CHR equivalent/mapping works unexpectedly with "ie"
Summary: Zebra CHR equivalent/mapping works unexpectedly with "ie"
Status: CLOSED WONTFIX
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: Bugs List
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-10-14 10:22 UTC by delaye
Modified: 2023-12-28 20:43 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
searchingtab_syspref (160.30 KB, image/png)
2016-11-08 12:19 UTC, Marjorie Barry-Vila
Details

Note You need to log in before you can comment on or make changes to this bug.
Description delaye 2011-10-14 10:22:24 UTC
When I search the word 'chien' in the results I have records without the word 'chien' but with the word 'Chine'

QueryFuzzy is disabled
and
 QueryStemming is not lemmatise

so I have too much results...

example with this url : http://catalogue.lamediathequedumarsan.fr/cgi-bin/koha/opac-search.pl?q=chien&limit=
Comment 1 Adrien SAURAT 2012-01-06 10:20:47 UTC
No big problem actually. There was a mistake in the French translation of the QueryFuzzy values.

"Don't try" was replaced with "Activer"
"Try" was replaced with "Désactiver"

It should be the opposite.


The translation in the Koha Translation Project has been updated.
Comment 2 delaye 2012-08-17 10:18:30 UTC
in the relase 3.8.2

i have the same problem

when i search the terme 'chien' koha display the bib. notice about the 'chine'

i test with the systeme preference 
QueryFuzzy  disabled or not desabled

and
 QueryStemming lemmatise or / not lemmatise
Comment 3 Marjorie Barry-Vila 2016-03-10 16:32:41 UTC
Hi,
I have the same problem with 3.22.

Even if i disable QueryFuzzy i have the same result for research Chien, i have record with Chine in the list result.

I try with different word with letter "ien" and i have same problem, It looks like the letter e is omitted from the syllable "ien"

paien = record with pain http://v322-dev.koha-dev.ccsr.qc.ca/cgi-bin/koha/opac-search.pl?idx=&q=paien (4th record)
sapien = record with sapin http://v322-dev.koha-dev.ccsr.qc.ca/cgi-bin/koha/opac-search.pl?idx=&q=sapien (2th record)
rien = record with rin http://v322-dev.koha-dev.ccsr.qc.ca/cgi-bin/koha/opac-search.pl?idx=&q=rien (2th record)

???

an idea of the problem with research in french?


Regards,


MArjorie
Comment 4 Fridolin Somers 2016-11-03 08:35:05 UTC
(In reply to Marjorie Barry-Vila from comment #3)
> Hi,
> I have the same problem with 3.22.
> 
> an idea of the problem with research in french?
> 
> 
> Regards,
> 
> 
> MArjorie

There may be a problem with syspref translation.
Can you test unsing non-translated Koha (english) ?

Regards,
Comment 5 Marjorie Barry-Vila 2016-11-03 14:28:24 UTC
(In reply to Fridolin SOMERS from comment #4)
> (In reply to Marjorie Barry-Vila from comment #3)
> > Hi,
> > I have the same problem with 3.22.
> > 
> > an idea of the problem with research in french?
> > 
> > 
> > Regards,
> > 
> > 
> > MArjorie
> 
> There may be a problem with syspref translation.
> Can you test unsing non-translated Koha (english) ?
> 
> Regards,

Hi Fridolin,
I tested with English translation and it is the same problem.
You can try here http://v322-dev.koha-dev.ccsr.qc.ca/cgi-bin/koha/opac-search.pl?idx=ti&q=paien&branch_group_limit=
Third result contains word "pain" while search was "paien".

Is this the test you want me to do?

Regards,
Marjorie
Comment 6 Fridolin Somers 2016-11-07 14:56:52 UTC
(In reply to Marjorie Barry-Vila from comment #5)
> 
> Hi Fridolin,
> I tested with English translation and it is the same problem.
> You can try here
> http://v322-dev.koha-dev.ccsr.qc.ca/cgi-bin/koha/opac-search.
> pl?idx=ti&q=paien&branch_group_limit=
> Third result contains word "pain" while search was "paien".
> 
> Is this the test you want me to do?
> 
> Regards,
> Marjorie

Indeed.

What values do you have for the preferences in tab "Searching" and in "Features" ?
(in english)
Comment 7 Marjorie Barry-Vila 2016-11-08 12:19:30 UTC
Created attachment 57310 [details]
searchingtab_syspref

Hi Fridolin,

Here is a screenshot for sys pref.

Regards,
Marjorie
Comment 8 Fridolin Somers 2017-03-17 08:03:12 UTC
You have the syspref IncludeSeeFormInSearches on, it may impact the search results on worlds not visible in the biblio record.
Comment 9 Marjorie Barry-Vila 2017-07-06 15:24:46 UTC
(In reply to Fridolin SOMERS from comment #8)
> You have the syspref IncludeSeeFormInSearches on, it may impact the search
> results on worlds not visible in the biblio record.

Sorry for my late reply.
Yes i know for syspref IncludeSeeFormInSearches but if i look in complete marc notice, even in rejected forms of authority, words are not present at all.

I do not know how to show you the problem but it still exists, even in 16.11.
There is a video with search "paien" and word "pain" is found in a title when there is not at all the word "pain" in the record and there is no authority either.
https://owncloud.ccsr.qc.ca/index.php/s/4yCofUUQjuV1k7S

Regards,

Marjorie
Comment 10 Fridolin Somers 2020-03-27 13:17:41 UTC
Really too tricky bug, I move on.
Comment 11 Katrin Fischer 2020-05-23 14:20:02 UTC
I think it might actually be something else at play here:

chine and chiene

i = ie + auto-truncate

pain and paien + auto-truncate

Can you check if turning of QueryAutoTruncate?

Thinking about something like this:

  18 equivalent iíìîịĩĭįǐȉȋ
  19 equivalent ï(ie)
Comment 12 Katrin Fischer 2020-05-23 14:21:55 UTC
This is from: /etc/zebradb/etc/word-phrase-utf.chr

Seeing if there is a difference between ICU and grs would also be interesting.
Comment 13 David Cook 2020-05-24 23:36:48 UTC
I love indexing problems...

In the Web UI, I just tried "kw, wrdl: chien" and "kw, wrdl: chine" in an English/Chinese Koha instance I have and got back 117 and 115 results respectively. 

In yaz-client, I tried "kw,wrdl=chien" and "kw,wrdl=chine" and got 0 results both times. However "chinese" and "chienese" both got 115 results...

I add "format xml" and "elements zebra::snippet" and retry "chienese", and I get the following:

Z> show 1
Sent presentRequest (1+1).
Records: 1
Record type: XML
<record xmlns="http://www.indexdata.com/zebra/">
  <snippet name="Any" type="w"><s>Chinese</s></snippet>
</record>nextResultSetPosition = 2
Elapsed: 0.032032

Looking at the Zebra setup, that's using CHR indexing (via word-phrase-utf.chr) rather than ICU indexing. 

In word-phrase-utf.chr, I see the following:
equivalent ï(ie)
map ï           i

My *guess* is that "ie" also gets searched as "ï" which is treated as "i", so "chien" would also search "chin". 

This corresponds with what Katrin and Marjorie are saying.

A bit odd that "chien/chin" would match "chine", but QueryAutoTruncation would cause that to happen, as "chien" would be treated as "chin" and auto truncated as "chin*".

So... that's probably that one explained. 

As for the solution... probably update "map ï i" to "map ï (ie)".
Comment 14 David Cook 2022-12-06 01:59:00 UTC
(In reply to David Cook from comment #13)
> I love indexing problems...

Looking back, it looks like I only tested with CHR indexing. 

I imagine that this isn't an issue for ICU indexing.
Comment 15 David Cook 2022-12-06 02:00:57 UTC
Just tested on a English/French/Arabic Koha using ICU and "chine" and "chien" get very different (and correct) results.

Going to close this one as we know that CHR indexing really isn't good enough in most non-English contexts. The answer is to switch to ICU indexing.