Bug 21331 - [Elasticsearch] Concatenated subfields are not indexed correctly
Summary: [Elasticsearch] Concatenated subfields are not indexed correctly
Status: RESOLVED DUPLICATE of bug 19893
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Alex Arnaud
QA Contact:
URL:
Keywords: Manual
Depends on: 19893
Blocks:
  Show dependency treegraph
 
Reported: 2018-09-11 15:42 UTC by Alex Arnaud
Modified: 2018-11-13 19:46 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 21331: Elasticsearch - fix indexing of concatenated subfields (2.00 KB, patch)
2018-09-11 15:51 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 21331: Elasticsearch - fix indexing of concatenated subfields (2.07 KB, patch)
2018-10-08 13:09 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Arnaud 2018-09-11 15:42:21 UTC
I have the following unimarc data for a biblio:

200 $f Hanna Johansen
700 $a Johansen, $b Hanna
701 $a Maquet, $b Yves-Marie
701 $a Behnd, $b Kathi

And the following mappings in search engine configuration:

author => 200f
author => 700ba
author => 701ba

Data that results of indexing in Elasticsearch is like:

"author__facet" : [ 
  "Hanna Johansen",
  "JohansenHanna"
  "MaquetYves-Marie",
  "BehndKathi"
],
"author" : [
  "Hanna Johansen",
  "JohansenHanna",
  "MaquetYves-Marie",
  "BehndKathi"
],

This can cause troubles when searching and with facets. What we want is:

"author__facet" : [ 
  "Hanna Johansen",
  "Johansen Hanna"
  "Maquet Yves-Marie",
  "Behnd Kathi"
],
"author" : [
  "Hanna Johansen",
  "Johansen Hanna",
  "Maquet Yves-Marie",
  "Behnd Kathi"
],
Comment 1 Alex Arnaud 2018-09-11 15:51:31 UTC
Created attachment 78560 [details] [review]
Bug 21331: Elasticsearch - fix indexing of concatenated subfields
Comment 2 Martin Renvoize 2018-10-08 13:09:34 UTC
Created attachment 80217 [details] [review]
Bug 21331: Elasticsearch - fix indexing of concatenated subfields

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 3 Martin Renvoize 2018-10-08 13:10:21 UTC
This one seemed to be a good bug to cut my teeth on as it were with my new found elastic search setup for testing.

Looks good, does as expected. Signing off.
Comment 4 Alex Arnaud 2018-10-08 13:22:53 UTC
(In reply to Martin Renvoize from comment #3)
> This one seemed to be a good bug to cut my teeth on as it were with my new
> found elastic search setup for testing.
> 
> Looks good, does as expected. Signing off.

Thanks Martin,

But we'll have to throw this patch away since bug 19893 will completely change the code, and, fix this bug i hope.
Comment 5 Martin Renvoize 2018-11-13 13:48:43 UTC

*** This bug has been marked as a duplicate of bug 19893 ***
Comment 6 Martin Renvoize 2018-11-13 13:50:47 UTC
The functionality here indeed appears to have been folded into bug 19893 so I'm marking as RESOLVED DUPLICATE.

However, it does rely on updated available configuration options which should be documented somewhere.

Fields may now be properly concatenated by specifying them within brackets `()`...  Marking as 'Manual' to bring a maintainers attention.
Comment 7 David Gustafsson 2018-11-13 17:25:42 UTC
(In reply to Martin Renvoize from comment #6)
> The functionality here indeed appears to have been folded into bug 19893 so
> I'm marking as RESOLVED DUPLICATE.
> 
> However, it does rely on updated available configuration options which
> should be documented somewhere.
> 
> Fields may now be properly concatenated by specifying them within brackets
> `()`...  Marking as 'Manual' to bring a maintainers attention.

Thanks! This has been brought up before in this issue and agree this needs to be fixed. I failed to find any existing section for the mappings.yaml format in the Koha Online manual where this could be put, but I just took a quick glance.

Another thing that I also think has been brought up is that there perhaps should be added some defaults for fields that should be concatenated, like author and full title. This could perhaps be handled in a follow up patch though.
Comment 8 David Gustafsson 2018-11-13 17:27:54 UTC
Oops, I thought I responded in bug 19893, and were referring to the discussion in that issue. Will link to this comment in that issue.
Comment 9 Martin Renvoize 2018-11-13 19:03:19 UTC
(In reply to David Gustafsson from comment #7)
> Another thing that I also think has been brought up is that there perhaps
> should be added some defaults for fields that should be concatenated, like
> author and full title. This could perhaps be handled in a follow up patch
> though.

See the last patch on bug 19365, does this cover the defaults scenario?
Comment 10 David Gustafsson 2018-11-13 19:46:31 UTC
(In reply to Martin Renvoize from comment #9)
> (In reply to David Gustafsson from comment #7)
> > Another thing that I also think has been brought up is that there perhaps
> > should be added some defaults for fields that should be concatenated, like
> > author and full title. This could perhaps be handled in a follow up patch
> > though.
> 
> See the last patch on bug 19365, does this cover the defaults scenario?

It seems to use the new syntax, but for me it seems a little bit odd to  categorically concatenate all the subfield for most fields. That defeats the purpose of the special syntax (to just concatenate the subfields where it makes sense to do so, when the result will be something like full title, or full author name like in the previous examples).