Bug 22339

Summary: Elasticsearch - fixed field mappings should match MARC ranges
Product: Koha Reporter: Nick Clemens (kidclamp) <nick>
Component: Searching - ElasticsearchAssignee: Ere Maijala <ere.maijala>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P5 - low CC: ere.maijala, frank.hansen, glasklas, josef.moravec, lucas, martin.renvoize
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
19.05.00, 18.11.05
Attachments: Bug 22339: Fix search field mappings of MARC fixed fields
Bug 22339: Fix search field mappings of MARC fixed fields
Bug 22339: Fix search field mappings of MARC fixed fields

Description Nick Clemens (kidclamp) 2019-02-14 13:27:27 UTC
ff7-00 is set to: 007_/1
it should be mapped to position 0 I believe
any other specific character positions should be checked to ensure they are corect as well

To recreate:
1 - Find an item with an 007 in your catalog
2 - Make sure it is indexed in ES
3 - View the ES record:
  kohadevbox
    curl 'localhost:9200/koha_kohadev_biblios/data/19?pretty=yes' | grep -A7 ff7
  koha testing docker
    curl 'es:9200/koha_kohadev_biblios/data/19?pretty=yes' | grep -A7 ff7
  adjust for other setups
4 - Note the 00 indexes the 01 position, etc
Comment 1 Ere Maijala 2019-03-14 11:24:11 UTC
*** Bug 22256 has been marked as a duplicate of this bug. ***
Comment 2 Ere Maijala 2019-03-14 11:37:28 UTC
Created attachment 86597 [details] [review]
Bug 22339: Fix search field mappings of MARC fixed fields

Fixes handling of /0 selector in addition to several fixed field positions. Note that ff7-01-02 is in fact 00-01 in Zebra and that has been replicated here.

Test plan:
1. Before applying a patch, check from Elasticsearch (e.g. localhost:9200/koha_biblios/_search?q=_id:123) what is indexed in Elasticsearch for a record for date-entered-on-file, ff7-00, ff7-01, ff7-02 and lleader.
2. Apply the patch, update database and save the record again.
3. Verify that the contents of the forementioned fields is now correct in Elasticsearch.
4. Verify that tests pass:  prove -v t/Koha/SearchEngine/Elasticsearch.t
Comment 3 Ere Maijala 2019-03-14 11:38:36 UTC
Note that it seems ff7-01-02 was misnamed in Zebra, and that is reflected here. It contains positions 00-01.
Comment 4 Frank Hansen 2019-03-29 13:57:53 UTC
Note that leader-fields should be spelled "LDR".
Comment 5 Ere Maijala 2019-04-01 07:31:46 UTC
Frank, sorry, I'm not following. Can you explain with a bit more detail?
Comment 6 Frank Hansen 2019-04-03 08:06:55 UTC
It is a little bit confusing because according to the MARCXML standard, leader field is called "leader". But I discovered that in ES, leader fields obviously is marked up as "LDR" instead of "leader". I have no idea why. Maybe it's a Koha or a Catmandu thing. The thing is, ES fields that comes from content in the leader will not be indexed unless the mapping is corrected to "LDR".
Comment 7 Björn Nylén 2019-04-12 13:00:24 UTC
Created attachment 87905 [details] [review]
Bug 22339: Fix search field mappings of MARC fixed fields

Fixes handling of /0 selector in addition to several fixed field positions. Note that ff7-01-02 is in fact 00-01 in Zebra and that has been replicated here.

Test plan:
1. Before applying a patch, check from Elasticsearch (e.g. localhost:9200/koha_biblios/_search?q=_id:123) what is indexed in Elasticsearch for a record for date-entered-on-file, ff7-00, ff7-01, ff7-02 and lleader.
2. Apply the patch, update database and save the record again.
3. Verify that the contents of the forementioned fields is now correct in Elasticsearch.
4. Verify that tests pass:  prove -v t/Koha/SearchEngine/Elasticsearch.t

Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Comment 8 Frank Hansen 2019-04-12 13:06:23 UTC
Yes, it seems to work after all. 
Leader fields is indexed as it should. Perfekt!
Comment 9 Martin Renvoize 2019-04-25 08:02:39 UTC
Created attachment 88706 [details] [review]
Bug 22339: Fix search field mappings of MARC fixed fields

Fixes handling of /0 selector in addition to several fixed field positions. Note that ff7-01-02 is in fact 00-01 in Zebra and that has been replicated here.

Test plan:
1. Before applying a patch, check from Elasticsearch (e.g. localhost:9200/koha_biblios/_search?q=_id:123) what is indexed in Elasticsearch for a record for date-entered-on-file, ff7-00, ff7-01, ff7-02 and lleader.
2. Apply the patch, update database and save the record again.
3. Verify that the contents of the forementioned fields is now correct in Elasticsearch.
4. Verify that tests pass:  prove -v t/Koha/SearchEngine/Elasticsearch.t

Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 10 Martin Renvoize 2019-04-25 08:03:38 UTC
I'll confess to not being a total expert when it comes to MARC fields, but these changes appear to be solid and I can't see any obvious regressions.

I really like that we're indexing by '0' now too.

Passing QA.
Comment 11 Nick Clemens (kidclamp) 2019-04-25 11:53:05 UTC
Awesome work all!

Pushed to master for 19.05
Comment 12 Martin Renvoize 2019-04-26 15:23:19 UTC
Pushed to 18.11.x for 18.11.05
Comment 13 Lucas Gass 2019-05-24 15:33:43 UTC
not backporting ES bugs/fixes to 18.05.x series