Bug 36269

Summary: Elasticsearch: publisher-location (pl) index should include 260a/264a (MARC21)
Product: Koha Reporter: Thomas Klausner <domm>
Component: Searching - ElasticsearchAssignee: Thomas Klausner <domm>
Status: Pushed to oldstable --- QA Contact:
Severity: normal    
Priority: P5 - low CC: caroline.cyr-la-rose, fridolin.somers, lucas, nick
Version: unspecified   
Hardware: All   
OS: All   
Change sponsored?: Sponsored Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
This enhancement adds 260$a and 264$a to the publisher-location (pl) Elasticsearch index for MARC21 records. Values in those two fields will be findable using the Publisher location option in the advanced search. Note: for existing installations, the index needs to be rebuilt using -r (reset mappings) in order for this information to be taken into account.
Version(s) released in:
24.05.00,23.11.06,23.05.12
Bug Depends on:    
Bug Blocks: 36578    
Attachments: Bug 36269: ElasticSearch: publisher-location (pl) should use260a/264a
Bug 36269: ElasticSearch: publisher-location (pl) should use260a/264a
Bug 36269: ElasticSearch: publisher-location (pl) should use260a/264a

Description Thomas Klausner 2024-03-07 10:47:02 UTC
Currently the index `pl` points to 008/15-17. It should (additionally/instead?) point to 260a and/or 264a.

* Fix the default mappings in admin/searchengine/elasticsearch/mappings.yaml

Notes from chat:

cait: maybe having the index broader is not a bad thing, I am not sure if it's confusing or not
cait: like put 260a (non-RDA), 264 (RDA) and 008 15-17 in it?
Comment 1 Katrin Fischer 2024-03-07 10:51:52 UTC
Not sure if we should aim for matching the Zebra config, but I think it might be a good starting point. I definitely think we should put 260 and 264 $a in. 

Not sure about the controlled value from 008 - but a 3 letter country code might not hurt things.
Comment 2 Thomas Klausner 2024-03-07 16:59:48 UTC
Created attachment 162913 [details] [review]
Bug 36269: ElasticSearch: publisher-location (pl) should use260a/264a

Currently the index `pl` points to 008/15-17. It should
(additionally/instead?) point to 260a and/or 264a.

Test plan (for koha-testing-docker with ElasticSearch via `ktd --es7 up`)

Verify the old/broken behaviour:

* Go to Staff/Advanced Search
* Select "Publisher Location" and enter "cau", start search
* You will get some hits (~16), eg "Perl best practices / Damian Conway", which has 008 of "051222s2005 caua b 001 0 eng d" with "cau" on 15-17
* Edit this record (cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=5)
* Inspect 260$a, which should containt "Sebastopol, CA :"
* Go back to Advanced Search and search for  "Publisher Location" = "Sebastopol"
* No hits!

Apply the patch!

* exit ktd and stop it (ktd --es7 down; ktd --es7 stop)
* start it again
* Go to Staff / Administration / Search Engine Config (Elasticsearch)
* Click on the Tab "Bibliographic records" and search/filter for "pl"
* you should see 3 entries for "pl", with Mapping values of "008_/15-17", "260a" and "264a"
* Go to Search,
* Select "Publisher Location" and enter "cau", start search
* same hits as befor
* Search again, but for "Sebastopol"
* Now you'll get 9 hits!!

Bonus: Test 264a

* Got to Admin / MARC bibl framework, select BKS -> MARC structure
* search for tag "264", edit subfields
* in tabs a, b, c: Check the "Editor" Checkbox (Visibility)
* Save changes
* find a book, eg again "Perl best practices" and edit it
* find field 264 and enter "Test" into  264a, Save
* Depending on your setup, you might have to manually re-index the book:
  * enter ktd: ktd --shell
  * reindex the one book (via --bn) or all (might also need a -d)
    /usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl -v -b -d
* Go again to Advanced Search, Publisher Location = "Test"
* You should find the book

If your NOT running ktd, you might be able to just edit the ElasticSearch Mappings to add / change the mapping for "pl" to point to "260a".

One rather harsh way to do this (which I needed to do, because the ES Mappings UI did not work for me) is via this SQL:
update search_marc_map set marc_field='260a' where marc_field='008_/15-17';

Sponsored-by: Steiermärkische Landesbibliothek
Comment 3 Caroline Cyr La Rose 2024-03-07 21:07:17 UTC
Created attachment 162932 [details] [review]
Bug 36269: ElasticSearch: publisher-location (pl) should use260a/264a

Currently the index `pl` points to 008/15-17. It should
(additionally/instead?) point to 260a and/or 264a.

Test plan (for koha-testing-docker with ElasticSearch via `ktd --es7 up`)

Verify the old/broken behaviour:

* Go to Staff/Advanced Search
* Select "Publisher Location" and enter "cau", start search
* You will get some hits (~16), eg "Perl best practices / Damian Conway", which has 008 of "051222s2005 caua b 001 0 eng d" with "cau" on 15-17
* Edit this record (cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=5)
* Inspect 260$a, which should containt "Sebastopol, CA :"
* Go back to Advanced Search and search for  "Publisher Location" = "Sebastopol"
* No hits!

Apply the patch!

* exit ktd and stop it (ktd --es7 down; ktd --es7 stop)
* start it again
* Go to Staff / Administration / Search Engine Config (Elasticsearch)
* Click on the Tab "Bibliographic records" and search/filter for "pl"
* you should see 3 entries for "pl", with Mapping values of "008_/15-17", "260a" and "264a"
* Go to Search,
* Select "Publisher Location" and enter "cau", start search
* same hits as befor
* Search again, but for "Sebastopol"
* Now you'll get 9 hits!!

Bonus: Test 264a

* Got to Admin / MARC bibl framework, select BKS -> MARC structure
* search for tag "264", edit subfields
* in tabs a, b, c: Check the "Editor" Checkbox (Visibility)
* Save changes
* find a book, eg again "Perl best practices" and edit it
* find field 264 and enter "Test" into  264a, Save
* Depending on your setup, you might have to manually re-index the book:
  * enter ktd: ktd --shell
  * reindex the one book (via --bn) or all (might also need a -d)
    /usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl -v -b -d
* Go again to Advanced Search, Publisher Location = "Test"
* You should find the book

If your NOT running ktd, you might be able to just edit the ElasticSearch Mappings to add / change the mapping for "pl" to point to "260a".

One rather harsh way to do this (which I needed to do, because the ES Mappings UI did not work for me) is via this SQL:
update search_marc_map set marc_field='260a' where marc_field='008_/15-17';

Sponsored-by: Steiermärkische Landesbibliothek
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Comment 4 Caroline Cyr La Rose 2024-03-07 21:08:20 UTC
I was able to rebuild and get the new mappings using the following command in ktd --shell

koha-elasticsearch --rebuild -r -d -b -v kohadev
Comment 5 Katrin Fischer 2024-03-09 11:26:38 UTC
I believe this is more of a bug fix really, as we offer "publisher location" as a search option in the advanced search and have search links in the detail pages. Thanks, Thomas!
Comment 6 Nick Clemens (kidclamp) 2024-04-01 17:56:38 UTC
Created attachment 164224 [details] [review]
Bug 36269: ElasticSearch: publisher-location (pl) should use260a/264a

Currently the index `pl` points to 008/15-17. It should
(additionally/instead?) point to 260a and/or 264a.

Test plan (for koha-testing-docker with ElasticSearch via `ktd --es7 up`)

Verify the old/broken behaviour:

* Go to Staff/Advanced Search
* Select "Publisher Location" and enter "cau", start search
* You will get some hits (~16), eg "Perl best practices / Damian Conway", which has 008 of "051222s2005 caua b 001 0 eng d" with "cau" on 15-17
* Edit this record (cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=5)
* Inspect 260$a, which should containt "Sebastopol, CA :"
* Go back to Advanced Search and search for  "Publisher Location" = "Sebastopol"
* No hits!

Apply the patch!

* exit ktd and stop it (ktd --es7 down; ktd --es7 stop)
* start it again
* Go to Staff / Administration / Search Engine Config (Elasticsearch)
* Click on the Tab "Bibliographic records" and search/filter for "pl"
* you should see 3 entries for "pl", with Mapping values of "008_/15-17", "260a" and "264a"
* Go to Search,
* Select "Publisher Location" and enter "cau", start search
* same hits as befor
* Search again, but for "Sebastopol"
* Now you'll get 9 hits!!

Bonus: Test 264a

* Got to Admin / MARC bibl framework, select BKS -> MARC structure
* search for tag "264", edit subfields
* in tabs a, b, c: Check the "Editor" Checkbox (Visibility)
* Save changes
* find a book, eg again "Perl best practices" and edit it
* find field 264 and enter "Test" into  264a, Save
* Depending on your setup, you might have to manually re-index the book:
  * enter ktd: ktd --shell
  * reindex the one book (via --bn) or all (might also need a -d)
    /usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl -v -b -d
* Go again to Advanced Search, Publisher Location = "Test"
* You should find the book

If your NOT running ktd, you might be able to just edit the ElasticSearch Mappings to add / change the mapping for "pl" to point to "260a".

One rather harsh way to do this (which I needed to do, because the ES Mappings UI did not work for me) is via this SQL:
update search_marc_map set marc_field='260a' where marc_field='008_/15-17';

Sponsored-by: Steiermärkische Landesbibliothek
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 7 Nick Clemens (kidclamp) 2024-04-01 17:57:16 UTC
I must admin, we have had it this way in our mappings internally for some time - been on the list to reconcile what we have against community version
Comment 8 Katrin Fischer 2024-04-04 13:25:33 UTC
Pushed for 24.05!

Well done everyone, thank you!
Comment 9 Caroline Cyr La Rose 2024-04-04 13:31:01 UTC
For upgrades, I think this will need a complete rebuild, correct?
Comment 10 Katrin Fischer 2024-04-04 18:55:43 UTC
(In reply to Caroline Cyr La Rose from comment #9)
> For upgrades, I think this will need a complete rebuild, correct?

I think any change to the indexing configuration would require a reindex, but you will also need to configure these or reset your mappings on update - we are not changing existing mappings.
Comment 11 Fridolin Somers 2024-05-23 09:24:59 UTC
Pushed to 23.11.x for 23.11.06
Comment 12 Lucas Gass 2024-05-28 16:50:40 UTC
Backported to 23.05.x for upcoming 23.05.12