Bug 36853

Summary: Enhance Elasticsearch Notes/Comments MARC21 index mappings
Product: Koha Reporter: Aleisha Amohia <aleisha>
Component: Searching - ElasticsearchAssignee: Wainui Witika-Park <wainuiwitikapark>
Status: Signed Off --- QA Contact: Thomas Klausner <domm>
Severity: enhancement    
Priority: P5 - low CC: david, dcook, domm, wainuiwitikapark
Version: unspecified   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: Sponsored
Patch complexity: --- Documentation contact:
Documentation submission: Text to go in the release notes:
This enhancement adds these note subfields to the Notes/Comments index for new installations using Elasticsearch: 501$a, 503$a, 504$a, 508$a, 511$a, 521$a, 538$a, and 547$a. This makes them notes findable in the staff interface and OPAC, and in the advanced search for when the Notes/Comments option is selected. Note: Existing installations will need to either manually add these, or reset the mappings.
Version(s) released in:
Circulation function:
Attachments: Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Description Aleisha Amohia 2024-05-13 23:56:57 UTC
We would like to add the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option:

* 501a
* 503a
* 504a
* 508a
* 511a
* 521a
* 538a
* 547a
* 591a
* 592a
* 593a
* 594a
* 595a
* 596a
* 597a
* 598a
* 599ab
Comment 1 Wainui Witika-Park 2024-05-28 04:06:10 UTC
Created attachment 167204 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a, 591a, 592a, 593a, 594a, 595a, 596a, 597a, 598a, 599ab

Sponsored-by: Educational Services Australia SCIS
Comment 2 David Nind 2024-05-31 04:57:46 UTC
Created attachment 167300 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a, 591a, 592a, 593a, 594a, 595a, 596a, 597a, 598a, 599ab

Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Comment 3 David Nind 2024-05-31 04:59:37 UTC
Testing notes using KTD:

1. Start up KTD with Elasticsearch: ktd --es8 up

2. Edit a record and add unique values as follows (I edited Programming Perl, 262):
   . Add a unique value to 500$a (this is currently included in the index)
   . Add unique values to a sample of the notes fields not currently mapped
   . Note: Need to edit BKS framework to make 503$a, 508$a, 511$a, 547$a, 591$a, 592$a, $593a, 594$a, 595$a, 596$a, 597$a, 598$a, 599$a and $b subfields visible in the editor for KTD)
   Mappings - none of these subfields are currently mapped:
   * 501a
   * 503a
   * 504a
   * 508a
   * 511a
   * 521a
   * 538a
   * 547a
   * 591a
   * 592a
   * 593a
   * 594a
   * 595a
   * 596a
   * 597a
   * 598a
   * 599ab

3. Do a search for the terms in the subfields using the 'Notes/Comments' index in the staff interface advanced search.
   ==> Result: Should get a result when using values in the subfields that are mapped, but no result where the subfields are not mapped.

4. Apply this patch and restart all services (restart_all).

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all then repeat step 2

8. Repeat step 3
   ==> Result: Should now get a result when using values in the subfields that are now mapped (that weren't previously).
Comment 4 Thomas Klausner 2024-07-16 19:57:35 UTC
Created attachment 169067 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a, 591a, 592a, 593a, 594a, 595a, 596a, 597a, 598a, 599ab

Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Comment 5 David Cook 2024-07-17 00:08:23 UTC
At a glance, this shouldn't have the same issue as bug 36850, bug 36851, and bug 36852, because these fields aren't faceted, so I think it's OK at a glance.

--

As an aside, I have a library that want to add 530 and 533 too. Part of me wonders if there's a better way we could be managing these index mappings...

In Zebra, there are no explicit note mappings for 530 and 533, but they are included in the "Any" index, which doesn't exist in our Elasticsearch implementation.

I don't know...
Comment 6 Katrin Fischer 2024-07-18 16:21:33 UTC
These are custom fields that are not present in the MARC frameworks:

https://www.loc.gov/marc/bibliographic/bd59x.html

+    - facet: ''
+      marc_field: '591a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '592a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '593a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '594a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '595a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '596a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '597a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '598a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '599ab'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''

I feel we should not add them to the standard mappings, as we cannot tell what information might be stored in these fields.
Comment 7 David Cook 2024-07-22 06:34:48 UTC
(In reply to Katrin Fischer from comment #6)
> I feel we should not add them to the standard mappings, as we cannot tell
> what information might be stored in these fields.

Mmm good call. 

I really need to look more at how custom mappings are done for Elasticsearch on a per-instance basis...
Comment 8 Wainui Witika-Park 2024-08-14 23:28:22 UTC
Created attachment 170333 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a, 591a, 592a, 593a, 594a, 595a, 596a, 597a, 598a, 599ab

Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Comment 9 Wainui Witika-Park 2024-08-14 23:29:38 UTC
Created attachment 170334 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a

Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Comment 10 David Nind 2025-06-30 02:12:25 UTC
Created attachment 183638 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a

Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: David Nind <david@davidnind.com>
Comment 11 David Nind 2025-06-30 02:19:38 UTC
Testing notes (using KTD):

1. Start up KTD with Elasticsearch: ktd --es7 up

2. Go to Koha administration > Catalog > Search engine configuration (Elasticsearch).

3. In the bibliographic records tab, search the page and note that none of 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a fields are mapped to the note search field.

4. Apply the patch and reset everything (reset_all) (alternative: reset the mappings, reindex everything (koha-elasticsearch --rebuild -d -b -a kohadev), then restart everything).

5. Repeat step 3 and note that the 501a, 503a, 504a, 508a, 511a, 521a, 538a, and 547a fields are now mapped to the note search field.

6. Bonus testing: 
   6.1 Update the bibliographic framework for BKS and make the 501a, 503a, 508a, 511a, and 547a fields editable in the staff interface (504a, 521a, 538a are already editable).
   6.2 Add some unique text to the fields 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a - I used TEST501, TEST503 and so on.
   6.3 Staff interface search: Search > Advanced search > select "Notes/Comments" and search for the unique terms you added to a record - you should only get one result each time.
   6.4 OPAC search: repeat the advanced search with more options selected in the OPAC - you should get the same results as for the staff interface.
   6.5 Add one of the unique terms to another record, for example TEST501 to 501$a, and you should now get two results.