Bug 36853

Created attachment 167204 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a, 591a, 592a, 593a, 594a, 595a, 596a, 597a, 598a, 599ab

Sponsored-by: Educational Services Australia SCIS

Created attachment 167300 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a, 591a, 592a, 593a, 594a, 595a, 596a, 597a, 598a, 599ab

Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>

Testing notes using KTD:

1. Start up KTD with Elasticsearch: ktd --es8 up

2. Edit a record and add unique values as follows (I edited Programming Perl, 262):
   . Add a unique value to 500$a (this is currently included in the index)
   . Add unique values to a sample of the notes fields not currently mapped
   . Note: Need to edit BKS framework to make 503$a, 508$a, 511$a, 547$a, 591$a, 592$a, $593a, 594$a, 595$a, 596$a, 597$a, 598$a, 599$a and $b subfields visible in the editor for KTD)
   Mappings - none of these subfields are currently mapped:
   * 501a
   * 503a
   * 504a
   * 508a
   * 511a
   * 521a
   * 538a
   * 547a
   * 591a
   * 592a
   * 593a
   * 594a
   * 595a
   * 596a
   * 597a
   * 598a
   * 599ab

3. Do a search for the terms in the subfields using the 'Notes/Comments' index in the staff interface advanced search.
   ==> Result: Should get a result when using values in the subfields that are mapped, but no result where the subfields are not mapped.

4. Apply this patch and restart all services (restart_all).

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all then repeat step 2

8. Repeat step 3
   ==> Result: Should now get a result when using values in the subfields that are now mapped (that weren't previously).

Created attachment 169067 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a, 591a, 592a, 593a, 594a, 595a, 596a, 597a, 598a, 599ab

Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Thomas Klausner <domm@plix.at>

At a glance, this shouldn't have the same issue as bug 36850, bug 36851, and bug 36852, because these fields aren't faceted, so I think it's OK at a glance.

--

As an aside, I have a library that want to add 530 and 533 too. Part of me wonders if there's a better way we could be managing these index mappings...

In Zebra, there are no explicit note mappings for 530 and 533, but they are included in the "Any" index, which doesn't exist in our Elasticsearch implementation.

I don't know...

These are custom fields that are not present in the MARC frameworks:

https://www.loc.gov/marc/bibliographic/bd59x.html

+    - facet: ''
+      marc_field: '591a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '592a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '593a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '594a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '595a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '596a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '597a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '598a'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''
+    - facet: ''
+      marc_field: '599ab'
+      marc_type: marc21
+      sort: 1
+      suggestible: ''

I feel we should not add them to the standard mappings, as we cannot tell what information might be stored in these fields.

(In reply to Katrin Fischer from comment #6)
> I feel we should not add them to the standard mappings, as we cannot tell
> what information might be stored in these fields.

Mmm good call. 

I really need to look more at how custom mappings are done for Elasticsearch on a per-instance basis...

Created attachment 170333 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a, 591a, 592a, 593a, 594a, 595a, 596a, 597a, 598a, 599ab

Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Thomas Klausner <domm@plix.at>

Created attachment 170334 [details] [review]
Bug 36853: Enhance Elasticsearch Notes/Comments MARC21 index mappings

Added the following tags and subfields to the existing note index for Elasticsearch MARC21, as used by the "Notes/Comments" advanced search dropdown option: 501a, 503a, 504a, 508a, 511a, 521a, 538a, 547a

Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Thomas Klausner <domm@plix.at>