Bug 31693

Summary: Remove non-existent fields from the See-also-from index (Elasticsearch, MARC21)
Product: Koha Reporter: Caroline Cyr La Rose <caroline.cyr-la-rose>
Component: Searching - ElasticsearchAssignee: Caroline Cyr La Rose <caroline.cyr-la-rose>
Status: CLOSED FIXED QA Contact: Kyle M Hall (khall) <kyle>
Severity: enhancement    
Priority: P5 - low CC: david, kyle, lucas
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31691
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
This enhancement removes fields from the See-from index that don't exist in MARC21. The existing fields can be found here: https://www.loc.gov/marc/authority/ad5xx.html
Version(s) released in:
22.11.00
Circulation function:
Attachments: Bug 31693 Remove non-existent fields from the See-also-from index (Elasticsearch, MARC21)
Bug 31693 Remove non-existent fields from the See-also-from index (Elasticsearch, MARC21)
Bug 31693 Remove non-existent fields from the See-also-from index (Elasticsearch, MARC21)
Bug 31693 Remove non-existent fields from the See-also-from index (Elasticsearch, MARC21)

Description Caroline Cyr La Rose 2022-10-05 16:20:58 UTC
The See-also-from index for MARC21 indexes fields that don't exist in the MARC21 format.

The existing fields can be found here: https://www.loc.gov/marc/authority/ad5xx.html

Here are the fields currently indexed that are not part of MARC21.

- 511$b
- 511$m
- 511$o
- 511$r
- 530$b
- 530$c
- 530$e
- 547$b
- 547$e
- 547$f
- 547$h
- 547$j
- 547$k
- 547$l
- 547$m
- 547$n
- 547$o
- 547$p
- 547$q
- 547$r
- 547$s
- 547$t
- 548$b
- 548$c
- 548$d
- 548$e
- 548$f
- 548$g
- 548$h
- 548$k
- 548$l
- 548$m
- 548$n
- 548$o
- 548$p
- 548$r
- 548$s
- 548$t
- 550$c
- 550$d
- 550$e
- 550$f
- 550$h
- 550$j
- 550$k
- 550$l
- 550$m
- 550$n
- 550$o
- 550$p
- 550$q
- 550$r
- 550$s
- 550$t
- 551$b
- 551$c
- 551$d
- 551$e
- 551$f
- 551$h
- 551$k
- 551$l
- 551$m
- 551$n
- 551$o
- 551$p
- 551$r
- 551$s
- 551$t
- 555$b
- 555$c
- 555$d
- 555$e
- 555$f
- 555$g
- 555$h
- 555$j
- 555$k
- 555$l
- 555$m
- 555$n
- 555$o
- 555$p
- 555$q
- 555$r
- 555$s
- 555$t
- 562$b
- 562$c
- 562$d
- 562$e
- 562$f
- 562$g
- 562$h
- 562$k
- 562$l
- 562$m
- 562$n
- 562$o
- 562$p
- 562$r
- 562$s
- 562$t

Also, the mapping for 511 is written differently than other mappings, with the a outside the parentheses. 

511a(bcdefghjklmnopqrstvxyz)
Comment 1 Caroline Cyr La Rose 2022-10-05 16:23:00 UTC
Created attachment 141389 [details] [review]
Bug 31693 Remove non-existent fields from the See-also-from index (Elasticsearch, MARC21)

This patch removes fields from the See-from index that don't exist in
MARC21.

The existing fields can be found here: https://www.loc.gov/marc/authority/ad5xx.html

The following fields are removed:

- 511$b
- 511$m
- 511$o
- 511$r
- 530$b
- 530$c
- 530$e
- 547$b
- 547$e
- 547$f
- 547$h
- 547$j
- 547$k
- 547$l
- 547$m
- 547$n
- 547$o
- 547$p
- 547$q
- 547$r
- 547$s
- 547$t
- 548$b
- 548$c
- 548$d
- 548$e
- 548$f
- 548$g
- 548$h
- 548$k
- 548$l
- 548$m
- 548$n
- 548$o
- 548$p
- 548$r
- 548$s
- 548$t
- 550$c
- 550$d
- 550$e
- 550$f
- 550$h
- 550$j
- 550$k
- 550$l
- 550$m
- 550$n
- 550$o
- 550$p
- 550$q
- 550$r
- 550$s
- 550$t
- 551$b
- 551$c
- 551$d
- 551$e
- 551$f
- 551$h
- 551$k
- 551$l
- 551$m
- 551$n
- 551$o
- 551$p
- 551$r
- 551$s
- 551$t
- 555$b
- 555$c
- 555$d
- 555$e
- 555$f
- 555$g
- 555$h
- 555$j
- 555$k
- 555$l
- 555$m
- 555$n
- 555$o
- 555$p
- 555$q
- 555$r
- 555$s
- 555$t
- 562$b
- 562$c
- 562$d
- 562$e
- 562$f
- 562$g
- 562$h
- 562$k
- 562$l
- 562$m
- 562$n
- 562$o
- 562$p
- 562$r
- 562$s
- 562$t

Furthermore, the format of the mapping for 511 has been corrected.
Comment 2 Caroline Cyr La Rose 2022-10-05 16:24:30 UTC
Created attachment 141391 [details] [review]
Bug 31693 Remove non-existent fields from the See-also-from index (Elasticsearch, MARC21)

This patch removes fields from the See-also-from index that don't exist in
MARC21.

The existing fields can be found here: https://www.loc.gov/marc/authority/ad5xx.html

The following fields are removed:

- 511$b
- 511$m
- 511$o
- 511$r
- 530$b
- 530$c
- 530$e
- 547$b
- 547$e
- 547$f
- 547$h
- 547$j
- 547$k
- 547$l
- 547$m
- 547$n
- 547$o
- 547$p
- 547$q
- 547$r
- 547$s
- 547$t
- 548$b
- 548$c
- 548$d
- 548$e
- 548$f
- 548$g
- 548$h
- 548$k
- 548$l
- 548$m
- 548$n
- 548$o
- 548$p
- 548$r
- 548$s
- 548$t
- 550$c
- 550$d
- 550$e
- 550$f
- 550$h
- 550$j
- 550$k
- 550$l
- 550$m
- 550$n
- 550$o
- 550$p
- 550$q
- 550$r
- 550$s
- 550$t
- 551$b
- 551$c
- 551$d
- 551$e
- 551$f
- 551$h
- 551$k
- 551$l
- 551$m
- 551$n
- 551$o
- 551$p
- 551$r
- 551$s
- 551$t
- 555$b
- 555$c
- 555$d
- 555$e
- 555$f
- 555$g
- 555$h
- 555$j
- 555$k
- 555$l
- 555$m
- 555$n
- 555$o
- 555$p
- 555$q
- 555$r
- 555$s
- 555$t
- 562$b
- 562$c
- 562$d
- 562$e
- 562$f
- 562$g
- 562$h
- 562$k
- 562$l
- 562$m
- 562$n
- 562$o
- 562$p
- 562$r
- 562$s
- 562$t

Furthermore, the format of the mapping for 511 has been corrected.
Comment 3 Caroline Cyr La Rose 2022-10-05 16:25:09 UTC
I corrected the index name in my commit message...
Comment 4 David Nind 2022-10-10 19:17:49 UTC
Created attachment 141584 [details] [review]
Bug 31693 Remove non-existent fields from the See-also-from index (Elasticsearch, MARC21)

This patch removes fields from the See-from index that don't exist in
MARC21.

The existing fields can be found here: https://www.loc.gov/marc/authority/ad5xx.html

The following fields are removed:

- 511$b
- 511$m
- 511$o
- 511$r
- 530$b
- 530$c
- 530$e
- 547$b
- 547$e
- 547$f
- 547$h
- 547$j
- 547$k
- 547$l
- 547$m
- 547$n
- 547$o
- 547$p
- 547$q
- 547$r
- 547$s
- 547$t
- 548$b
- 548$c
- 548$d
- 548$e
- 548$f
- 548$g
- 548$h
- 548$k
- 548$l
- 548$m
- 548$n
- 548$o
- 548$p
- 548$r
- 548$s
- 548$t
- 550$c
- 550$d
- 550$e
- 550$f
- 550$h
- 550$j
- 550$k
- 550$l
- 550$m
- 550$n
- 550$o
- 550$p
- 550$q
- 550$r
- 550$s
- 550$t
- 551$b
- 551$c
- 551$d
- 551$e
- 551$f
- 551$h
- 551$k
- 551$l
- 551$m
- 551$n
- 551$o
- 551$p
- 551$r
- 551$s
- 551$t
- 555$b
- 555$c
- 555$d
- 555$e
- 555$f
- 555$g
- 555$h
- 555$j
- 555$k
- 555$l
- 555$m
- 555$n
- 555$o
- 555$p
- 555$q
- 555$r
- 555$s
- 555$t
- 562$b
- 562$c
- 562$d
- 562$e
- 562$f
- 562$g
- 562$h
- 562$k
- 562$l
- 562$m
- 562$n
- 562$o
- 562$p
- 562$r
- 562$s
- 562$t

Furthermore, the format of the mapping for 511 has been corrected.

Signed-off-by: David Nind <david@davidnind.com>
Comment 5 David Nind 2022-10-10 19:19:15 UTC
"Tested" by comparing the list of fields removed with the Library of Congress definitions to make sure they don't exist.
Comment 6 Kyle M Hall (khall) 2022-10-21 11:39:04 UTC
Created attachment 142336 [details] [review]
Bug 31693 Remove non-existent fields from the See-also-from index (Elasticsearch, MARC21)

This patch removes fields from the See-from index that don't exist in
MARC21.

The existing fields can be found here: https://www.loc.gov/marc/authority/ad5xx.html

The following fields are removed:

- 511$b
- 511$m
- 511$o
- 511$r
- 530$b
- 530$c
- 530$e
- 547$b
- 547$e
- 547$f
- 547$h
- 547$j
- 547$k
- 547$l
- 547$m
- 547$n
- 547$o
- 547$p
- 547$q
- 547$r
- 547$s
- 547$t
- 548$b
- 548$c
- 548$d
- 548$e
- 548$f
- 548$g
- 548$h
- 548$k
- 548$l
- 548$m
- 548$n
- 548$o
- 548$p
- 548$r
- 548$s
- 548$t
- 550$c
- 550$d
- 550$e
- 550$f
- 550$h
- 550$j
- 550$k
- 550$l
- 550$m
- 550$n
- 550$o
- 550$p
- 550$q
- 550$r
- 550$s
- 550$t
- 551$b
- 551$c
- 551$d
- 551$e
- 551$f
- 551$h
- 551$k
- 551$l
- 551$m
- 551$n
- 551$o
- 551$p
- 551$r
- 551$s
- 551$t
- 555$b
- 555$c
- 555$d
- 555$e
- 555$f
- 555$g
- 555$h
- 555$j
- 555$k
- 555$l
- 555$m
- 555$n
- 555$o
- 555$p
- 555$q
- 555$r
- 555$s
- 555$t
- 562$b
- 562$c
- 562$d
- 562$e
- 562$f
- 562$g
- 562$h
- 562$k
- 562$l
- 562$m
- 562$n
- 562$o
- 562$p
- 562$r
- 562$s
- 562$t

Furthermore, the format of the mapping for 511 has been corrected.

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 7 Tomás Cohen Arazi (tcohen) 2022-10-21 14:47:04 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!