|
Description
Nick Clemens (kidclamp)
2019-10-16 15:28:07 UTC
Part of this issue is resolved by tweaking the mappings for authorities in ES. The current community authority mappings are indexing MARC fields and subfields as strings. Example: 110(abcdefghjklnpqstvxyz) is mapped to Heading. When "Catholic Church" is used for "Search main heading," the result list looks like this (subfields added for clarity): - $a Catholic Church $x Clergy $v Biography - $a Catholic Church $z Italy $x Clergy $v Biography - $a Catholic Church $x Doctrines - $a Catholic Church $x Fiction - $a Catholic Church $v Pastoral letters and charges - $a Catholic Church $x Adult Education $v Periodicals It is ordering them this way because they are indexed with the subfields in alphabetical order -- despite the fact that the subfields do not appear in alphabetical order in the record. When 110 is mapped to Heading, without the (abcd...), the result list looks like this: - Catholic Church Adult Education Periodicals - Catholic Church Adult Education United States - Catholic Church Clergy Biography - Catholic Church Doctrines - Catholic Church Fiction - Catholic Church Italy Clergy Biography WHAT IS STILL A BUG > If a heading has a period in it, it will still appear after all the headings without a period. In the Catholic Church example, some authority records use a period: Catholic Church. Archdiocese of San Salvador (El Salvador). Archbishop (1977-1980 : Romero) That particular heading and all others that follow a similar format will not appear in the results list alphabetically. It appears at the end after all "Catholic Church" headings that do not use a period. We should probably be escaping punctuation when searching and/or indexing authorities. Created attachment 110175 [details] [review] Bug 23828: Unit tests Created attachment 110176 [details] [review] Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly Note: these patches do not address punctuation, we should file a new bug for that issue Note: these patches do not address punctuation, we should file a new bug for that issue I put in Bug 26472 to address punctuation in general in the filing/ordering of authorities search results, having just come across a good example of the problem in our catalog!:) Created attachment 110547 [details] [review] Bug 23828: Unit tests Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Created attachment 110548 [details] [review] Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> There are lots of merge markers in the test file - this hasn't applied cleanly when tested. Created attachment 111329 [details] [review] Bug 23828: Unit tests Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Bug 23828: (follow-up) fix unit test merge Created attachment 111330 [details] [review] Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> (In reply to Katrin Fischer from comment #9) > There are lots of merge markers in the test file - this hasn't applied > cleanly when tested. Only the unit tests failed, the testing should still be valid as the code itself applied and worked Created attachment 111813 [details] [review] Bug 23828: Unit tests Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Bug 23828: (follow-up) fix unit test merge Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Created attachment 111814 [details] [review] Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Created attachment 111815 [details] [review] Bug 23828: (QA follow-up) Fix number of tests Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Created attachment 112142 [details] [review] Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> JD amended patch: Fix FAIL spelling combind ==> combined Created attachment 112143 [details] [review] Bug 23828: Unit tests Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Bug 23828: (follow-up) fix unit test merge Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 23828: (QA follow-up) Fix number of tests Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Pushed to master for 20.11, thanks to everybody involved! backported to 20.05.x for 20.05.06 enhancement, not backported to 19.11.x |