Bug 33206 - Bad title__sort made of multisubfield 245
Summary: Bad title__sort made of multisubfield 245
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P3 normal (vote)
Assignee: Janusz Kaczmarek
QA Contact: Katrin Fischer
URL:
Keywords:
Depends on:
Blocks: 33594
  Show dependency treegraph
 
Reported: 2023-03-11 16:34 UTC by Janusz Kaczmarek
Modified: 2023-07-20 04:03 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.05.00,22.11.06,22.05.14


Attachments
Bug 33206 - Bad title__sort made of multifield 245 (4.06 KB, patch)
2023-03-11 17:05 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 33206: Bad title__sort made of multifield 245 (4.10 KB, patch)
2023-04-17 19:27 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 33206: Unit test (3.13 KB, patch)
2023-04-17 19:28 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 33206: Bad title__sort made of multifield 245 (4.16 KB, patch)
2023-04-23 21:40 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 33206: Unit test (3.18 KB, patch)
2023-04-23 21:40 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Janusz Kaczmarek 2023-03-11 16:34:28 UTC
In MARC 21, having title search field defined in mappings.yaml as 245abp (more than one subfield), the Koha generated title__sort will have not only $a, but also other subfields cut by ind2 characters, i.e.:

245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

becomes as title__sort:

Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen

instead of:

Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

This is because copying an array of pointers in Perl causes you to point to the same (anonymous) element.  Therefore, the intended modification only for subfield "a" (Koha/SearchEngine/Elasticsearch.pm, ~line 1157) causes in fact modification for all the subfields sharing the same mapping definition and introduction of the nonfiling_characters_indicator to all subfields 245 when processing 245abp.  And this is obviously not what is intended.
Comment 1 Janusz Kaczmarek 2023-03-11 17:05:41 UTC
Created attachment 148103 [details] [review]
Bug 33206 - Bad title__sort made of multifield 245

In MARC 21, having title search field defined in mappings.yaml as 245abp
(more than one subfield), the Koha generated title__sort will have not
only $a, but also other subfields cut by ind2 characters, i.e.:

245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

becomes as title__sort:

Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen

instead of:

Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

This is because copying an array of pointers in Perl causes you to
point to the same (anonymous) element.  Therefore, the intended
modification only for subfield "a"
(Koha/SearchEngine/Elasticsearch.pm, ~line 1157) causes in fact
modification for all the subfields sharing the same mapping
definition and introduction of the nonfiling_characters_indicator to
all subfields 245 when processing 245abp.  And this is obviously not
what is intended and results (in some cases) with bad sort order 
when sorting by titles.

Test plan
=========
1. Have a standard Koha ES installation. In mappings.yaml
   you should have 245abp for title search field.

2. Insert / download / modify a record with 245 $b or $p,
    an article at the beginning of the title, and correctly
    set 2nd indicator. For instance:

245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

3. In ES, with Kibana or culr, control the content of
   title__sort field for the related document.  It will be:

Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen

4. Apply the patch, full reindex ES (--rebuild -r -d -a -b)

5. In ES, with Kibana or culr, control the content of
   title__sort field for the related document.  It should
   be now correct, i.e.:

Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
Comment 2 Nick Clemens 2023-04-17 19:27:45 UTC
Created attachment 149772 [details] [review]
Bug 33206: Bad title__sort made of multifield 245

In MARC 21, having title search field defined in mappings.yaml as 245abp
(more than one subfield), the Koha generated title__sort will have not
only $a, but also other subfields cut by ind2 characters, i.e.:

245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

becomes as title__sort:

Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen

instead of:

Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

This is because copying an array of pointers in Perl causes you to
point to the same (anonymous) element.  Therefore, the intended
modification only for subfield "a"
(Koha/SearchEngine/Elasticsearch.pm, ~line 1157) causes in fact
modification for all the subfields sharing the same mapping
definition and introduction of the nonfiling_characters_indicator to
all subfields 245 when processing 245abp.  And this is obviously not
what is intended and results (in some cases) with bad sort order
when sorting by titles.

Test plan
=========
1. Have a standard Koha ES installation. In mappings.yaml
   you should have 245abp for title search field.

2. Insert / download / modify a record with 245 $b or $p,
    an article at the beginning of the title, and correctly
    set 2nd indicator. For instance:

245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

3. In ES, with Kibana or culr, control the content of
   title__sort field for the related document.  It will be:

Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen

4. Apply the patch, full reindex ES (--rebuild -r -d -a -b)

5. In ES, with Kibana or culr, control the content of
   title__sort field for the related document.  It should
   be now correct, i.e.:

Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

Signed-off-by: Nick <nick@bywatersolutions.com>
Comment 3 Nick Clemens 2023-04-17 19:28:17 UTC
Created attachment 149773 [details] [review]
Bug 33206: Unit test

Signed-off-by: Nick <nick@bywatersolutions.com>
Comment 4 Nick Clemens 2023-04-17 19:29:07 UTC
Added a unit test, correct commit title, make sure the format is:
Bug ####: Description
it had a dash not a colon
Comment 5 Katrin Fischer 2023-04-23 12:43:11 UTC
I just ran into this in training and glad to see there is already a fix underway.
Comment 6 Katrin Fischer 2023-04-23 21:40:54 UTC
Created attachment 150138 [details] [review]
Bug 33206: Bad title__sort made of multifield 245

In MARC 21, having title search field defined in mappings.yaml as 245abp
(more than one subfield), the Koha generated title__sort will have not
only $a, but also other subfields cut by ind2 characters, i.e.:

245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

becomes as title__sort:

Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen

instead of:

Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

This is because copying an array of pointers in Perl causes you to
point to the same (anonymous) element.  Therefore, the intended
modification only for subfield "a"
(Koha/SearchEngine/Elasticsearch.pm, ~line 1157) causes in fact
modification for all the subfields sharing the same mapping
definition and introduction of the nonfiling_characters_indicator to
all subfields 245 when processing 245abp.  And this is obviously not
what is intended and results (in some cases) with bad sort order
when sorting by titles.

Test plan
=========
1. Have a standard Koha ES installation. In mappings.yaml
   you should have 245abp for title search field.

2. Insert / download / modify a record with 245 $b or $p,
    an article at the beginning of the title, and correctly
    set 2nd indicator. For instance:

245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

3. In ES, with Kibana or culr, control the content of
   title__sort field for the related document.  It will be:

Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen

4. Apply the patch, full reindex ES (--rebuild -r -d -a -b)

5. In ES, with Kibana or culr, control the content of
   title__sort field for the related document.  It should
   be now correct, i.e.:

Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen

Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 7 Katrin Fischer 2023-04-23 21:40:57 UTC
Created attachment 150139 [details] [review]
Bug 33206: Unit test

Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 8 Katrin Fischer 2023-04-23 21:41:28 UTC
Signed-off-by: Nick <nick@bywatersolutions.com> ?

Found some other 'oddness' with the title sorting, filed as bug 33594.
Comment 9 Tomás Cohen Arazi 2023-05-09 14:48:01 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 10 Matt Blenkinsop 2023-05-15 12:10:05 UTC
Nice work everyone!

Pushed to stable for 22.11.x
Comment 11 Lucas Gass 2023-06-15 16:53:24 UTC
Backported to 22.05.x for upcoming 22.05.14
Comment 12 Danyon Sewell 2023-07-20 04:03:34 UTC
Not backporting for 21.11.x