Created attachment 171219 [details]
Screenshot

Screenshot of my record with the title field showing (Title, Number of part, Name of part, Remainder of title) and the "Duplicate record suspected" warning message at the top saying this is a possible duplicate of Giulio Cesare.

Why? That record has 246 $iOther title on container:$aGiulio Cesare in Egitto, and Koha indexes 246$i (and for that matter $h, or any imaginary subfield you add to 246 in a framework) in the title index.

The "fun" thing about ElasticSearch is you can fix that: Administration - Search engine configuration (Elasticsearch) - Bibliographic records tab, find the title line for 246 and change it from just 246 to 246ab and koha-elasticsearch --rebuild -d -b kohadev.

Thanks Phil! 

I feel like 246$i should not be indexed in the title, is it just me? We could change the default mapping to 246abnp.

Opinions?

(In reply to Caroline Cyr La Rose from comment #3)
> Thanks Phil! 
> 
> I feel like 246$i should not be indexed in the title, is it just me? We
> could change the default mapping to 246abnp.
> 
> Opinions?

We have had libraries complaining about the title index being too wide in the past. I am not sure if we can make everyone happy here, but maybe removing the $i by default from 246 would be a good start.

Created attachment 171488 [details] [review]
Bug 37871: Remove extraneous 246 subfields from the title mappings (Elasticsearch, MARC21)

This patch limits indexing of field 246 to $a, $b, $n, and $p in
various title indexes.
Previously, all 246 subfields were indexed, including non-title
subfields such as $i (Display text), $g (Miscellaneous information),
and linking subfields, making the title index very large and giving
false results.

To test:

1. Add all the 246 subfields to the default bibliographic framework
   1.1. Go to Administration > MARC bibliographic framework
   1.2. Click Actions > MARC structure next to the Default framework
   1.3. Search for 246
   1.4. Click Actions > Edit subfields
   1.5. For each subfield, make sure Editor is checked
   1.6. Click Save changes

2. Create a new record and fill out all the 246 subfields
   2.1. Go to Cataloging
   2.2. Click New record
   2.3. Fill out the mandatory fields (000, 003, 005, 008, 040$c,
        245$a, 942$c)
   2.4. Fill out all the subfields in 246 (I simply write the name of
        the subfield in the text field e.g. Display text in 246$i)
   2.5. Click Save
   2.6. Click Normal to access the detailed record

3. View the Elastic search record
   --> All the subfields (including "Display text", "Miscellaneous
       information" and other non-title subfields) should be indexed in
       - title
       - title__suggestion
       - title-abbreviated
       - title-abbreviated__sort
       - title expanded
       - title-expanded__sort
       - title-former
       - title-former__sort

4. Make a title or keyword search for "Display" (or whatever you wrote
   in 246$i)
   --> Your record should be in the results

5. Apply patch

6. Rebuild the index using -r

koha-elasticsearch --rebuild -d -r -v kohadev

7. Redo step 3
   --> Only 246 $a, $b, $n, and $p should be indexed

8. Redo step 4
   --> Your record should NOT be in the results

Created attachment 171497 [details] [review]
Bug 37871: Remove extraneous 246 subfields from the title mappings (Elasticsearch, MARC21)

This patch limits indexing of field 246 to $a, $b, $n, and $p in
various title indexes.
Previously, all 246 subfields were indexed, including non-title
subfields such as $i (Display text), $g (Miscellaneous information),
and linking subfields, making the title index very large and giving
false results.

To test:

1. Add all the 246 subfields to the default bibliographic framework
   1.1. Go to Administration > MARC bibliographic framework
   1.2. Click Actions > MARC structure next to the Default framework
   1.3. Search for 246
   1.4. Click Actions > Edit subfields
   1.5. For each subfield, make sure Editor is checked
   1.6. Click Save changes

2. Create a new record and fill out all the 246 subfields
   2.1. Go to Cataloging
   2.2. Click New record
   2.3. Fill out the mandatory fields (000, 003, 005, 008, 040$c,
        245$a, 942$c)
   2.4. Fill out all the subfields in 246 (I simply write the name of
        the subfield in the text field e.g. Display text in 246$i)
   2.5. Click Save
   2.6. Click Normal to access the detailed record

3. View the Elastic search record
   --> All the subfields (including "Display text", "Miscellaneous
       information" and other non-title subfields) should be indexed in
       - title
       - title__suggestion
       - title-abbreviated
       - title-abbreviated__sort
       - title expanded
       - title-expanded__sort
       - title-former
       - title-former__sort

4. Make a title or keyword search for "Display" (or whatever you wrote
   in 246$i)
   --> Your record should be in the results

5. Apply patch

6. Rebuild the index using -r

koha-elasticsearch --rebuild -d -r -v kohadev

7. Redo step 3
   --> Only 246 $a, $b, $n, and $p should be indexed

8. Redo step 4
   --> Your record should NOT be in the results

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>

Created attachment 172165 [details] [review]
Bug 37871: Remove extraneous 246 subfields from the title mappings (Elasticsearch, MARC21)

This patch limits indexing of field 246 to $a, $b, $n, and $p in
various title indexes.
Previously, all 246 subfields were indexed, including non-title
subfields such as $i (Display text), $g (Miscellaneous information),
and linking subfields, making the title index very large and giving
false results.

To test:

1. Add all the 246 subfields to the default bibliographic framework
   1.1. Go to Administration > MARC bibliographic framework
   1.2. Click Actions > MARC structure next to the Default framework
   1.3. Search for 246
   1.4. Click Actions > Edit subfields
   1.5. For each subfield, make sure Editor is checked
   1.6. Click Save changes

2. Create a new record and fill out all the 246 subfields
   2.1. Go to Cataloging
   2.2. Click New record
   2.3. Fill out the mandatory fields (000, 003, 005, 008, 040$c,
        245$a, 942$c)
   2.4. Fill out all the subfields in 246 (I simply write the name of
        the subfield in the text field e.g. Display text in 246$i)
   2.5. Click Save
   2.6. Click Normal to access the detailed record

3. View the Elastic search record
   --> All the subfields (including "Display text", "Miscellaneous
       information" and other non-title subfields) should be indexed in
       - title
       - title__suggestion
       - title-abbreviated
       - title-abbreviated__sort
       - title expanded
       - title-expanded__sort
       - title-former
       - title-former__sort

4. Make a title or keyword search for "Display" (or whatever you wrote
   in 246$i)
   --> Your record should be in the results

5. Apply patch

6. Rebuild the index using -r

koha-elasticsearch --rebuild -d -r -v kohadev

7. Redo step 3
   --> Only 246 $a, $b, $n, and $p should be indexed

8. Redo step 4
   --> Your record should NOT be in the results

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Thanks for all the hard work!

Pushed to main for the next 24.11.00 release as RM Assistant

I am all OK with the change and the push, but I feel the bug description no longer really matches what we did here and maybe a short explanation in release notes would be good as well.

Is the title better, and release notes clear?

I like it, thanks :)

Backported to 24.05.x for upcoming 24.05.06