Bug 36727

Summary:	Add incorrect ISSN to Elasticsearch index mappings
Product:	Koha	Reporter:	Aleisha Amohia <aleisha>
Component:	Searching - Elasticsearch	Assignee:	Alex Buckley <alexbuckley>
Status:	Needs documenting ---	QA Contact:
Severity:	enhancement
Priority:	P5 - low	CC:	alexbuckley, david, januszop, lucas, m.de.rooy
Version:	unspecified
Hardware:	All
OS:	All
Change sponsored?:	Sponsored	Patch complexity:	Trivial patch
Documentation contact:		Documentation submission:
Text to go in the release notes:	This enhancement makes changes to International Standard Serial Number (ISSN) related index mapping when using Elasticsearch or Open Search. A new issn-all search field enables finding all records with any ISSN related to that content. For example, you are able to find the online and paper version of a serial with one search. The issn search field remains a precise search - you get the exact record with the ISSN in 022$a (MARC21) or 011$a (UNIMARC). MARC21: - Adds these subfields to the new issn-all search field: . 022$a (International Standard Serial Number) . 022$y (Incorrect ISSN) . 022$z (Canceled ISSN) . 023$a (Cluster ISSN) . 023$y (Incorrect Cluster ISSN) . 023$z (Canceled Cluster ISSN) - Adds 022$y, 022$z, 023$a, 023$y, and 023$z to the identifier-standard search field UNIMARC: - Adds these subfields to the new issn-all search field: . 011$a (Number (ISSN) . 011$y (Cancelled ISSN) . 011$z (Erroneous ISSN or ISSN-L) - Removes 011$y and 011$z from the issn search field These changes will only take effect for existing installations if the index mappings are reset (caution: existing customizations are lost) and records are re-indexed.	Version(s) released in:	24.11.00
Circulation function:
Bug Depends on:
Bug Blocks:	36798, 36854
Attachments:	Bug 36727: Add new issn-all Elasticsearch index for MARC21 Bug 36727: Add new issn-all Elasticsearch index for MARC21 Bug 36727: Add new issn-all Elasticsearch index for MARC21 Bug 36727: Add new issn-all Elasticsearch index for MARC21 Bug 36727: Add new issn-all Elasticsearch index for MARC21 Bug 36727: Add new issn-all Elasticsearch index for MARC21

Description Aleisha Amohia 2024-04-30 05:22:15 UTC

Can currently search on the issn with the 022$a mapping

It would be useful to have another index on the incorrect-issn mapping to the 022$y only https://www.loc.gov/marc/bibliographic/bd022.html

Comment 1 Katrin Fischer 2024-04-30 07:42:38 UTC

I believe we had discussed a solution/bug for this at hackfest - adding Janusz.

Comment 2 Janusz Kaczmarek 2024-04-30 12:18:26 UTC

(In reply to Katrin Fischer from comment #1)
> I believe we had discussed a solution/bug for this at hackfest - adding
> Janusz.

In fact, we have discussed ISBNs (020) -- cf. bug 36574, but ISSNs (022) could be resolved in an analogous way.  By the way, there is a new 023 field also involved (https://www.loc.gov/marc/bibliographic/bd023.html).

Comment 3 Alex Buckley 2024-05-13 18:44:38 UTC

Created attachment 166653 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

This index searches 022ay, 023ay (MARC21) and 011ayz (UNIMARC).
011y and 011z have been removed from issn search field for UNIMARC.

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Apply this patch and restart all services (restart_all)

3. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

4. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

5. Alternative to steps 3 and 4: reset_all

6. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y / 023y for MARC21), or if using UNIMARC then 011z

7. Make a biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS

Comment 4 Janusz Kaczmarek 2024-05-13 19:20:22 UTC

Just a doubt: don't we want to have 022 $z and 023 $z indexed with issn-all search field as well?  And don't we want to add 022/023 $y and $z subfields to identifier-standard search field?

Comment 5 Alex Buckley 2024-05-13 19:47:29 UTC

Created attachment 166654 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Apply this patch and restart all services (restart_all)

3. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

4. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

5. Alternative to steps 3 and 4: reset_all

6. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y / 023y for MARC21), or if using UNIMARC then 011z

7. Make a biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS

Comment 6 Alex Buckley 2024-05-13 19:48:17 UTC

(In reply to Janusz Kaczmarek from comment #4)
> Just a doubt: don't we want to have 022 $z and 023 $z indexed with issn-all
> search field as well?  And don't we want to add 022/023 $y and $z subfields
> to identifier-standard search field?

Hi Janusz, 

Thanks for your reply. You're right and I have implemented those changes now.

This is now ready to test.

Thanks,
Alex

Comment 7 Janusz Kaczmarek 2024-05-13 20:04:13 UTC

There are problems in mappings.yaml: should be marc_field instead of marcfield (three spots).

Comment 8 Janusz Kaczmarek 2024-05-13 20:06:03 UTC

And I would also add an initial point to the Test plan to demonstrate the issue.

Comment 9 Alex Buckley 2024-05-13 20:16:30 UTC

(In reply to Janusz Kaczmarek from comment #7)
> There are problems in mappings.yaml: should be marc_field instead of
> marcfield (three spots).

Will do thanks Janusz. Making those changes now.

Comment 10 Alex Buckley 2024-05-13 20:21:36 UTC

Created attachment 166655 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y /023y for MARC21, or 011z for UNIMARC)

3. Do a biblio search for '0046-2254' and confirm no results are returned

4. Apply this patch and restart all services (restart_all)

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all

8. Repeat the biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS

Comment 11 Alex Buckley 2024-05-13 20:22:55 UTC

I've made those changes Janusz and have added two new steps in the test plan (steps 2 and 3) demonstrating the issue before the patch is applied. 

Could you please take another look now?

Thanks!

Comment 12 Janusz Kaczmarek 2024-05-13 20:36:05 UTC

Created attachment 166656 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y /023y for MARC21, or 011z for UNIMARC)

3. Do a biblio search for '0046-2254' and confirm no results are returned

4. Apply this patch and restart all services (restart_all)

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all

8. Repeat the biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>

Comment 13 Alex Buckley 2024-05-13 20:58:49 UTC

Thanks for your testing Janusz.

As you signed off the patch I'll change the status of this bug report to 'Signed off'.

Comment 14 Janusz Kaczmarek 2024-05-13 21:05:56 UTC

(In reply to Alex Buckley from comment #13)
> As you signed off the patch I'll change the status of this bug report to
> 'Signed off'.

Sure! My fault... Thanks!

Comment 15 Marcel de Rooy 2024-05-17 09:35:16 UTC

  issn:
    label: issn
    mandatory: ~
    mappings:
    - facet: ''
      marc_field: 022a
      marc_type: marc21
      sort: 1
      suggestible: ''
    - facet: ''
      marc_field: 011a
      marc_type: unimarc
      sort: 1
      suggestible: ''
    opac: 1
    staff_client: 1
    type: stdno
  issn-all:
    label: issn-all
    mandatory: ~
    mappings:
    - facet: ''
      marc_field: 022ayz
      marc_type: marc21
      sort: 1
      suggestible: ''
    - facet: ''
      marc_field: 023ayz
      marc_type: marc21
      sort: 1
      suggestible: ''
    - facet: ''
      marc_field: 011ayz
      marc_type: unimarc
      sort: 1
      suggestible: ''
    opac: 1
    staff_client: 1
    type: stdno

Seems that we create an inconsistency here? 023 is not in issn but it is in issn-all?

Comment 16 Janusz Kaczmarek 2024-05-17 11:00:41 UTC

(In reply to Marcel de Rooy from comment #15)
> Seems that we create an inconsistency here? 023 is not in issn but it is in
> issn-all?

If I may comment on this: I wouldn't say that.  The 023 field is a new field introduced only recently instead of the removed 022 $l subfield (a linking ISSN). 

https://www.loc.gov/marc/bibliographic/bd023.html
https://www.loc.gov/marc/bibliographic/bd022.html (infra)

So, the logic is: with issn-all search field we want to be able to find a record/all records with any ISSN related to that content, e.g. we want to be able to find the online and paper version of a serial with one search.

Whereas issn search field should remain for the precise search: I want to get exactly that record with exactly this ISSN in 022 $a.

With this logic I do not see a possibility nor need to put any 023 subfield in issn search field.

Comment 17 Alex Buckley 2024-05-30 01:17:10 UTC

(In reply to Janusz Kaczmarek from comment #16)
> (In reply to Marcel de Rooy from comment #15)
> > Seems that we create an inconsistency here? 023 is not in issn but it is in
> > issn-all?
> 
> If I may comment on this: I wouldn't say that.  The 023 field is a new field
> introduced only recently instead of the removed 022 $l subfield (a linking
> ISSN). 
> 
> https://www.loc.gov/marc/bibliographic/bd023.html
> https://www.loc.gov/marc/bibliographic/bd022.html (infra)
> 
> So, the logic is: with issn-all search field we want to be able to find a
> record/all records with any ISSN related to that content, e.g. we want to be
> able to find the online and paper version of a serial with one search.
> 
> Whereas issn search field should remain for the precise search: I want to
> get exactly that record with exactly this ISSN in 022 $a.
> 
> With this logic I do not see a possibility nor need to put any 023 subfield
> in issn search field.

This makes sense to us!

Comment 18 David Nind 2024-05-30 03:49:30 UTC

Created attachment 167257 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y /023y for MARC21, or 011z for UNIMARC)

3. Do a biblio search for '0046-2254' and confirm no results are returned

4. Apply this patch and restart all services (restart_all)

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all

8. Repeat the biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: David Nind <david@davidnind.com>

Comment 19 David Nind 2024-05-30 03:51:01 UTC

I have added my sign off and changed the status.

I wasn't sure from the comments whether more work is required or not. Please change the status back to whatever is appropriate if more work is required.

Comment 20 Nick Clemens (kidclamp) 2024-06-24 22:58:50 UTC

Created attachment 168046 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y /023y for MARC21, or 011z for UNIMARC)

3. Do a biblio search for '0046-2254' and confirm no results are returned

4. Apply this patch and restart all services (restart_all)

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all

8. Repeat the biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Comment 21 Katrin Fischer 2024-06-25 14:55:28 UTC

I am not sure if this is a bug fix or an enh. I tend to mark this as an enh, as it will mostly apply to new installations or libraries resetting their index configuration.

Comment 22 Katrin Fischer 2024-06-25 16:36:58 UTC

Pushed for 24.11!

Well done everyone, thank you!

Comment 23 Lucas Gass (lukeg) 2024-07-18 19:46:08 UTC

Enhancement will not be included in 24.05.x

Comment 24 David Nind 2024-07-22 09:54:18 UTC

I've attempted to write a release note - I got a bit mixed up with indexes and search fields (I'm not sure index or search field - the UI says search field in the column heading). Please correct what I may have got wrong.

I've changed the status to "Needs documenting", as the manual includes the list of indexes in https://koha-community.org/manual/latest/en/html/searching.html#elasticsearch-indexes