Bug 36727 - Add incorrect ISSN to Elasticsearch index mappings
Summary: Add incorrect ISSN to Elasticsearch index mappings
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Alex Buckley
QA Contact:
URL:
Keywords:
Depends on:
Blocks: 36798 36854
  Show dependency treegraph
 
Reported: 2024-04-30 05:22 UTC by Aleisha Amohia
Modified: 2024-06-25 16:36 UTC (History)
5 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.11.00


Attachments
Bug 36727: Add new issn-all Elasticsearch index for MARC21 (1.97 KB, patch)
2024-05-13 18:44 UTC, Alex Buckley
Details | Diff | Splinter Review
Bug 36727: Add new issn-all Elasticsearch index for MARC21 (2.36 KB, patch)
2024-05-13 19:47 UTC, Alex Buckley
Details | Diff | Splinter Review
Bug 36727: Add new issn-all Elasticsearch index for MARC21 (2.39 KB, patch)
2024-05-13 20:21 UTC, Alex Buckley
Details | Diff | Splinter Review
Bug 36727: Add new issn-all Elasticsearch index for MARC21 (2.44 KB, patch)
2024-05-13 20:36 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 36727: Add new issn-all Elasticsearch index for MARC21 (2.48 KB, patch)
2024-05-30 03:49 UTC, David Nind
Details | Diff | Splinter Review
Bug 36727: Add new issn-all Elasticsearch index for MARC21 (2.54 KB, patch)
2024-06-24 22:58 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Aleisha Amohia 2024-04-30 05:22:15 UTC
Can currently search on the issn with the 022$a mapping

It would be useful to have another index on the incorrect-issn mapping to the 022$y only https://www.loc.gov/marc/bibliographic/bd022.html
Comment 1 Katrin Fischer 2024-04-30 07:42:38 UTC
I believe we had discussed a solution/bug for this at hackfest - adding Janusz.
Comment 2 Janusz Kaczmarek 2024-04-30 12:18:26 UTC
(In reply to Katrin Fischer from comment #1)
> I believe we had discussed a solution/bug for this at hackfest - adding
> Janusz.

In fact, we have discussed ISBNs (020) -- cf. bug 36574, but ISSNs (022) could be resolved in an analogous way.  By the way, there is a new 023 field also involved (https://www.loc.gov/marc/bibliographic/bd023.html).
Comment 3 Alex Buckley 2024-05-13 18:44:38 UTC
Created attachment 166653 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

This index searches 022ay, 023ay (MARC21) and 011ayz (UNIMARC).
011y and 011z have been removed from issn search field for UNIMARC.

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Apply this patch and restart all services (restart_all)

3. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

4. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

5. Alternative to steps 3 and 4: reset_all

6. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y / 023y for MARC21), or if using UNIMARC then 011z

7. Make a biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS
Comment 4 Janusz Kaczmarek 2024-05-13 19:20:22 UTC
Just a doubt: don't we want to have 022 $z and 023 $z indexed with issn-all search field as well?  And don't we want to add 022/023 $y and $z subfields to identifier-standard search field?
Comment 5 Alex Buckley 2024-05-13 19:47:29 UTC
Created attachment 166654 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Apply this patch and restart all services (restart_all)

3. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

4. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

5. Alternative to steps 3 and 4: reset_all

6. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y / 023y for MARC21), or if using UNIMARC then 011z

7. Make a biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS
Comment 6 Alex Buckley 2024-05-13 19:48:17 UTC
(In reply to Janusz Kaczmarek from comment #4)
> Just a doubt: don't we want to have 022 $z and 023 $z indexed with issn-all
> search field as well?  And don't we want to add 022/023 $y and $z subfields
> to identifier-standard search field?

Hi Janusz, 

Thanks for your reply. You're right and I have implemented those changes now.

This is now ready to test.

Thanks,
Alex
Comment 7 Janusz Kaczmarek 2024-05-13 20:04:13 UTC
There are problems in mappings.yaml: should be marc_field instead of marcfield (three spots).
Comment 8 Janusz Kaczmarek 2024-05-13 20:06:03 UTC
And I would also add an initial point to the Test plan to demonstrate the issue.
Comment 9 Alex Buckley 2024-05-13 20:16:30 UTC
(In reply to Janusz Kaczmarek from comment #7)
> There are problems in mappings.yaml: should be marc_field instead of
> marcfield (three spots).

Will do thanks Janusz. Making those changes now.
Comment 10 Alex Buckley 2024-05-13 20:21:36 UTC
Created attachment 166655 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y /023y for MARC21, or 011z for UNIMARC)

3. Do a biblio search for '0046-2254' and confirm no results are returned

4. Apply this patch and restart all services (restart_all)

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all

8. Repeat the biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS
Comment 11 Alex Buckley 2024-05-13 20:22:55 UTC
I've made those changes Janusz and have added two new steps in the test plan (steps 2 and 3) demonstrating the issue before the patch is applied. 

Could you please take another look now?

Thanks!
Comment 12 Janusz Kaczmarek 2024-05-13 20:36:05 UTC
Created attachment 166656 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y /023y for MARC21, or 011z for UNIMARC)

3. Do a biblio search for '0046-2254' and confirm no results are returned

4. Apply this patch and restart all services (restart_all)

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all

8. Repeat the biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Comment 13 Alex Buckley 2024-05-13 20:58:49 UTC
Thanks for your testing Janusz.

As you signed off the patch I'll change the status of this bug report to 'Signed off'.
Comment 14 Janusz Kaczmarek 2024-05-13 21:05:56 UTC
(In reply to Alex Buckley from comment #13)
> As you signed off the patch I'll change the status of this bug report to
> 'Signed off'.

Sure! My fault... Thanks!
Comment 15 Marcel de Rooy 2024-05-17 09:35:16 UTC
  issn:
    label: issn
    mandatory: ~
    mappings:
    - facet: ''
      marc_field: 022a
      marc_type: marc21
      sort: 1
      suggestible: ''
    - facet: ''
      marc_field: 011a
      marc_type: unimarc
      sort: 1
      suggestible: ''
    opac: 1
    staff_client: 1
    type: stdno
  issn-all:
    label: issn-all
    mandatory: ~
    mappings:
    - facet: ''
      marc_field: 022ayz
      marc_type: marc21
      sort: 1
      suggestible: ''
    - facet: ''
      marc_field: 023ayz
      marc_type: marc21
      sort: 1
      suggestible: ''
    - facet: ''
      marc_field: 011ayz
      marc_type: unimarc
      sort: 1
      suggestible: ''
    opac: 1
    staff_client: 1
    type: stdno

Seems that we create an inconsistency here? 023 is not in issn but it is in issn-all?
Comment 16 Janusz Kaczmarek 2024-05-17 11:00:41 UTC
(In reply to Marcel de Rooy from comment #15)
> Seems that we create an inconsistency here? 023 is not in issn but it is in
> issn-all?

If I may comment on this: I wouldn't say that.  The 023 field is a new field introduced only recently instead of the removed 022 $l subfield (a linking ISSN). 

https://www.loc.gov/marc/bibliographic/bd023.html
https://www.loc.gov/marc/bibliographic/bd022.html (infra)

So, the logic is: with issn-all search field we want to be able to find a record/all records with any ISSN related to that content, e.g. we want to be able to find the online and paper version of a serial with one search.

Whereas issn search field should remain for the precise search: I want to get exactly that record with exactly this ISSN in 022 $a.

With this logic I do not see a possibility nor need to put any 023 subfield in issn search field.
Comment 17 Alex Buckley 2024-05-30 01:17:10 UTC
(In reply to Janusz Kaczmarek from comment #16)
> (In reply to Marcel de Rooy from comment #15)
> > Seems that we create an inconsistency here? 023 is not in issn but it is in
> > issn-all?
> 
> If I may comment on this: I wouldn't say that.  The 023 field is a new field
> introduced only recently instead of the removed 022 $l subfield (a linking
> ISSN). 
> 
> https://www.loc.gov/marc/bibliographic/bd023.html
> https://www.loc.gov/marc/bibliographic/bd022.html (infra)
> 
> So, the logic is: with issn-all search field we want to be able to find a
> record/all records with any ISSN related to that content, e.g. we want to be
> able to find the online and paper version of a serial with one search.
> 
> Whereas issn search field should remain for the precise search: I want to
> get exactly that record with exactly this ISSN in 022 $a.
> 
> With this logic I do not see a possibility nor need to put any 023 subfield
> in issn search field.

This makes sense to us!
Comment 18 David Nind 2024-05-30 03:49:30 UTC
Created attachment 167257 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y /023y for MARC21, or 011z for UNIMARC)

3. Do a biblio search for '0046-2254' and confirm no results are returned

4. Apply this patch and restart all services (restart_all)

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all

8. Repeat the biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: David Nind <david@davidnind.com>
Comment 19 David Nind 2024-05-30 03:51:01 UTC
I have added my sign off and changed the status.

I wasn't sure from the comments whether more work is required or not. Please change the status back to whatever is appropriate if more work is required.
Comment 20 Nick Clemens (kidclamp) 2024-06-24 22:58:50 UTC
Created attachment 168046 [details] [review]
Bug 36727: Add new issn-all Elasticsearch index for MARC21

- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field

Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up

2. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y /023y for MARC21, or 011z for UNIMARC)

3. Do a biblio search for '0046-2254' and confirm no results are returned

4. Apply this patch and restart all services (restart_all)

5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)

6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

7. Alternative to steps 5 and 6: reset_all

8. Repeat the biblio search for '0046-2254' and confirm you get a result

Sponsored-by: Education Services Australia SCIS
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 21 Katrin Fischer 2024-06-25 14:55:28 UTC
I am not sure if this is a bug fix or an enh. I tend to mark this as an enh, as it will mostly apply to new installations or libraries resetting their index configuration.
Comment 22 Katrin Fischer 2024-06-25 16:36:58 UTC
Pushed for 24.11!

Well done everyone, thank you!