Bug 18984

Summary: Remove support for NORMARC
Product: Koha Reporter: Magnus Enger <magnus>
Component: MARC Bibliographic data supportAssignee: Jonathan Druart <jonathan.druart>
Status: CLOSED FIXED QA Contact: Tomás Cohen Arazi <tomascohen>
Severity: enhancement    
Priority: P5 - low CC: david, dcook, jonathan.druart, josef.moravec, kyle, m.de.rooy, magnus, martin.renvoize, tomascohen, victor
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15140
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
The National library of Norway has replaced NORMARC with MARC21. Koha instances that use NORMARC have either been converted to MARC21, or will need to convert as part of any upgrade (from Koha 21.11 onwards).
Version(s) released in:
21.11.00
Bug Depends on:    
Bug Blocks: 31581, 27622, 30476    
Attachments: Bug 18984: Remove NORMARC support
Bug 18984: Remove ES mapping for NORMARC
Bug 18984: Remove NORMARC support
Bug 18984: Remove ES mapping for NORMARC
Bug 18984: Remove NORMARC support
Bug 18984: Remove ES mapping for NORMARC
Bug 18984: Remove NORMARC support
Bug 18984: Remove ES mapping for NORMARC
Bug 18984: Remove NORMARC support
Bug 18984: Remove ES mapping for NORMARC
Bug 18984: Remove NORMARC support
Bug 18984: Remove ES mapping for NORMARC
Bug 18984: DB changes
Bug 18984: Remove NORMARC support
Bug 18984: Remove ES mapping for NORMARC
Bug 18984: DB changes
Bug 18984: Remove NORMARC support
Bug 18984: Remove ES mapping for NORMARC
Bug 18984: DB changes

Description Magnus Enger 2017-07-25 20:34:40 UTC
The National library of Norway has (finally) decided to replace NORMARC with MARC21. The timeframe for this is a bit vague yet, but I will provide a patch to remove all traces of NORMARC in Koha once all known Koha-instances that use NORMARC have been converted to MARC21. 

As soon as I can I will close all NORMARC-specific bugs in Bugzilla. 

More info about the decision of the National library, in Norwegian:
https://bibliotekutvikling.no/ressurser/kunnskapsorganisering/verktoykasse-for-kunnskapsorganisering/marc-formater/marc-21/nytt-utvekslingsformat-bibliografiske-data/

See also:
https://bibliotekutvikling.no/ressurser/kunnskapsorganisering/verktoykasse-for-kunnskapsorganisering/marc-formater/
Comment 1 Jonathan Druart 2021-01-21 09:34:42 UTC
Created attachment 115514 [details] [review]
Bug 18984: Remove NORMARC support
Comment 2 Jonathan Druart 2021-01-21 09:34:45 UTC
Created attachment 115515 [details] [review]
Bug 18984: Remove ES mapping for NORMARC

perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
Comment 3 Jonathan Druart 2021-01-21 09:35:00 UTC
Would that work, Magnus?
Comment 4 Jonathan Druart 2021-05-03 09:58:29 UTC
Created attachment 120397 [details] [review]
Bug 18984: Remove NORMARC support
Comment 5 Jonathan Druart 2021-05-03 09:58:35 UTC
Created attachment 120398 [details] [review]
Bug 18984: Remove ES mapping for NORMARC

perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
Comment 6 Jonathan Druart 2021-09-28 15:10:04 UTC
From IRC - http://irc.koha-community.org/koha/2021-09-22

07:31 	Joubu 	magnuse: Hi! Can I finally push bug 18984?
07:31 	Joubu 	magnuse: I need it for bug 27622
11:20 	magnuse	Joubu: hm, we have not converted our normarc libraries to marc 21 yet. on the other hand, having to convert before upgrading to 21.11 just might be the stick we need to get the conversions done...
Comment 7 Jonathan Druart 2021-09-28 15:12:43 UTC
Created attachment 125417 [details] [review]
Bug 18984: Remove NORMARC support
Comment 8 Jonathan Druart 2021-09-28 15:12:47 UTC
Created attachment 125418 [details] [review]
Bug 18984: Remove ES mapping for NORMARC

perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
Comment 9 Jonathan Druart 2021-09-28 15:13:10 UTC
Patches rebased.
Comment 10 Tomás Cohen Arazi 2021-10-06 13:19:56 UTC
Please rebase.

I also have a concern about the z39.50 targets for the nb-NO language. If they are still going to be supported, we probably need crosswalk xslts?
Comment 11 Jonathan Druart 2021-10-06 14:18:23 UTC
Created attachment 125819 [details] [review]
Bug 18984: Remove NORMARC support
Comment 12 Jonathan Druart 2021-10-06 14:18:28 UTC
Created attachment 125820 [details] [review]
Bug 18984: Remove ES mapping for NORMARC

perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
Comment 13 Marcel de Rooy 2021-10-06 14:32:41 UTC Comment hidden (obsolete)
Comment 14 Marcel de Rooy 2021-10-06 14:33:02 UTC Comment hidden (obsolete)
Comment 15 Marcel de Rooy 2021-10-06 14:36:46 UTC
Lots of qa warns here for DBIx::Class::Relationship::ManyToMany::many_to_many():
Comment 16 Marcel de Rooy 2021-10-06 14:48:51 UTC
Created attachment 125826 [details] [review]
Bug 18984: Remove NORMARC support

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 17 Marcel de Rooy 2021-10-06 14:48:54 UTC
Created attachment 125827 [details] [review]
Bug 18984: Remove ES mapping for NORMARC

perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 18 Marcel de Rooy 2021-10-06 14:49:54 UTC
DBIx::Class::Relationship::ManyToMany::many_to_many():
***************************************************************************
The many-to-many relationship 'search_fields' is trying to create a utility method
called add_to_search_fields.
This will completely overwrite one such already existing method on class
Koha::Schema::Result::SearchMarcMap.

You almost certainly want to rename your method or the many-to-many
relationship, as the functionality of the original method will not be
accessible anymore.

To disable this warning set to a true value the environment variable
DBIC_OVERWRITE_HELPER_METHODS_OK
Comment 19 Marcel de Rooy 2021-10-06 14:52:18 UTC
diff --git a/Koha/Schema/Result/SearchMarcMap.pm b/Koha/Schema/Result/SearchMarcMap.pm
index 03bc94304c..6b700c65ff 100644
--- a/Koha/Schema/Result/SearchMarcMap.pm
+++ b/Koha/Schema/Result/SearchMarcMap.pm
@@ -40,7 +40,7 @@ what storage index this map is for
 =head2 marc_type

   data_type: 'enum'
-  extra: {list => ["marc21","unimarc","normarc"]}
+  extra: {list => ["marc21","unimarc"]}
   is_nullable: 0

 what MARC type this map is for
@@ -67,7 +67,7 @@ __PACKAGE__->add_columns(
   "marc_type",
   {
     data_type => "enum",
-    extra => { list => ["marc21", "unimarc", "normarc"] },
+    extra => { list => ["marc21", "unimarc"] },
     is_nullable => 0,
   },
   "marc_field",

You dont want this in your regular patch.
Comment 20 Marcel de Rooy 2021-10-06 14:53:31 UTC
The signoff stands, but it needs attention still. Switch back to SO afterwards.
Comment 21 Marcel de Rooy 2021-10-06 14:54:38 UTC
Dont forget to add a dbrev for search_marc_map
Comment 22 Jonathan Druart 2021-10-06 14:56:39 UTC
Created attachment 125828 [details] [review]
Bug 18984: Remove NORMARC support
Comment 23 Jonathan Druart 2021-10-06 14:56:45 UTC
Created attachment 125829 [details] [review]
Bug 18984: Remove ES mapping for NORMARC

perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
Comment 24 Jonathan Druart 2021-10-06 14:57:01 UTC
Koha/Schema/Result/SearchMarcMap.pm changes removed from last patches.
Comment 25 Marcel de Rooy 2021-10-06 14:58:40 UTC
How about removing:

misc/translator/po/am-Ethi-marc-NORMARC.po
misc/translator/po/ar-Arab-marc-NORMARC.po
misc/translator/po/as-IN-marc-NORMARC.po
misc/translator/po/az-AZ-marc-NORMARC.po
misc/translator/po/be-BY-marc-NORMARC.po
misc/translator/po/bg-Cyrl-marc-NORMARC.po
misc/translator/po/bn-IN-marc-NORMARC.po
misc/translator/po/ca-ES-marc-NORMARC.po
misc/translator/po/cs-CZ-marc-NORMARC.po
misc/translator/po/cy-GB-marc-NORMARC.po
misc/translator/po/da-DK-marc-NORMARC.po
misc/translator/po/de-CH-marc-NORMARC.po
misc/translator/po/de-DE-marc-NORMARC.po
misc/translator/po/dz-marc-NORMARC.po
misc/translator/po/el-GR-marc-NORMARC.po
misc/translator/po/en-GB-marc-NORMARC.po
misc/translator/po/en-NZ-marc-NORMARC.po
misc/translator/po/eo-marc-NORMARC.po
misc/translator/po/es-ES-marc-NORMARC.po
misc/translator/po/eu-marc-NORMARC.po
misc/translator/po/fa-Arab-marc-NORMARC.po
misc/translator/po/fi-FI-marc-NORMARC.po
misc/translator/po/fo-FO-marc-NORMARC.po
misc/translator/po/fr-CA-marc-NORMARC.po
misc/translator/po/fr-FR-marc-NORMARC.po
misc/translator/po/ga-marc-NORMARC.po
misc/translator/po/gd-marc-NORMARC.po
misc/translator/po/gl-marc-NORMARC.po
misc/translator/po/he-Hebr-marc-NORMARC.po
misc/translator/po/hi-marc-NORMARC.po
misc/translator/po/hr-HR-marc-NORMARC.po
misc/translator/po/hu-HU-marc-NORMARC.po
misc/translator/po/hy-Armn-marc-NORMARC.po
misc/translator/po/hyw-Armn-marc-NORMARC.po
misc/translator/po/ia-marc-NORMARC.po
misc/translator/po/id-ID-marc-NORMARC.po
misc/translator/po/iq-CA-marc-NORMARC.po
misc/translator/po/is-IS-marc-NORMARC.po
misc/translator/po/it-IT-marc-NORMARC.po
misc/translator/po/iu-CA-marc-NORMARC.po
misc/translator/po/ja-Jpan-JP-marc-NORMARC.po
misc/translator/po/ka-marc-NORMARC.po
misc/translator/po/km-KH-marc-NORMARC.po
misc/translator/po/kn-Knda-marc-NORMARC.po
misc/translator/po/ko-Kore-KP-marc-NORMARC.po
misc/translator/po/ku-Arab-marc-NORMARC.po
misc/translator/po/lo-Laoo-marc-NORMARC.po
misc/translator/po/lv-marc-NORMARC.po
misc/translator/po/mi-NZ-marc-NORMARC.po
misc/translator/po/ml-marc-NORMARC.po
misc/translator/po/mon-marc-NORMARC.po
misc/translator/po/mr-marc-NORMARC.po
misc/translator/po/ms-MY-marc-NORMARC.po
misc/translator/po/my-marc-NORMARC.po
misc/translator/po/nb-NO-marc-NORMARC.po
misc/translator/po/ne-NE-marc-NORMARC.po
misc/translator/po/nl-BE-marc-NORMARC.po
misc/translator/po/nl-NL-marc-NORMARC.po
misc/translator/po/nn-NO-marc-NORMARC.po
misc/translator/po/oc-marc-NORMARC.po
misc/translator/po/pbr-marc-NORMARC.po
misc/translator/po/pl-PL-marc-NORMARC.po
misc/translator/po/prs-marc-NORMARC.po
misc/translator/po/pt-BR-marc-NORMARC.po
misc/translator/po/pt-PT-marc-NORMARC.po
misc/translator/po/ro-RO-marc-NORMARC.po
misc/translator/po/ru-RU-marc-NORMARC.po
misc/translator/po/rw-RW-marc-NORMARC.po
misc/translator/po/sd-PK-marc-NORMARC.po
misc/translator/po/sk-SK-marc-NORMARC.po
misc/translator/po/sl-SI-marc-NORMARC.po
misc/translator/po/sq-AL-marc-NORMARC.po
misc/translator/po/sr-Cyrl-marc-NORMARC.po
misc/translator/po/sv-SE-marc-NORMARC.po
misc/translator/po/sw-KE-marc-NORMARC.po
misc/translator/po/ta-LK-marc-NORMARC.po
misc/translator/po/ta-marc-NORMARC.po
misc/translator/po/te-marc-NORMARC.po
misc/translator/po/tet-marc-NORMARC.po
misc/translator/po/th-TH-marc-NORMARC.po
misc/translator/po/tl-PH-marc-NORMARC.po
misc/translator/po/tr-TR-marc-NORMARC.po
misc/translator/po/tvl-marc-NORMARC.po
misc/translator/po/uk-UA-marc-NORMARC.po
misc/translator/po/ur-Arab-marc-NORMARC.po
misc/translator/po/vi-VN-marc-NORMARC.po
misc/translator/po/zh-Hans-CN-marc-NORMARC.po
misc/translator/po/zh-Hant-TW-marc-NORMARC.po
root@master:/usr/share/koha# git grep -l normarc > zz
root@master:/usr/share/koha# cat zz
installer/data/mysql/updatedatabase.pl
misc/release_notes/release_notes_3_12_0.txt
misc/translator/po/am-Ethi-marc-NORMARC.po
misc/translator/po/ar-Arab-marc-NORMARC.po
misc/translator/po/as-IN-marc-NORMARC.po
misc/translator/po/az-AZ-marc-NORMARC.po
misc/translator/po/be-BY-marc-NORMARC.po
misc/translator/po/bg-Cyrl-marc-NORMARC.po
misc/translator/po/bn-IN-marc-NORMARC.po
misc/translator/po/ca-ES-marc-NORMARC.po
misc/translator/po/cs-CZ-marc-NORMARC.po
misc/translator/po/cy-GB-marc-NORMARC.po
misc/translator/po/da-DK-marc-NORMARC.po
misc/translator/po/de-CH-marc-NORMARC.po
misc/translator/po/de-DE-marc-NORMARC.po
misc/translator/po/dz-marc-NORMARC.po
misc/translator/po/el-GR-marc-NORMARC.po
misc/translator/po/en-GB-marc-NORMARC.po
misc/translator/po/en-NZ-marc-NORMARC.po
misc/translator/po/eo-marc-NORMARC.po
misc/translator/po/es-ES-marc-NORMARC.po
misc/translator/po/eu-marc-NORMARC.po
misc/translator/po/fa-Arab-marc-NORMARC.po
misc/translator/po/fi-FI-marc-NORMARC.po
misc/translator/po/fo-FO-marc-NORMARC.po
misc/translator/po/fr-CA-marc-NORMARC.po
misc/translator/po/fr-FR-marc-NORMARC.po
misc/translator/po/ga-marc-NORMARC.po
misc/translator/po/gd-marc-NORMARC.po
misc/translator/po/gl-marc-NORMARC.po
misc/translator/po/he-Hebr-marc-NORMARC.po
misc/translator/po/hi-marc-NORMARC.po
misc/translator/po/hr-HR-marc-NORMARC.po
misc/translator/po/hu-HU-marc-NORMARC.po
misc/translator/po/hy-Armn-marc-NORMARC.po
misc/translator/po/hyw-Armn-marc-NORMARC.po
misc/translator/po/ia-marc-NORMARC.po
misc/translator/po/id-ID-marc-NORMARC.po
misc/translator/po/iq-CA-marc-NORMARC.po
misc/translator/po/is-IS-marc-NORMARC.po
misc/translator/po/it-IT-marc-NORMARC.po
misc/translator/po/iu-CA-marc-NORMARC.po
misc/translator/po/ja-Jpan-JP-marc-NORMARC.po
misc/translator/po/ka-marc-NORMARC.po
misc/translator/po/km-KH-marc-NORMARC.po
misc/translator/po/kn-Knda-marc-NORMARC.po
misc/translator/po/ko-Kore-KP-marc-NORMARC.po
misc/translator/po/ku-Arab-marc-NORMARC.po
misc/translator/po/lo-Laoo-marc-NORMARC.po
misc/translator/po/lv-marc-NORMARC.po
misc/translator/po/mi-NZ-marc-NORMARC.po
misc/translator/po/ml-marc-NORMARC.po
misc/translator/po/mon-marc-NORMARC.po
misc/translator/po/mr-marc-NORMARC.po
misc/translator/po/ms-MY-marc-NORMARC.po
misc/translator/po/my-marc-NORMARC.po
misc/translator/po/nb-NO-marc-NORMARC.po
misc/translator/po/ne-NE-marc-NORMARC.po
misc/translator/po/nl-BE-marc-NORMARC.po
misc/translator/po/nl-NL-marc-NORMARC.po
misc/translator/po/nn-NO-marc-NORMARC.po
misc/translator/po/oc-marc-NORMARC.po
misc/translator/po/pbr-marc-NORMARC.po
misc/translator/po/pl-PL-marc-NORMARC.po
misc/translator/po/prs-marc-NORMARC.po
misc/translator/po/pt-BR-marc-NORMARC.po
misc/translator/po/pt-PT-marc-NORMARC.po
misc/translator/po/ro-RO-marc-NORMARC.po
misc/translator/po/ru-RU-marc-NORMARC.po
misc/translator/po/rw-RW-marc-NORMARC.po
misc/translator/po/sd-PK-marc-NORMARC.po
misc/translator/po/sk-SK-marc-NORMARC.po
misc/translator/po/sl-SI-marc-NORMARC.po
misc/translator/po/sq-AL-marc-NORMARC.po
misc/translator/po/sr-Cyrl-marc-NORMARC.po
misc/translator/po/sv-SE-marc-NORMARC.po
misc/translator/po/sw-KE-marc-NORMARC.po
misc/translator/po/ta-LK-marc-NORMARC.po
misc/translator/po/ta-marc-NORMARC.po
misc/translator/po/te-marc-NORMARC.po
misc/translator/po/tet-marc-NORMARC.po
misc/translator/po/th-TH-marc-NORMARC.po
misc/translator/po/tl-PH-marc-NORMARC.po
misc/translator/po/tr-TR-marc-NORMARC.po
misc/translator/po/tvl-marc-NORMARC.po
misc/translator/po/uk-UA-marc-NORMARC.po
misc/translator/po/ur-Arab-marc-NORMARC.po
misc/translator/po/vi-VN-marc-NORMARC.po
misc/translator/po/zh-Hans-CN-marc-NORMARC.po
misc/translator/po/zh-Hant-TW-marc-NORMARC.po
Comment 26 Marcel de Rooy 2021-10-06 14:59:20 UTC
hmm bit too much text
Comment 27 Jonathan Druart 2021-10-06 15:00:27 UTC
Created attachment 125830 [details] [review]
Bug 18984: DB changes
Comment 28 Jonathan Druart 2021-10-06 15:02:20 UTC
(In reply to Marcel de Rooy from comment #25)
> How about removing:

Hum good question. That needs to be done with Bernardo at release time I'd say, otherwise he is going to re-introduce them.

I don't know, to be honest.
Comment 29 Jonathan Druart 2021-10-06 15:03:04 UTC
(In reply to Jonathan Druart from comment #28)
> (In reply to Marcel de Rooy from comment #25)
> > How about removing:
> 
> Hum good question. That needs to be done with Bernardo at release time I'd
> say, otherwise he is going to re-introduce them.
> 
> I don't know, to be honest.

It would make sense to remove them now, but make sure we are not going to reintroduce them later.
Comment 30 Jonathan Druart 2021-10-06 15:07:31 UTC
Patch is too big
See https://gitlab.com/joubu/Koha/-/commits/bug_18984

For "Bug 18984: Remove *-NORMARC.po files"
Comment 31 Marcel de Rooy 2021-10-07 06:14:26 UTC
Created attachment 125850 [details] [review]
Bug 18984: Remove NORMARC support

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 32 Marcel de Rooy 2021-10-07 06:14:31 UTC
Created attachment 125851 [details] [review]
Bug 18984: Remove ES mapping for NORMARC

perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 33 Marcel de Rooy 2021-10-07 06:14:35 UTC
Created attachment 125852 [details] [review]
Bug 18984: DB changes

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 34 Tomás Cohen Arazi 2021-10-07 13:08:13 UTC
Created attachment 125876 [details] [review]
Bug 18984: Remove NORMARC support

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 35 Tomás Cohen Arazi 2021-10-07 13:08:24 UTC
Created attachment 125877 [details] [review]
Bug 18984: Remove ES mapping for NORMARC

perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 36 Tomás Cohen Arazi 2021-10-07 13:08:33 UTC
Created attachment 125878 [details] [review]
Bug 18984: DB changes

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 37 Jonathan Druart 2021-10-07 13:48:08 UTC
Pushed to master for 21.11, thanks to everybody involved!