The National library of Norway has (finally) decided to replace NORMARC with MARC21. The timeframe for this is a bit vague yet, but I will provide a patch to remove all traces of NORMARC in Koha once all known Koha-instances that use NORMARC have been converted to MARC21. As soon as I can I will close all NORMARC-specific bugs in Bugzilla. More info about the decision of the National library, in Norwegian: https://bibliotekutvikling.no/ressurser/kunnskapsorganisering/verktoykasse-for-kunnskapsorganisering/marc-formater/marc-21/nytt-utvekslingsformat-bibliografiske-data/ See also: https://bibliotekutvikling.no/ressurser/kunnskapsorganisering/verktoykasse-for-kunnskapsorganisering/marc-formater/
Created attachment 115514 [details] [review] Bug 18984: Remove NORMARC support
Created attachment 115515 [details] [review] Bug 18984: Remove ES mapping for NORMARC perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
Would that work, Magnus?
Created attachment 120397 [details] [review] Bug 18984: Remove NORMARC support
Created attachment 120398 [details] [review] Bug 18984: Remove ES mapping for NORMARC perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
From IRC - http://irc.koha-community.org/koha/2021-09-22 07:31 Joubu magnuse: Hi! Can I finally push bug 18984? 07:31 Joubu magnuse: I need it for bug 27622 11:20 magnuse Joubu: hm, we have not converted our normarc libraries to marc 21 yet. on the other hand, having to convert before upgrading to 21.11 just might be the stick we need to get the conversions done...
Created attachment 125417 [details] [review] Bug 18984: Remove NORMARC support
Created attachment 125418 [details] [review] Bug 18984: Remove ES mapping for NORMARC perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
Patches rebased.
Please rebase. I also have a concern about the z39.50 targets for the nb-NO language. If they are still going to be supported, we probably need crosswalk xslts?
Created attachment 125819 [details] [review] Bug 18984: Remove NORMARC support
Created attachment 125820 [details] [review] Bug 18984: Remove ES mapping for NORMARC perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
FAIL t/db_dependent/selenium/purchase_workflow.t OK critic FAIL file permissions File must have the exec flag OK forbidden patterns OK git manipulation OK pod SKIP spelling OK valid Processing additional checks * Commit title does not start with 'Bug XXXXX: ' - 75d22621e8
(In reply to Marcel de Rooy from comment #13) > FAIL t/db_dependent/selenium/purchase_workflow.t > OK critic > FAIL file permissions > File must have the exec flag > OK forbidden patterns > OK git manipulation > OK pod > SKIP spelling > OK valid > > > Processing additional checks > > * Commit title does not start with 'Bug XXXXX: ' - 75d22621e8 wrong bug
Lots of qa warns here for DBIx::Class::Relationship::ManyToMany::many_to_many():
Created attachment 125826 [details] [review] Bug 18984: Remove NORMARC support Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 125827 [details] [review] Bug 18984: Remove ES mapping for NORMARC perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1' Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
DBIx::Class::Relationship::ManyToMany::many_to_many(): *************************************************************************** The many-to-many relationship 'search_fields' is trying to create a utility method called add_to_search_fields. This will completely overwrite one such already existing method on class Koha::Schema::Result::SearchMarcMap. You almost certainly want to rename your method or the many-to-many relationship, as the functionality of the original method will not be accessible anymore. To disable this warning set to a true value the environment variable DBIC_OVERWRITE_HELPER_METHODS_OK
diff --git a/Koha/Schema/Result/SearchMarcMap.pm b/Koha/Schema/Result/SearchMarcMap.pm index 03bc94304c..6b700c65ff 100644 --- a/Koha/Schema/Result/SearchMarcMap.pm +++ b/Koha/Schema/Result/SearchMarcMap.pm @@ -40,7 +40,7 @@ what storage index this map is for =head2 marc_type data_type: 'enum' - extra: {list => ["marc21","unimarc","normarc"]} + extra: {list => ["marc21","unimarc"]} is_nullable: 0 what MARC type this map is for @@ -67,7 +67,7 @@ __PACKAGE__->add_columns( "marc_type", { data_type => "enum", - extra => { list => ["marc21", "unimarc", "normarc"] }, + extra => { list => ["marc21", "unimarc"] }, is_nullable => 0, }, "marc_field", You dont want this in your regular patch.
The signoff stands, but it needs attention still. Switch back to SO afterwards.
Dont forget to add a dbrev for search_marc_map
Created attachment 125828 [details] [review] Bug 18984: Remove NORMARC support
Created attachment 125829 [details] [review] Bug 18984: Remove ES mapping for NORMARC perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1'
Koha/Schema/Result/SearchMarcMap.pm changes removed from last patches.
How about removing: misc/translator/po/am-Ethi-marc-NORMARC.po misc/translator/po/ar-Arab-marc-NORMARC.po misc/translator/po/as-IN-marc-NORMARC.po misc/translator/po/az-AZ-marc-NORMARC.po misc/translator/po/be-BY-marc-NORMARC.po misc/translator/po/bg-Cyrl-marc-NORMARC.po misc/translator/po/bn-IN-marc-NORMARC.po misc/translator/po/ca-ES-marc-NORMARC.po misc/translator/po/cs-CZ-marc-NORMARC.po misc/translator/po/cy-GB-marc-NORMARC.po misc/translator/po/da-DK-marc-NORMARC.po misc/translator/po/de-CH-marc-NORMARC.po misc/translator/po/de-DE-marc-NORMARC.po misc/translator/po/dz-marc-NORMARC.po misc/translator/po/el-GR-marc-NORMARC.po misc/translator/po/en-GB-marc-NORMARC.po misc/translator/po/en-NZ-marc-NORMARC.po misc/translator/po/eo-marc-NORMARC.po misc/translator/po/es-ES-marc-NORMARC.po misc/translator/po/eu-marc-NORMARC.po misc/translator/po/fa-Arab-marc-NORMARC.po misc/translator/po/fi-FI-marc-NORMARC.po misc/translator/po/fo-FO-marc-NORMARC.po misc/translator/po/fr-CA-marc-NORMARC.po misc/translator/po/fr-FR-marc-NORMARC.po misc/translator/po/ga-marc-NORMARC.po misc/translator/po/gd-marc-NORMARC.po misc/translator/po/gl-marc-NORMARC.po misc/translator/po/he-Hebr-marc-NORMARC.po misc/translator/po/hi-marc-NORMARC.po misc/translator/po/hr-HR-marc-NORMARC.po misc/translator/po/hu-HU-marc-NORMARC.po misc/translator/po/hy-Armn-marc-NORMARC.po misc/translator/po/hyw-Armn-marc-NORMARC.po misc/translator/po/ia-marc-NORMARC.po misc/translator/po/id-ID-marc-NORMARC.po misc/translator/po/iq-CA-marc-NORMARC.po misc/translator/po/is-IS-marc-NORMARC.po misc/translator/po/it-IT-marc-NORMARC.po misc/translator/po/iu-CA-marc-NORMARC.po misc/translator/po/ja-Jpan-JP-marc-NORMARC.po misc/translator/po/ka-marc-NORMARC.po misc/translator/po/km-KH-marc-NORMARC.po misc/translator/po/kn-Knda-marc-NORMARC.po misc/translator/po/ko-Kore-KP-marc-NORMARC.po misc/translator/po/ku-Arab-marc-NORMARC.po misc/translator/po/lo-Laoo-marc-NORMARC.po misc/translator/po/lv-marc-NORMARC.po misc/translator/po/mi-NZ-marc-NORMARC.po misc/translator/po/ml-marc-NORMARC.po misc/translator/po/mon-marc-NORMARC.po misc/translator/po/mr-marc-NORMARC.po misc/translator/po/ms-MY-marc-NORMARC.po misc/translator/po/my-marc-NORMARC.po misc/translator/po/nb-NO-marc-NORMARC.po misc/translator/po/ne-NE-marc-NORMARC.po misc/translator/po/nl-BE-marc-NORMARC.po misc/translator/po/nl-NL-marc-NORMARC.po misc/translator/po/nn-NO-marc-NORMARC.po misc/translator/po/oc-marc-NORMARC.po misc/translator/po/pbr-marc-NORMARC.po misc/translator/po/pl-PL-marc-NORMARC.po misc/translator/po/prs-marc-NORMARC.po misc/translator/po/pt-BR-marc-NORMARC.po misc/translator/po/pt-PT-marc-NORMARC.po misc/translator/po/ro-RO-marc-NORMARC.po misc/translator/po/ru-RU-marc-NORMARC.po misc/translator/po/rw-RW-marc-NORMARC.po misc/translator/po/sd-PK-marc-NORMARC.po misc/translator/po/sk-SK-marc-NORMARC.po misc/translator/po/sl-SI-marc-NORMARC.po misc/translator/po/sq-AL-marc-NORMARC.po misc/translator/po/sr-Cyrl-marc-NORMARC.po misc/translator/po/sv-SE-marc-NORMARC.po misc/translator/po/sw-KE-marc-NORMARC.po misc/translator/po/ta-LK-marc-NORMARC.po misc/translator/po/ta-marc-NORMARC.po misc/translator/po/te-marc-NORMARC.po misc/translator/po/tet-marc-NORMARC.po misc/translator/po/th-TH-marc-NORMARC.po misc/translator/po/tl-PH-marc-NORMARC.po misc/translator/po/tr-TR-marc-NORMARC.po misc/translator/po/tvl-marc-NORMARC.po misc/translator/po/uk-UA-marc-NORMARC.po misc/translator/po/ur-Arab-marc-NORMARC.po misc/translator/po/vi-VN-marc-NORMARC.po misc/translator/po/zh-Hans-CN-marc-NORMARC.po misc/translator/po/zh-Hant-TW-marc-NORMARC.po root@master:/usr/share/koha# git grep -l normarc > zz root@master:/usr/share/koha# cat zz installer/data/mysql/updatedatabase.pl misc/release_notes/release_notes_3_12_0.txt misc/translator/po/am-Ethi-marc-NORMARC.po misc/translator/po/ar-Arab-marc-NORMARC.po misc/translator/po/as-IN-marc-NORMARC.po misc/translator/po/az-AZ-marc-NORMARC.po misc/translator/po/be-BY-marc-NORMARC.po misc/translator/po/bg-Cyrl-marc-NORMARC.po misc/translator/po/bn-IN-marc-NORMARC.po misc/translator/po/ca-ES-marc-NORMARC.po misc/translator/po/cs-CZ-marc-NORMARC.po misc/translator/po/cy-GB-marc-NORMARC.po misc/translator/po/da-DK-marc-NORMARC.po misc/translator/po/de-CH-marc-NORMARC.po misc/translator/po/de-DE-marc-NORMARC.po misc/translator/po/dz-marc-NORMARC.po misc/translator/po/el-GR-marc-NORMARC.po misc/translator/po/en-GB-marc-NORMARC.po misc/translator/po/en-NZ-marc-NORMARC.po misc/translator/po/eo-marc-NORMARC.po misc/translator/po/es-ES-marc-NORMARC.po misc/translator/po/eu-marc-NORMARC.po misc/translator/po/fa-Arab-marc-NORMARC.po misc/translator/po/fi-FI-marc-NORMARC.po misc/translator/po/fo-FO-marc-NORMARC.po misc/translator/po/fr-CA-marc-NORMARC.po misc/translator/po/fr-FR-marc-NORMARC.po misc/translator/po/ga-marc-NORMARC.po misc/translator/po/gd-marc-NORMARC.po misc/translator/po/gl-marc-NORMARC.po misc/translator/po/he-Hebr-marc-NORMARC.po misc/translator/po/hi-marc-NORMARC.po misc/translator/po/hr-HR-marc-NORMARC.po misc/translator/po/hu-HU-marc-NORMARC.po misc/translator/po/hy-Armn-marc-NORMARC.po misc/translator/po/hyw-Armn-marc-NORMARC.po misc/translator/po/ia-marc-NORMARC.po misc/translator/po/id-ID-marc-NORMARC.po misc/translator/po/iq-CA-marc-NORMARC.po misc/translator/po/is-IS-marc-NORMARC.po misc/translator/po/it-IT-marc-NORMARC.po misc/translator/po/iu-CA-marc-NORMARC.po misc/translator/po/ja-Jpan-JP-marc-NORMARC.po misc/translator/po/ka-marc-NORMARC.po misc/translator/po/km-KH-marc-NORMARC.po misc/translator/po/kn-Knda-marc-NORMARC.po misc/translator/po/ko-Kore-KP-marc-NORMARC.po misc/translator/po/ku-Arab-marc-NORMARC.po misc/translator/po/lo-Laoo-marc-NORMARC.po misc/translator/po/lv-marc-NORMARC.po misc/translator/po/mi-NZ-marc-NORMARC.po misc/translator/po/ml-marc-NORMARC.po misc/translator/po/mon-marc-NORMARC.po misc/translator/po/mr-marc-NORMARC.po misc/translator/po/ms-MY-marc-NORMARC.po misc/translator/po/my-marc-NORMARC.po misc/translator/po/nb-NO-marc-NORMARC.po misc/translator/po/ne-NE-marc-NORMARC.po misc/translator/po/nl-BE-marc-NORMARC.po misc/translator/po/nl-NL-marc-NORMARC.po misc/translator/po/nn-NO-marc-NORMARC.po misc/translator/po/oc-marc-NORMARC.po misc/translator/po/pbr-marc-NORMARC.po misc/translator/po/pl-PL-marc-NORMARC.po misc/translator/po/prs-marc-NORMARC.po misc/translator/po/pt-BR-marc-NORMARC.po misc/translator/po/pt-PT-marc-NORMARC.po misc/translator/po/ro-RO-marc-NORMARC.po misc/translator/po/ru-RU-marc-NORMARC.po misc/translator/po/rw-RW-marc-NORMARC.po misc/translator/po/sd-PK-marc-NORMARC.po misc/translator/po/sk-SK-marc-NORMARC.po misc/translator/po/sl-SI-marc-NORMARC.po misc/translator/po/sq-AL-marc-NORMARC.po misc/translator/po/sr-Cyrl-marc-NORMARC.po misc/translator/po/sv-SE-marc-NORMARC.po misc/translator/po/sw-KE-marc-NORMARC.po misc/translator/po/ta-LK-marc-NORMARC.po misc/translator/po/ta-marc-NORMARC.po misc/translator/po/te-marc-NORMARC.po misc/translator/po/tet-marc-NORMARC.po misc/translator/po/th-TH-marc-NORMARC.po misc/translator/po/tl-PH-marc-NORMARC.po misc/translator/po/tr-TR-marc-NORMARC.po misc/translator/po/tvl-marc-NORMARC.po misc/translator/po/uk-UA-marc-NORMARC.po misc/translator/po/ur-Arab-marc-NORMARC.po misc/translator/po/vi-VN-marc-NORMARC.po misc/translator/po/zh-Hans-CN-marc-NORMARC.po misc/translator/po/zh-Hant-TW-marc-NORMARC.po
hmm bit too much text
Created attachment 125830 [details] [review] Bug 18984: DB changes
(In reply to Marcel de Rooy from comment #25) > How about removing: Hum good question. That needs to be done with Bernardo at release time I'd say, otherwise he is going to re-introduce them. I don't know, to be honest.
(In reply to Jonathan Druart from comment #28) > (In reply to Marcel de Rooy from comment #25) > > How about removing: > > Hum good question. That needs to be done with Bernardo at release time I'd > say, otherwise he is going to re-introduce them. > > I don't know, to be honest. It would make sense to remove them now, but make sure we are not going to reintroduce them later.
Patch is too big See https://gitlab.com/joubu/Koha/-/commits/bug_18984 For "Bug 18984: Remove *-NORMARC.po files"
Created attachment 125850 [details] [review] Bug 18984: Remove NORMARC support Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 125851 [details] [review] Bug 18984: Remove ES mapping for NORMARC perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1' Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 125852 [details] [review] Bug 18984: DB changes Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 125876 [details] [review] Bug 18984: Remove NORMARC support Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 125877 [details] [review] Bug 18984: Remove ES mapping for NORMARC perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml grep -v 'mandatory: ~' admin/searchengine/elasticsearch/mappings.yaml|grep -v 'opac: 1'|grep -v 'staff_client: 1' Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 125878 [details] [review] Bug 18984: DB changes Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Pushed to master for 21.11, thanks to everybody involved!