Bug 20709 - Update German MARC frameworks to Updates 23-26 (Nov 2016, May and Apr 2018)
Summary: Update German MARC frameworks to Updates 23-26 (Nov 2016, May and Apr 2018)
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic data support (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Katrin Fischer
QA Contact: Testopia
URL:
Keywords:
Depends on: 19835
Blocks:
  Show dependency treegraph
 
Reported: 2018-05-03 22:12 UTC by Katrin Fischer
Modified: 2019-10-14 19:57 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 20709: Update GERMAN MARC frameworks to Updates 23+24+25 (Nov 2016, May and Dec 2017) (288.73 KB, patch)
2018-05-03 22:37 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 20709: (follow-up) Update GERMAN MARC frameworks to Update 26 (April 2018) (6.91 KB, patch)
2018-05-11 05:56 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 20709: Update GERMAN MARC frameworks to Updates 23+24+25 (Nov 2016, May and Dec 2017) (287.88 KB, patch)
2018-05-17 00:01 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 20709: (follow-up) Update GERMAN MARC frameworks to Update 26 (April 2018) (7.03 KB, patch)
2018-05-17 00:01 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 20709: Update GERMAN MARC frameworks to Updates 23+24+25 (Nov 2016, May and Dec 2017) (287.90 KB, patch)
2018-06-07 14:30 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 20709: (follow-up) Update GERMAN MARC frameworks to Update 26 (April 2018) (7.05 KB, patch)
2018-06-07 14:31 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Katrin Fischer 2018-05-03 22:12:20 UTC

    
Comment 1 Katrin Fischer 2018-05-03 22:37:05 UTC
Created attachment 75044 [details] [review]
Bug 20709: Update GERMAN MARC frameworks to Updates 23+24+25 (Nov 2016, May and Dec 2017)

Translation of bibliographic and authority frameworks can be done
using translate.koha-community.org.

Updated and translated files will then be provided for download.

See: http://lists.koha-community.org/pipermail/koha-translate/2018-April/002721.html

To test:
- Verify SQL is correct and frameworks load without problem
Comment 2 Mark Tompsett 2018-05-04 13:15:19 UTC
Comment on attachment 75044 [details] [review]
Bug 20709: Update GERMAN MARC frameworks to Updates 23+24+25 (Nov 2016, May and Dec 2017)

Review of attachment 75044 [details] [review]:
-----------------------------------------------------------------

::: installer/data/mysql/de-DE/marcflavour/marc21/optional/marc21_sample_fastadd_framework.sql
@@ +86,4 @@
>  UPDATE marc_subfield_structure SET hidden ='0' WHERE tagfield = '952' AND tagsubfield = 'r' AND frameworkcode = 'FA';
>  UPDATE marc_subfield_structure SET hidden ='0' WHERE tagfield = '952' AND tagsubfield = 's' AND frameworkcode = 'FA';
>  UPDATE marc_subfield_structure SET hidden ='0' WHERE tagfield = '952' AND tagsubfield = 'x' AND frameworkcode = 'FA';
> +

If a blank line is the only difference, perhaps it should not be part of the patch?

::: installer/data/mysql/de-DE/marcflavour/marc21/optional/marc21_simple_bib_frameworks.sql
@@ -520,5 @@
> -INSERT INTO marc_subfield_structure(tagfield, tagsubfield, liblibrarian, libopac, repeatable, mandatory, kohafield, tab, authorised_value, authtypecode, value_builder, isurl, hidden, frameworkcode, seealso, link, defaultvalue, maxlength)
> -SELECT tagfield, tagsubfield, liblibrarian, libopac, repeatable, mandatory, kohafield, tab, authorised_value, authtypecode, value_builder, isurl, hidden, 'ACQ', seealso, link, defaultvalue, maxlength
> -FROM marc_subfield_structure
> -WHERE frameworkcode=""
> -AND kohafield IN ("biblio.title", "biblio.author", "biblioitems.publishercode", "biblioitems.editionstatement", "biblio.copyrightdate", "biblioitems.isbn", "biblio.seriestitle" );

Why remove this?
Comment 3 Mark Tompsett 2018-05-04 13:17:43 UTC
Also, why no atomic update to correct current systems doing an upgrade?
Comment 4 Katrin Fischer 2018-05-04 15:20:46 UTC
Because we don't do atomic updates in this case, we don't want to mess with what people have set up for their use.

Note: these files have been auto-generated so I am just copying them into the right spot. If that changes a blank line, I am still ok with it :)
If you take a closer look you can see that the ACQ framework has been moved into another file.
Comment 5 Bernardo Gonzalez Kriegel 2018-05-04 23:53:09 UTC
Hi Katrin, Bug 27010 adds a few new subfields.
Translated files have been updated accordingly.

Maybe you could consider updating the patch (23+24+25+26)
Comment 6 Katrin Fischer 2018-05-11 05:56:15 UTC
Created attachment 75248 [details] [review]
Bug 20709: (follow-up) Update GERMAN MARC frameworks to Update 26 (April 2018)

A follow-up in line with bug 20710.

Again, the files used here have been generated automatically
from the po files on translate.koha-community.org.

To test:
- Make sure the de-DE installer can install all mandatory files cleanly.
Comment 7 Katrin Fischer 2018-05-11 05:57:56 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #5)
> Hi Katrin, Bug 27010 adds a few new subfields.
> Translated files have been updated accordingly.
> 
> Maybe you could consider updating the patch (23+24+25+26)

Done, thx Bernardo!
Comment 8 Bernardo Gonzalez Kriegel 2018-05-17 00:01:27 UTC
Created attachment 75386 [details] [review]
Bug 20709: Update GERMAN MARC frameworks to Updates 23+24+25 (Nov 2016, May and Dec 2017)

Translation of bibliographic and authority frameworks can be done
using translate.koha-community.org.

Updated and translated files will then be provided for download.

See: http://lists.koha-community.org/pipermail/koha-translate/2018-April/002721.html

To test:
- Verify SQL is correct and frameworks load without problem

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 9 Bernardo Gonzalez Kriegel 2018-05-17 00:01:31 UTC
Created attachment 75387 [details] [review]
Bug 20709: (follow-up) Update GERMAN MARC frameworks to Update 26 (April 2018)

A follow-up in line with bug 20710.

Again, the files used here have been generated automatically
from the po files on translate.koha-community.org.

To test:
- Make sure the de-DE installer can install all mandatory files cleanly.

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Tested using de-DE installer, all files load Ok
No errors
Comment 10 Bernardo Gonzalez Kriegel 2018-05-17 00:07:29 UTC
I did a little change on marc21 bib framework file, line 4596

from

INSERT IGNORE INTO biblio_framework VALUES( 'ACQ', 'Acquisition framework' );

to

INSERT IGNORE INTO biblio_framework VALUES
                ( 'ACQ', 'Erwerbung' );

Original EN file need a little update to allow translation.
The one used to translate is fixed now.
Comment 11 Katrin Fischer 2018-05-17 00:12:34 UTC
Thx Bernardo!
Comment 12 Jonathan Druart 2018-06-05 16:46:38 UTC
About the changes to installer/data/mysql/de-DE/marcflavour/marc21/mandatory/authorities_normal_marc21.sql, should not we added them on bug 19835?

We are going to have these files differ, again.
Comment 13 Bernardo Gonzalez Kriegel 2018-06-05 18:40:49 UTC
(In reply to Jonathan Druart from comment #12)
> About the changes to
> installer/data/mysql/de-DE/marcflavour/marc21/mandatory/
> authorities_normal_marc21.sql, should not we added them on bug 19835?
> 
> We are going to have these files differ, again.

Hi Jonathan

On master+19835+20710
en/marcflavour/marc21/mandatory/authorities_normal_marc21.sql will have 2385 lines

On master+20709
de-DE/marcflavour/marc21/mandatory/authorities_normal_marc21.sql will have 2385 lines too

They will differ in that one is the translation of the other, but they are basically the same file.

The only thing is that translations are made using master+19835+20710,
so they will differ (a little) until 19835 & 20710 are pushed.
Comment 14 Jonathan Druart 2018-06-05 18:46:57 UTC
What about other languages?
Comment 15 Katrin Fischer 2018-06-05 19:01:40 UTC
I think some have translated manually, so might not be so easy to switch them to the generated files?
Comment 16 Bernardo Gonzalez Kriegel 2018-06-05 19:31:15 UTC
(In reply to Jonathan Druart from comment #14)
> What about other languages?

Not all languages that have translated sql files (marc21 project on translation server) have custom install dirs, so is up to each language maintainer what to do with those sql files.

It would be better if we could 'translate' sql files using the normal way, but it's no easy task. I tried a while ago using XML files...
Comment 17 Jonathan Druart 2018-06-06 14:37:26 UTC
I am not sure you got my point.

This patch adds  '034', '1', 'Real World Object URI

After bug 19835 and bug 20709:
% git grep -l "'034', '1'"
installer/data/mysql/de-DE/marcflavour/marc21/mandatory/authorities_normal_marc21.sql
installer/data/mysql/de-DE/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql
installer/data/mysql/en/marcflavour/marc21/mandatory/authorities_normal_marc21.sql
installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql

We usually apply the changes done on 'en' to all other languages (not translated), then language maintainers provide patches for the translation.
Comment 18 Katrin Fischer 2018-06-06 14:47:52 UTC
(In reply to Jonathan Druart from comment #17)
> I am not sure you got my point.
> 
> This patch adds  '034', '1', 'Real World Object URI
> 
> After bug 19835 and bug 20709:
> % git grep -l "'034', '1'"
> installer/data/mysql/de-DE/marcflavour/marc21/mandatory/
> authorities_normal_marc21.sql
> installer/data/mysql/de-DE/marcflavour/marc21/mandatory/
> marc21_framework_DEFAULT.sql
> installer/data/mysql/en/marcflavour/marc21/mandatory/
> authorities_normal_marc21.sql
> installer/data/mysql/en/marcflavour/marc21/mandatory/
> marc21_framework_DEFAULT.sql
> 
> We usually apply the changes done on 'en' to all other languages (not
> translated), then language maintainers provide patches for the translation.

That's not how we do it anymore for the frameworks. Bernardo provides po files for translating, which is much more convenient and less error prone then translating a gigantig sql file. So we generate the translated files from the files in master. 

See here:
http://translate.koha-community.org/projects/marc21/

Bernardo automatically generates the files regularly and provides them here:
http://translate.koha-community.org/files/

For the frameworks it's not super bad if a field is missing, I think the new workflow is worth it.
Comment 19 Jonathan Druart 2018-06-06 15:00:37 UTC
Fields will be missing and files will differ if language maintainers do not provide files for their language. That sounds weird. It does not seems difficult to provide the changes for all languages on bug 19835.

Where was the announce about this change? Where is describe the workflow? On the koha translate list?
Comment 20 Katrin Fischer 2018-06-06 15:14:33 UTC
See first comment:
http://lists.koha-community.org/pipermail/koha-translate/2018-April/002721.html

But it's been done for a while, Bernardo might know when we started. I've update the files a few times already. I think they are already out of sync. Some files might also have different data because they were translted manually and renamed/restructured/changed.

Bugzilla search:
ALL german update frameworks
bug 16601
bug 16472
bug 10964

Italian and I think ru-RU have also been updated a while ago.
Comment 21 Bernardo Gonzalez Kriegel 2018-06-06 16:24:32 UTC
(In reply to Jonathan Druart from comment #17)
> I am not sure you got my point.

May be not :)

> This patch adds  '034', '1', 'Real World Object URI
> ...
> We usually apply the changes done on 'en' to all other languages (not
> translated), then language maintainers provide patches for the translation.

Not for frameworks, or more exactly not be me.
I only check for updates/changes on EN 'default' framework, bib and auth.

Bug 10963 and Bug 10488 guarantees that you only need to add/update things once per file, before them changes were needed on many places in the same file which I did not do either.

But I never tried to update frameworks for other languages.

(In reply to Jonathan Druart from comment #19)
> Fields will be missing and files will differ if language maintainers do not
> provide files for their language. That sounds weird. It does not seems
> difficult to provide the changes for all languages on bug 19835.

Yes, there will be differences. But it's not easy for me to do that job.
That was the reason to introduce 'marc21' project on translation server as a way to enable the translation of 5 sql files.

The po file for that is built in a very non standard way, and the translations generated by an ugly script. But it's doing it's job since mid 2013.

> Where was the announce about this change? Where is describe the workflow? On
> the koha translate list?

It was made in the translation list, 5 years ago. Languages have been added upon request. There are even languages that do not have SQL installers.

But you are quite right, there is a lack of documentation on the workflow.

Finally, I promise never to update the sql translations again until the corresponding changes are in master :)

PS: I need to resurrect Bug 13897 or find a way to teach SQL to translate script.
Comment 22 Jonathan Druart 2018-06-07 14:30:57 UTC
Created attachment 75884 [details] [review]
Bug 20709: Update GERMAN MARC frameworks to Updates 23+24+25 (Nov 2016, May and Dec 2017)

Translation of bibliographic and authority frameworks can be done
using translate.koha-community.org.

Updated and translated files will then be provided for download.

See: http://lists.koha-community.org/pipermail/koha-translate/2018-April/002721.html

To test:
- Verify SQL is correct and frameworks load without problem

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 23 Jonathan Druart 2018-06-07 14:31:02 UTC
Created attachment 75885 [details] [review]
Bug 20709: (follow-up) Update GERMAN MARC frameworks to Update 26 (April 2018)

A follow-up in line with bug 20710.

Again, the files used here have been generated automatically
from the po files on translate.koha-community.org.

To test:
- Make sure the de-DE installer can install all mandatory files cleanly.

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Tested using de-DE installer, all files load Ok
No errors
Comment 24 Nick Clemens 2018-07-02 13:17:32 UTC
Awesome work all!

Pushed to master for 18.11
Comment 25 Martin Renvoize 2018-07-03 10:04:37 UTC
Pushed to 18.05.x for 18.05.02, I felt that although this is marked as an enhancement, many librarians would see it as a bug not being in sync with MARC21 guidelines.
Comment 26 Fridolin Somers 2018-07-04 16:01:31 UTC
(In reply to Martin Renvoize from comment #25)
> Pushed to 18.05.x for 18.05.02, I felt that although this is marked as an
> enhancement, many librarians would see it as a bug not being in sync with
> MARC21 guidelines.

Same for 17.11.x, will be in 17.11.08