|
Description
François Pichenot
2018-03-16 10:37:39 UTC
Patch is too big for Nginx (413 Request Entity Too Large), so I pushed it to a branch on git.biblibre.com: http://git.biblibre.com/biblibre/kohac/commits/bug/20434 http://git.biblibre.com/biblibre/kohac/commit/457524cef965faf96a9516c17e9e0a8bbdec19aa I copy the commit message here: Bug 20434: Add a script to apply UNIMARC update from IFLA Test plan: 1. Take a look at misc/migration_tools/unimarc_ifla.yml. It contains all data that will be inserted into Koha. Its content is based on the previous patches. The language of labels are in french but they are translatable (see below) 2. Run the script misc/migration_tools/unimarc_ifla.pl and verify that it effectively added the new fields, subfields, authorised values and authority types. 3. Run the script again and see that it doesn't update existing fields 4. Run with --force and verify that it update existing fields (you can modify unimarc_ifla.yml to see changes) 5. Run with --force --translations-file misc/migration_tools/unimarc_ifla.en.yml and verify that the labels are now in english (not all labels are translated yet) 6. Run with --dump-translations and redirect the output to a file. It should contain a YAML document containing all translatable strings (useful to start translating for another language) If you don't want to add a new remote to your git repository you can apply the patch with the following command: wget -q -O- http://git.biblibre.com/biblibre/kohac/commit/457524cef965faf96a9516c17e9e0a8bbdec19aa.patch | git am -3 - Hi Julian, this is an interesting approach. Are you aware that we can currently already translate the SQL files for the frameworks using pootle? For an example: http://translate.koha-community.org/de/marc21/ Bernardo provides the auto-generated SQL files for creating patches for the web installer. Could there be a way to bring this together? Use po files to create the .yml file? Created attachment 93415 [details] [review] Bug 20434: Update UNIMARC framework - defaults Created attachment 93416 [details] [review] Bug 20434: Update UNIMARC framework - authorised values Created attachment 93417 [details] [review] Bug 20434: Update UNIMARC framework - biblio Created attachment 93418 [details] [review] Bug 20434: Update UNIMARC framework - authtypes Created attachment 93419 [details] [review] Bug 20434: Update UNIMARC framework - auth (default) Created attachment 93420 [details] [review] Bug 20434: Update UNIMARC framework - auth (CLASS) Created attachment 93421 [details] [review] Bug 20434: Update UNIMARC framework - auth (CO) Created attachment 93422 [details] [review] Bug 20434: Update UNIMARC framework - auth (EXP) Created attachment 93423 [details] [review] Bug 20434: Update UNIMARC framework - auth (FAM) Created attachment 93424 [details] [review] Bug 20434: Update UNIMARC framework - auth (GENRE/FORM) Created attachment 93425 [details] [review] Bug 20434: Update UNIMARC framework - auth (NP) Created attachment 93426 [details] [review] Bug 20434: Update UNIMARC framework - auth (NTEXP) Created attachment 93427 [details] [review] Bug 20434: Update UNIMARC framework - auth (NTWORK) Created attachment 93428 [details] [review] Bug 20434: Update UNIMARC framework - auth (PA) Created attachment 93429 [details] [review] Bug 20434: Update UNIMARC framework - auth (PERS) Created attachment 93430 [details] [review] Bug 20434: Update UNIMARC framework - auth (PUB) Created attachment 93431 [details] [review] Bug 20434: Update UNIMARC framework - auth (SAUTTIT) Created attachment 93432 [details] [review] Bug 20434: Update UNIMARC framework - auth (SNC) Created attachment 93433 [details] [review] Bug 20434: Update UNIMARC framework - auth (SNG) Created attachment 93434 [details] [review] Bug 20434: Update UNIMARC framework - auth (TM) Created attachment 93435 [details] [review] Bug 20434: Update UNIMARC framework - auth (TU) Created attachment 93436 [details] [review] Bug 20434: Update UNIMARC framework - auth (WORK) Created attachment 93437 [details] [review] Bug 20434: Update UNIMARC framework - translations Created attachment 93438 [details] [review] Bug 20434: Update UNIMARC framework - script Test plan: 1. Take a look at files in misc/migration_tools/ifla/data. It contains all data that will be inserted into Koha. Its content is based on the previous patches. 2. Run the script misc/migration_tools/ifla/update.pl and verify that it effectively added the new fields, subfields, authorised values and authority types. 3. Run the script again and see that it doesn't update existing fields 4. Run with --force and verify that it update existing fields (you can modify unimarc_ifla.yml to see changes) 5. Run with --force --po-file misc/migration_tools/ifla/language/fr.po and verify that the labels are now in french There is a POT file in misc/migration_tools/ifla/language/template.pot, use it to create PO files for other languages. I split the big patch into smaller patches so that they can be attached on bugzilla. Also, I converted the translations file from YAML to PO. Strings are in english by default, french translations are available as a PO file. So now, if needed, the POT file can be integrated into Pootle. Created attachment 93449 [details] [review] Bug 20434: Add UNIMARC field 214 and its subfields I have an error while executing misc/migration_tools/ifla/update.pl DBD::mysql::st execute failed: Cannot add or update a child row: a foreign key c onstraint fails (`koha_kohadev`.`auth_tag_structure`, CONSTRAINT `auth_tag_struc ture_ibfk_1` FOREIGN KEY (`authtypecode`) REFERENCES `auth_types` (`authtypecode `) ON DELETE CASCADE ON UPDATE CASCADE) [for Statement "INSERT INTO `auth_tag_st ructure` ( `authorised_value`, `authtypecode`, `liblibrarian`, `libopac`, `manda tory`, `repeatable`, `tagfield`) VALUES ( ?, ?, ?, ?, ?, ?, ? )" with ParamValue s: 0=undef, 1="CO", 2="PPN", 3="", 4="0", 5="0", 6="009"] at /usr/share/perl5/DB Ix/Class/Storage/DBI.pm line 1832. DBIx::Class::Storage::DBI::_dbh_execute(): Cannot add or update a child row: a f oreign key constraint fails (`koha_kohadev`.`auth_tag_structure`, CONSTRAINT `au th_tag_structure_ibfk_1` FOREIGN KEY (`authtypecode`) REFERENCES `auth_types` (` authtypecode`) ON DELETE CASCADE ON UPDATE CASCADE) at misc/migration_tools/ifla /update.pl line 285 Created attachment 93614 [details] [review] Bug 20434: Add missing authority types (In reply to Koha Team University Lyon 3 from comment #30) > I have an error while executing misc/migration_tools/ifla/update.pl The last patch should fix this. Please test again Created attachment 93658 [details] [review] Bug 20434: Add missing fields 018 - [Reserved for other International Standard Numbers] 033 - Other System Persistent Record Identifier 183 - Coded Data Field: Type of Carrier 203 - Content Form and Media Type 231 - Digital File Characteristics 283 - Carrier Type 338 - Funding Information Note Created attachment 93683 [details] [review] Bug 20434: Update UNIMARC framework - defaults Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93684 [details] [review] Bug 20434: Update UNIMARC framework - authorised values Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93685 [details] [review] Bug 20434: Update UNIMARC framework - biblio Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93686 [details] [review] Bug 20434: Update UNIMARC framework - authtypes Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93687 [details] [review] Bug 20434: Update UNIMARC framework - auth (default) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93688 [details] [review] Bug 20434: Update UNIMARC framework - auth (CLASS) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93689 [details] [review] Bug 20434: Update UNIMARC framework - auth (CO) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93690 [details] [review] Bug 20434: Update UNIMARC framework - auth (EXP) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93691 [details] [review] Bug 20434: Update UNIMARC framework - auth (FAM) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93692 [details] [review] Bug 20434: Update UNIMARC framework - auth (GENRE/FORM) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93693 [details] [review] Bug 20434: Update UNIMARC framework - auth (NP) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93694 [details] [review] Bug 20434: Update UNIMARC framework - auth (NTEXP) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93695 [details] [review] Bug 20434: Update UNIMARC framework - auth (NTWORK) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93696 [details] [review] Bug 20434: Update UNIMARC framework - auth (PA) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93698 [details] [review] Bug 20434: Update UNIMARC framework - auth (PERS) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93701 [details] [review] Bug 20434: Update UNIMARC framework - auth (PUB) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93702 [details] [review] Bug 20434: Update UNIMARC framework - auth (SAUTTIT) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93703 [details] [review] Bug 20434: Update UNIMARC framework - auth (SNC) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93704 [details] [review] Bug 20434: Update UNIMARC framework - auth (SNG) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93705 [details] [review] Bug 20434: Update UNIMARC framework - auth (TM) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93706 [details] [review] Bug 20434: Update UNIMARC framework - auth (TU) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93707 [details] [review] Bug 20434: Update UNIMARC framework - auth (WORK) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93708 [details] [review] Bug 20434: Update UNIMARC framework - translations Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93709 [details] [review] Bug 20434: Update UNIMARC framework - script Test plan: 1. Take a look at files in misc/migration_tools/ifla/data. It contains all data that will be inserted into Koha. Its content is based on the previous patches. 2. Run the script misc/migration_tools/ifla/update.pl and verify that it effectively added the new fields, subfields, authorised values and authority types. 3. Run the script again and see that it doesn't update existing fields 4. Run with --force and verify that it update existing fields (you can modify unimarc_ifla.yml to see changes) 5. Run with --force --po-file misc/migration_tools/ifla/language/fr.po and verify that the labels are now in french There is a POT file in misc/migration_tools/ifla/language/template.pot, use it to create PO files for other languages. Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93710 [details] [review] Bug 20434: Add UNIMARC field 214 and its subfields Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93711 [details] [review] Bug 20434: Add missing authority types Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Created attachment 93712 [details] [review] Bug 20434: Add missing fields 018 - [Reserved for other International Standard Numbers] 033 - Other System Persistent Record Identifier 183 - Coded Data Field: Type of Carrier 203 - Content Form and Media Type 231 - Digital File Characteristics 283 - Carrier Type 338 - Funding Information Note Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> I have followed the test plan and it works as expected. 1.I first launched the update.pl : it adds the new unimarc fields and subfields. 2. I modified one of the subfield added by the script in the default framework in Koha and launched the script again -> We can see in console that the field is not updated because it already exists 3. I launched the script with --force option : the field that I have changed is now updated by the script 4. I have launched the script with --force and -po options : the labels are now in french Sonia Update script fails on a SQL foreign key constraint if there is no default framework. So the question is: Should we consider a DB without a default auth type as invalidd or should the script handle that case ? I noticed that it's not possible to remove the default type via UI. Created attachment 94040 [details] [review] Bug 20434: Add default authority type Created attachment 94045 [details] [review] Bug 20434: Update UNIMARC framework - defaults Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94046 [details] [review] Bug 20434: Update UNIMARC framework - authorised values Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94047 [details] [review] Bug 20434: Update UNIMARC framework - biblio Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94048 [details] [review] Bug 20434: Update UNIMARC framework - authtypes Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94049 [details] [review] Bug 20434: Update UNIMARC framework - auth (default) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94050 [details] [review] Bug 20434: Update UNIMARC framework - auth (CLASS) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94051 [details] [review] Bug 20434: Update UNIMARC framework - auth (CO) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94052 [details] [review] Bug 20434: Update UNIMARC framework - auth (EXP) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94053 [details] [review] Bug 20434: Update UNIMARC framework - auth (FAM) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94054 [details] [review] Bug 20434: Update UNIMARC framework - auth (GENRE/FORM) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94055 [details] [review] Bug 20434: Update UNIMARC framework - auth (NP) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94056 [details] [review] Bug 20434: Update UNIMARC framework - auth (NTEXP) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94057 [details] [review] Bug 20434: Update UNIMARC framework - auth (NTWORK) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94058 [details] [review] Bug 20434: Update UNIMARC framework - auth (PA) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94059 [details] [review] Bug 20434: Update UNIMARC framework - auth (PERS) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94060 [details] [review] Bug 20434: Update UNIMARC framework - auth (PUB) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94061 [details] [review] Bug 20434: Update UNIMARC framework - auth (SAUTTIT) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94062 [details] [review] Bug 20434: Update UNIMARC framework - auth (SNC) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94063 [details] [review] Bug 20434: Update UNIMARC framework - auth (SNG) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94064 [details] [review] Bug 20434: Update UNIMARC framework - auth (TM) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94065 [details] [review] Bug 20434: Update UNIMARC framework - auth (TU) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94066 [details] [review] Bug 20434: Update UNIMARC framework - auth (WORK) Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94067 [details] [review] Bug 20434: Update UNIMARC framework - translations Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94068 [details] [review] Bug 20434: Update UNIMARC framework - script Test plan: 1. Take a look at files in misc/migration_tools/ifla/data. It contains all data that will be inserted into Koha. Its content is based on the previous patches. 2. Run the script misc/migration_tools/ifla/update.pl and verify that it effectively added the new fields, subfields, authorised values and authority types. 3. Run the script again and see that it doesn't update existing fields 4. Run with --force and verify that it update existing fields (you can modify unimarc_ifla.yml to see changes) 5. Run with --force --po-file misc/migration_tools/ifla/language/fr.po and verify that the labels are now in french There is a POT file in misc/migration_tools/ifla/language/template.pot, use it to create PO files for other languages. Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94069 [details] [review] Bug 20434: Add UNIMARC field 214 and its subfields Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94070 [details] [review] Bug 20434: Add missing authority types Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94071 [details] [review] Bug 20434: Add missing fields 018 - [Reserved for other International Standard Numbers] 033 - Other System Persistent Record Identifier 183 - Coded Data Field: Type of Carrier 203 - Content Form and Media Type 231 - Digital File Characteristics 283 - Carrier Type 338 - Funding Information Note Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Created attachment 94072 [details] [review] Bug 20434: Add default authority type Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> Nice work! Pushed to master for 19.11.00 |