1. CSV 1. Export default MARC framework in CSV 2. Create a new framework 3. import the CSV 4. See MARC structure of the new framework 5. View subfield of tag 240 6. Compare with default framework 7. Subfields are missing 2. ODS 1. Export default MARC framework in .ods 2. Open the .ods with LibreOffice 3. Save it 4. Create a new framework 5. import the .ods 6. See MARC structure of the new framework 7. View subfield of tag 240 8. Compare with default framework 9. See Also: 3, and Link:3 have appeared (the code of my new framework code is 3) For ODS, it seems a cell is offset and the framework code ends up in other columns. The raw ODS from Koha and the one saved by LibreOffice don't seem to differ on the 240 subfields. Both issues happen on master and on 19.11 (with bug 27569 applied) Not sure about severity.
[2021/03/17 07:47:02] [WARN] C4::ImportExportFramework::_processRow_DB(): DBI Exception: DBD::mysql::st execute failed: Incorrect integer value: '' for column 'isurl' at row 1 [for Statement "INSERT INTO marc_subfield_structure (tagfield,tagsubfield,liblibrarian,libopac,repeatable,mandatory,important,kohafield,tab,authorised_value,authtypecode,value_builder,isurl,hidden,frameworkcode,seealso,link,defaultvalue,maxlength,display_order) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) ON DUPLICATE KEY UPDATE tagfield=?,tagsubfield=?,liblibrarian=?,libopac=?,repeatable=?,mandatory=?,important=?,kohafield=?,tab=?,authorised_value=?,authtypecode=?,value_builder=?,isurl=?,hidden=?,frameworkcode=?,seealso=?,link=?,defaultvalue=?,maxlength=?,display_order=?" with ParamValues: 0='999', 1='c', 2='Koha biblionumber', 3='Koha biblionumber', 4='0', 5='0', 6='0', 7='biblio.biblionumber', 8='-1', 9='', 10='', 11='', 12='', 13='-5', 14="X", 15='', 16='', 17='', 18='9999', 19='0', 20='999', 21='c', 22='Koha biblionumber', 23='Koha biblionumber', 24='0', 25='0', 26='0', 27='biblio.biblionumber', 28='-1', 29='', 30='', 31='', 32='', 33='-5', 34="X", 35='', 36='', 37='', 38='9999', 39='0'] at /kohadevbox/koha/C4/ImportExportFramework.pm line 986
Only happens when SQL strict mode is on.
Created attachment 118357 [details] [review] Bug 27968: Fix framework import if isurl has an incorrect value If isurl is set to undef in the DB, it will be exported as an empty string, which is an invalid value for isurl (int) Incorrect integer value: '' for column 'isurl' Test plan: Export framework structure in CSV and ODS, then reimport it and check that all the subfields are important correctly
MariaDB [koha_kohadev]> select tagsubfield, isurl from marc_subfield_structure where tagfield=240 and frameworkcode="" limit 6; +-------------+-------+ | tagsubfield | isurl | +-------------+-------+ | 0 | NULL | | 1 | NULL | | 2 | NULL | | 6 | 0 | | 8 | 0 | | 9 | 0 | +-------------+-------+ 6 rows in set (0.01 sec)
Likely not related but, is that expected that 1. Create new framework 2. Marc structure 3. Internal server error CGI::Compile::ROOT::kohadevbox_koha_admin_marctagstructure_2epl::marctagstructure_2epl(): DBI Exception: DBD::mysql::st execute failed: 'koha_kohadev.biblio_framework.frameworktext' isn't in GROUP BY [for Statement "select count(*),marc_tag_structure.frameworkcode,frameworktext from marc_tag_structure,biblio_framework where biblio_framework.frameworkcode=marc_tag_structure.frameworkcode group by marc_tag_structure.frameworkcode"] at /usr/share/perl5/CGI/Emulate/PSGI.pm line 30 at /usr/share/perl5/DBIx/Class/Exception.pm line 77
Found issue, test scenario 1 passes but not if I add a save with LibreOffice 1. Export default MARC framework in CSV 2. Open the .csv with LibreOffice and save it 3. Create a new framework 4. import the CSV 5. See MARC structure of the new framework 6. Tags are missing the trailing zeros (10 instead of 010) 7. View subfields of tag 10/010 8. Compare with default framework 9. Subfields are missing note, why does on the default framework, these two URLs don't yield the same result? /koha/admin/marc_subfields_structure.pl?tagfield=010&frameworkcode= /koha/admin/marc_subfields_structure.pl?tagfield=10&frameworkcode=3 It would also make sense that 10 doesn't yield anything. 10. Same with 031/31 default framework still has different output for 31 and 031 ODS issue when saved by LibreOffice is still there 1. Export default MARC framework in .ods 2. Open the .ods with LibreOffice 3. Save it 4. Create a new framework 5. import the .ods 6. See MARC structure of the new framework 7. View subfield of tag 010 8. Compare with default framework 9. See Also: 4, and Link:4 have appeared (the code of my new framework code is 4) Testing notes: Tags with a lot or diverse subfields: 010 100 240 995 031 110 600 700
(In reply to Victor Grousset/tuxayo from comment #6) > Found issue, test scenario 1 passes but not if I add a save with LibreOffice > > 1. Export default MARC framework in CSV > 2. Open the .csv with LibreOffice and save it > 3. Create a new framework > 4. import the CSV > 5. See MARC structure of the new framework > 6. Tags are missing the trailing zeros (10 instead of 010) I don't recreate that. Certainly a setting in the libreoffice config. > 7. View subfields of tag 10/010 > 8. Compare with default framework > 9. Subfields are missing > note, why does on the default framework, these two URLs don't yield the > same result? > /koha/admin/marc_subfields_structure.pl?tagfield=010&frameworkcode= > /koha/admin/marc_subfields_structure.pl?tagfield=10&frameworkcode=3 > It would also make sense that 10 doesn't yield anything. > 10. Same with 031/31 > default framework still has different output for 31 and 031 10 picked the 100's subfields, that's definitely a bug, should be reported separately. > ODS issue when saved by LibreOffice is still there > > 1. Export default MARC framework in .ods > 2. Open the .ods with LibreOffice > 3. Save it > 4. Create a new framework > 5. import the .ods > 6. See MARC structure of the new framework > 7. View subfield of tag 010 > 8. Compare with default framework > 9. See Also: 4, and Link:4 have appeared (the code of my new framework code > is 4) Does it work if you don't open/save it?
(In reply to Jonathan Druart from comment #7) > (In reply to Victor Grousset/tuxayo from comment #6) > > 1. Export default MARC framework in CSV > > 2. Open the .csv with LibreOffice and save it > > 3. Create a new framework > > 4. import the CSV > > 5. See MARC structure of the new framework > > 6. Tags are missing the trailing zeros (10 instead of 010) > > I don't recreate that. > Certainly a setting in the libreoffice config. Next is attached the CSV saved by LibreOffice, 010 => 10 happens there. When opening the CSV, one of the options is "Format quoted fields as text.", it's off by default. Hence 010 => 10. Should the Koha import handle the spreadsheet software removing the leading zeros? Since one of the common spreadsheet software does this by default, a fair share of usage of this feature will be with this data. > > 7. View subfields of tag 10/010 > > 8. Compare with default framework > > 9. Subfields are missing > > note, why does on the default framework, these two URLs don't yield the > > same result? > > /koha/admin/marc_subfields_structure.pl?tagfield=010&frameworkcode= > > /koha/admin/marc_subfields_structure.pl?tagfield=10&frameworkcode=3 > > It would also make sense that 10 doesn't yield anything. > > 10. Same with 031/31 > > default framework still has different output for 31 and 031 > > 10 picked the 100's subfields, that's definitely a bug, should be reported > separately. Ok, see bug 27989. About the subfields missing it's related to the above issue. Because when reexporting they are in the CSV even if I can't find a way to see them in the interface. > > ODS issue when saved by LibreOffice is still there > > > > 1. Export default MARC framework in .ods > > 2. Open the .ods with LibreOffice > > 3. Save it > > 4. Create a new framework > > 5. import the .ods > > 6. See MARC structure of the new framework > > 7. View subfield of tag 010 > > 8. Compare with default framework > > 9. See Also: 4, and Link:4 have appeared (the code of my new framework code > > is 4) > > Does it work if you don't open/save it? Yes, if the ODS export is directly imported, no issues
Created attachment 118470 [details] Default-framework-export-resaved-libreoffice
While there are still issues, this patch solves part of the problem and makes a few workflows work again. Shall the ticket be split for the remaining failing cases so the current patch can be merged?
Yes, I think we should go ahead with this patch and open a separate bug report for the other bug.
Created attachment 120036 [details] [review] Bug 27968: Fix framework import if isurl has an incorrect value If isurl is set to undef in the DB, it will be exported as an empty string, which is an invalid value for isurl (int) Incorrect integer value: '' for column 'isurl' Test plan: Export framework structure in CSV and ODS, then reimport it and check that all the subfields are important correctly Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
It works. Tested successfully CSV and ODS without going through LibreOffice.
Looking here
Created attachment 120038 [details] [review] Bug 27968: Fix framework import if isurl has an incorrect value If isurl is set to undef in the DB, it will be exported as an empty string, which is an invalid value for isurl (int) Incorrect integer value: '' for column 'isurl' Test plan: Export framework structure in CSV and ODS, then reimport it and check that all the subfields are important correctly Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Passed QA Note that I would not call this a major bug. Devaluates severity ;)
Pushed to master for 21.05, thanks to everybody involved!
Pushed to 20.11.x for 20.11.06
What are our chances of having this pushed to 20.05?
Pushed to 20.05.x for 20.05.12
Backported: Pushed to 19.11.x branch for 19.11.18