Bug 6727 - can't import frameworks
Summary: can't import frameworks
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: 3.6
Hardware: All All
: PATCH-Sent (DO NOT USE) critical (vote)
Assignee: Salva Zaragoza Rubio
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-08-16 12:51 UTC by Nicole C. Engard
Modified: 2019-06-27 09:24 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:


Attachments
framework exported (12.96 KB, text/csv)
2011-08-16 12:52 UTC, Nicole C. Engard
Details
my framework (135.66 KB, image/png)
2011-08-17 10:53 UTC, Nicole C. Engard
Details
long field (42.48 KB, image/png)
2011-08-17 10:54 UTC, Nicole C. Engard
Details
CVS with LF ok (12.95 KB, application/vnd.ms-excel)
2011-08-17 11:38 UTC, Salva Zaragoza Rubio
Details
Proposed patch with changes on exporting and importing csv to deal with spurious CR/LF (2.11 KB, patch)
2011-08-19 10:43 UTC, Salva Zaragoza Rubio
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 6727 - Change on exporting and importing csv to deal with spurious CR/LF (2.15 KB, patch)
2011-08-25 00:02 UTC, Nicole C. Engard
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nicole C. Engard 2011-08-16 12:51:56 UTC
This needs to be tested by someone else, but I exported a framework from my system and imported it in to another system and it did not import the entire framework, only one field.  Attached is the exported framework.
Comment 1 Nicole C. Engard 2011-08-16 12:52:15 UTC
Created attachment 5030 [details]
framework exported
Comment 2 Liz Rea 2011-08-16 20:05:43 UTC
I was unable to replicate this issue on current master.

What I did:

Exported a framework from my dev server
looked at the CSV - looks ok
Imported the same CSV into the ByWater Demo server 

Framework looked ok, seemed to work ok.
Comment 3 Nicole C. Engard 2011-08-16 23:29:48 UTC
Liz,

Did you try it with my file? Which I exported and then tried to upload on someone else's server and it came up with only one field. The CSV did look fine.

I'm saying you should try exporting from one server and importing to another - which is what I did when this error occurred. 

Nicole
Comment 4 Nicole C. Engard 2011-08-16 23:34:01 UTC
Okay, I tested on another server, again other than the one I exported from and the same thing happened.  I've now exported one file from one server and imported it in to two different other servers and in both cases the imported file only had one field in it instead of all the fields in the CSV file -- see the attached CSV maybe it's a character issue.
Comment 5 Salva Zaragoza Rubio 2011-08-17 07:22:17 UTC
Hi Nicole,

I've seen the csv file and inside the content of some fields exist line feeds characters that breaks what is supposed to be one line into multiples lines.

Two possible causes:
1. The field contents have line feed characters so if you export the framework the csv will be malformed. This will have to be corrected, so the LN characters inside data fields are taking into account.
Can you check the framework and see if it has some LN in its content?

2. The export process fails for this framework and adds LN in the middle of a field.
Send me the framework in another format to dump into my database and test the export to csv.


Thanks
Comment 6 Nicole C. Engard 2011-08-17 10:53:52 UTC
Created attachment 5042 [details]
my framework

Here's what the framework listing looks like on the screen - none of the descriptions go to two lines so I don't think there is a line feed in there.

Also, this framework came straight from an install of Koha before the upgrade to 3.4 - so it's possible there used to be and as upgrades were done things changed.

Can you tell me what lines to look at? I can edit their descriptions by hand and see if that solves it.
Comment 7 Nicole C. Engard 2011-08-17 10:54:27 UTC
Created attachment 5043 [details]
long field

here's a long field - there is a hyphen in there, but no line feed that I see.
Comment 8 Nicole C. Engard 2011-08-17 11:25:54 UTC
This might be related to bug 6723.
Comment 9 Nicole C. Engard 2011-08-17 11:33:50 UTC
I tested the patch for that other bug and it didn't solve my problem so maybe it's not related.

Nicole
Comment 10 Salva Zaragoza Rubio 2011-08-17 11:38:41 UTC
Created attachment 5044 [details]
CVS with LF ok

Opening the csv with a text editor this is what I see (only the
marc_tag_structure is affected):

"tagfield","liblibrarian","libopac","repeatable","mandatory","authorised_value","frameworkcode"
"000","LEADER","LEADER","0","1","
","FA"
"020","INTERNATIONAL STANDARD BOOK
NUMBER","INTERNATIONAL STANDARD BOOK
NUMBER","1","0","","FA"
"022","INTERNATIONAL STANDARD SERIAL
NUMBER","INTERNATIONAL STANDARD SERIAL NUMBER","1","0","","FA"
"082","DEWEY DECIMAL CLASSIFICATION NUMBER ","DEWEY
DECIMAL CLASSIFICATION NUMBER ","0","0","","FA"
"090","LOCALLY ASSIGNED LC-TYPE
CALL NUMBER","LOCALLY ASSIGNED LC-TYPE CALL NUMBER","1","0","","FA"
"092","Local Dewey","Local Dewey","0","0","","FA"
"100","MAIN
ENTRY--PERSONAL NAME","MAIN ENTRY--PERSONAL NAME","0","0","","FA"
"245","TITLE
STATEMENT","TITLE STATEMENT","0","1","","FA"
"250","EDITION STATEMENT","EDITION
STATEMENT","0","0","","FA"
"260","PUBLICATION, DISTRIBUTION, ETC.
(IMPRINT)","PUBLICATION, DISTRIBUTION, ETC.
(IMPRINT)","1","0","","FA"
"500","GENERAL NOTE","GENERAL
NOTE","1","0","","FA"
"942","ADDED ENTRY ELEMENTS (KOHA)","ADDED ENTRY ELEMENTS
(KOHA)","0","0","","FA"
"952","LOCATION AND ITEM INFORMATION (KOHA)","LOCATION
AND ITEM INFORMATION (KOHA)","1","0","","FA"
"999","SYSTEM CONTROL NUMBERS
(KOHA)","SYSTEM CONTROL NUMBERS (KOHA)","1","0","","FA"


This is what it should be (deleting the spurious LF):
I attach the file or the textarea will add again LF where it shouldn't.


With this new format I've imported and exported correctly the csv file.

I don't know where those LF characters have come from.
The code from exporting in csv format is (C4::ImportExportFramework, line 319,
only takes data from the database):

$query = 'SELECT * FROM ' . $table . ' WHERE frameworkcode=?';
        $sth = $dbh->prepare($query);
        $sth->execute($frameworkcode);
        my $data;
        while (my $hashRef = $sth->fetchrow_hashref) {
            for (@fields) {
                $$strCSV .= '"' . $hashRef->{$_} . '",';
            }
            chop $$strCSV;
            $$strCSV .= chr(10);
        }
        $$strCSV .= chr(10);

So I thinks they have come from $hashRef->{$_}, the result from the database
query. Look at your marc_tag_structure table from the mysql command and query
the framework to check whether there are LF or not.
Comment 11 Liz Rea 2011-08-17 21:56:35 UTC
That is what I did, but today it's not working... and I think I know why.

This happens when you EDIT the framework in between the export and the import.
Comment 12 Nicole C. Engard 2011-08-19 04:49:42 UTC
So, do we have an idea of how to fix it? Or is this something I need to change on my end?
Comment 13 Salva Zaragoza Rubio 2011-08-19 07:29:52 UTC
(In reply to comment #12)
> So, do we have an idea of how to fix it? Or is this something I need to change
> on my end?

To avoid introduction of spurious LF/CR on exporting and processing with spreadsheet applications (MSExcel, OpenOffice/LibreOffice Calc, Gnumeric, ...) in CSV files.

I'll make a patch with the next modifications:

* On export_csv: all the LF/CR will be remove of the data from the database query so the spreadsheet will be well formed.

* On import_csv: all the LF/CR that it's inside a data field, i.e., between quotes will be remove, so there won't be multiples lines where only should be one.

I hope these changes will fix the issue.
Comment 14 Salva Zaragoza Rubio 2011-08-19 10:43:07 UTC Comment hidden (obsolete)
Comment 15 Nicole C. Engard 2011-08-25 00:02:28 UTC
Created attachment 5134 [details] [review]
[SIGNED-OFF] Bug 6727 - Change on exporting and importing csv to deal with spurious CR/LF

Function _export_table_csv modified to remove CR/LF in the data
from the database to built a correct csv file.

Function _import_table_csv modified to deal with CR/LF in the data
field. When a line is found with an unproper end (a data field has not
end quote), it's concatenated to a temporary buffer until the next
line ends ok and then it's proccessed.

Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Comment 16 Ian Walls 2011-08-25 01:24:20 UTC
Commit message and comments are clear.  Only effects import and export of frameworks (a very specific area of code), and has been tested and found to work.  Limited chances of any regressions, so I'm marking as Passed QA.
Comment 17 Chris Cormack 2011-08-25 22:29:24 UTC
Pushed, please test
Comment 18 Chris Nighswonger 2011-10-26 18:02:16 UTC
The fix for this bug was published in the 3.4.5 release. If you were the reporter of this bug, please take time to verify the fix and update the status of this bug report accordingly. If the bug is fixed to your satisfaction, please close this report.