Summary: | can't import frameworks | ||
---|---|---|---|
Product: | Koha | Reporter: | Nicole C. Engard <nengard> |
Component: | Cataloging | Assignee: | Salva Zaragoza Rubio <salva> |
Status: | CLOSED FIXED | QA Contact: | Bugs List <koha-bugs> |
Severity: | critical | ||
Priority: | PATCH-Sent (DO NOT USE) | CC: | chris, koha.sekjal, paul.poulain, salva, wizzyrea |
Version: | 3.6 | ||
Hardware: | All | ||
OS: | All | ||
Change sponsored?: | --- | Patch complexity: | --- |
Documentation contact: | Documentation submission: | ||
Text to go in the release notes: | Version(s) released in: | ||
Circulation function: | |||
Attachments: |
framework exported
my framework long field CVS with LF ok Proposed patch with changes on exporting and importing csv to deal with spurious CR/LF [SIGNED-OFF] Bug 6727 - Change on exporting and importing csv to deal with spurious CR/LF |
Description
Nicole C. Engard
2011-08-16 12:51:56 UTC
Created attachment 5030 [details]
framework exported
I was unable to replicate this issue on current master. What I did: Exported a framework from my dev server looked at the CSV - looks ok Imported the same CSV into the ByWater Demo server Framework looked ok, seemed to work ok. Liz, Did you try it with my file? Which I exported and then tried to upload on someone else's server and it came up with only one field. The CSV did look fine. I'm saying you should try exporting from one server and importing to another - which is what I did when this error occurred. Nicole Okay, I tested on another server, again other than the one I exported from and the same thing happened. I've now exported one file from one server and imported it in to two different other servers and in both cases the imported file only had one field in it instead of all the fields in the CSV file -- see the attached CSV maybe it's a character issue. Hi Nicole, I've seen the csv file and inside the content of some fields exist line feeds characters that breaks what is supposed to be one line into multiples lines. Two possible causes: 1. The field contents have line feed characters so if you export the framework the csv will be malformed. This will have to be corrected, so the LN characters inside data fields are taking into account. Can you check the framework and see if it has some LN in its content? 2. The export process fails for this framework and adds LN in the middle of a field. Send me the framework in another format to dump into my database and test the export to csv. Thanks Created attachment 5042 [details]
my framework
Here's what the framework listing looks like on the screen - none of the descriptions go to two lines so I don't think there is a line feed in there.
Also, this framework came straight from an install of Koha before the upgrade to 3.4 - so it's possible there used to be and as upgrades were done things changed.
Can you tell me what lines to look at? I can edit their descriptions by hand and see if that solves it.
Created attachment 5043 [details]
long field
here's a long field - there is a hyphen in there, but no line feed that I see.
I tested the patch for that other bug and it didn't solve my problem so maybe it's not related. Nicole Created attachment 5044 [details]
CVS with LF ok
Opening the csv with a text editor this is what I see (only the
marc_tag_structure is affected):
"tagfield","liblibrarian","libopac","repeatable","mandatory","authorised_value","frameworkcode"
"000","LEADER","LEADER","0","1","
","FA"
"020","INTERNATIONAL STANDARD BOOK
NUMBER","INTERNATIONAL STANDARD BOOK
NUMBER","1","0","","FA"
"022","INTERNATIONAL STANDARD SERIAL
NUMBER","INTERNATIONAL STANDARD SERIAL NUMBER","1","0","","FA"
"082","DEWEY DECIMAL CLASSIFICATION NUMBER ","DEWEY
DECIMAL CLASSIFICATION NUMBER ","0","0","","FA"
"090","LOCALLY ASSIGNED LC-TYPE
CALL NUMBER","LOCALLY ASSIGNED LC-TYPE CALL NUMBER","1","0","","FA"
"092","Local Dewey","Local Dewey","0","0","","FA"
"100","MAIN
ENTRY--PERSONAL NAME","MAIN ENTRY--PERSONAL NAME","0","0","","FA"
"245","TITLE
STATEMENT","TITLE STATEMENT","0","1","","FA"
"250","EDITION STATEMENT","EDITION
STATEMENT","0","0","","FA"
"260","PUBLICATION, DISTRIBUTION, ETC.
(IMPRINT)","PUBLICATION, DISTRIBUTION, ETC.
(IMPRINT)","1","0","","FA"
"500","GENERAL NOTE","GENERAL
NOTE","1","0","","FA"
"942","ADDED ENTRY ELEMENTS (KOHA)","ADDED ENTRY ELEMENTS
(KOHA)","0","0","","FA"
"952","LOCATION AND ITEM INFORMATION (KOHA)","LOCATION
AND ITEM INFORMATION (KOHA)","1","0","","FA"
"999","SYSTEM CONTROL NUMBERS
(KOHA)","SYSTEM CONTROL NUMBERS (KOHA)","1","0","","FA"
This is what it should be (deleting the spurious LF):
I attach the file or the textarea will add again LF where it shouldn't.
With this new format I've imported and exported correctly the csv file.
I don't know where those LF characters have come from.
The code from exporting in csv format is (C4::ImportExportFramework, line 319,
only takes data from the database):
$query = 'SELECT * FROM ' . $table . ' WHERE frameworkcode=?';
$sth = $dbh->prepare($query);
$sth->execute($frameworkcode);
my $data;
while (my $hashRef = $sth->fetchrow_hashref) {
for (@fields) {
$$strCSV .= '"' . $hashRef->{$_} . '",';
}
chop $$strCSV;
$$strCSV .= chr(10);
}
$$strCSV .= chr(10);
So I thinks they have come from $hashRef->{$_}, the result from the database
query. Look at your marc_tag_structure table from the mysql command and query
the framework to check whether there are LF or not.
That is what I did, but today it's not working... and I think I know why. This happens when you EDIT the framework in between the export and the import. So, do we have an idea of how to fix it? Or is this something I need to change on my end? (In reply to comment #12) > So, do we have an idea of how to fix it? Or is this something I need to change > on my end? To avoid introduction of spurious LF/CR on exporting and processing with spreadsheet applications (MSExcel, OpenOffice/LibreOffice Calc, Gnumeric, ...) in CSV files. I'll make a patch with the next modifications: * On export_csv: all the LF/CR will be remove of the data from the database query so the spreadsheet will be well formed. * On import_csv: all the LF/CR that it's inside a data field, i.e., between quotes will be remove, so there won't be multiples lines where only should be one. I hope these changes will fix the issue. Created attachment 5057 [details] [review] Proposed patch with changes on exporting and importing csv to deal with spurious CR/LF Function _export_table_csv modified to remove CR/LF in the data from the database to built a correct csv file. Function _import_table_csv modified to deal with CR/LF in the data field. When a line is found with an unproper end (a data field has not end quote), it's concatenated to a temporary buffer until the next line ends ok and then it's proccessed. Created attachment 5134 [details] [review] [SIGNED-OFF] Bug 6727 - Change on exporting and importing csv to deal with spurious CR/LF Function _export_table_csv modified to remove CR/LF in the data from the database to built a correct csv file. Function _import_table_csv modified to deal with CR/LF in the data field. When a line is found with an unproper end (a data field has not end quote), it's concatenated to a temporary buffer until the next line ends ok and then it's proccessed. Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com> Commit message and comments are clear. Only effects import and export of frameworks (a very specific area of code), and has been tested and found to work. Limited chances of any regressions, so I'm marking as Passed QA. Pushed, please test The fix for this bug was published in the 3.4.5 release. If you were the reporter of this bug, please take time to verify the fix and update the status of this bug report accordingly. If the bug is fixed to your satisfaction, please close this report. |