I'm noticing "Data too long for column 'title'" and "Data too long for column 'issn'" database errors in the logs. These are preventing import_biblios rows from being added to the system, although import_records rows are still being added. On koha-testing-docker, these errors are being silently ignored, which is not good.
One option is to try to match the import_biblios columns with the biblio and biblioitems tables... I notice import_biblios has an index on title, so I don't know if that will truncate the title for the index, or if it will just fail. If people don't clean their import_biblios table, this could also be a massive database update later, which would not be fun...
Another option would be to truncate the data before trying to insert it into the database. But considering that things like Koha::MetaSearcher use those fields... maybe we should fix them.
Created attachment 112656 [details] [review] Bug 26853: Throw a fatal error if import_biblios insert fails
Created attachment 112657 [details] [review] Bug 26853: Harmonize import_biblios with biblio and biblioitems This patch harmonizes the column datatypes of import_biblios with biblio and biblioitems to prevent database errors which cause staged MARC uploads to sometimes fail to update 100% correctly. To test: 1. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 2. Upload a file with MARC records with titles longer than 128 characters long 3. Click "Stage for import" 4. Monitor /var/log/koha/kohadev/intranet-error.log for database errors 5. Note that there are no database errors
Created attachment 112658 [details] [review] Bug 26853: Throw a fatal error if import_biblios insert fails
Hi David. Are you able to provide a sample file for import with records that will trigger this? I tried by editing a record and making the title longer, then exporting; but it still worked. Otherwise, I will have another go this evening. David
(In reply to David Nind from comment #6) > Hi David. > > Are you able to provide a sample file for import with records that will > trigger this? > > I tried by editing a record and making the title longer, then exporting; but > it still worked. > > Otherwise, I will have another go this evening. > > David Were you testing the patch or trying to reproduce the error? The problem is really difficult to spot, but I'll look at putting together a sample set of data.
Created attachment 112884 [details] Record with long title
I've added a record with a long title. Without the patch, when I go to view the import batch (at /cgi-bin/koha/tools/manage-marc-import.pl?import_batch_id=1), I see that the "Citation" says "null". If I go into the database, I can see 1 record in import_records, but there are 0 records in import_biblios.
Created attachment 112885 [details] [review] Bug 26853: Harmonize import_biblios with biblio and biblioitems This patch harmonizes the column datatypes of import_biblios with biblio and biblioitems to prevent database errors which cause staged MARC uploads to sometimes fail to update 100% correctly. To test: 1. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 2. Upload a file with MARC records with titles longer than 128 characters long 3. Click "Stage for import" 4. Monitor /var/log/koha/kohadev/intranet-error.log for database errors 5. Note that there are no database errors Signed-off-by: David Nind <david@davidnind.com>
Created attachment 112886 [details] [review] Bug 26853: Throw a fatal error if import_biblios insert fails Signed-off-by: David Nind <david@davidnind.com>
Thanks David! I had tried to recreate the error first, but I had no luck. The record you provided really helped! I hadn't spotted that a database was required as well. Everything appears to work now for me so I have signed off. I did notice another error in the database - it happened both before and after the patch was applied, but I don't think this is related to this bug: [Tue Nov 03 02:57:55.761624 2020] [cgi:error] [pid 3980] [client 172.18.0.1: 58304] AH01215: [Tue Nov 3 02:57:55 2020] stage-marc-import.pl: Filehandle STDOUT reopened as $fh only for input at /usr/share/perl5/DateTime/TimeZone/ Local/Unix.pm line 146.: /kohadevbox/koha/tools/stage-marc-import.pl, refere r: http://127.0.0.1:8081/cgi-bin/koha/tools/stage-marc-import.pl David
(In reply to David Nind from comment #12) > > Everything appears to work now for me so I have signed off. Thanks, David. Much appreciated! I think is a really nasty bug that is easy to miss, especially when RaiseError is turned off. When Bug 25026 is pushed, it will make issues like this more obvious. > I did notice another error in the database - it happened both before and > after the patch was applied, but I don't think this is related to this bug: > > [Tue Nov 03 02:57:55.761624 2020] [cgi:error] [pid 3980] [client 172.18.0.1: > 58304] AH01215: [Tue Nov 3 02:57:55 2020] stage-marc-import.pl: Filehandle > STDOUT reopened as $fh only for input at /usr/share/perl5/DateTime/TimeZone/ > Local/Unix.pm line 146.: /kohadevbox/koha/tools/stage-marc-import.pl, refere > r: http://127.0.0.1:8081/cgi-bin/koha/tools/stage-marc-import.pl That's more of a warning than an error. It's not related to this bug, but I have been thinking of looking into it more.
Created attachment 112888 [details] [review] Bug 26853: Harmonize import_biblios with biblio and biblioitems This patch harmonizes the column datatypes of import_biblios with biblio and biblioitems to prevent database errors which cause staged MARC uploads to sometimes fail to update 100% correctly. To test: 1. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 2. Upload a file with MARC records with titles longer than 128 characters long 3. Click "Stage for import" 4. Monitor /var/log/koha/kohadev/intranet-error.log for database errors 5. Note that there are no database errors Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 112889 [details] [review] Bug 26853: Throw a fatal error if import_biblios insert fails Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
It makes sense for these tables to be harmonized and I'm happy with the error being thrown on failures. Passing QA
Pushed to master for 20.11, thanks to everybody involved!
backported to 20.05.x for 20.05.06
I can't reproduce the errors in the logs in 19.11.x. not backported
(In reply to Aleisha Amohia from comment #19) > I can't reproduce the errors in the logs in 19.11.x. not backported That's interesting since 19.11.something is where I discovered the bug.