Bug 19687

Summary: Recent upgrade to 17.05.04.000 bulkmarcimport started to fail
Product: Koha Reporter: Mason James <mtj>
Component: Architecture, internals, and plumbingAssignee: Bugs List <koha-bugs>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: normal    
Priority: P3 CC: fridolin.somers, glasklas, gveranis, ian.bays, jonathan.druart, katrin.fischer, martin.renvoize, nick, tomascohen, victor
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 18927    
Bug Blocks:    
Attachments: Bug 19687: Undefined subroutine in bulkmarcimport
Bug 19687: Undefined subroutine in bulkmarcimport
Bug 19687: Undefined subroutine in bulkmarcimport
Bug 19687: Undefined subroutine in bulkmarcimport

Description Mason James 2017-11-23 21:53:27 UTC
After recent upgrade to 17.05.04.000 bulkmarcimport started to fail on all items attempted to be added.

ERROR: Adding items to bib 1 failed: Undefined subroutine &C4::Items::TransformMarcToKoha called at /usr/share/koha/lib/C4/Items.pm line 382, <GEN46> line 1

So I manually added the prefix and the next error came up:

ERROR: Adding items to bib 74925 failed: Undefined subroutine &C4::Items::GetMarcStructure called at /usr/share/koha/lib/C4/Items.pm line 2307, <GEN46> line 14.

Manually adding the prefix to that line seems to have worked, but it does not feel like the right solution.
Comment 1 Ian Bays 2018-06-07 14:53:42 UTC
Also the command line commit (after stage) has similar issues:

Undefined subroutine &C4::Items::TransformMarcToKoha called at /usr/share/koha/lib/C4/Items.pm line 243.

/usr/share/koha/bin/commit_file.pl --batch-number 5
... importing MARC records -- please wait
Undefined subroutine &C4::Items::ModZebra called at /usr/share/koha/lib/C4/Items.pm line 302.
Comment 2 Katrin Fischer 2018-07-03 09:26:11 UTC
This is evil, someone working on a fix?
Comment 3 Jonathan Druart 2018-07-04 16:50:38 UTC
How do you run the command? Which version of perl? Which OS?
I do not recreate with
  perl misc/migration_tools/bulkmarcimport.pl -b --file koha.mrc
on Debian 8.10 with perl v5.20.2

If you have a patch that fixes your issue, please submit it.
Comment 4 Katrin Fischer 2018-07-08 18:48:19 UTC
Testing without items bulkmarcimport seems to mostly work (apart from the update option, see bug 20811).

I also tested with a record exported from Koha with 3 items:

/home/vagrant/kohaclone/misc$ ./stage_file.pl --match 3 --file ../data/bib-69.utf8 --item-action replace --add-items
... staging MARC records -- please wait
... finished staging MARC records
... looking for matches with records already in database
Invalid normalization routine required (Biblionumber) at /home/vagrant/kohaclone/C4/Matcher.pm line 874.
... finished looking for matches

MARC record staging report
------------------------------------
Input file:                 ../data/bib-69.utf8
Record type:                biblio
Number of input records:    1
Number of valid records:    1
Number of invalid records:  0
Number of records matched:  1
Number of items parsed:  3

Batch number assigned:  2


kohadev-koha@kohadevbox:/home/vagrant/kohaclone/misc$ ./commit_file.pl --batch-number 2
... importing MARC records -- please wait
... finished importing MARC records

MARC record import report
----------------------------------------
Batch number:                    2
Number of new records added:     0
Number of records replaced:      1
Number of records ignored:       0
Number of items added:           0
Number of items replaced:        3
Number of items ignored:         0


Then I deleted the record and tried another import:

./stage_file.pl --match 3 --file ../data/bib-69.utf8 --item-action always_add --add-items
./commit_file.pl --batch-number 4

Record and items were added.

Tested on master. I think we need more information to track this problem down.
Comment 5 Ian Bays 2018-07-09 10:50:41 UTC
For the bulkmarcimport error:

install is a "standard install" at version 17.11.04.000

Command line used:

~/kohaclone/misc/migration_tools/bulkmarcimport.pl -file Results/$x.mrc -v > Logs/$x.txt

Repeating error is:

ERROR: Adding items to bib 4026 failed: Undefined subroutine &C4::Items::TransformMarcToKoha called at /usr/share/koha/lib/C4/Items.pm line 379, <GEN46> line 4026.

Koha About information:

Koha version: 	17.11.04.000
OS version ('uname -a'): 	Linux ncol.dh.bytemark.co.uk 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64
Perl interpreter: 	/usr/bin/perl
Perl version: 	5.024001
Perl @INC: 	/usr/share/koha/lib
/etc/perl
/usr/local/lib/x86_64-linux-gnu/perl/5.24.1
/usr/local/share/perl/5.24.1
/usr/lib/x86_64-linux-gnu/perl5/5.24
/usr/share/perl5
/usr/lib/x86_64-linux-gnu/perl/5.24
/usr/share/perl/5.24
/usr/local/lib/site_perl
/usr/lib/x86_64-linux-gnu/perl-base
MySQL version: 	mysql Ver 15.1 Distrib 10.1.26-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
Comment 6 Ian Bays 2018-07-09 11:03:02 UTC
The commit error was on the same system at the same version with the commit line described earlier. If it helps the stage command used was:

/usr/share/koha/bin/stage_file.pl --file Results/ncol_bib.mrc --match 1 --add-items --item-action always_add --no-replace

where match 1 is match on ISBN and aggressive ISBN matching is turned on.

PERL5LIB is:

echo $PERL5LIB
/usr/share/koha/lib
Comment 7 Katrin Fischer 2018-07-12 08:49:37 UTC
Hi Ian and others,

we have run several tests with 17.11.07 as we plan to update soon and could not see any errors. We are using a package installation with Debian. 

The problem could only exist in 17.05 or there is another difference we haven't figured out yet.
Comment 8 Katrin Fischer 2018-07-15 13:49:32 UTC
Lowering severity until we have more information.
Comment 9 Victor Grousset/tuxayo 2018-09-10 10:18:05 UTC
Aren't "just" prefixes missing? (C4::Biblio::)

We are having the same problem with
Undefined subroutine &C4::Items::TransformMarcToKoha called at /home/koha/src/C4/Items.pm line 243.

On a 17.11.06 on a Debian 8 with perl v5.20.2
Comment 10 Victor Grousset/tuxayo 2018-09-10 12:30:14 UTC
An hypothesis from a colleague is that there might be some circular dependency that prevents the prefixless-resolution to work.
Comment 11 David Gustafsson 2018-09-26 13:39:25 UTC
Probably caused by circular dependencies and incorrect use of Exporter (https://www.perlmonks.org/?node_id=778639). I have tried moving the exports so appears before other module imports in C4/Biblio.pm and seams to solve the problem. I will try to prepare a patch for this and create a separate issue, probably today or tomorrow.
Comment 12 David Gustafsson 2018-09-26 15:32:20 UTC
Created attachment 79435 [details] [review]
Bug 19687: Undefined subroutine in bulkmarcimport

Bulkmarcimport produces undefined subroutine error for
C4::Items::TransformMarcToKoha and C4::Items::GetMarcStructure
probably caused by mutually recursive modules. Put exports in
BEGIN clause before all the other imports.

1) Run bulkmarcimport.pl before applying patch and verify
that undefined subroutine error occurs
2) Apply patch
3) Run bulkmarcimport.pl again and verify that no errors are
produced

Sponsored-by: Gothenburg University Library
Comment 13 David Gustafsson 2018-09-26 15:35:45 UTC
I decided to attach the patch to this issue instead of creating a new. Have not had time to test it for the bulkmarcimport-case but this solved the exact same issue I was having with a plugin. My current koha-setup is a little bit broken with regards to bulkmarcimport, so if anyone could test if this fixes the problem there also that would be great. Can test this myself eventually, but will probably be a couple of days.
Comment 14 David Gustafsson 2018-09-26 15:46:16 UTC
Created attachment 79436 [details] [review]
Bug 19687: Undefined subroutine in bulkmarcimport

Bulkmarcimport produces undefined subroutine error for
C4::Items::TransformMarcToKoha and C4::Items::GetMarcStructure
probably caused by mutually recursive modules. Put exports in
BEGIN clause before all the other imports.

To test:
1) Run bulkmarcimport.pl before applying patch and verify
that undefined subroutine error occurs
2) Apply patch
3) Run bulkmarcimport.pl again and verify that no errors are
produced

Sponsored-by: Gothenburg University Library
Comment 15 George Veranis 2018-10-04 06:10:25 UTC
I apply your patch on 
Koha version:	18.05.04.000
OS version ('uname -a'):	Linux Debian-95-stretch-64-minimal 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64
Perl version:	5.024001

and it works !
Comment 16 Katrin Fischer 2018-10-04 20:36:21 UTC
Hi George, would you like to add your sign off and set this to 'Signed off' then?
Comment 17 George Veranis 2018-10-05 20:10:23 UTC
I try to sign-off this patch on koha-sandbox but the patch not apply. So I changed the status. I apply the patch on my koha manually and the bug solved. I try maybe to rewrite the patch later.
Comment 18 Jonathan Druart 2018-10-09 14:15:58 UTC
Created attachment 80276 [details] [review]
Bug 19687: Undefined subroutine in bulkmarcimport

Bulkmarcimport produces undefined subroutine error for
C4::Items::TransformMarcToKoha and C4::Items::GetMarcStructure
probably caused by mutually recursive modules. Put exports in
BEGIN clause before all the other imports.

To test:
1) Run bulkmarcimport.pl before applying patch and verify
that undefined subroutine error occurs
2) Apply patch
3) Run bulkmarcimport.pl again and verify that no errors are
produced

Sponsored-by: Gothenburg University Library

Signed-off-by: George Veranis <gveranis@gmail.com>
Comment 19 Jonathan Druart 2018-10-09 14:16:52 UTC
The patch applies ok for me, I have attached your signed-off-by line to the patch. I let you adjust the status to Signed Off to get the credit :)
Comment 20 George Veranis 2018-10-09 14:29:53 UTC
Thanks, next time I will manage it.
Comment 21 Chris Cormack 2018-10-09 23:52:36 UTC
Created attachment 80314 [details] [review]
Bug 19687: Undefined subroutine in bulkmarcimport

Bulkmarcimport produces undefined subroutine error for
C4::Items::TransformMarcToKoha and C4::Items::GetMarcStructure
probably caused by mutually recursive modules. Put exports in
BEGIN clause before all the other imports.

To test:
1) Run bulkmarcimport.pl before applying patch and verify
that undefined subroutine error occurs
2) Apply patch
3) Run bulkmarcimport.pl again and verify that no errors are
produced

Sponsored-by: Gothenburg University Library

Signed-off-by: George Veranis <gveranis@gmail.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 22 Nick Clemens 2018-10-15 16:57:59 UTC
Awesome work all!

Pushed to master for 18.11
Comment 23 Martin Renvoize 2018-10-16 13:39:24 UTC
Pushed to 18.05.x for 18.05.05
Comment 24 Fridolin Somers 2018-11-09 07:50:54 UTC
Does not apply easily on 17.11.x.