Bug 41681

Summary: bulkmarcimport.pl reports an incorrect number of MARC records processed
Product: Koha Reporter: Hammat wele <hammat.wele>
Component: Architecture, internals, and plumbingAssignee: Hammat wele <hammat.wele>
Status: Needs Signoff --- QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: bibliothek, caroline.cyr-la-rose, nick, philippe.blouin, roman.dolny
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41147
GIT URL: Initiative type: ---
Sponsorship status: --- Comma delimited list of Sponsors:
Crowdfunding goal: 0 Crowdfunding committed: 0
Crowdfunding contact: Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: A marc file containing 5 records
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed
File with 2 MARC records where the first is not valid
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed
Bug 41681: (follow-up) Fix record number count when broken records are processed

Description Hammat wele 2026-01-21 23:53:58 UTC
Created attachment 191849 [details]
A marc file containing 5 records

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

To reproduce:

1. Download the joined MARC file containing 5 records.
2. Run the import command:
   ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc
3. Check the final script report
   ==> 11 MARC records done in 0.566349983215332 seconds
       The reported number is incorrect
Comment 1 Hammat wele 2026-01-22 00:01:12 UTC
Created attachment 191850 [details] [review]
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

To test:

1. Download the joined MARC file containing 5 records.
2. Run the import command:
   ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc
3. Check the final script report
   ==> 11 MARC records done in 0.566349983215332 seconds
   Expected output should show 5 records
4. Apply the patch
5. Repeat step 2, 3
   ==> The correct number is shown
Comment 2 Janusz Kaczmarek 2026-02-18 14:39:29 UTC
Created attachment 193360 [details] [review]
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

To test:

1. Download the joined MARC file containing 5 records.
2. Run the import command:
   ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc
3. Check the final script report
   ==> 11 MARC records done in 0.566349983215332 seconds
   Expected output should show 5 records
4. Apply the patch
5. Repeat step 2, 3
   ==> The correct number is shown

Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Comment 3 Jan Kissig 2026-03-09 08:13:24 UTC
*** Bug 41147 has been marked as a duplicate of this bug. ***
Comment 4 Jan Kissig 2026-03-09 08:25:49 UTC
Created attachment 194964 [details]
File with 2 MARC records where the first is not valid
Comment 5 Jan Kissig 2026-03-09 08:40:17 UTC
From bug 41147 there is an important comment from Nick in his patch:

# We increment here, if a record is skipped because we can't load form the batch or can't be converted
# it still counts against the limit if a number to process is passed

And so I created a file with the first record to be broken to test against param n=1 which would mean that only 1 record should be imported from the given file.
As the first record is broken the script then should say 

0 records imported (though 1 record processed)

But the output is still like  '1 MARC records done in ...' so I think there needs to be done a bit more or am I misinterpreting the number param to count only successful records?


Tested via: perl misc/migration_tools/bulkmarcimport.pl -b -v -n=1 -m MARCXML --file broken.xml
Comment 6 Jan Kissig 2026-03-10 14:23:35 UTC
Created attachment 195081 [details] [review]
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

To test:

1. Download the joined MARC file containing 5 records.
2. Run the import command:
   ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc
3. Check the final script report
   ==> 11 MARC records done in 0.566349983215332 seconds
   Expected output should show 5 records
4. Apply the patch
5. Repeat step 2, 3
   ==> The correct number is shown

Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Comment 7 Jan Kissig 2026-03-10 14:23:37 UTC
Created attachment 195082 [details] [review]
Bug 41681: (follow-up) Fix record number count when broken records are processed

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

This patch fixes 3 issues
- broken (invalid) records from batch get now counted (record_number)
- when using the number param (n) to limit the number of records to be processed valid and invalid records get counted
- when using th offset param (o) the scipt errored when broken records were part of the offset.

To test:

1. Download the MARC file with the broken record
2. Run the following import commands and compare the counted import numbers:
  a) all records
     perl misc/migration_tools/bulkmarcimport.pl -b -v -m MARCXML --file 2_records_1st_broken.xml
     4 MARC records done in 0.0421221256256104 seconds
  b) limit to 1 record in total
     perl misc/migration_tools/bulkmarcimport.pl -b -v -n 1 -m MARCXML --file 2_records_1st_broken.xml
     4 MARC records done in 0.0420489311218262 seconds
  c) limit to 1 record  offset 1 record
     perl misc/migration_tools/bulkmarcimport.pl -b -v -n 1 -o 1 -m MARCXML --file 2_records_1st_broken.xml
     :2: parser error : Opening and ending tag mismatch: leader2 line 2 and leader
  <leader2>00216nam a22001097a 4500</leader>

3. Apply the patch
4. Repeat steps in 2 and compare the results
  a) 2 MARC records done in 0.0405838489532471 seconds
  b) 1 MARC records done in 0.00948596000671387 seconds  *
  c) 1 MARC records done in 0.0294840335845947 seconds

*though this one was skipped