Bug 41681 - bulkmarcimport.pl reports an incorrect number of MARC records processed
Summary: bulkmarcimport.pl reports an incorrect number of MARC records processed
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: Hammat wele
QA Contact: Testopia
URL:
Keywords:
: 41147 (view as bug list)
Depends on:
Blocks:
 
Reported: 2026-01-21 23:53 UTC by Hammat wele
Modified: 2026-03-10 14:23 UTC (History)
5 users (show)

See Also:
GIT URL:
Initiative type: ---
Sponsorship status: ---
Comma delimited list of Sponsors:
Crowdfunding goal: 0
Crowdfunding committed: 0
Crowdfunding contact:
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
A marc file containing 5 records (1.19 KB, text/plain)
2026-01-21 23:53 UTC, Hammat wele
Details
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed (1.25 KB, patch)
2026-01-22 00:01 UTC, Hammat wele
Details | Diff | Splinter Review
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed (1.30 KB, patch)
2026-02-18 14:39 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
File with 2 MARC records where the first is not valid (1.72 KB, application/xml)
2026-03-09 08:25 UTC, Jan Kissig
Details
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed (1.30 KB, patch)
2026-03-10 14:23 UTC, Jan Kissig
Details | Diff | Splinter Review
Bug 41681: (follow-up) Fix record number count when broken records are processed (5.17 KB, patch)
2026-03-10 14:23 UTC, Jan Kissig
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Hammat wele 2026-01-21 23:53:58 UTC
Created attachment 191849 [details]
A marc file containing 5 records

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

To reproduce:

1. Download the joined MARC file containing 5 records.
2. Run the import command:
   ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc
3. Check the final script report
   ==> 11 MARC records done in 0.566349983215332 seconds
       The reported number is incorrect
Comment 1 Hammat wele 2026-01-22 00:01:12 UTC
Created attachment 191850 [details] [review]
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

To test:

1. Download the joined MARC file containing 5 records.
2. Run the import command:
   ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc
3. Check the final script report
   ==> 11 MARC records done in 0.566349983215332 seconds
   Expected output should show 5 records
4. Apply the patch
5. Repeat step 2, 3
   ==> The correct number is shown
Comment 2 Janusz Kaczmarek 2026-02-18 14:39:29 UTC
Created attachment 193360 [details] [review]
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

To test:

1. Download the joined MARC file containing 5 records.
2. Run the import command:
   ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc
3. Check the final script report
   ==> 11 MARC records done in 0.566349983215332 seconds
   Expected output should show 5 records
4. Apply the patch
5. Repeat step 2, 3
   ==> The correct number is shown

Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Comment 3 Jan Kissig 2026-03-09 08:13:24 UTC
*** Bug 41147 has been marked as a duplicate of this bug. ***
Comment 4 Jan Kissig 2026-03-09 08:25:49 UTC
Created attachment 194964 [details]
File with 2 MARC records where the first is not valid
Comment 5 Jan Kissig 2026-03-09 08:40:17 UTC
From bug 41147 there is an important comment from Nick in his patch:

# We increment here, if a record is skipped because we can't load form the batch or can't be converted
# it still counts against the limit if a number to process is passed

And so I created a file with the first record to be broken to test against param n=1 which would mean that only 1 record should be imported from the given file.
As the first record is broken the script then should say 

0 records imported (though 1 record processed)

But the output is still like  '1 MARC records done in ...' so I think there needs to be done a bit more or am I misinterpreting the number param to count only successful records?


Tested via: perl misc/migration_tools/bulkmarcimport.pl -b -v -n=1 -m MARCXML --file broken.xml
Comment 6 Jan Kissig 2026-03-10 14:23:35 UTC
Created attachment 195081 [details] [review]
Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

To test:

1. Download the joined MARC file containing 5 records.
2. Run the import command:
   ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc
3. Check the final script report
   ==> 11 MARC records done in 0.566349983215332 seconds
   Expected output should show 5 records
4. Apply the patch
5. Repeat step 2, 3
   ==> The correct number is shown

Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Comment 7 Jan Kissig 2026-03-10 14:23:37 UTC
Created attachment 195082 [details] [review]
Bug 41681: (follow-up) Fix record number count when broken records are processed

When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file.

This patch fixes 3 issues
- broken (invalid) records from batch get now counted (record_number)
- when using the number param (n) to limit the number of records to be processed valid and invalid records get counted
- when using th offset param (o) the scipt errored when broken records were part of the offset.

To test:

1. Download the MARC file with the broken record
2. Run the following import commands and compare the counted import numbers:
  a) all records
     perl misc/migration_tools/bulkmarcimport.pl -b -v -m MARCXML --file 2_records_1st_broken.xml
     4 MARC records done in 0.0421221256256104 seconds
  b) limit to 1 record in total
     perl misc/migration_tools/bulkmarcimport.pl -b -v -n 1 -m MARCXML --file 2_records_1st_broken.xml
     4 MARC records done in 0.0420489311218262 seconds
  c) limit to 1 record  offset 1 record
     perl misc/migration_tools/bulkmarcimport.pl -b -v -n 1 -o 1 -m MARCXML --file 2_records_1st_broken.xml
     :2: parser error : Opening and ending tag mismatch: leader2 line 2 and leader
  <leader2>00216nam a22001097a 4500</leader>

3. Apply the patch
4. Repeat steps in 2 and compare the results
  a) 2 MARC records done in 0.0405838489532471 seconds
  b) 1 MARC records done in 0.00948596000671387 seconds  *
  c) 1 MARC records done in 0.0294840335845947 seconds

*though this one was skipped