Bug 23766

Summary: Allow MARC record importer to handle large batches without causing memory issues
Product: Koha Reporter: Kyle M Hall (khall) <kyle>
Component: Architecture, internals, and plumbingAssignee: Kyle M Hall (khall) <kyle>
Status: NEW --- QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: nick
Version: unspecified   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: Bug 23766: Upate RecordsFromISO2709File and RecordsFromMARCXMLFile to accept an offset and count

Description Kyle M Hall (khall) 2019-10-07 14:49:12 UTC 
Right now, no matter how large a marc batch file is, the marc record stager loads all the records into memory ( see http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=C4/ImportBatch.pm;h=8d39ed20347132e34ab5661dedd15548df672708;hb=HEAD#l1545 ).

We should break this up and process chunks of records at a time to keep the memory footprint low for servers without enough RAM to handle this memory load.

My proposal is to modify process_batch in stage_file to loop over the records processing X number of records at a time. X should be configurable via a syspref.

In addition, this will require modifying C4::ImporBatch::RecordsFromMARCXMLFile to accept an offset and a count such that it will start at the offset and return the given count of records.
Comment 1 Kyle M Hall (khall) 2019-10-07 15:54:50 UTC 
Created attachment 93826 [details] [review]
Bug 23766: Upate RecordsFromISO2709File and RecordsFromMARCXMLFile to accept an offset and count
Comment 2 Nick Clemens (kidclamp) 2019-10-07 16:10:54 UTC 
When looking at this I wondered if instead of the  RecordsFromISO2709File/RecordsFromMARCXMLFile looping the entire file we could loop in the script and pass each record to the subs, changing them to convert a single record, then have the script process the records in chunks, that way we don't need to handle offsets
Comment 3 Katrin Fischer 2023-12-09 11:42:55 UTC 
Was this solved by using the background jobs?