Right now, no matter how large a marc batch file is, the marc record stager loads all the records into memory ( see http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=C4/ImportBatch.pm;h=8d39ed20347132e34ab5661dedd15548df672708;hb=HEAD#l1545 ). We should break this up and process chunks of records at a time to keep the memory footprint low for servers without enough RAM to handle this memory load. My proposal is to modify process_batch in stage_file to loop over the records processing X number of records at a time. X should be configurable via a syspref. In addition, this will require modifying C4::ImporBatch::RecordsFromMARCXMLFile to accept an offset and a count such that it will start at the offset and return the given count of records.
Created attachment 93826 [details] [review] Bug 23766: Upate RecordsFromISO2709File and RecordsFromMARCXMLFile to accept an offset and count
When looking at this I wondered if instead of the RecordsFromISO2709File/RecordsFromMARCXMLFile looping the entire file we could loop in the script and pass each record to the subs, changing them to convert a single record, then have the script process the records in chunks, that way we don't need to handle offsets
Was this solved by using the background jobs?