Bug 37020

Is there anything in the other logs maybe that could give us some more information on the issue?

Are you sure it's a regression and wouldn't happen with the older version of the script? How much swap space does your installation have?

Generally back when I ran Koha VM with very low RAM (like 2 or 4 GB), some older version like 23.05, I'd have issues with various cron tasks crashing due to OOM (out-of-memory), up until I gave it some significant swap space (like 16 or 24 GB), which I think was even also a recommendation somewhere.

Just thinking if it could perchance be simply a matter of that rather than a regression per se (though it might be both!).

Hi Michał

(In reply to Michał from comment #2)
> Are you sure it's a regression and wouldn't happen with the older version of
> the script? 

I just checked it, I ran bulkmarcimport around 20 times before on the same machine, with the same amount of records (~120k) on versions < 24.05. Everytime it worked.

> How much swap space does your installation have?

4GB

> Generally back when I ran Koha VM with very low RAM (like 2 or 4 GB), some
> older version like 23.05, I'd have issues with various cron tasks crashing
> due to OOM (out-of-memory), up until I gave it some significant swap space
> (like 16 or 24 GB), which I think was even also a recommendation somewhere.
> 
> Just thinking if it could perchance be simply a matter of that rather than a
> regression per se (though it might be both!).

I tried also on ktd with 8GB RAM and it seems that docker is crashing because of bulkmarcimport. I am trying to give it more ressources.

I also gave the script a little extra output in both loops and it always stopped in the first while-loop (https://git.koha-community.org/Koha-community/Koha/src/branch/main/misc/migration_tools/bulkmarcimport.pl#L317)
Maybe that @marc_records array is causing my error.

I can confirm on ktd with 24.06.:

- 8GB RAM, 4GB swap, 120k Records -> Killed
- 12GB RAM, 4GB swap, 120k Records -> Worked

No entries in ktd koha logs. 

Like mentioned before. The script seems to be hanging in the first loop, there is no commit of records in my output found. 

Shall i link to my marcxml for someone else to test?

I am having even bigger problems due to the mysql warnings. I have used koha-dump to backup koha 23.11 in server 1. I have installed latest Koha 24.05 in server 2 and ran koha-restore. As it's a version upgrade, it's expected to have web installer. 

When I am going through database upgrade step, I am getting errors and can't proceed further: "WARNING: MYSQL_OPT_RECONNECT is deprecated and will be removed in a future version.". 

I have used "apt install --reinstall koha-common", it seemed like the issue got fixed with the same warning in terminal, but the database doesn't seem to be upgraded as I am now getting "Error 500" in Circulation and Fine Rules page, and empty fields in "Identity Provider" edit buttons, which was working perfectly fine in version 23.11. 

I have checked through installer codebase, it seems like whenever DBD::mysql generates any warning logs, the Koha's error handling mechanism mistakes it as error instead of warnings (line 441 of "/usr/share/koha/intranet/cgi-bin/installer/install.pl"). And I believe, the same error mechanism is reflected either throughout the Koha codebase or in DBD::mysql package. 

Possible fix: 
1. Change logic for error handling
2. Add "SET sql_notes=0" everywhere before having any query of SQL
3. Bundle error-free DBD::mysql perl package with Koha that has no such issue

(In reply to Aditya from comment #5)
> I am having even bigger problems due to the mysql warnings. I have used
> koha-dump to backup koha 23.11 in server 1. I have installed latest Koha
> 24.05 in server 2 and ran koha-restore. As it's a version upgrade, it's
> expected to have web installer. 
> 
> When I am going through database upgrade step, I am getting errors and can't
> proceed further: "WARNING: MYSQL_OPT_RECONNECT is deprecated and will be
> removed in a future version.". 
> 
> I have used "apt install --reinstall koha-common", it seemed like the issue
> got fixed with the same warning in terminal, but the database doesn't seem
> to be upgraded as I am now getting "Error 500" in Circulation and Fine Rules
> page, and empty fields in "Identity Provider" edit buttons, which was
> working perfectly fine in version 23.11. 
> 
> I have checked through installer codebase, it seems like whenever DBD::mysql
> generates any warning logs, the Koha's error handling mechanism mistakes it
> as error instead of warnings (line 441 of
> "/usr/share/koha/intranet/cgi-bin/installer/install.pl"). And I believe, the
> same error mechanism is reflected either throughout the Koha codebase or in
> DBD::mysql package. 
> 
> Possible fix: 
> 1. Change logic for error handling
> 2. Add "SET sql_notes=0" everywhere before having any query of SQL
> 3. Bundle error-free DBD::mysql perl package with Koha that has no such issue

Hi, this bug report is about the import script bulkmarcimport.pl. I think you might be experiencing different bugs, like bug 37533 (error 500 on circulation rules). Please check on the mailing list or Mattermost chat first if you are experiencing different issues like this. We will often be able to point you to the right bugs or solution.