Bug 37545

Summary: worker-output log - Size grows every minute
Product: Koha Reporter: José Anjos <joseanjos>
Component: Architecture, internals, and plumbingAssignee: Bugs List <koha-bugs>
Status: NEW --- QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: dcook
Version: 24.05   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37714
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:

Description José Anjos 2024-08-01 12:56:25 UTC
The disk was full...
The problem is the log file: worker-output.log
After emptying the file it grew by 400Mb in one morning!

It's full of these messages:
main::catch {...}  /usr/share/koha/bin/workers/background_jobs_worker.pl (129)
[2024/08/01 10:41:16] [WARN] Frame does not have correct args, ignoring it main:: /usr/share/koha/bin/workers/background_jobs_worker.pl (134)
[2024/08/01 10:41:16] [WARN] Frame not processed - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "You must log in usin...") at /usr/share/koha/bin/workers/background_jobs_worker.pl line 127.
Comment 1 Katrin Fischer 2024-08-01 13:10:38 UTC
Hm, we see the same since the security update on Wednesday. 22.11.19. Could it be related?

I was wondering about the "you must log in usin..."
Comment 2 Katrin Fischer 2024-08-01 13:11:09 UTC
Jose - since when does this occur, did you also update? (24.05.02?)
Comment 3 José Anjos 2024-08-01 13:44:04 UTC
(In reply to Katrin Fischer from comment #2)
> Jose - since when does this occur, did you also update? (24.05.02?)

I had version 23.11 already with this problem. Maybe I had it before but I only noticed when the disk was completely full.
I've updated to 24.05 and the problem persists.
Comment 4 José Anjos 2024-08-12 09:03:31 UTC
Disk full again... worker-output.log with 120Gb

From what I read, the bugs mentioned below are related to this problem:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26742
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25674

I file: /etc/koha/sites/koha/koha-conf.xml I have this:

<message_broker>
   <hostname>localhost</hostname>
   <port>61613</port>
   <username>guest</username>
   <password>guest</password>
   <vhost></vhost>
 </message_broker>


In: About:
Message broker: Using RabbitMQ

I have millions of this in log:
[2024/08/12 09:10:36] [WARN] Frame does not have correct args, ignoring it main:: /usr/share/koha/bin/workers/background_jobs_worker.pl (134)
[2024/08/12 09:10:36] [WARN] Frame not processed - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "You must log in usin...") at /usr/share/koha/bin/workers/background_jobs_worker.pl line 127.
 main::catch {...}  /usr/share/koha/bin/workers/background_jobs_worker.pl (129)
[2024/08/12 09:10:36] [WARN] Frame does not have correct args, ignoring it main:: /usr/share/koha/bin/workers/background_jobs_worker.pl (134)
Cannot connect to the message broker, the jobs will be processed anyway (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27.
) at /usr/share/koha/bin/workers/background_jobs_worker.pl line 98.
Comment 5 José Anjos 2024-08-12 10:02:51 UTC
I've found this bug:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34070
He suggest to do some commands from here:
https://stackoverflow.com/questions/58689551/rabbitmq-vhost-is-down-for-user-xyz-even-after-user-has-all-access

I've empty the log and did:
sudo rabbitmqctl list_queues --offline state name
[sudo] password for koha: 
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
state	name
down	koha_koha-long_tasks
down	koha_koha-batch_authority_record_modification
down	koha_koha-default
down	koha_koha-batch_biblio_record_modification


sudo rabbitmqctl restart_vhost
Trying to restart vhost '/' on node 'rabbit@biblioteca' ...
Error:
Failed to start vhost '/' on node 'rabbit@biblioteca'Reason: {:shutdown, {:failed_to_start_child, :rabbit_vhost_process, {:badmatch, {:error, {{{:badmatch, {:error, {:not_a_dets_file, '/var/lib/rabbitmq/mnesia/rabbit@biblioteca/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets'}}}, [{:rabbit_recovery_terms, :open_table, 1, [file: 'src/rabbit_recovery_terms.erl', line: 199]}, {:rabbit_recovery_terms, :init, 1, [file: 'src/rabbit_recovery_terms.erl', line: 179]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 374]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 342]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}, {:child, :undefined, :rabbit_recovery_terms, {:rabbit_recovery_terms, :start_link, ["/"]}, :transient, 30000, :worker, [:rabbit_recovery_terms]}}}}}}

sudo rabbitmqctl stop_app
Stopping rabbit application on node rabbit@biblioteca ...

koha@biblioteca:/var/log/koha/koha$ sudo rabbitmqctl reset
Resetting node rabbit@biblioteca ...

koha@biblioteca:/var/log/koha/koha$ sudo rabbitmqctl start_app
Starting node rabbit@biblioteca ...
 completed with 1 plugins.

koha@biblioteca:/var/log/koha/koha$ sudo rabbitmqctl restart_vhost
Trying to restart vhost '/' on node 'rabbit@biblioteca' ...
Vhost '/' is already running on node 'rabbit@biblioteca'

sudo rabbitmqctl list_permissions --vhost /
Listing permissions for vhost "/" ...
user	configure	write	read
guest	.*	.*	.*


Now I will wait to see if the log keep growing.
Comment 6 José Anjos 2024-08-14 08:41:53 UTC
Since last post the log stopped to grow.