Bug 37164 - worker-output.log file grows to incredible sizes and causing the server to crash
Summary: worker-output.log file grows to incredible sizes and causing the server to crash
Status: RESOLVED DUPLICATE of bug 34070
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: 23.05
Hardware: All Linux
: P5 - low critical
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-06-22 18:45 UTC by Yuriy Kotsyuk
Modified: 2024-08-01 08:01 UTC (History)
1 user (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
scrinchot worker-output.log.2 (fragment) (197.73 KB, image/png)
2024-06-22 18:45 UTC, Yuriy Kotsyuk
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Yuriy Kotsyuk 2024-06-22 18:45:37 UTC
Created attachment 168002 [details]
scrinchot worker-output.log.2 (fragment)

The worker-output.log file grows to incredible sizes.  While checking one of the files (5 GB), a warning entry was found and this entry was duplicated a huge number of times over a short period of time.

The first two records are as follows:
DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('database=koha_library;host=localhost;port=3306','koha_library',...) failed: Lost connection to MySQL server at 'reading initial communication packet', system error: 104 at /usr/share/koha/lib/Koha/Database.pm line 91. at /usr/share/koha/lib/Koha/Objects.pm line 312
DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('database=koha_library;host=localhost;port=3306','koha_library',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) at /usr/share/koha/lib/Koha/Database.pm line 91. at /usr/share/koha/lib/Koha/Objects.pm line 312

 

Line number three indicates the date and time 2024/05/20 17:35:52
In general, the record looks like this:
 [2024/05/20 17:35:52] [WARN] Frame not processed - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "You must log in usin...") at /usr/share/koha/bin/workers/background_jobs_worker.pl line 120.
 main::catch {...}  /usr/share/koha/bin/workers/background_jobs_worker.pl (122)
This entry occurs 985 times in a row.

Recording in the next second:
[2024/05/20 17:35:53] [WARN] Frame not processed - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "You must log in usin...") at /usr/share/koha/bin/workers/background_jobs_worker.pl line 120.
 main::catch {...}  /usr/share/koha/bin/workers/background_jobs_worker.pl (122)
occurs 1483 times in a row.

Recording in the next second:
[2024/05/20 17:35:54] [WARN] Frame not processed - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "You must log in usin...") at /usr/share/koha/bin/workers/background_jobs_worker.pl line 120.
 main::catch {...}  /usr/share/koha/bin/workers/background_jobs_worker.pl (122)
occurs 6952 times

A total of 16,709,173 messages of the same type were generated from 2024/05/20 17:35:52 to 2024/05/21 10:59:08



After this series of messages, other messages have already appeared:
Cannot connect to the message broker, the jobs will be processed anyway (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27.
) at /usr/share/koha/bin/workers/background_jobs_worker.pl line 92.
Cannot connect to the message broker, the jobs will be processed anyway (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27.
) at /usr/share/koha/bin/workers/background_jobs_worker.pl line 92.


Today, worker-output.log was 108 GB in size and took up all free disk space, causing the server to crash.

Perhaps someone had the same problems and was able to overcome them
Comment 1 Martin Renvoize (ashimema) 2024-08-01 08:01:57 UTC
Please see discussion on bug 34070.. if none of the options there resolve your issue please re-open this bug with more details of your specific case.

*** This bug has been marked as a duplicate of bug 34070 ***