Summary: | worker-output.log file grows to incredible sizes and causing the server to crash | ||
---|---|---|---|
Product: | Koha | Reporter: | Yuriy Kotsyuk <yuriy.kotsyuk> |
Component: | Architecture, internals, and plumbing | Assignee: | Bugs List <koha-bugs> |
Status: | RESOLVED DUPLICATE | QA Contact: | Testopia <testopia> |
Severity: | critical | ||
Priority: | P5 - low | CC: | martin.renvoize |
Version: | 23.05 | ||
Hardware: | All | ||
OS: | Linux | ||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | --- | Documentation contact: | |
Documentation submission: | Text to go in the release notes: | ||
Version(s) released in: | Circulation function: | ||
Attachments: | scrinchot worker-output.log.2 (fragment) |
Created attachment 168002 [details] scrinchot worker-output.log.2 (fragment) The worker-output.log file grows to incredible sizes. While checking one of the files (5 GB), a warning entry was found and this entry was duplicated a huge number of times over a short period of time. The first two records are as follows: DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('database=koha_library;host=localhost;port=3306','koha_library',...) failed: Lost connection to MySQL server at 'reading initial communication packet', system error: 104 at /usr/share/koha/lib/Koha/Database.pm line 91. at /usr/share/koha/lib/Koha/Objects.pm line 312 DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('database=koha_library;host=localhost;port=3306','koha_library',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) at /usr/share/koha/lib/Koha/Database.pm line 91. at /usr/share/koha/lib/Koha/Objects.pm line 312 Line number three indicates the date and time 2024/05/20 17:35:52 In general, the record looks like this: [2024/05/20 17:35:52] [WARN] Frame not processed - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "You must log in usin...") at /usr/share/koha/bin/workers/background_jobs_worker.pl line 120. main::catch {...} /usr/share/koha/bin/workers/background_jobs_worker.pl (122) This entry occurs 985 times in a row. Recording in the next second: [2024/05/20 17:35:53] [WARN] Frame not processed - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "You must log in usin...") at /usr/share/koha/bin/workers/background_jobs_worker.pl line 120. main::catch {...} /usr/share/koha/bin/workers/background_jobs_worker.pl (122) occurs 1483 times in a row. Recording in the next second: [2024/05/20 17:35:54] [WARN] Frame not processed - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "You must log in usin...") at /usr/share/koha/bin/workers/background_jobs_worker.pl line 120. main::catch {...} /usr/share/koha/bin/workers/background_jobs_worker.pl (122) occurs 6952 times A total of 16,709,173 messages of the same type were generated from 2024/05/20 17:35:52 to 2024/05/21 10:59:08 After this series of messages, other messages have already appeared: Cannot connect to the message broker, the jobs will be processed anyway (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at /usr/share/koha/bin/workers/background_jobs_worker.pl line 92. Cannot connect to the message broker, the jobs will be processed anyway (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at /usr/share/koha/bin/workers/background_jobs_worker.pl line 92. Today, worker-output.log was 108 GB in size and took up all free disk space, causing the server to crash. Perhaps someone had the same problems and was able to overcome them