Bug 30013 - Limit/throttle/retry SMTP connections
Summary: Limit/throttle/retry SMTP connections
Status: RESOLVED DUPLICATE of bug 33964
Alias: None
Product: Koha
Classification: Unclassified
Component: Notices (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-02-03 11:54 UTC by Magnus Enger
Modified: 2023-07-20 13:45 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Magnus Enger 2022-02-03 11:54:20 UTC
We have a setup where Koha is sending email through the SMTP server of the library's mother institution. We are seeing a problem with sending lots of emails in one go, where the first emails are sent ok, but then the rest fail with an error like this:

unable to establish SMTP connection to (mail.x.y) port 25

It looks like this is due to some limit on the server, making it unresponsive if there are too many connections from the same source in a short time. 

I don't know a lot about SMTP but I can imagine a couple of solutions:

- Introduce a configurable "sleep" period between each call to the external SMTP server (or batches of a configurable number, with a configurable sleep between each batch) 

- Looking at the response from the server and if "unable to establish SMTP connection" is received, wait for a configurable amount of time before trying to resend the failed message and the rest of the queue
Comment 1 Magnus Enger 2022-02-03 13:53:01 UTC
Is this the sort of functionality that was taken care of when we used postfix or similar on the server, and that we loose when moving to using the SMTP functionality in Koha?
Comment 2 Magnus Enger 2022-02-04 07:13:50 UTC
From RFC5321, page 66:

The sender MUST delay retrying a particular destination after one
attempt has failed.  In general, the retry interval SHOULD be at
least 30 minutes;
Comment 3 Kyle M Hall 2023-03-20 15:11:02 UTC
How do we know what the error is? Koha::Email uses Email::Stuffer as a base with uses Email::Sender::Simple which throws an Email::Sender::Failure exception. If we know what the code for that is, we add a new 'retry' status, and a 'retry_after' timestamp such that process_message_queue.pl will only attempt to send retries after that timestamp.
Comment 4 Liz Rea 2023-04-07 16:15:29 UTC
This also happens when using google as a provider of smtp.
Comment 5 Magnus Enger 2023-04-14 09:01:48 UTC
Not sure if this will be solved by 33360.
Comment 6 Marcel de Rooy 2023-04-14 09:10:08 UTC
(In reply to Liz Rea from comment #4)
> This also happens when using google as a provider of smtp.

Yes, I wrote 33360 with mail deferrals from google in mind.
Comment 7 Marcel de Rooy 2023-04-14 09:10:43 UTC
(In reply to Magnus Enger from comment #5)
> Not sure if this will be solved by 33360.

Yes. But the current -limit is already doing that partially. The domain limits will be additional.
Comment 8 Marcel de Rooy 2023-04-14 09:14:17 UTC
(In reply to Kyle M Hall from comment #3)
> How do we know what the error is? Koha::Email uses Email::Stuffer as a base
> with uses Email::Sender::Simple which throws an Email::Sender::Failure
> exception. If we know what the code for that is, we add a new 'retry'
> status, and a 'retry_after' timestamp such that process_message_queue.pl
> will only attempt to send retries after that timestamp.

In many cases Koha will just send to port 25. If the local mta receives it, Koha marks it as sent. But the mail is not yet outside. E.g. My local mta is now trying to send it to my mail container. Which also receives it and the problem comes up there.

Bug 33360 tries to throttle a bit smarter than the current limit does by looking at recipient domains too.
Comment 9 Marcel de Rooy 2023-04-14 09:14:35 UTC

*** This bug has been marked as a duplicate of bug 33360 ***
Comment 10 Liz Rea 2023-06-08 15:16:05 UTC
I don't believe 33360 fixes this issue - this is about using google as a smtp server and the fact that our message queue opens a connection per message to the smtp server. Near as I can tell 33360 has to do with the recipients and sender limits of the emails, not the behaviour of the smtp server. Google wants each connection to send several mails, right now we only send one message per connection.

Google does not publish the limits, but does say that smtp server connection limits are some number per rolling 24 hour period. They prefer the connection be kept open rather than making many individual connections.
Comment 11 Kyle M Hall 2023-06-08 15:19:51 UTC
I think we need to implement https://metacpan.org/pod/Email::Sender::Transport::SMTP::Persistent and keep the instances of it in memory during each call to SendQueuedMessages
Comment 12 David Cook 2023-06-09 00:26:47 UTC
(In reply to Kyle M Hall from comment #11)
> I think we need to implement
> https://metacpan.org/pod/Email::Sender::Transport::SMTP::Persistent and keep
> the instances of it in memory during each call to SendQueuedMessages

+1

Since different libraries can have different SMTP servers, I reckon SendQueuedMessages should have an empty hashref that gets passed through to _send_message_by_email. (Looks like _send_message_by_email is only used in SendQueuedMessages so changes to the arguments should be easy.)

Then the $stmp_server can be cached in that hashref for the life of SendQueuedMessages.
Comment 13 David Cook 2023-06-09 05:19:48 UTC
(In reply to David Cook from comment #12)
> (In reply to Kyle M Hall from comment #11)
> > I think we need to implement
> > https://metacpan.org/pod/Email::Sender::Transport::SMTP::Persistent and keep
> > the instances of it in memory during each call to SendQueuedMessages
> 
> +1
> 
> Since different libraries can have different SMTP servers, I reckon
> SendQueuedMessages should have an empty hashref that gets passed through to
> _send_message_by_email. (Looks like _send_message_by_email is only used in
> SendQueuedMessages so changes to the arguments should be easy.)
> 
> Then the $stmp_server can be cached in that hashref for the life of
> SendQueuedMessages.

And that's exactly what you've already done. Just saw the other report heh.
Comment 14 Marcel de Rooy 2023-07-20 13:45:39 UTC
So we can close this one and look at 33964 instead..

*** This bug has been marked as a duplicate of bug 33964 ***