Bug 38338

Summary: Add support for XOAUTH2 authentication for SMTP servers
Product: Koha Reporter: David Cook <dcook>
Component: Architecture, internals, and plumbingAssignee: Bugs List <koha-bugs>
Status: In Discussion --- QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: andreadup, blawlor, domm, schodkowy.omegi-0r, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40153
GIT URL: Change sponsored?: ---
Patch complexity: --- Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
Circulation function:

Description David Cook 2024-11-04 01:27:00 UTC
Both Google and Microsoft have been pushing XOAUTH2 for SMTP AUTH over the traditional basic username/password authentication for some time now and we haven't really had a good answer for Koha admins. 

For SMTP connections in Koha, we use Email::Sender::Transport::SMTP::Persistent which uses Authen::SASL which uses libsasl2 (if available) or a pure Perl fallback. 

Out of the box, neither libsasl2 nor Authen::SASL::Perl provides XOAUTH2 support.

However, there are a number of SASL plugins out there. I'll explain more in the comments.
Comment 1 David Cook 2024-11-04 01:32:17 UTC
I don't want to link to them here for fear of Bugzilla autobans but big ones are "sasl-xoauth2" by tarickb and "cyrus-sasl-xoauth2" by "moriyoshi". The one by tarickb is quite self-contained. The documentation isn't brilliant, but it's not bad. The author developed it for Postfix, but it should be usable by anything. 

In theory, for "sasl-xoauth2" you'd pass the username (ie email address) known by the upstream SMTP server, and then for the password you give it a filepath to a JSON object which contains things like an initial OAUTH2 access token, refresh token, etc. There are a couple different configuration options depending on your Koha server setup. 

In theory, Email::Sender::Transport::SMTP::Persistent should auto-detect the XOAUTH2 login method when it's offered by the SMTP server, so if a SASL plugin for XOAUTH2 is installed, then it should use it. 

I haven't tested this, but in theory that means we could gain XOAUTH2 support with Koha just by installing "sasl-xoauth2" on the Koha server. (Note that there is an experimental package available in Debian for this. Our feedback could help get that moved into stable I reckon.)

This does need testing... but if we put in a bit of work and confirmed that this method works, then we could put up a wiki page for this and point to it in the manual and in the Koha code.

--

Note that Email::Sender::Transport::SMTP::Persistent also allows you to provide a "sasl_authenticator", so if we wanted to provide a way to force XOAUTH2 as the choice, we could develop some code for that. But I think we can try the auto-detection path first as a proof-of-concept at the very least.
Comment 2 David Cook 2024-11-04 01:37:16 UTC
With the "sasl-xoauth2" plugin, I believe that it handles auto-refreshing of the access token, so while there is a bit of overhead for the backend sysadmin at first, once it's set up, it should be fairly set and forget. (Although if your refresh token expires, then you'll need to set it up again.)

--

Now I also mentioned "cyrus-sasl-xoauth2". I don't understand it as well and its documentation is not very good at all. However, my understanding is that you have to pass the access_token value directly as the sasl_password. 

That means no auto-refreshing of access tokens or anything like that. 

If we were to support that SASL plugin, then we'd have to handle a lot of the OAuth2 process ourselves, which is not impossible, but it would involve significantly more work.

So... at the moment I'm inclined to go with "sasl-xoauth2". 

--

Additionally, on the "perl-authen-sasl" github by "gbarr", I raised an issue called "Supporting XOAUTH2 authentication?", and there I think the repo maintainer would be open to accepting a patch for a pure Perl XOAUTH2 implementation. 

I don't know that any of us have the time/expertise to do that, but that's a opportunity open to folks as well. (We could also instantiate our own Authen::SASL plugin and pass that to Email::Sender::Transport::SMTP::Persistent using sasl_authenticator.)
Comment 3 David Cook 2024-11-04 01:56:49 UTC
I tried to test out the experimental sasl-xoauth2 package...

But I got stuck on dependencies:
The following packages have unmet dependencies:
 sasl-xoauth2 : Depends: libc6 (>= 2.38) but 2.36-9+deb12u8 is to be installed
                Depends: libcurl3t64-gnutls (>= 7.16.2) but it is not installable
                Depends: libjsoncpp26 (>= 1.9.6) but it is not installable
                Recommends: sasl-xoauth2-tool but it is not installable

The libcurl3t64-gnutls and libjsoncpp26 dependencies aren't available in Debian 12 Bookworm, but they're a bit weird anyway, since libcurl3-gnutls and libjsoncpp25 are available.

So I'm going to get that package maintainer to double-check those dependencies.
Comment 4 David Cook 2024-11-04 02:28:59 UTC
For testing purposes, I might end up building it from source just to get it running, but I'm going to wait to hear back from tarickb and dkg first about the packages, since it would be great to have a known pathway to using this in prod without too much messing about...
Comment 5 Michał 2024-11-04 08:03:15 UTC
For auto-refresh, there's also a challenge in that the refreshed credentials will have to be retrieved and saved back to the database (unless you say it's specifically filepath as password, and then the library would just write to that file instead on its own, then that'd be covered already I suppose).

There's also the open question about whether Koha should offer some kind of way in the interface to open up login page for some common providers (or a configurable custom one). If yes, then whether it should provide some kind of default community OAuth app client ID to make it easiest (I vote yes, apps like Thunderbird do just that) (of course these app IDs could and should be made configurable to allow flexibility there if need be).

I imagine someone could click a button to add server, choose a preconfigured provider like Google Mail or Microsoft, be directed to login page, after login the new server with all data would be populated, easy. Similarly an existing server with oauth login specified should have an option to trigger a login and generation of new access+refresh token data afresh as well.
Comment 6 David Cook 2024-11-04 22:24:57 UTC
I think for now we just focus on the minimum viable product (MVP), but I opened this bug report, so that we could gather ideas for the longer term. 

At the moment, for OAuth2, Koha uses Mojolicious::Plugin::OAuth2 which requires a sysadmin to restart Koha for it to take effect. 

But I like the idea of there being a OAuth2 client in Koha so that people could get that initial OAuth2 token response, which can then be stored and passed to the library. 

But first we have to see if the SASL plugin works as expected hehe.
Comment 7 David Cook 2024-11-05 02:53:15 UTC
I just noticed we're using Email::Sender::Transport::SMTP 2.600 and the sasl_authenticator parameter only appears in 2.601...

I've installed sasl-xoauth2 thanks to tarickb's Ubuntu PPA, and I was able to set it up, but... Email::Sender::Transport::SMTP seems to only be attempting the "LOGIN" SMTP AUTH mechanism...

I'll hack /usr/share/perl5/Email/Sender/Transport/SMTP.pm to see if I can at least test if it would work with 2.601...
Comment 8 David Cook 2024-11-05 02:54:16 UTC
Note that libemail-sender-perl is currently 2.601 in Debian trixie (testing).
Comment 9 David Cook 2024-11-05 03:04:13 UTC
Ahhh... we only have Authen::SASL::Perl available, so it can't make use of the SASL plugin, since it's a pure Perl implementation.

libauthen-sasl-xs-perl is in Debian trixie (testing).

Layers upon layers...
Comment 10 David Cook 2024-11-05 03:11:18 UTC
(In reply to David Cook from comment #9)
> Ahhh... we only have Authen::SASL::Perl available, so it can't make use of
> the SASL plugin, since it's a pure Perl implementation.
> 
> libauthen-sasl-xs-perl is in Debian trixie (testing).
> 
> Layers upon layers...

Unfortunately, libauthen-sasl-xs-perl isn't an easy backport. 

--

At the moment, the barriers appear to be:

1. Installing a sasl-xoauth2 package, which is only available in a Ubuntu PPA
2. Getting the initial access token (I had to hack /usr/bin/sasl-xoauth2-tool to use a particular port which I had my docker forward. For Koha, we'd need to think more about how we'd want to get the token response.)
3. Install libemail-sender-perl 2.601, which isn't available in stable Debian
4. Install libauthen-sasl-xs-perl, which isn't available in stable Debian

There's probably other barriers but those are the immediate one jumping out at me.
Comment 11 David Cook 2024-11-05 03:21:45 UTC
(In reply to David Cook from comment #10)
> At the moment, the barriers appear to be:
> 
> 1. Installing a sasl-xoauth2 package, which is only available in a Ubuntu PPA
> 2. Getting the initial access token (I had to hack
> /usr/bin/sasl-xoauth2-tool to use a particular port which I had my docker
> forward. For Koha, we'd need to think more about how we'd want to get the
> token response.)
> 3. Install libemail-sender-perl 2.601, which isn't available in stable Debian
> 4. Install libauthen-sasl-xs-perl, which isn't available in stable Debian
> 
> There's probably other barriers but those are the immediate one jumping out
> at me.

Note that Ubuntu 24.04 has both libauthen-sasl-xs-perl and libemail-sender-perl 2.601

I'd be tempted to try that out. 

--

I do wonder a bit if libemail-sender-perl 2.601 is totally necessary or if it's just libauthen-sasl-xs-perl, and that using the XS module will allow support for XOAUTH2 even when an explicit SASL authenticator isn't provided, but I'm not sure...
Comment 12 Michał 2024-11-05 08:57:02 UTC
(In reply to David Cook from comment #6)
> But first we have to see if the SASL plugin works as expected hehe.

True true

> At the moment, for OAuth2, Koha uses Mojolicious::Plugin::OAuth2 which
> requires a sysadmin to restart Koha for it to take effect. 
> 
> But I like the idea of there being a OAuth2 client in Koha so that people
> could get that initial OAuth2 token response, which can then be stored and
> passed to the library. 

Hmm, I don't think a dedicated OAuth2 library should be needed for this at all.

I think that for most popular providers out there at least, one should get away with a purely JavaScript solution that'd fill up the field in the UI, that might be the cleanest and most lightweight solution.

But if it's impossible, then I think it could be possible to just construct some special redirect URL that'd update the auth data for given SMTP server (or add new one depending on what user does) with the data it gets in response, and just redirect to server list. There should be no need for any kind of OAuth support library there whatsoever I imagine, just copy over the fields, which should just be standard access_token, refresh_token, expires_in etc. in an object.
Comment 13 David Cook 2024-11-06 00:39:28 UTC
At this point, we don't have a viable SASL solution, so I wouldn't worry too much about it.
Comment 14 David Cook 2024-11-07 01:47:52 UTC
Note that tarickb provides a "debian" directory in other branches: https://github.com/tarickb/sasl-xoauth2/tree/packaging/ubuntu

So if we wanted to build this ourselves technically we could.

Technically, I suppose we could always write a pure Perl Authen::SASL::Perl module for XOAUTH2 as well, although that is more work than I have time for.

Of course, we need libemail-sender-perl 2.601 anyway, so a few factors there.

--

This is quite low priority, but hopefully I'll find some time at some point to put together a little Ubuntu 24.04 test...
Comment 15 David Cook 2025-03-26 01:58:59 UTC
*** Bug 35775 has been marked as a duplicate of this bug. ***
Comment 16 David Cook 2025-04-28 04:03:11 UTC
(In reply to David Cook from comment #14)
> Technically, I suppose we could always write a pure Perl Authen::SASL::Perl
> module for XOAUTH2 as well, although that is more work than I have time for.

Luckily, someone else on the Internet took it upon themselves to write that code! 

Authen::SASL 2.18 now contains a Authen::SASL::Perl::XOAUTH2 module. 

Coupled with Email::Sender::Transport::SMTP 2.601, we'd have a way of authenticating using XOAUTH2 with a OAuth2 access token.

--

Now... that being said... that's only part of the solution. Koha would still be responsible for getting the initial access token and refresh token, and then Koha would be responsible for getting new access tokens using refresh tokens. 

So even if we don't have "Authen::SASL 2.18" and "Email::Sender::Transport::SMTP 2.601" yet, we could work on the other mechanisms required to make this work for real.

When we do send_or_die() we'll need to check for a failed auth. Hopefully we'll get an error message flowing through about an expired token, so that we can do a token refresh and then retry.

We'll need a way to get the initial token response, store it in Koha, and then use it on an ongoing basis.
Comment 17 David Cook 2025-05-21 23:52:00 UTC
I don't think we've mentioned it here, but there are workarounds to this problem.

On Mattermost, tadzik and @domm have mentioned using https://github.com/simonrob/email-oauth2-proxy as an email proxy which gets them around this problem
Comment 18 Thomas Klausner 2025-06-10 09:49:11 UTC
We have a customer that will need XOAUTH support for mailing soon (as they are using MS365, and they will disable basic SMTP).

They might go with the external proxy solution (simonrob/email-oauth2-proxy), but they are actually interested in getting proper XOAUTH support into Koha. And are (maybe) willing to sponsor the development (depending on how much it will be).

We are currently preparing an offer for the following steps:

* First step: Add XOAUTH support to the Koha SMTP sender
  * all the hard parts are already available on CPAN: LWP::Authen::OAuth2, Authen::SASL::Perl::XOAUTH2 and Email::Sender
  * we "just" need to add some new config settings to SMTP to allow using an access-token (instead of username/password)
  * we will also need to set up a cronjob to refresh the access tokens and the refresh token (better via cronjob, because if the mail sending happens infrequently even the refresh token could be invalidated in the interval where nothing happend)
  * The XOAUTH setup (i.e. the OAuth handshake to login the service account and create a first refresh token) will be done via a command line script
* Second step (optional):
  * Add a GUI to the Staff interface to setup XOAUTH and do the OAuth handshake (so system librarians can do the setup without needing access to the shell)
  * Probably do this via a plugin

The second step will be very likely outside the budget of the client, and even the first step might be too much. In that case we (HKS3) would try to get further funding by other interested libraries.

Of course we would very much like to get this changes into the Koha core or at least into a plugin, though AFAIK the Notifications-Area does not support any plugins/hooks as of now. We would need some feedback / discussion on how to actually add this feature to Koha: Just add it to Core, or add a way to Koha core to add this (and later other protocols) via Plugin.

Looking forward to some feedback to getting this feature (that will be needed more often in the future, as more and more email providers move away from plain SMTP) into Koha!
Comment 19 Tomás Cohen Arazi (tcohen) 2025-06-10 11:51:37 UTC
(In reply to Thomas Klausner from comment #18)
> We have a customer that will need XOAUTH support for mailing soon (as they
> are using MS365, and they will disable basic SMTP).
> 
> They might go with the external proxy solution
> (simonrob/email-oauth2-proxy), but they are actually interested in getting
> proper XOAUTH support into Koha. And are (maybe) willing to sponsor the
> development (depending on how much it will be).
> 
> We are currently preparing an offer for the following steps:
> 
> * First step: Add XOAUTH support to the Koha SMTP sender
>   * all the hard parts are already available on CPAN: LWP::Authen::OAuth2,
> Authen::SASL::Perl::XOAUTH2 and Email::Sender
>   * we "just" need to add some new config settings to SMTP to allow using an
> access-token (instead of username/password)
>   * we will also need to set up a cronjob to refresh the access tokens and
> the refresh token (better via cronjob, because if the mail sending happens
> infrequently even the refresh token could be invalidated in the interval
> where nothing happend)
>   * The XOAUTH setup (i.e. the OAuth handshake to login the service account
> and create a first refresh token) will be done via a command line script
> * Second step (optional):
>   * Add a GUI to the Staff interface to setup XOAUTH and do the OAuth
> handshake (so system librarians can do the setup without needing access to
> the shell)
>   * Probably do this via a plugin
> 
> The second step will be very likely outside the budget of the client, and
> even the first step might be too much. In that case we (HKS3) would try to
> get further funding by other interested libraries.
> 
> Of course we would very much like to get this changes into the Koha core or
> at least into a plugin, though AFAIK the Notifications-Area does not support
> any plugins/hooks as of now. We would need some feedback / discussion on how
> to actually add this feature to Koha: Just add it to Core, or add a way to
> Koha core to add this (and later other protocols) via Plugin.
> 
> Looking forward to some feedback to getting this feature (that will be
> needed more often in the future, as more and more email providers move away
> from plain SMTP) into Koha!

The token should be retrieved (if a refresh is needed) when it's going to be used. I don't think we need a cronjob here.

Making the API call to retrieve a fresh token, if it has expired or absent, could be done during the sending session.

I initially thought of implementing an Email::Sender::Transport class based on the persistent SMTP one. So all the session shares the same token and it gets retrieved in a more generic way. The caveat is you wouldn't have a 'Koha context' in such generic transport, thus not possible to store it on the DB, etc.

Excited someone can devote time to accomplish this.
Comment 20 David Cook 2025-06-11 02:01:14 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #19)
> The token should be retrieved (if a refresh is needed) when it's going to be
> used. I don't think we need a cronjob here.
> 
> Making the API call to retrieve a fresh token, if it has expired or absent,
> could be done during the sending session.

Theoretically yes but in practice if you're a system that rarely sends emails your refresh token might expire before you have a chance to send another email, and then your email authentication is broken until someone can re-configure it. That's why a cronjob can help keep your refresh token refreshing in the background, so that it's never expired. 

It's not quite clear how long refresh tokens live for major providers. I think it's 90 days for Microsoft, but no idea for Google. 
 
> Excited someone can devote time to accomplish this.

Agreed. Or rather... excited that multiple people are interested in this topic and willing to move forward on it.
Comment 21 David Cook 2025-06-11 02:46:27 UTC
(In reply to Thomas Klausner from comment #18)

> * First step: Add XOAUTH support to the Koha SMTP sender
>   * all the hard parts are already available on CPAN: LWP::Authen::OAuth2,
> Authen::SASL::Perl::XOAUTH2 and Email::Sender
>   * we "just" need to add some new config settings to SMTP to allow using an
> access-token (instead of username/password)
>   * we will also need to set up a cronjob to refresh the access tokens and
> the refresh token (better via cronjob, because if the mail sending happens
> infrequently even the refresh token could be invalidated in the interval
> where nothing happend)
>   * The XOAUTH setup (i.e. the OAuth handshake to login the service account
> and create a first refresh token) will be done via a command line script

> * Second step (optional):
>   * Add a GUI to the Staff interface to setup XOAUTH and do the OAuth
> handshake (so system librarians can do the setup without needing access to
> the shell)
>   * Probably do this via a plugin

In theory, I think that this all sounds good to me.

That said, I haven't tried the XOAUTH setup before, so that's the thing that I still need to wrap my head around. Still just looking around on Google at how to get the initial access token.

I find the run throughs like https://github.com/google/gmail-oauth2-tools/wiki/OAuth2DotPyRunThrough to be interesting because they use the "Installed application" type and not the "service account" type, but surely we'd want to use service accounts. 

Anyway, it's just this process that I still need to spend some time on...
Comment 22 David Cook 2025-06-11 03:05:14 UTC
Keycloak provides some useful info for Microsoft... https://www.keycloak.org/docs/26.2.5/server_admin/#xoauth2-email-configuration-with-third-party-vendors