Bug 8897

Summary: Optional GnuPG encryption of outgoing emails
Product: Koha Reporter: Mirko Tietgen <mirko>
Component: NoticesAssignee: Mirko Tietgen <mirko>
Status: RESOLVED WISHLIST QA Contact:
Severity: enhancement    
Priority: P5 - low CC: dcook, gitbot, gmcharlt, mfuerst, mirko, robin, veron, wizzyrea
Version: master   
Hardware: All   
OS: All   
See Also: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15303
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15540
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 8897 [ENH] Optional GnuPG encryption of outgoing emails
Bug 8897 [ENH] Optional GnuPG encryption of outgoing emails
Bug 8897 [ENH] Optional GnuPG encryption of outgoing emails
Bug 8897 [ENH] Optional GnuPG encryption of outgoing emails
Bug 8897 [ENH] Optional GnuPG encryption of outgoing emails
Bug 8897 [ENH] Followup: add Icons to indicate encryption
Bug 8897 [ENH] GPG Mail encryption -- New version
Bug 8897 [ENH] GPG Mail encryption -- New version
Bug 8897 [ENH] GPG Mail encryption -- New version
Bug 8897 [ENH] GPG Mail encryption -- New version
Bug 8897 Follow-up: add Crypt:GPG dependency
Bug 8897 [ENH] GPG Mail encryption -- New version
Bug 8897 [ENH] GPG Mail encryption -- New version
Bug 8897 [ENH] GPG Mail encryption -- New version
Bug 8897 [ENH] GPG Mail encryption -- New version

Description Mirko Tietgen 2012-10-10 18:23:19 UTC
Koha should have an option to encrypt outgoing emails with GnuPG.
Comment 1 Mirko Tietgen 2012-10-10 18:33:25 UTC Comment hidden (obsolete)
Comment 2 Mirko Tietgen 2012-10-10 18:52:49 UTC
To try the initial patch you would need

- install gnupg and libcrypt-gpg-perl
- add a keypair to GnuPG for your KohaAdminEmailAddress
- add a key for your test patron (or use the same email as KohaAdminEmailAddress)
- set EnhancedMessagingPreferences to 'allow'
- set messaging preferences for your test patron to send emails for check-in and checkout
- set GnuPG prefs:
    GnuPGPassphrase is the passphrase for your admin key, also enables the feature
    GnuPGPath is the path to gnupg. Standard on Debian is default: '/usr/bin/gpg'
- check one thing out
- check SQL: SELECT * FROM message_queue WHERE to_address = <testpatronemailaddress>;
- observe that 'content' is encrypted, 'metadata' (next field) shows one entry (encryption of add)
- check in and out more stuff
- check SQL again, observe that 'content' is still encrypted while 'metadata' is gettign filled with more items checked in and checked out (encryption of append)
Comment 3 Mirko Tietgen 2012-10-10 19:08:28 UTC
I forgot to say you need to add the keys as the Koha linux user (like koha-koha in a package installation) to be available to Koha. Also your admin key must be trusted. You could just copy over your .gnupg folder.

Or you can do it the hard way:

export keypair from your installation, copy to a file. Add on the command line with

'gpg --import filename'

then get it's 'long key id' with

'gpg --list-keys --with-colons <key_id>'

and trust it with

'gpg --trustedkey <long key id>'

Or you wait until Koha can do that ;)
Comment 4 Mirko Tietgen 2012-10-21 19:29:08 UTC Comment hidden (obsolete)
Comment 5 Mirko Tietgen 2012-10-21 19:38:48 UTC Comment hidden (obsolete)
Comment 6 Mirko Tietgen 2012-10-21 19:43:31 UTC Comment hidden (obsolete)
Comment 7 Mirko Tietgen 2012-12-08 01:01:46 UTC Comment hidden (obsolete)
Comment 8 Mirko Tietgen 2012-12-08 01:08:52 UTC Comment hidden (obsolete)
Comment 9 Robin Sheat 2012-12-13 22:43:25 UTC
This is interesting and I like it.

However, why is it necessary to set a passphrase? To encrypt a message to a person, all you need is their key, you don't need your own one.

To sign a message, you need a passphrase, but it's not documented as to whether that's happening or not.

Also, the C4::GnuPG module should probably be moved to the Koha:: namespace and made nice and OO-ey.

I'd like to see (in the future, this is a good beginning) the ability for users to attach their own keys.
Comment 10 Mirko Tietgen 2012-12-13 23:34:51 UTC
Hey Robin,

thanks a lot for your comment. Good point about the passphrase, that seems worth considering. I am not too happy about putting it in the database.

Some form of key management (let users enter their public keys) is supposed to be the next thing I'll be doing when I have some time to work on this.
Comment 11 Galen Charlton 2013-07-03 20:35:39 UTC
(In reply to Robin Sheat from comment #9)
> This is interesting and I like it.

Question: is there any evidence that there actually exist patron communities that care enough to have actively requested encryption of outgoing library emails?

I understand the reasons for adding this feature, particularly in light of recent political events, but I must confess to some doubt as to whether this would be used by anybody other than other Koha hackers.

My question shouldn't be construed as a blocker.
Comment 12 Mirko Tietgen 2013-07-03 21:11:10 UTC
Hi Galen,

thanks for having a look at this bug.

I do not know of communities requesting this so far. But
- as a library user, I would like to have the option
- as a Koha developer, I would like to be able to offer people the option

What I would like to look into once I have the time (will take a little while unfortunately) is
- possibly change to openpgpjs (http://openpgpjs.org/)
- see if outgoing encryption without a library key (as discussed above) is feasible

That would make it possible for the individual user to decide if s/he prefers to receive encrypted emails without any interaction with or setup needs of the library (no key generation etc.). -- Just upload your public key to opt in.

I do have libraries in mind that I assume would like to offer this feature to their users, but since this is still in an early stage I have not approached them regarding testing yet.
Comment 13 Robin Sheat 2013-07-04 09:50:30 UTC
I don't think you could use a JS library for this unless you ran the JS server-side. You can't send an email to their browser, have it encrypt it, and send it back.

As for the passphrase stuff, you definitely can encrypt without a passphrase:
$ gpg -a --encrypt --trust-model always -r 4BE030B3
This is a message.
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.12 (GNU/Linux)

hQIOA07aRfuO76ueEAgAnjpAVlVtyfOpEUSest9ESwNDkwANu6DGNBEygkClBmDj
ACCU22Yx8IIgV6zPfnT0W4rfndR3oUpel9mBnmGOFLKHPEY+FlZbCXtzHj6jqMGA
...

You could have the server having its own passphrase if you wanted to sign things, but I'd consider that a less important thing, at least initially.

I think as libraries tend to be fairly privacy-minded, this is something that they probably don't know they want. It's also a good argument against other software, "unlike that one, Koha really cares about user's privacy!" :)
Comment 14 Mirko Tietgen 2013-07-04 10:09:01 UTC
I was thinking to have the JS server-side, like we have jquery in Koha. Would that be a problem? I think it does not break the "OPAC needs to work without JS" rule as it is not a core functionality. We allow local JS in the OPAC too.

About using a library key/passphrase, I'm not sure if you would have situations where you need to decrypt messages, add stuff and encrypt them again before sending the email. I vaguely remember that is done, have not looked at the code in a while. You should be able to circumvent that with encryption at the very last moment of course if it does not matter the messages are stored unencrypted in the DB until then.
Comment 15 Robin Sheat 2013-07-04 10:13:17 UTC
(In reply to Mirko Tietgen from comment #14)
> I was thinking to have the JS server-side, like we have jquery in Koha.

But jquery runs client side. It doesn't make sense to run this client side, as email generation happens without the interaction of a browser.

> Would that be a problem? I think it does not break the "OPAC needs to work
> without JS" rule as it is not a core functionality. We allow local JS in the
> OPAC too.

I just can't see a place where encryption from the OPAC is something that's generally useful. We don't send emails from there.

> About using a library key/passphrase, I'm not sure if you would have
> situations where you need to decrypt messages, add stuff and encrypt them
> again before sending the email. 

No, once they're added to the message queue, I don't think anything more happens to them. They wouldn't be encrypted while working their way through the running process, they'd presumably get encrypted just before putting them in the queue (but I haven't looked at your code in forever.)
Comment 16 Mirko Tietgen 2013-07-04 10:21:42 UTC
(In reply to Robin Sheat from comment #15)
> (In reply to Mirko Tietgen from comment #14)
> > I was thinking to have the JS server-side, like we have jquery in Koha.
> 
> But jquery runs client side. It doesn't make sense to run this client side,
> as email generation happens without the interaction of a browser.

True. I will look into that when i have more time to see if it is possible. I have only used the JS an afternoon so far.

> 
> > Would that be a problem? I think it does not break the "OPAC needs to work
> > without JS" rule as it is not a core functionality. We allow local JS in the
> > OPAC too.
> 
> I just can't see a place where encryption from the OPAC is something that's
> generally useful. We don't send emails from there.

Oh you are right of course, the only thing that needs to be done from the OPAC is to offer patrons a field to enter the public key. My bad.

> 
> > About using a library key/passphrase, I'm not sure if you would have
> > situations where you need to decrypt messages, add stuff and encrypt them
> > again before sending the email. 
> 
> No, once they're added to the message queue, I don't think anything more
> happens to them. They wouldn't be encrypted while working their way through
> the running process, they'd presumably get encrypted just before putting
> them in the queue (but I haven't looked at your code in forever.)

I think there is an update sometimes when the same action is repeated, to only send one email about it covering all instances of that actions.
Comment 17 Robin Sheat 2013-07-04 11:10:04 UTC
(In reply to Mirko Tietgen from comment #16)
> I think there is an update sometimes when the same action is repeated, to
> only send one email about it covering all instances of that actions.

Ah right, I assume that it doesn't go and read the email back though, because that'd be weird. Really, it'd be best if it put events in to a queue and when the newest was 15 minutes old or more, it batched them into a digest and sent them. That might be what it does, I don't know. A conversation for a different bug, anyway.
Comment 18 Liz Rea 2013-07-05 06:54:36 UTC
Yes, the issue and return emails function on a delay - they could not be encrypted until they were leaving the server, as they wouldn't be complete yet.
Comment 19 Mirko Tietgen 2013-07-20 18:19:32 UTC Comment hidden (obsolete)
Comment 20 Mirko Tietgen 2013-07-20 21:32:58 UTC Comment hidden (obsolete)
Comment 21 Robin Sheat 2013-07-21 09:24:51 UTC
The path shouldn't be a syspref, it doesn't make sense. Library staff shouldn't be touching it.
Comment 22 Mirko Tietgen 2013-07-21 11:10:08 UTC Comment hidden (obsolete)
Comment 23 Mirko Tietgen 2013-07-21 11:18:09 UTC
Robin: I was thinking of a case where the binary is in another place for some reasons. I agree that staff should not interfere with it. What other options are there? Make the Sysadmin put a symlink?

Found the problem I had (needed to allow encryption with untrusted keys) and fixed some other small errors, I think it is ready for the first tests.
Comment 24 Mirko Tietgen 2013-07-21 11:24:16 UTC Comment hidden (obsolete)
Comment 25 Mirko Tietgen 2013-07-21 13:13:03 UTC Comment hidden (obsolete)
Comment 26 Robin Sheat 2013-07-21 23:09:08 UTC
The best option would probably be to search $PATH.


For dependencies, don't forget debian/control.in.
Comment 27 Mirko Tietgen 2013-07-23 14:37:30 UTC Comment hidden (obsolete)
Comment 28 Mirko Tietgen 2013-07-23 22:08:38 UTC
You can also test the encryption of subscription alerts if you apply bug 10621 before this patch. See description there to trigger a subscription alert.
Comment 29 Jared Camins-Esakov 2013-09-04 19:42:13 UTC
There is a small conflict in C4::Letter that I am not sure how to resolve. Does the from e-mail address have to be the patron's e-mail address for GPG to work? It seems to me we'd prefer it to be the library's e-mail address, if one is set, but I presume there's a reason for the change in the patch, so I wanted to ask for feedback on it.
Comment 30 Jared Camins-Esakov 2013-09-04 19:46:29 UTC
Additionally, it looks like there are no unit tests. Is it possible to unit test C4::GnuPG?

Finally, and this is a definite blocker, it looks like C4::GnuPG is initialized with file-level my variables:

+my $gpg = new Crypt::GPG;
+$gpg->gpgbin('/usr/bin/gpg');
+$gpg->encryptsafe(0); # we have to allow untrusted keys

This will not work well with persistent environments like Plack. The my could be changed to our, but even better would be changing the way initialization is handled such that a file-level variable is unnecessary.
Comment 31 Mirko Tietgen 2013-09-04 19:52:47 UTC
(In reply to Jared Camins-Esakov from comment #29)
> There is a small conflict in C4::Letter that I am not sure how to resolve.
> Does the from e-mail address have to be the patron's e-mail address for GPG
> to work? It seems to me we'd prefer it to be the library's e-mail address,
> if one is set, but I presume there's a reason for the change in the patch,
> so I wanted to ask for feedback on it.

I found the problem with the wrong address while I was working on this and fixed it in Bug 10621 instead of here. I had initially fixed it in this bug and then un-did it to have the seperate entry. I will change the patch when I have a little more time, the old behaviour should not be reimplemented by this patch.
Comment 32 Mirko Tietgen 2013-09-19 12:01:32 UTC Comment hidden (obsolete)
Comment 33 Mirko Tietgen 2013-09-19 12:04:10 UTC
> Finally, and this is a definite blocker, it looks like C4::GnuPG is
> initialized with file-level my variables:
> 
> +my $gpg = new Crypt::GPG;
> +$gpg->gpgbin('/usr/bin/gpg');
> +$gpg->encryptsafe(0); # we have to allow untrusted keys
> 
> This will not work well with persistent environments like Plack. The my
> could be changed to our, but even better would be changing the way
> initialization is handled such that a file-level variable is unnecessary.

The initialization id required by the perl dependency. I have changed it to our. I have also rebased and resolved the conflict in Letters.pm.
Comment 34 Mirko Tietgen 2013-09-19 12:04:55 UTC
"The initialization is required"… (typo)
Comment 35 I'm just a bot 2013-09-24 05:46:26 UTC
Patch applied cleanly, go forth and signoff
Comment 36 I'm just a bot 2013-10-24 19:32:36 UTC
Applying: Bug 8897 [ENH] GPG Mail encryption -- New version
Using index info to reconstruct a base tree...
M	C4/Installer/PerlDependencies.pm
M	C4/Letters.pm
M	debian/control
M	install_misc/debian.packages
M	opac/opac-sendshelf.pl
Falling back to patching base and 3-way merge...
Auto-merging opac/opac-sendshelf.pl
CONFLICT (content): Merge conflict in opac/opac-sendshelf.pl
Auto-merging install_misc/debian.packages
Auto-merging debian/control
CONFLICT (content): Merge conflict in debian/control
Auto-merging C4/Letters.pm
CONFLICT (content): Merge conflict in C4/Letters.pm
Auto-merging C4/Installer/PerlDependencies.pm
CONFLICT (content): Merge conflict in C4/Installer/PerlDependencies.pm
Patch failed at 0001 Bug 8897 [ENH] GPG Mail encryption -- New version
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Comment 37 David Cook 2014-01-09 23:21:42 UTC
I think this would be a great addition to Koha.

As Galen mentioned, I'm not sure how many libraries would want this, but as Robin (I think) mentioned, they probably just don't know they want this yet.

I suppose I rather Koha be a forerunner rather than playing catch-up when it comes to encryption.

P.S. It does give me an excuse to finally play with GPG...
Comment 38 Mark Tompsett 2014-04-19 03:51:15 UTC Comment hidden (obsolete)
Comment 39 Mirko Tietgen 2015-07-10 12:49:43 UTC
(In reply to M. Tompsett from comment #38)

> NOTE: I rebased this so it applies nicely. I'm setting into
> 'In Discussion', so the original author can rebase himself or
> confirm this is good.

Thanks Mark for the rebase! I wasn't aware you did that and wondered why my rebase was much easier than expected now :) I will add an up-to-date version with changes for bootstrap once I figure out how to make git bz and the new bugzilla version friends again, can't submit anything atm.
Comment 40 Mirko Tietgen 2016-01-14 08:44:47 UTC
Created attachment 46640 [details] [review]
Bug 8897 [ENH] GPG Mail encryption -- New version

Koha should offer patrons the option to receive their emails encrypted.
This patch adds a dialog to the user interface (OPAC side) for patrons to add or delete a GPG public key. The key is directly added to or deleted from gpg (gnupg needs to be installed, binary expected at /usr/bin/gpg).

This feature does not require sysadmins/librarians to do anything as long as the gpg binary if found.

So far, mails are encrypted if a key for the recipient is avaliable
- when sending lists or carts (encryption of email text + attachment)
- when mails are sent through the message queue
- ...

Missing so far:
- hide public key interface from OPAC/userdetails when no binary is found
- de-hardcode path to gpg (syspref)
- deal with BCC mails (send message explaining that an encrypted email has been send to the patron instead of a BCC)
- ...

Maybe later:
- library-side key management: secret key for signing
- encryption of emails from library to vendors
- ...

Test plan:

-apply patch

* Add/delete key:
- in OPAC, log in as a user you got GPG kes for (primary email needs to match the key)
- go to user details, email encryption
- paste your public key into the form, save. when the page reloads, your key should be displayed in the form and a delete button at the bottom. in the terminal (as the koha-user!), do 'gpg --list-keys' and verify the key you just added is present
- press the delete button under the form. on reload the form should be empty and your key gone. verify with (as the koha user) 'gpg --list-keys' in the terminal.

- paste your key into the form again and save to do the following tests.

* Encrypted cart emails
- do a search, save a record in the the cart. open the cart, send it to the adress you added the key for.
- check if the mail you receive has encrypted text and an encrypted attachment
- decrypt both the text and the attachment, check if they are what you would expect

* Encrypted list emails
- if you do not have any lists, set up one with one record.
- open the list from the opac, send it to you. enter the address you just added the key for
- check that the mail you receive has encrypted text and an encrypted attachment.
- decrypt both the text and the attachment, check if they are what you would expect

* Encrypted message queue emails
- check some items in and out for the patron you added the key for
- wait for the message queue cronjob to run or run it manually
- check if the email(s) you receive are encrypted and after decryption contain what you would expect

Check all this again with another patron (without a gpg key) or use the same but delete the key before. Check that all mails and attachments are unencrypted and contain what you would expect (nothing is broken).

If all that works for you, sign off the patch.

Squashed a few minor changes.

Bug 8897 Follow-up: plack fix

NOTE: I rebased this so it applies nicely. I'm setting into
'In Discussion', so the original author can rebase himself or
confirm this is good. I found the following:

- Bug 8368 (only a couple days after the last rebase) patch
(recoded the line, for the first conflict -- New version is correct)
(renamed shelf to list, for the second conflict -- Second version is this patch)
- Bug 11124 triggers a conflict in C4/Installer/PerlDependencies.pm
- Bug 9611 also affected C4/Installer/PerlDependencies.pm and debian/control
- Bug 5544 added code which caused a conflict for C4/Letter.
Comment 41 Mirko Tietgen 2016-01-14 08:46:46 UTC
Rebased. I will look into this soon, probably make a few changes and set it back to needs signoff
Comment 42 Mirko Tietgen 2016-01-30 19:41:34 UTC
Working on this. Ported the template to Bootstrap, was broken before. Struggling with PGP design flaws. If this is supposed to work without any knowledge on the librarian side, there needs to be a way to prevent people from "stealing a patrons identity".

Right now, patron uploads key, key will be used to encrypt emails to patron. However, the connection 'patron' => 'key' is done by gpg:

- patron uploads public key
- (if previous key for email address exists, it is deleted)
- key is stored in gpg (keyring of the koha system user)

Let's say an attacker generates a key pair with patron's email address in the UID and uploads it to the attacker's own user account. Patron's old key gets deleted, the new one is used for patron's account.

Patron will find out once they cannot open their emails from the library. If the attacker managed to get the email(s) up to this point, they are able to open them, because they got the relevant private key.

There needs to be a way to connect patron and key in Koha. The easiest and best for our use case would be to have the public keys in the Koha DB only and encrypt emails using what the DB has, without the gpg key management. Unfortunately that does not seem to be possible at all, at least I could not find it yet. I mean something like
"gpg, encrypt text x with public key $public_key_as_string_from_db"

Workarounds either need interference on the librarian side or prevent updating patron's public key. I am still hoping to get this working for libraries with zero interest in crypto. I will look into what could be done with some kind of Key ID verification by the patron, but it would be overly complicated compared to storing keys in the db per patron.

Sorry, just thinking aloud. If you accidentally receive this transmission and think I am missing something, feel free to interrupt.
Comment 43 Mirko Tietgen 2016-01-31 11:56:45 UTC
I will change the patch to store a keyid per user email address in Koha and make GnuPG.pm use that for all relevant operations.
Comment 44 David Cook 2016-04-12 00:08:57 UTC
(This is pretty unrelated to this bug, but I was randomly looking at the CPAN GPG modules one day and found that they all seem to use IPC to use the GPG binary itself, which seems suboptimal... so I started thinking how it would be interesting to write Perl bindings for GPG via GPGME [ttps://www.gnupg.org/related_software/swlist.html#gpgme], but sadly I haven't had the time. I've been looking a lot more at XS over the past year and C over the past month. It would be a fun pet project, but alas...time.)
Comment 45 David Cook 2017-09-21 00:05:17 UTC
Another unrelated note... I wonder if there's anything we could do to help people use TLS for email. Alas, Let's Encrypt doesn't provide certificates for email, so that doesn't help. I think maybe it is all up to the sysadmin on the final system to set that up...

But that's a layer of security that I could see all libraries using quite readily.
Comment 46 Martha Fuerst 2018-03-14 17:40:13 UTC
Huntsville-Madison County is interested in this as an added layer of security.
Comment 47 David Cook 2018-03-15 00:48:15 UTC
(In reply to David Cook from comment #45)
> Another unrelated note... I wonder if there's anything we could do to help
> people use TLS for email. Alas, Let's Encrypt doesn't provide certificates
> for email, so that doesn't help. I think maybe it is all up to the sysadmin
> on the final system to set that up...
> 
> But that's a layer of security that I could see all libraries using quite
> readily.

Ohhh... looks like I misunderstood the Let's Encrypt certificate thing: https://community.letsencrypt.org/t/how-to-get-certificate-for-email-server/50702/2

So we could be using TLS for email.
Comment 48 David Cook 2022-08-22 02:48:32 UTC
Mirko disappeared from Koha a long time ago, so I don't think he's going to revisit these patches. 

It's a really interesting idea, but realistically I don't think that it would ever be used in production.