Bug 23975

Summary: Add ability to search and install plugins from GitHub
Product: Koha Reporter: Kyle M Hall <kyle>
Component: Plugin architectureAssignee: Kyle M Hall <kyle>
Status: CLOSED FIXED QA Contact: Tomás Cohen Arazi <tomascohen>
Severity: new feature    
Priority: P5 - low CC: 1joynelson, caroline.cyr-la-rose, david, dcook, egpetridis, jonathan.druart, martin.renvoize, mtj, paul.derscheid, tomascohen, ujjwalguptax202
Version: master   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25672
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
This patch makes public plugins discoverable from within Koha itself via a search box at the top of the plugin management page. One can search for plugins and install them directly from their source. WARNING: Plugins are not yet verified by the community, use at your own risk. The feature is **disabled** by default; to enable it an administrator must uncomment, or add new lines to, the relevant configuration lines inside the `plugin_repos` config block within koha-conf.xml.
Version(s) released in:
20.05.00
Bug Depends on:    
Bug Blocks: 24631, 24632, 24634, 24635, 25952, 24633, 26751, 33776    
Attachments: Bug 23975: [PoC] Add ability to search and install plugins from GitHub
Bug 23975: Add ability to search and install plugins from GitHub
Bug 23975: Add 'Install' support for github results
Bug 23975: Add ability to search and install plugins from GitHub
Bug 23975: Add 'Install' support for github results
Bug 23975: (follow-up) Don't rely on save_to being present
Bug 23975: Add ability to search and install plugins from GitHub
Bug 23975: Add 'Install' support for github results
Bug 23975: (follow-up) Don't rely on save_to being present
Bug 23975: Add ability to search and install plugins from GitHub
Bug 23975: Add 'Install' support for github results
Bug 23975: (follow-up) Don't rely on save_to being present
Bug 23975: (QA follow-up) Fix stylingof install buttons
Bug 23975: Add ability to search and install plugins from GitHub
Bug 23975: Add 'Install' support for github results
Bug 23975: (follow-up) Don't rely on save_to being present
Bug 23975: (QA follow-up) Fix stylingof install buttons
Bug 23975: (QA follow-up) Search button style and functionality
Bug 23975: (QA follow-up) Add missing TT filters

Description Kyle M Hall 2019-11-05 17:30:20 UTC
It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.
Comment 1 Kyle M Hall 2019-11-05 17:32:28 UTC
Created attachment 95067 [details] [review]
Bug 23975: [PoC] Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!
Comment 2 Martin Renvoize 2019-11-08 16:29:31 UTC
This is an awesome PoC :)

Wish list wise I'd love to see.

1) Metadata that's currently embedded inside the plugins main class pulled out into a json (or yaml) file in the repository as a standard so we could read that and get things like supported versions etc into this.
2) Perhaps some 'Changelog.md' or 'Readme.md' support somehow at some point
3) Support for gitlab added.

But, this is a great first proof we can refine :)
Comment 3 Martin Renvoize 2019-11-08 17:09:18 UTC
Going further with the moving metadata outside of the file, this could allow us to adopt signed manifest approach and make our plugins that bit more secure and reliable.

Let's give this a real push for 20.05 and give the best plugins another bug lift in that release. That should give us ample time to work it out.
Comment 4 Tomás Cohen Arazi 2019-12-30 10:58:52 UTC
(In reply to Martin Renvoize from comment #2)
> This is an awesome PoC :)
> 
> Wish list wise I'd love to see.
> 
> 1) Metadata that's currently embedded inside the plugins main class pulled
> out into a json (or yaml) file in the repository as a standard so we could
> read that and get things like supported versions etc into this.
> 2) Perhaps some 'Changelog.md' or 'Readme.md' support somehow at some point
> 3) Support for gitlab added.
> 
> But, this is a great first proof we can refine :)

And Gitlab support :-D
Comment 5 Martin Renvoize 2020-02-11 16:59:40 UTC
Lets get this one moving again.. lots of great enhancements suggested, but lets start with this bug and build a tree of enhancements later.

So.. I think all that's missing is the 'install' action.. I'll take a look
Comment 6 Martin Renvoize 2020-02-12 14:43:45 UTC
Created attachment 98757 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 7 Martin Renvoize 2020-02-12 14:43:47 UTC
Created attachment 98758 [details] [review]
Bug 23975: Add 'Install' support for github results
Comment 8 David Nind 2020-02-13 05:24:01 UTC
The search worked fine!

Should I be able to install a plugin yet? I can install plugins the standard way.

This is the start of the error message I got clicking the Install button from the search results:

Can't locate object method "save_to" via package "Mojo::Message::Response" at /kohadevbox/koha/plugins/plugins-upload.pl line 78
in (eval) at /kohadevbox/koha/plugins/plugins-upload.pl line 78
   75:         if ( $uploadlocation ) {
   76:             my $ua = Mojo::UserAgent->new(max_redirects => 5);
   77:             my $tx = $ua->get($uploadlocation);
   78:             $tx->result->save_to($tempfile);
   79:         } else {
   80:             $errors{'EMPTYUPLOAD'}    = 1 unless ( length($uploadfile) > 0 );
   81:         }
in CGI::Compile::ROOT::kohadevbox_koha_plugins_plugins_2dupload_2epl::__ANON__ at /kohadevbox/koha/plugins/plugins-upload.pl line 2
Comment 9 Martin Renvoize 2020-02-13 07:06:02 UTC
Ah, 'save_to' was added to mojo in version 8.02.   I'll have a look at how we can support this without using that call.
Comment 10 Martin Renvoize 2020-02-13 08:50:14 UTC
Created attachment 98791 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 11 Martin Renvoize 2020-02-13 08:50:18 UTC
Created attachment 98792 [details] [review]
Bug 23975: Add 'Install' support for github results
Comment 12 Martin Renvoize 2020-02-13 08:50:21 UTC
Created attachment 98793 [details] [review]
Bug 23975: (follow-up) Don't rely on save_to being present

The `save_to` shortcut method was introduced to Mojolicious in version
8.02 but we still support 7.21 as our minimum dependancy and as such we
cannot yet use it.
Comment 13 Martin Renvoize 2020-02-13 08:54:42 UTC
The followup should get Install working more consistently now.. thanks for testing David.
Comment 14 David Nind 2020-02-13 10:06:03 UTC
Created attachment 98800 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: David Nind <david@davidnind.com>
Comment 15 David Nind 2020-02-13 10:06:07 UTC
Created attachment 98801 [details] [review]
Bug 23975: Add 'Install' support for github results

Signed-off-by: David Nind <david@davidnind.com>
Comment 16 David Nind 2020-02-13 10:06:10 UTC
Created attachment 98802 [details] [review]
Bug 23975: (follow-up) Don't rely on save_to being present

The `save_to` shortcut method was introduced to Mojolicious in version
8.02 but we still support 7.21 as our minimum dependancy and as such we
cannot yet use it.

Signed-off-by: David Nind <david@davidnind.com>
Comment 17 David Nind 2020-02-13 10:07:32 UTC
You may also need to add this to your Apache configuration (in the staff interface section - this folder location works for koha-testing-docker):

<Directory "/var/lib/koha/kohadev/plugins">
Require all granted
</Directory>
Alias /plugin "/var/lib/koha/kohadev/plugins/"
Comment 18 Kyle M Hall 2020-02-13 11:31:23 UTC
Created attachment 98813 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 19 Kyle M Hall 2020-02-13 11:31:31 UTC
Created attachment 98814 [details] [review]
Bug 23975: Add 'Install' support for github results

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 20 Kyle M Hall 2020-02-13 11:31:34 UTC
Created attachment 98815 [details] [review]
Bug 23975: (follow-up) Don't rely on save_to being present

The `save_to` shortcut method was introduced to Mojolicious in version
8.02 but we still support 7.21 as our minimum dependancy and as such we
cannot yet use it.

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 21 Jonathan Druart 2020-02-26 15:07:15 UTC
My feeling is that we should maintain a list of "accepted"/"validated" plugins, and so have a WS serving (on a Koha community server) that list.
Comment 22 Martin Renvoize 2020-02-28 16:19:11 UTC
(In reply to Jonathan Druart from comment #21)
> My feeling is that we should maintain a list of "accepted"/"validated"
> plugins, and so have a WS serving (on a Koha community server) that list.

I did initially think the same and intended to write such a service.. however I have come around to this approach as it reduces the need build, host and maintain another system, not to mention maintaining the data.

I have added followup bugs that request we add signatures to plugin releases and a way from inside Koha itself to validate those.

Once we have those I think we pretty much have what you are suggesting.. we can update the default config we ship to only include a pointer to the community gitlab and as we as a community deem a plugin to have reached a level of maturity we are happy with we can suggest an adoption of it into our gitlab and thus sign subsequent releases.

Hope that makes sense.
Comment 23 Kyle M Hall 2020-03-02 17:49:59 UTC
> Once we have those I think we pretty much have what you are suggesting.. we
> can update the default config we ship to only include a pointer to the
> community gitlab and as we as a community deem a plugin to have reached a
> level of maturity we are happy with we can suggest an adoption of it into
> our gitlab and thus sign subsequent releases.

+1
Comment 24 Martin Renvoize 2020-03-02 18:58:50 UTC
Created attachment 99958 [details] [review]
Bug 23975: (QA follow-up) Fix stylingof install buttons
Comment 25 Tomás Cohen Arazi 2020-03-03 03:10:09 UTC
Created attachment 99970 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 26 Tomás Cohen Arazi 2020-03-03 03:10:14 UTC
Created attachment 99971 [details] [review]
Bug 23975: Add 'Install' support for github results

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 27 Tomás Cohen Arazi 2020-03-03 03:10:18 UTC
Created attachment 99972 [details] [review]
Bug 23975: (follow-up) Don't rely on save_to being present

The `save_to` shortcut method was introduced to Mojolicious in version
8.02 but we still support 7.21 as our minimum dependancy and as such we
cannot yet use it.

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 28 Tomás Cohen Arazi 2020-03-03 03:10:22 UTC
Created attachment 99973 [details] [review]
Bug 23975: (QA follow-up) Fix stylingof install buttons

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 29 Tomás Cohen Arazi 2020-03-03 03:10:26 UTC
Created attachment 99974 [details] [review]
Bug 23975: (QA follow-up) Search button style and functionality

This patch makes the search form button match the other buttons style,
and also adds the submit() action on click.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 30 Tomás Cohen Arazi 2020-03-03 03:10:42 UTC
Created attachment 99975 [details] [review]
Bug 23975: (QA follow-up) Add missing TT filters

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 31 Jonathan Druart 2020-03-03 08:47:07 UTC
I am sorry but I have concerns about this patch. Feel free to ignore them.

To me this feature is not ready as it.
With those patches, the default will be to search on theke and bws github repo. Nothing says that the plugins are fetched from "outside".
Did anyone review those plugins?
Why only those 2 companies? What about others (ptfs-e, biblibre, inlibro, etc)?
What if a plugin that have security issues is pushed to those repos?

When are you going to work on follow-up bugs? Is the plan to have them ready for 20.05?
Comment 32 Martin Renvoize 2020-03-03 09:23:03 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 33 Martin Renvoize 2020-03-03 09:39:56 UTC
I intend to start working on those bugs, and encourage Kyle and Tomas to help very soon.. I've left the default config in on master for now so that developers get exposed to the feature whilst we're in development, but I intend to drop those as defaults just before release... hopefully to have been replaced with a default of the community gitlab with hopefully at least a few reviewed plugins within it.
Comment 34 Joy Nelson 2020-04-01 21:12:44 UTC
new feature not backported to 19.11.x
Comment 35 David Cook 2020-04-30 23:09:47 UTC
(In reply to Jonathan Druart from comment #31)
> I am sorry but I have concerns about this patch. Feel free to ignore them.
> 
> To me this feature is not ready as it.
> With those patches, the default will be to search on theke and bws github
> repo. Nothing says that the plugins are fetched from "outside".
> Did anyone review those plugins?
> Why only those 2 companies? What about others (ptfs-e, biblibre, inlibro,
> etc)?
> What if a plugin that have security issues is pushed to those repos?
> 

I have those same concerns. At the very least, there should be transparency about where the plugins are being fetched from.

> When are you going to work on follow-up bugs? Is the plan to have them ready
> for 20.05?

I've done the plugin signing work on https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24632, but I only did it for manually uploaded plugins. I only noticed this bug's functionality as I was wrapping up my initial work. I'll have to revisit it...
Comment 36 David Cook 2020-07-08 10:35:43 UTC
I'm noticing that the plugin_repos isn't appearing in /etc/koha/sites/kohadev/koha-conf.xml in koha-testing-docker.

I'm guessing that it just hasn't been added into a Docker image yet?
Comment 37 David Cook 2020-07-08 10:46:08 UTC
Trying this out for the first time and I really like the idea of fetching plugins from a repository and installing them, but I don't like how you have to know exactly what you're looking for.

I think it would be nicer if you could get a list of all available plugins.

Hmm searching for '%' or 'koha-plugin' is generating errors. I'll create a new issue for that.
Comment 38 David Cook 2020-07-08 10:52:06 UTC
Wow... looks like it may have been due to rate limiting. You're supposed to get 60 requests per hour. I don't think that I've done that many...
Comment 39 David Cook 2020-07-08 10:55:53 UTC
Interestingly, I can do lots of queries to the search API...

https://api.github.com/search/repositories?q=coverflow+user:bywatersolutions+in:name,description

But I am blocked for this:

https://api.github.com/repos/bywatersolutions/koha-plugin-coverflow/releases/latest

I was just messing around for 5 minutes.

This seems like it would never be feasible for production?
Comment 40 David Cook 2020-07-08 11:17:06 UTC
I'll stop spamming in a moment...

I'm actually thinking a bit about making a plugin that manages plugins.

I was inspired by the "Game Package Registry for Unity". It's a package you can add to the Unity IDE, and it lets you then import all Google packages into Unity IDE without having to manually find, download, and import into Unity IDE...
Comment 41 Vihaan Gupta 2020-10-28 11:12:02 UTC
i am not able to find usekohaplugin in koha version 20.05 in system perfrences can any one tell me where to i have enable in new koha version?????
Comment 42 Kyle M Hall 2020-10-28 11:21:08 UTC
(In reply to Vihaan Gupta from comment #41)
> i am not able to find usekohaplugin in koha version 20.05 in system
> perfrences can any one tell me where to i have enable in new koha
> version?????

That system preference was removed by bug 20415
Comment 43 Manos PETRIDIS 2021-11-18 21:56:20 UTC
Is the "new block for plugin_repos" described somewhere? I see that the the system preference for plugins has been removed, but existing koha-conf.xmls have not been updated with the new block, even in commented form. Where is one to find it?
Comment 44 Jonathan Druart 2021-11-19 10:14:09 UTC
(In reply to Manos PETRIDIS from comment #43)
> Is the "new block for plugin_repos" described somewhere? I see that the the
> system preference for plugins has been removed, but existing koha-conf.xmls
> have not been updated with the new block, even in commented form. Where is
> one to find it?

Hi Manos, it's in the source:

https://git.koha-community.org/Koha-community/Koha/src/commit/7bddfeb412e427ee8c02c700295382fe4a7340d6/debian/templates/koha-conf-site.xml.in#L406
Comment 45 Paul Derscheid 2022-08-26 09:36:11 UTC
The patches don't apply on the current master branch anymore. Lots of merge conflicts.
Comment 46 Tomás Cohen Arazi 2022-08-26 11:48:32 UTC
(In reply to Paul Derscheid from comment #45)
> The patches don't apply on the current master branch anymore. Lots of merge
> conflicts.

This is already in the codebase since 2020.

Best regards