Created attachment 95067 [details] [review]
Bug 23975: [PoC] Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

This is an awesome PoC :)

Wish list wise I'd love to see.

1) Metadata that's currently embedded inside the plugins main class pulled out into a json (or yaml) file in the repository as a standard so we could read that and get things like supported versions etc into this.
2) Perhaps some 'Changelog.md' or 'Readme.md' support somehow at some point
3) Support for gitlab added.

But, this is a great first proof we can refine :)

Going further with the moving metadata outside of the file, this could allow us to adopt signed manifest approach and make our plugins that bit more secure and reliable.

Let's give this a real push for 20.05 and give the best plugins another bug lift in that release. That should give us ample time to work it out.

(In reply to Martin Renvoize from comment #2)
> This is an awesome PoC :)
> 
> Wish list wise I'd love to see.
> 
> 1) Metadata that's currently embedded inside the plugins main class pulled
> out into a json (or yaml) file in the repository as a standard so we could
> read that and get things like supported versions etc into this.
> 2) Perhaps some 'Changelog.md' or 'Readme.md' support somehow at some point
> 3) Support for gitlab added.
> 
> But, this is a great first proof we can refine :)

And Gitlab support :-D

Lets get this one moving again.. lots of great enhancements suggested, but lets start with this bug and build a tree of enhancements later.

So.. I think all that's missing is the 'install' action.. I'll take a look

Created attachment 98757 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Created attachment 98758 [details] [review]
Bug 23975: Add 'Install' support for github results

The search worked fine!

Should I be able to install a plugin yet? I can install plugins the standard way.

This is the start of the error message I got clicking the Install button from the search results:

Can't locate object method "save_to" via package "Mojo::Message::Response" at /kohadevbox/koha/plugins/plugins-upload.pl line 78
in (eval) at /kohadevbox/koha/plugins/plugins-upload.pl line 78
   75:         if ( $uploadlocation ) {
   76:             my $ua = Mojo::UserAgent->new(max_redirects => 5);
   77:             my $tx = $ua->get($uploadlocation);
   78:             $tx->result->save_to($tempfile);
   79:         } else {
   80:             $errors{'EMPTYUPLOAD'}    = 1 unless ( length($uploadfile) > 0 );
   81:         }
in CGI::Compile::ROOT::kohadevbox_koha_plugins_plugins_2dupload_2epl::__ANON__ at /kohadevbox/koha/plugins/plugins-upload.pl line 2

Ah, 'save_to' was added to mojo in version 8.02.   I'll have a look at how we can support this without using that call.

Created attachment 98791 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Created attachment 98792 [details] [review]
Bug 23975: Add 'Install' support for github results

Created attachment 98793 [details] [review]
Bug 23975: (follow-up) Don't rely on save_to being present

The `save_to` shortcut method was introduced to Mojolicious in version
8.02 but we still support 7.21 as our minimum dependancy and as such we
cannot yet use it.

The followup should get Install working more consistently now.. thanks for testing David.

Created attachment 98800 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 98801 [details] [review]
Bug 23975: Add 'Install' support for github results

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 98802 [details] [review]
Bug 23975: (follow-up) Don't rely on save_to being present

The `save_to` shortcut method was introduced to Mojolicious in version
8.02 but we still support 7.21 as our minimum dependancy and as such we
cannot yet use it.

Signed-off-by: David Nind <david@davidnind.com>

You may also need to add this to your Apache configuration (in the staff interface section - this folder location works for koha-testing-docker):

<Directory "/var/lib/koha/kohadev/plugins">
Require all granted
</Directory>
Alias /plugin "/var/lib/koha/kohadev/plugins/"

Created attachment 98813 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Created attachment 98814 [details] [review]
Bug 23975: Add 'Install' support for github results

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Created attachment 98815 [details] [review]
Bug 23975: (follow-up) Don't rely on save_to being present

The `save_to` shortcut method was introduced to Mojolicious in version
8.02 but we still support 7.21 as our minimum dependancy and as such we
cannot yet use it.

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

My feeling is that we should maintain a list of "accepted"/"validated" plugins, and so have a WS serving (on a Koha community server) that list.

(In reply to Jonathan Druart from comment #21)
> My feeling is that we should maintain a list of "accepted"/"validated"
> plugins, and so have a WS serving (on a Koha community server) that list.

I did initially think the same and intended to write such a service.. however I have come around to this approach as it reduces the need build, host and maintain another system, not to mention maintaining the data.

I have added followup bugs that request we add signatures to plugin releases and a way from inside Koha itself to validate those.

Once we have those I think we pretty much have what you are suggesting.. we can update the default config we ship to only include a pointer to the community gitlab and as we as a community deem a plugin to have reached a level of maturity we are happy with we can suggest an adoption of it into our gitlab and thus sign subsequent releases.

Hope that makes sense.

> Once we have those I think we pretty much have what you are suggesting.. we
> can update the default config we ship to only include a pointer to the
> community gitlab and as we as a community deem a plugin to have reached a
> level of maturity we are happy with we can suggest an adoption of it into
> our gitlab and thus sign subsequent releases.

+1

Created attachment 99958 [details] [review]
Bug 23975: (QA follow-up) Fix stylingof install buttons

Created attachment 99970 [details] [review]
Bug 23975: Add ability to search and install plugins from GitHub

It would be nice to be able to search and install plugins directly from GitHub ( and GitLab if possible ). This would make it easier to verify the authenticity of plugins and make discovering new plugins easier as well as making installation of plugins simpler.

Test Plan:
1) Apply this patch
2) Add the new block for plugin_repos to your koha-conf.xml
3) Restart all the things!
4) Browse to the plugins home page
5) Try searching for 'patrons' in the new search box
6) Verify you get results!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Created attachment 99971 [details] [review]
Bug 23975: Add 'Install' support for github results

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Created attachment 99972 [details] [review]
Bug 23975: (follow-up) Don't rely on save_to being present

The `save_to` shortcut method was introduced to Mojolicious in version
8.02 but we still support 7.21 as our minimum dependancy and as such we
cannot yet use it.

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Created attachment 99973 [details] [review]
Bug 23975: (QA follow-up) Fix stylingof install buttons

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Created attachment 99974 [details] [review]
Bug 23975: (QA follow-up) Search button style and functionality

This patch makes the search form button match the other buttons style,
and also adds the submit() action on click.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Created attachment 99975 [details] [review]
Bug 23975: (QA follow-up) Add missing TT filters

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

I am sorry but I have concerns about this patch. Feel free to ignore them.

To me this feature is not ready as it.
With those patches, the default will be to search on theke and bws github repo. Nothing says that the plugins are fetched from "outside".
Did anyone review those plugins?
Why only those 2 companies? What about others (ptfs-e, biblibre, inlibro, etc)?
What if a plugin that have security issues is pushed to those repos?

When are you going to work on follow-up bugs? Is the plan to have them ready for 20.05?

Nice work everyone!

Pushed to master for 20.05

I intend to start working on those bugs, and encourage Kyle and Tomas to help very soon.. I've left the default config in on master for now so that developers get exposed to the feature whilst we're in development, but I intend to drop those as defaults just before release... hopefully to have been replaced with a default of the community gitlab with hopefully at least a few reviewed plugins within it.

new feature not backported to 19.11.x

(In reply to Jonathan Druart from comment #31)
> I am sorry but I have concerns about this patch. Feel free to ignore them.
> 
> To me this feature is not ready as it.
> With those patches, the default will be to search on theke and bws github
> repo. Nothing says that the plugins are fetched from "outside".
> Did anyone review those plugins?
> Why only those 2 companies? What about others (ptfs-e, biblibre, inlibro,
> etc)?
> What if a plugin that have security issues is pushed to those repos?
> 

I have those same concerns. At the very least, there should be transparency about where the plugins are being fetched from.

> When are you going to work on follow-up bugs? Is the plan to have them ready
> for 20.05?

I've done the plugin signing work on https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24632, but I only did it for manually uploaded plugins. I only noticed this bug's functionality as I was wrapping up my initial work. I'll have to revisit it...

I'm noticing that the plugin_repos isn't appearing in /etc/koha/sites/kohadev/koha-conf.xml in koha-testing-docker.

I'm guessing that it just hasn't been added into a Docker image yet?

Trying this out for the first time and I really like the idea of fetching plugins from a repository and installing them, but I don't like how you have to know exactly what you're looking for.

I think it would be nicer if you could get a list of all available plugins.

Hmm searching for '%' or 'koha-plugin' is generating errors. I'll create a new issue for that.

Wow... looks like it may have been due to rate limiting. You're supposed to get 60 requests per hour. I don't think that I've done that many...

Interestingly, I can do lots of queries to the search API...

https://api.github.com/search/repositories?q=coverflow+user:bywatersolutions+in:name,description

But I am blocked for this:

https://api.github.com/repos/bywatersolutions/koha-plugin-coverflow/releases/latest

I was just messing around for 5 minutes.

This seems like it would never be feasible for production?

I'll stop spamming in a moment...

I'm actually thinking a bit about making a plugin that manages plugins.

I was inspired by the "Game Package Registry for Unity". It's a package you can add to the Unity IDE, and it lets you then import all Google packages into Unity IDE without having to manually find, download, and import into Unity IDE...

i am not able to find usekohaplugin in koha version 20.05 in system perfrences can any one tell me where to i have enable in new koha version?????

(In reply to Vihaan Gupta from comment #41)
> i am not able to find usekohaplugin in koha version 20.05 in system
> perfrences can any one tell me where to i have enable in new koha
> version?????

That system preference was removed by bug 20415

Is the "new block for plugin_repos" described somewhere? I see that the the system preference for plugins has been removed, but existing koha-conf.xmls have not been updated with the new block, even in commented form. Where is one to find it?

(In reply to Manos PETRIDIS from comment #43)
> Is the "new block for plugin_repos" described somewhere? I see that the the
> system preference for plugins has been removed, but existing koha-conf.xmls
> have not been updated with the new block, even in commented form. Where is
> one to find it?

Hi Manos, it's in the source:

https://git.koha-community.org/Koha-community/Koha/src/commit/7bddfeb412e427ee8c02c700295382fe4a7340d6/debian/templates/koha-conf-site.xml.in#L406

The patches don't apply on the current master branch anymore. Lots of merge conflicts.

(In reply to Paul Derscheid from comment #45)
> The patches don't apply on the current master branch anymore. Lots of merge
> conflicts.

This is already in the codebase since 2020.

Best regards