It looks like it's due to rate limiting.

I can use the search API:
https://api.github.com/search/repositories?q=koha-plugin+user:bywatersolutions+in:name,description

But I can't get the releases:
https://api.github.com/repos/bywatersolutions/koha-plugin-coverflow/releases/latest

I guess that's because I've done < 60 searches, but each search has to do X release API calls, so it would be easy to exceed 60 API calls quickly if a person was looking for "koha-plugin" which would retrieve everything.

Created attachment 106650 [details] [review]
Bug 25952: Add error-handling to prevent errors from rate limiting

This patch makes sure there is a successful response before trying to decode
the response.

If you search for "koha-plugin" in the Bywater and Theke repos,
you'll quickly exceed your API rate limit, and you'll stop getting
successful responses.

To test:
0) Don't apply patch yet
1) Enable plugins
2) Enable plugin_repos
3) Search for koha-plugin a few times
4) Notice how Plack crashes with a visible error in the browser
5) Apply the patch
6) restart_all (in koha-testing-docker)
7) Search for koha-plugin
8) Observe that no search results are returned and there are no errors
in the browser

I enabled the feature by adding this to koha-conf.xml:

 <plugin_repos>
        <repo>
                <name>ByWater Solutions</name>
                <org_name>bywatersolutions</org_name>
                <service>github</service>
            </repo>
        <repo>
                <name>Theke Solutions</name>
                <org_name>thekesolutions</org_name>
                <service>github</service>
            </repo>
 </plugin_repos>


Without the patch I always get this error:

malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm line 171

Is this the expected error we are trying to fix here?

After applying the patch I searched for "koha-plugin" and got a bunch of hits, but only from Bywater. When I did the search again straight away it chewed for a while, then I got nothing. That is better than an ugly error message, of course, but wouldn't a friendly message like "oops something went wrong, please adjust your search or try again later" be better?

Perhaps we could also intercept a search for "koha-plugin" and warn that it might exceed the rate limits and give unexepected results?

(In reply to Magnus Enger from comment #3)
> I enabled the feature by adding this to koha-conf.xml:
> 
>  <plugin_repos>
>         <repo>
>                 <name>ByWater Solutions</name>
>                 <org_name>bywatersolutions</org_name>
>                 <service>github</service>
>             </repo>
>         <repo>
>                 <name>Theke Solutions</name>
>                 <org_name>thekesolutions</org_name>
>                 <service>github</service>
>             </repo>
>  </plugin_repos>
> 
> 
> Without the patch I always get this error:
> 
> malformed JSON string, neither tag, array, object, number, string or atom,
> at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm
> line 171
> 
> Is this the expected error we are trying to fix here?
> 

It's been a while but I think so.

> After applying the patch I searched for "koha-plugin" and got a bunch of
> hits, but only from Bywater. When I did the search again straight away it
> chewed for a while, then I got nothing. That is better than an ugly error
> message, of course, but wouldn't a friendly message like "oops something
> went wrong, please adjust your search or try again later" be better?
> 
> Perhaps we could also intercept a search for "koha-plugin" and warn that it
> might exceed the rate limits and give unexepected results?

That's a good point, although at this point I think I'm abandoning/shelving my work on this, as I think using the GitHub/GitLab APIs is a flawed approach overall.

It does seem a bit brittle if we are already reaching the limits of the API. Do you have another approach in mind or do you think the whole “search and install” feature should be reverted?

Setting to “in discussion”.

(In reply to Magnus Enger from comment #5)
> It does seem a bit brittle if we are already reaching the limits of the API.
> Do you have another approach in mind or do you think the whole “search and
> install” feature should be reverted?
> 

I don't think that it should be reverted, but I think a manifest based approach would be more robust. It's something I plan to work on eventually one day in the future.

Basically, we'd point the Koha config at a set of URLs (which are either for  files or directories depending on the implementation) and we'd download manifests of available Koha plugins. (Basically, it's the same model as package managers.)

The neat part of that would also be that sysadmins could choose which manifests to point at or even point to a local one of curated plugins. It would be quite flexible.

The only downside I see with the manifests is that they would need to be updated, but I think that's really more of a plugin developer issue than a Koha issue.

(In reply to David Cook from comment #6)
> Basically, we'd point the Koha config at a set of URLs (which are either for
> files or directories depending on the implementation) and we'd download
> manifests of available Koha plugins. (Basically, it's the same model as
> package managers.)

Ah, sounds good to me!