Bug 25952 - Github search for plugin throwing errors due to API rate limits
Summary: Github search for plugin throwing errors due to API rate limits
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: Plugin architecture (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low minor (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on: 23975
Blocks:
  Show dependency treegraph
 
Reported: 2020-07-08 10:48 UTC by David Cook
Modified: 2022-04-04 23:21 UTC (History)
1 user (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 25952: Add error-handling to prevent errors from rate limiting (2.49 KB, patch)
2020-07-08 11:05 UTC, David Cook
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2020-07-08 10:48:07 UTC
I was trying to get a full list of the available plugins so I tried '%' and 'koha-plugin' as search terms, but they created errors in Koha.

Note that the 'koha-plugin' query works directly:
https://api.github.com/search/repositories?q=koha-plugin+user:thekesolutions+in:name,description
Comment 1 David Cook 2020-07-08 11:01:43 UTC
It looks like it's due to rate limiting.

I can use the search API:
https://api.github.com/search/repositories?q=koha-plugin+user:bywatersolutions+in:name,description

But I can't get the releases:
https://api.github.com/repos/bywatersolutions/koha-plugin-coverflow/releases/latest

I guess that's because I've done < 60 searches, but each search has to do X release API calls, so it would be easy to exceed 60 API calls quickly if a person was looking for "koha-plugin" which would retrieve everything.
Comment 2 David Cook 2020-07-08 11:05:48 UTC
Created attachment 106650 [details] [review]
Bug 25952: Add error-handling to prevent errors from rate limiting

This patch makes sure there is a successful response before trying to decode
the response.

If you search for "koha-plugin" in the Bywater and Theke repos,
you'll quickly exceed your API rate limit, and you'll stop getting
successful responses.

To test:
0) Don't apply patch yet
1) Enable plugins
2) Enable plugin_repos
3) Search for koha-plugin a few times
4) Notice how Plack crashes with a visible error in the browser
5) Apply the patch
6) restart_all (in koha-testing-docker)
7) Search for koha-plugin
8) Observe that no search results are returned and there are no errors
in the browser
Comment 3 Magnus Enger 2020-10-08 18:51:34 UTC
I enabled the feature by adding this to koha-conf.xml:

 <plugin_repos>
        <repo>
                <name>ByWater Solutions</name>
                <org_name>bywatersolutions</org_name>
                <service>github</service>
            </repo>
        <repo>
                <name>Theke Solutions</name>
                <org_name>thekesolutions</org_name>
                <service>github</service>
            </repo>
 </plugin_repos>


Without the patch I always get this error:

malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm line 171

Is this the expected error we are trying to fix here?

After applying the patch I searched for "koha-plugin" and got a bunch of hits, but only from Bywater. When I did the search again straight away it chewed for a while, then I got nothing. That is better than an ugly error message, of course, but wouldn't a friendly message like "oops something went wrong, please adjust your search or try again later" be better?

Perhaps we could also intercept a search for "koha-plugin" and warn that it might exceed the rate limits and give unexepected results?
Comment 4 David Cook 2020-10-08 22:31:06 UTC
(In reply to Magnus Enger from comment #3)
> I enabled the feature by adding this to koha-conf.xml:
> 
>  <plugin_repos>
>         <repo>
>                 <name>ByWater Solutions</name>
>                 <org_name>bywatersolutions</org_name>
>                 <service>github</service>
>             </repo>
>         <repo>
>                 <name>Theke Solutions</name>
>                 <org_name>thekesolutions</org_name>
>                 <service>github</service>
>             </repo>
>  </plugin_repos>
> 
> 
> Without the patch I always get this error:
> 
> malformed JSON string, neither tag, array, object, number, string or atom,
> at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm
> line 171
> 
> Is this the expected error we are trying to fix here?
> 

It's been a while but I think so.

> After applying the patch I searched for "koha-plugin" and got a bunch of
> hits, but only from Bywater. When I did the search again straight away it
> chewed for a while, then I got nothing. That is better than an ugly error
> message, of course, but wouldn't a friendly message like "oops something
> went wrong, please adjust your search or try again later" be better?
> 
> Perhaps we could also intercept a search for "koha-plugin" and warn that it
> might exceed the rate limits and give unexepected results?

That's a good point, although at this point I think I'm abandoning/shelving my work on this, as I think using the GitHub/GitLab APIs is a flawed approach overall.
Comment 5 Magnus Enger 2020-10-09 21:08:33 UTC
It does seem a bit brittle if we are already reaching the limits of the API. Do you have another approach in mind or do you think the whole “search and install” feature should be reverted?

Setting to “in discussion”.
Comment 6 David Cook 2020-10-11 22:16:35 UTC
(In reply to Magnus Enger from comment #5)
> It does seem a bit brittle if we are already reaching the limits of the API.
> Do you have another approach in mind or do you think the whole “search and
> install” feature should be reverted?
> 

I don't think that it should be reverted, but I think a manifest based approach would be more robust. It's something I plan to work on eventually one day in the future.

Basically, we'd point the Koha config at a set of URLs (which are either for  files or directories depending on the implementation) and we'd download manifests of available Koha plugins. (Basically, it's the same model as package managers.)

The neat part of that would also be that sysadmins could choose which manifests to point at or even point to a local one of curated plugins. It would be quite flexible.

The only downside I see with the manifests is that they would need to be updated, but I think that's really more of a plugin developer issue than a Koha issue.
Comment 7 Magnus Enger 2020-10-12 08:51:50 UTC
(In reply to David Cook from comment #6)
> Basically, we'd point the Koha config at a set of URLs (which are either for
> files or directories depending on the implementation) and we'd download
> manifests of available Koha plugins. (Basically, it's the same model as
> package managers.)

Ah, sounds good to me!