Bug 23653 - Plack fails when http://swagger.io/v2/schema.json is unavailable and schema cache missing
Summary: Plack fails when http://swagger.io/v2/schema.json is unavailable and schema c...
Status: Pushed to master
Alias: None
Product: Koha
Classification: Unclassified
Component: REST API (show other bugs)
Version: master
Hardware: Other Linux
: P5 - low major (vote)
Assignee: David Cook
QA Contact: Tomás Cohen Arazi
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-09-20 08:32 UTC by Rick Conrad
Modified: 2020-11-17 20:58 UTC (History)
14 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:
20.11.00


Attachments
Bug 23653: use local copy of swagger v2 schema (45.63 KB, patch)
2020-07-01 02:31 UTC, David Cook
Details | Diff | Splinter Review
Bug 23653: use local copy of swagger v2 schema (45.68 KB, patch)
2020-07-01 02:56 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 23653: Remove uneeded cond test (1009 bytes, patch)
2020-08-10 09:00 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23653: use local copy of swagger v2 schema (45.73 KB, patch)
2020-08-25 14:47 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 23653: Remove uneeded cond test (1.04 KB, patch)
2020-08-25 14:47 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 23653: Add license information in about.pl (1.60 KB, patch)
2020-08-25 15:01 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 23653: Add license information in about.pl (2.00 KB, patch)
2020-08-26 13:21 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Rick Conrad 2019-09-20 08:32:50 UTC
I had plack working fine before this morning, but now if I have plack enabled, Koha does not respond to web requests, and apache eventually shows me a 502. These are the entries from plack-error.log. I don't know what swagger.io is - seems there is such a website, but I can't `curl` it from the webserver, at least right now.

plack-error.log
````
Error while loading /etc/koha/plack.psgi: Can't load application from file "/usr/share/koha/api/v1/app.pl": [JSON::Validator] GET http://swagger.io/v2/schema.json == Inactivity timeout at /usr/share/perl5/JSON/Validator.pm line 274.
Compilation failed in require at (eval 389) line 1.
Child process 3159 exited with status 255
````

The only change I made this morning was changing the server's timezone, but I doubt that's related. I did put it back to UTC, but no improvement. Disabling plack works for me, but my site is a lot slower (and it wasn't real quick to begin with).
Comment 1 Rick Conrad 2019-09-20 10:10:11 UTC
I observed later that I was able to reach swagger.io from the server, so I tried reenabling plack and now it's working.

So, this may properly be a plack issue and not a koha issue, but it seems like my library site ought to work whether or not I can connect to swagger.io, whatever that is.
Comment 2 Tomás Cohen Arazi 2019-09-23 13:21:52 UTC
Can you give us more information on your setup? The 'swagger file' is bundled in Koha and there shouldn't be any external system reached out.

It would help if you additionally:

$ sudo apt install pmtools

and then we need the output for:

$ pmvers JSON::Validator
$ pmvers Mojolicious
$ pmvers Mojolicious::Plugin::OpenAPI
Comment 3 Rick Conrad 2019-09-23 14:45:49 UTC
````
rick@koha:~$ pmvers JSON::Validator
1.08
rick@koha:~$ pmvers Mojolicious
7.59
rick@koha:~$ pmvers Mojolicious::Plugin::OpenAPI
1.15
````

Also

````
rick@koha:~$ cd /
rick@koha:/$ sudo find . -name swagger
./usr/share/koha/api/v1/swagger
rick@koha:/$ cd /usr/share/koha/api/v1/swagger/
rick@koha:/usr/share/koha/api/v1/swagger$ ls
definitions       parameters       paths       swagger.json
definitions.json  parameters.json  paths.json  x-primitives.json
```
Comment 4 Chad Billman 2020-06-23 18:55:55 UTC
I was able to reproduce this on a fresh installation of koha 20.05 from packages on debian buster.

I have the webserver behind a firewall blocking all outbound traffic and could not get the koha installer to launch. After digging into the logs I saw the same connection timeouts to swagger.io

As soon as I opened up our firewall the installer loaded.

You can also reproduce this by dropping something bogus in your /etc/hosts file for swagger.io and restarting plack

For example - 

127.0.0.1 swagger.io
Comment 5 Mason James 2020-06-23 23:08:43 UTC
(In reply to Rick Conrad from comment #3)
> ````
> rick@koha:~$ pmvers JSON::Validator
> 1.08
> rick@koha:~$ pmvers Mojolicious
> 7.59
> rick@koha:~$ pmvers Mojolicious::Plugin::OpenAPI
> 1.15
> ````

hi Rick
which debian version is this output from? buster?
Comment 6 Rick Conrad 2020-06-23 23:21:02 UTC
I don't have it on-hand, but I believe it was Ubuntu 18.04.
Comment 7 Mason James 2020-06-23 23:54:56 UTC
(In reply to Rick Conrad from comment #6)
> I don't have it on-hand, but I believe it was Ubuntu 18.04.

for 18.04, try running these commands

$ sudo echo 'deb http://debian.koha-community.org/koha 20.05 main bionic' >  /etc/apt/sources.list.d/koha.list 
$ sudo apt-get update; apt-get -y upgrade
$ sudo apt install pmtools

you should have...
$ pmvers JSON::Validator
3.06
$ pmvers Mojolicious
8.12
$ pmvers Mojolicious::Plugin::OpenAPI
2.12
Comment 8 Tomás Cohen Arazi 2020-06-24 00:02:45 UTC
This looks really bad if Koha is trying to contact an external site on start-up
Comment 9 Chad Billman 2020-06-24 01:20:13 UTC
Here is the output from my fresh buster server with koha-common 20.05.00-1

$ pmvers JSON::Validator
3.06
$ pmvers Mojolicious
8.12
$ pmvers Mojolicious::Plugin::OpenAPI
2.12

From plack-error.og

Compilation failed in require at (eval 410) line 1.
Child process 25034 exited with status 255
Error while loading /etc/koha/plack.psgi: Can't load application from file "/usr/share/koha/api/v1/app.pl": [JSON::Validator] GET http://swagger.io/v2/schema.json == Not Found at /usr/share/perl5/JSON/Validator.pm line 343.
Compilation failed in require at (eval 410) line 1.
Child process 25042 exited with status 255
Error while loading /etc/koha/plack.psgi: Can't load application from file "/usr/share/koha/api/v1/app.pl": [JSON::Validator] GET http://swagger.io/v2/schema.json == Not Found at /usr/share/perl5/JSON/Validator.pm line 343.
Comment 10 Mason James 2020-06-24 03:15:02 UTC
hmm, this does look like a legitimate bug, caused by 22522

the 'try' block always fails with...

 Can't locate object method "bundle" via package "JSON::Validator::OpenAPI::Mojolicious" at /kohadevbox/koha/Koha/REST/V1.pm line 76.


...and the 'catch' block calls validator->load_and_validate_schema() which does a lookup to swagger.io


./Koha/REST/V1.pm
 72
 73     # Try to load and merge all schemas first and validate the result just once.
 74     my $spec;
 75     try {
 76         $spec = $validator->bundle(
 77             {
 78                 replace => 1,
 79                 schema => $self->home->rel_file("api/v1/swagger/swagger.json")
 80             }
 81         );
 82

100     }
101     catch {
102         # Validation of the complete spec failed. Resort to validation one-by-one
103         # to catch bad ones.
104         $validator->load_and_validate_schema(
105             $self->home->rel_file("api/v1/swagger/swagger.json"),
106             {
107                 allow_invalid_ref  => 1,
108             }
109         );
Comment 11 Mason James 2020-06-24 05:36:36 UTC
(In reply to Mason James from comment #10)
> hmm, this does look like a legitimate bug, caused by 22522

^ oops, ignore that - i was testing on jessie :)


the problem seems to be that the debian libjson-validator-perl package has the 'cache' dir omitted, which contains local copies of the various schemas


Rick, a quick solution to the problem is to restore the cached schemas

$ git clone https://github.com/mojolicious/json-validator
$ cd json-validator
$ git reset --hard 3.06
HEAD is now at 8b19ba0 Released version 3.06

$ sudo cp -a ./lib/JSON/Validator/cache /usr/share/perl5/JSON/Validator/
Comment 12 Chad Billman 2020-06-24 11:51:58 UTC
I followed the steps Jason provided to restore the cache folder on debian buster. 

Plack was able to start even when access to swagger.io was blocked.

Thanks!
Comment 13 Mason James 2020-06-25 00:56:57 UTC
(In reply to Mason James from comment #11)
> (In reply to Mason James from comment #10)
> > hmm, this does look like a legitimate bug, caused by 22522
> 
> ^ oops, ignore that - i was testing on jessie :)
> 
> 
> the problem seems to be that the debian libjson-validator-perl package has
> the 'cache' dir omitted, which contains local copies of the various schemas
> 

looks like the schemas were removed due to a license issue...

Source: https://metacpan.org/release/JSON-Validator
Upstream-Contact: Jan Henning Thorsen <jhthorsen@cpan.org>
Upstream-Name: JSON-Validator
Files-Excluded: lib/JSON/Validator/cache
Comment: The +dfsg version is created by removing the cache files from the
 lib/JSON/Validator directory, where the license is unknown.

https://salsa.debian.org/perl-team/modules/packages/libjson-validator-perl/-/raw/master/debian/copyright



i'll attempt to build a new package with the schemas included
Comment 14 Tomás Cohen Arazi 2020-06-25 02:44:21 UTC
(In reply to Mason James from comment #13)
> (In reply to Mason James from comment #11)
> > (In reply to Mason James from comment #10)
> > > hmm, this does look like a legitimate bug, caused by 22522
> > 
> > ^ oops, ignore that - i was testing on jessie :)
> > 
> > 
> > the problem seems to be that the debian libjson-validator-perl package has
> > the 'cache' dir omitted, which contains local copies of the various schemas
> > 
> 
> looks like the schemas were removed due to a license issue...
> 
> Source: https://metacpan.org/release/JSON-Validator
> Upstream-Contact: Jan Henning Thorsen <jhthorsen@cpan.org>
> Upstream-Name: JSON-Validator
> Files-Excluded: lib/JSON/Validator/cache
> Comment: The +dfsg version is created by removing the cache files from the
>  lib/JSON/Validator directory, where the license is unknown.
> 
> https://salsa.debian.org/perl-team/modules/packages/libjson-validator-perl/-/
> raw/master/debian/copyright
> 
> 
> 
> i'll attempt to build a new package with the schemas included

That's weird. The schema is clearly licensed on the OpenAPI repository. It maybe I got it is about the process to acknowledge that on Debian side:

https://github.com/OAI/OpenAPI-Specification/blob/master/LICENSE
Comment 15 Victor Grousset/tuxayo 2020-06-25 21:35:01 UTC
This bug is marked as only affecting version 19.05.x, but it seem to affect 20.05 (as likely master) also, right?
Comment 16 Victor Grousset/tuxayo 2020-06-30 02:36:01 UTC
== Test plan ==
1. In /etc/hosts , add «127.0.0.1    swagger.io»
2. try to load a page
3. it works
4. koha-plack --restart kohadev
5. try to load a page
6. endless or timeout loading
7. remove the sabotage from /etc/hosts
8. after a few seconds, the page will load

Confirmed on master.
It works on 19.05


(In reply to Mason James from comment #10)
> hmm, this does look like a legitimate bug, caused by 22522

Possibly. Because 19.05.x doesn't have bug 22522 patches
Comment 17 Ian Gordon 2020-06-30 21:10:38 UTC
Hi,

At about 21:00UTC today we also started to have this problem.

We're running 19.05.12 on Ubuntu 18.04.

The solution in comment 11 seems to work for now.

% pmvers JSON::Validator
1.08
% pmvers Mojolicious
7.59
% pmvers Mojolicious::Plugin::OpenAPI
1.15

Error message in plack-error.log is:

Error while loading /etc/koha/plack.psgi: Can't load application from file "/usr/share/koha/api/v1/app.pl": [JSON::Validator] GET http://swagger.io/v2/schema.json == Connect timeout at /usr/share/perl5/JSON/Validator.pm line 274.
Compilation failed in require at (eval 389) line 1.

Regards,

Ian Gordon.
Comment 18 Victor Grousset/tuxayo 2020-06-30 22:42:56 UTC
Issue reported to upstream's upstream: https://github.com/swagger-api/swagger.io/issues/281
Comment 19 David Cook 2020-07-01 00:37:52 UTC
A solution to this problem is to provide our own argument "schema" argument to the load_and_validate_schema() method. 

https://metacpan.org/pod/JSON::Validator#load_and_validate_schema

I've looked through the code and there's a variety of ways of providing the schema, which we can source from https://raw.githubusercontent.com/mojolicious/json-validator/3.06/lib/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df

I'd say we should provide it as a local file, so that we're not relying on any web servers to provide this crucial file.
Comment 20 Federico Paiz 2020-07-01 00:52:51 UTC
The same thing happened to me, I solved it by doing 

>cpan
> install Mojolicious::Plugin::OpenAPI
Comment 21 David Cook 2020-07-01 00:56:53 UTC
(In reply to Federico Paiz from comment #20)
> The same thing happened to me, I solved it by doing 
> 
> >cpan
> > install Mojolicious::Plugin::OpenAPI

Yeah, the CPAN version of that module comes with the "cache" which the Debian package excludes, so you'll be fine with that. 

But it's a short-term solution.
Comment 22 David Cook 2020-07-01 00:57:10 UTC
I'm about to start on a patch for this. Hopefully will have something very soon.
Comment 23 Tomás Cohen Arazi 2020-07-01 01:43:57 UTC
(In reply to David Cook from comment #22)
> I'm about to start on a patch for this. Hopefully will have something very
> soon.

We should just package the library with the correct cache
Comment 24 David Cook 2020-07-01 01:45:54 UTC
Note that this should only affect plugin-enabled instances
Comment 25 David Cook 2020-07-01 01:49:03 UTC
(In reply to Tomás Cohen Arazi from comment #23)
> (In reply to David Cook from comment #22)
> > I'm about to start on a patch for this. Hopefully will have something very
> > soon.
> 
> We should just package the library with the correct cache

Maybe. I think there is more to this than we think.

Consider ./Koha/REST/V1.pm:

$validator->bundle uses a schema of $self->home->rel_file("api/v1/swagger/swagger.json"), which is a drastically reduced schema file.

Calls to $validator->load_and_validate_schema (in ./Koha/REST/V1.pm and ./Koha/REST/Plugin/PluginRoutes.pm) don't use that same schema. They try to fetch http://swagger.io/v2/schema.json.

The question I have is whether they should all be using $self->home->rel_file("api/v1/swagger/swagger.json"), or if *some* calls should use our local swagger.json and others should use the full swagger 2.0 schema.
Comment 26 David Cook 2020-07-01 01:52:29 UTC
Oh wait I'm misunderstanding...

$validator->bundle is just loading the schema... and it doesn't validate it.

Whereas $validator->load_and_validate_schema loads the schema, but then validates it against the Swagger 2.0 API schema. 

Wow, JSON::Validator really overloads the word "schema" in its code...
Comment 27 David Cook 2020-07-01 01:54:23 UTC
From what I can see, we're not actually validating the schema in ./Koha/REST/V1.pm when we use bundle(). 

We only validate it in ./Koha/REST/V1.pm when we do the catch{} block.

And we always validate ./Koha/REST/Plugin/PluginRoutes.pm.

Ok... yeah re-packaging libjson-validator-perl to include the "cache" probably makes the most sense. 

I'm going to post a code-level patch anyway (partially just because I'm already this far anyway)
Comment 28 David Cook 2020-07-01 01:54:46 UTC
(In reply to David Cook from comment #27)
> I'm going to post a code-level patch anyway (partially just because I'm
> already this far anyway)

Plus, if I do that, then we wouldn't have to maintain another custom package dependency...
Comment 29 David Cook 2020-07-01 02:12:54 UTC
Actually, it looks like Koha::REST::Plugin::PluginRoutes doesn't receive the validator.

It looks like the OpenAPI plugin must be calling a validator and that's where it's failing...
Comment 30 Victor Grousset/tuxayo 2020-07-01 02:22:35 UTC
(In reply to Tomás Cohen Arazi from comment #23)
> We should just package the library with the correct cache

Indeed, that way, deployers only have to update the perl-deps, no patch to apply.
Comment 31 David Cook 2020-07-01 02:31:58 UTC
Created attachment 106444 [details] [review]
Bug 23653: use local copy of swagger v2 schema

By default, JSON::Validator::OpenAPI tries to fetch the
swagger v2 schema from http://swagger.io/v2/schema.json.

If you've installed from CPAN, JSON::Validator::OpenAPI will
come with a cached copy, so it won't try to fetch it over HTTP.

However, if you've installed from libjson-validator-perl
from Debian/Ubuntu, the Debian package excludes the cached copy,
so JSON::Validator::OpenAPI tries to fetch it over HTTP.

Unfortunately, today and other days in the past, the file at
http://swagger.io/v2/schema.json has been unavailable, and this causes
Koha to crash in a perpetual loop.

This patch includes a copy of the swagger v2 schema, and it loads
it locally rather than fetching over HTTP.

The changes to Koha/REST/Plugin/PluginRoutes.pm are not required,
since the validator isn't currently called there, but I've added
a patch to future proof it.

To Test:
0a) Remove /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df
if it exists
0b) Block external access to http://swagger.io/v2/schema.json or
test during an outage when it's unavailable
0c) Do not apply patch
1) koha-plack --restart kohadev
2) Note that it crashes in a loop and is unavailable in web browser
3) Apply patch
4) koha-plack --restart kohadev
5) Note that Koha comes up and there are no errors in the Plack logs
Comment 32 Victor Grousset/tuxayo 2020-07-01 02:56:48 UTC
Created attachment 106445 [details] [review]
Bug 23653: use local copy of swagger v2 schema

By default, JSON::Validator::OpenAPI tries to fetch the
swagger v2 schema from http://swagger.io/v2/schema.json.

If you've installed from CPAN, JSON::Validator::OpenAPI will
come with a cached copy, so it won't try to fetch it over HTTP.

However, if you've installed from libjson-validator-perl
from Debian/Ubuntu, the Debian package excludes the cached copy,
so JSON::Validator::OpenAPI tries to fetch it over HTTP.

Unfortunately, today and other days in the past, the file at
http://swagger.io/v2/schema.json has been unavailable, and this causes
Koha to crash in a perpetual loop.

This patch includes a copy of the swagger v2 schema, and it loads
it locally rather than fetching over HTTP.

The changes to Koha/REST/Plugin/PluginRoutes.pm are not required,
since the validator isn't currently called there, but I've added
a patch to future proof it.

To Test:
0a) Remove /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df
if it exists
0b) Block external access to http://swagger.io/v2/schema.json or
test during an outage when it's unavailable
0c) Do not apply patch
1) koha-plack --restart kohadev
2) Note that it crashes in a loop and is unavailable in web browser
3) Apply patch
4) koha-plack --restart kohadev
5) Note that Koha comes up and there are no errors in the Plack logs

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 33 Victor Grousset/tuxayo 2020-07-01 03:19:35 UTC
It works! :D


(In reply to Victor Grousset/tuxayo from comment #30)
> Indeed, that way, deployers only have to update the perl-deps, no patch to
> apply.

Should we actually expect a significant number of instances to have more difficulty applying such a simple patch (just this patch, not updating Koha) compared to updating the perl deps?

The question is whether or not packaging libjson-validator-perl during some time is worth it. Not sure now.

Also, if http://swagger.io/v2/schema.json would work again tomorrow that would help letting those instances count on that until their next Koha update.
Comment 34 David Cook 2020-07-01 03:54:00 UTC
(In reply to Victor Grousset/tuxayo from comment #33)

> Should we actually expect a significant number of instances to have more
> difficulty applying such a simple patch (just this patch, not updating Koha)
> compared to updating the perl deps?
> 
> The question is whether or not packaging libjson-validator-perl during some
> time is worth it. Not sure now.
> 
> Also, if http://swagger.io/v2/schema.json would work again tomorrow that
> would help letting those instances count on that until their next Koha
> update.

I'm curious about this as well. While we felt this in production in Australia, I'm guessing the rest of the world might not feel it for another few hours.

I don't know where Jonathan is physically located these days, but I'm guessing that we may have just missed him as well, so the Koha world is in for an interesting day...
Comment 35 David Cook 2020-07-01 04:00:06 UTC
Mason announced 30 minutes ago on koha-devel that he's updated the libjson-validator-perl package.

As per his email:

hi Folks
i've updated the 'libjson-validator-perl' package to include the cache dir

to update the package...
$ sudo apt update
$ sudo apt install libjson-validator-perl

run the following command, and confirm the result...
$ apt-cache policy  libjson-validator-perl
libjson-validator-perl:
   Installed: 3.06+dfsg-1+koha1
   Candidate: 3.06+dfsg-1+koha1
Comment 36 David Cook 2020-07-01 04:03:09 UTC
I've tested the new package in koha-testing-docker and it looks good.
Comment 37 David Cook 2020-07-01 07:11:57 UTC
I'd recommend to people starting their day to look at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23653#c11 for a quick fix to get through the worst of things right now.
Comment 38 David Cook 2020-07-01 07:22:17 UTC
I think Mason found that newer versions of Ubuntu weren't installing 3.06 as they were using newer versions, but it looks like he's now released a 3.23+dfsg-1+koha1 version.

I've installed that on Debian Jessie in koha-testing-docker, and it looks like it's working successfully.

So folk using the Debian packages shoudl be able to just do the following:

apt-get update
apt-get install libjson-validator-perl
Comment 39 David Cook 2020-07-01 07:40:10 UTC
Ok that 3.23 is still in the staging repo.

If you're running a Debian/Ubuntu package, you'll be vulnerable, but not if you're running a Koha Community package. 

So you should be fine on Debian Jessie and Debian Stretch and Ubuntu Xenial. 

If you're on anything newer than those, you'll need to upgrade to the Koha Community 3.06 (or 3.23 when it is available) as per the table below:

__Koha Community Apt Repo__
libjson-validator-perl_0.67+dfsg-1~koha1_all.deb
libjson-validator-perl_0.97-1~kohadev1_all.deb
libjson-validator-perl_3.06+dfsg-1+koha1_all.deb

__Debian__
Stretch 9: libjson-validator-perl (0.92+dfsg-1)
Buster 10: libjson-validator-perl (3.06+dfsg-1)
Bullseye 11: libjson-validator-perl (3.25+dfsg-1)

__Ubuntu__
Xenial 16.04: libjson-validator-perl (0.66+dfsg-1)
Bionic 18.04: libjson-validator-perl (1.08+dfsg-1)
Eoan 19.10: libjson-validator-perl (3.14+dfsg-1)
Focal 20.04: libjson-validator-perl (3.23+dfsg-1)
Groovy 20.10: libjson-validator-perl (3.25+dfsg-1)
Comment 40 David Cook 2020-07-01 07:57:49 UTC
Here's a condensed version of the above:

Good (don't worry):
- Debian Jessie 8
- Debian Stretch 9
- Ubuntu Xenial 16.04

Bad (do worry): 
- Debian Buster 10 (and above)
- Ubuntu Bionic 18.04 (and above)
Comment 41 Fridolin Somers 2020-07-01 12:05:12 UTC
For Ubuntu Debian, we at biblibre installed from CPAN JSON::Validator 4.01 and added the cache file.
It works without patch on sources.
Comment 42 Victor Grousset/tuxayo 2020-07-01 16:13:05 UTC
The schema hosting at swagger.io is back
https://github.com/swagger-api/swagger.io/issues/282
Comment 43 David Cook 2020-07-01 23:38:53 UTC
(In reply to Victor Grousset/tuxayo from comment #42)
> The schema hosting at swagger.io is back
> https://github.com/swagger-api/swagger.io/issues/282

This is great!

Although I hope this doesn't cause us to forget about the issue...
Comment 44 Mason James 2020-07-02 01:58:34 UTC
(In reply to David Cook from comment #35)
> Mason announced 30 minutes ago on koha-devel that he's updated the
> libjson-validator-perl package.
> 
> As per his email:
> 
> hi Folks
> i've updated the 'libjson-validator-perl' package to include the cache dir
> 
> to update the package...
> $ sudo apt update
> $ sudo apt install libjson-validator-perl
> 
> run the following command, and confirm the result...
> $ apt-cache policy  libjson-validator-perl
> libjson-validator-perl:
>    Installed: 3.06+dfsg-1+koha1
>    Candidate: 3.06+dfsg-1+koha1


I've updated the libjson-validator-perl and libmojolicious-plugin-openapi-perl packages again, to make them work on debian/ubuntu 'unstable'

$ apt-cache policy libjson-validator-perl libmojolicious-plugin-openapi-perl 

libjson-validator-perl:
  Installed: 3.25+dfsg-1+koha3
  Candidate: 3.25+dfsg-1+koha3
  Version table:
 *** 3.25+dfsg-1+koha3 500
        500 http://debian.koha-community.org/koha stable/main amd64 Packages
        100 /var/lib/dpkg/status
     ..

libmojolicious-plugin-openapi-perl:
  Installed: 2.16-1+koha1
  Candidate: 2.16-1+koha1
  Version table:
 *** 2.16-1+koha1 500
        500 http://debian.koha-community.org/koha stable/main amd64 Packages
        100 /var/lib/dpkg/status
Comment 45 Andrew Nugged 2020-07-02 07:51:47 UTC
(In reply to Fridolin SOMERS from comment #41)
> For Ubuntu Debian, we at biblibre installed from CPAN JSON::Validator 4.01
> and added the cache file.
> It works without patch on sources.

Confirming the same: just using cpanm with only one this updated all quickfixed, temporarily for libs, but now I am learning how to make this solution stable, reading all above & tracking :).
Comment 46 Marcel de Rooy 2020-07-02 09:15:00 UTC
How often this schema changes ? Need to check for updates ?
Comment 47 Victor Grousset/tuxayo 2020-07-02 17:31:31 UTC
From what I understand, it's a schema from a version of the OpenAPI Specification. Something like the draft 4 of the version 2.

Here we can compare a version from 2014 and from now.
https://web.archive.org/web/20141019005411/https://swagger.io/v2/schema.json
vs
https://web.archive.org/web/20200702172005/https://schema.swagger.io/v2/schema.json


Here is the actual source it seems:
https://github.com/OAI/OpenAPI-Specification/blob/c196cac8191069fa97028c2f1ae322d11a74f891/schemas/v2.0/schema.json
vs
https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v2.0/schema.json


I would *guess* it will stay backward compatible.
Comment 48 Andrew Nugged 2020-07-02 23:55:36 UTC
Note by the way, 

these changes with dependencies brought issues with fresh installations: 

on Ubuntu 18.04 with those regular steps which worked everywhere usually before,
(preps + "apt-get install koha-common" and voila) now fails with: 

  install 'koha-common' failed: Unable to correct problems, you have held broken packages.
  Depends: libmojolicious-plugin-openapi-perl but it is not going to be installed               
  Depends: libnet-oauth2-authorizationserver-perl but it is not going to be installed


This was when:

  "deb http://debian.koha-community.org/koha 20.05 main"

(https://wiki.koha-community.org/wiki/Koha_on_Debian#Follow_a_codename_.28version_number.29:_20.05.2C_19.11.2C_19.05)

but adding mentioned above explicit release name:

  "deb http://debian.koha-community.org/koha 20.05 main bionic"

made install seamless.
Comment 49 Julian Maurice 2020-07-03 05:48:34 UTC
Comment on attachment 106445 [details] [review]
Bug 23653: use local copy of swagger v2 schema

Review of attachment 106445 [details] [review]:
-----------------------------------------------------------------

::: Koha/REST/Plugin/PluginRoutes.pm
@@ +49,5 @@
>      {
> +        my $schema = $app->home->rel_file("api/swagger-v2-schema.json");
> +        if ($schema){
> +           $self->{'swagger-v2-schema'} = $schema;
> +        }

What is the test for ? rel_file returns something even if the file does not exist
Comment 50 Julian Maurice 2020-07-03 05:55:19 UTC
The cached copy of the schema is slightly different from the one at https://schema.swagger.io/v2/schema.json
Where does it come from ?
Comment 51 Victor Grousset/tuxayo 2020-07-03 06:09:09 UTC
(In reply to Julian Maurice from comment #50)
> The cached copy of the schema is slightly different from the one at
> https://schema.swagger.io/v2/schema.json
> Where does it come from ?

From what seems to be the canonical source: https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v2.0/schema.json
Comment 52 Jonathan Druart 2020-07-03 10:48:51 UTC
Do we need a unit test to compare the local and remote files? (ofc it won't fail if the remove file is not reachable)
Comment 53 Julian Maurice 2020-07-03 12:11:44 UTC
(In reply to Jonathan Druart from comment #52)
> Do we need a unit test to compare the local and remote files? (ofc it won't
> fail if the remove file is not reachable)

I think it's not worth it. Swagger/OpenAPI 2.0 doesn't change anymore (last change was in 2017)
Comment 54 Victor Grousset/tuxayo 2020-07-05 03:26:06 UTC
(In reply to Jonathan Druart from comment #52)
> Do we need a unit test to compare the local and remote files? (ofc it won't
> fail if the remove file is not reachable)

my 2 cents: don't think it's worth it also.
Because here we can see migration paths from the different draft versions:
https://json-schema.org/specification.html#migrating-from-older-drafts
So backward compatibility seem to be assured.
Comment 55 David Cook 2020-07-06 01:02:18 UTC
(In reply to Julian Maurice from comment #49)
> What is the test for ? rel_file returns something even if the file does not
> exist

That's interesting. I didn't think it did for me when I tested. I'd have to look gain.
Comment 56 David Cook 2020-07-06 01:03:36 UTC
(In reply to Jonathan Druart from comment #52)
> Do we need a unit test to compare the local and remote files? (ofc it won't
> fail if the remove file is not reachable)

I wouldn't think so. If our other API unit tests are passing, I'd say that's what would matter most.
Comment 57 Victor Grousset/tuxayo 2020-07-08 01:38:35 UTC
Ticket opened upstream (Debian) to have the cache (not really cache) files included in the package.
As upstream's(Debian) upstream's(json-validator) upstream (OpenAPI/Swagger) confirmed the license of these files.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496
Comment 58 David Cook 2020-07-08 03:02:01 UTC
(In reply to Victor Grousset/tuxayo from comment #57)
> Ticket opened upstream (Debian) to have the cache (not really cache) files
> included in the package.
> As upstream's(Debian) upstream's(json-validator) upstream (OpenAPI/Swagger)
> confirmed the license of these files.
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496

Nice one, Victor!
Comment 59 Julian Maurice 2020-07-09 07:13:07 UTC
Failing QA to get an answer to comment 49
Comment 60 Jonathan Druart 2020-07-09 09:53:51 UTC
IIUC this is no longer blocker, lowering severity.
Comment 61 David Cook 2020-07-09 23:48:41 UTC
(In reply to Julian Maurice from comment #59)
> Failing QA to get an answer to comment 49

Yeah that's fair. I'll put it on my TODO list.
Comment 62 Victor Grousset/tuxayo 2020-07-23 20:04:26 UTC
Upstream answer
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496#10

> Indeed, the OpenAPI schema files are Apache2-licensed [3], 
> thus at least them could be retained.
> However, copies of these schema files are already around in Debian [4],
> thus I would suggest packaging OpenAPI schema files in a separate
> Debian package, and symlinking them to JSON::Validator cache location.

Help welcome to try to find arguments to stop removing the schema files. The duplication concern is fair but I don't think someone here has the resources to push towards a common schema Debian package. Or maybe I'm wrong and it's not much work? hmmmm ^^"
Comment 63 Jonathan Druart 2020-08-10 09:00:50 UTC
Created attachment 107984 [details] [review]
Bug 23653: Remove uneeded cond test

rel_file returns the path anyway
Comment 64 Jonathan Druart 2020-08-10 09:01:35 UTC
(In reply to Julian Maurice from comment #49)
> Comment on attachment 106445 [details] [review] [review]
> Bug 23653: use local copy of swagger v2 schema
> 
> Review of attachment 106445 [details] [review] [review]:
> -----------------------------------------------------------------
> 
> ::: Koha/REST/Plugin/PluginRoutes.pm
> @@ +49,5 @@
> >      {
> > +        my $schema = $app->home->rel_file("api/swagger-v2-schema.json");
> > +        if ($schema){
> > +           $self->{'swagger-v2-schema'} = $schema;
> > +        }
> 
> What is the test for ? rel_file returns something even if the file does not
> exist

I removed the test. Ready for QA.
Comment 65 Jonathan Druart 2020-08-10 09:02:48 UTC
(In reply to Victor Grousset/tuxayo from comment #62)
> Upstream answer
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496#10
> 
> > Indeed, the OpenAPI schema files are Apache2-licensed [3], 
> > thus at least them could be retained.
> > However, copies of these schema files are already around in Debian [4],
> > thus I would suggest packaging OpenAPI schema files in a separate
> > Debian package, and symlinking them to JSON::Validator cache location.
> 
> Help welcome to try to find arguments to stop removing the schema files. The
> duplication concern is fair but I don't think someone here has the resources
> to push towards a common schema Debian package. Or maybe I'm wrong and it's
> not much work? hmmmm ^^"

We don't want to rely on a remote file/server to start up correctly anyway.
Comment 66 Jonathan Druart 2020-08-25 13:49:24 UTC
What are we waiting for exactly? Can I get feedback ASAP please?
Comment 67 Tomás Cohen Arazi 2020-08-25 14:47:50 UTC
Created attachment 109095 [details] [review]
Bug 23653: use local copy of swagger v2 schema

By default, JSON::Validator::OpenAPI tries to fetch the
swagger v2 schema from http://swagger.io/v2/schema.json.

If you've installed from CPAN, JSON::Validator::OpenAPI will
come with a cached copy, so it won't try to fetch it over HTTP.

However, if you've installed from libjson-validator-perl
from Debian/Ubuntu, the Debian package excludes the cached copy,
so JSON::Validator::OpenAPI tries to fetch it over HTTP.

Unfortunately, today and other days in the past, the file at
http://swagger.io/v2/schema.json has been unavailable, and this causes
Koha to crash in a perpetual loop.

This patch includes a copy of the swagger v2 schema, and it loads
it locally rather than fetching over HTTP.

The changes to Koha/REST/Plugin/PluginRoutes.pm are not required,
since the validator isn't currently called there, but I've added
a patch to future proof it.

To Test:
0a) Remove /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df
if it exists
0b) Block external access to http://swagger.io/v2/schema.json or
test during an outage when it's unavailable
0c) Do not apply patch
1) koha-plack --restart kohadev
2) Note that it crashes in a loop and is unavailable in web browser
3) Apply patch
4) koha-plack --restart kohadev
5) Note that Koha comes up and there are no errors in the Plack logs

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 68 Tomás Cohen Arazi 2020-08-25 14:47:56 UTC
Created attachment 109096 [details] [review]
Bug 23653: Remove uneeded cond test

rel_file returns the path anyway

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 69 Tomás Cohen Arazi 2020-08-25 14:49:58 UTC
Do we need to acknowledge the file is from another project, and its license?
Comment 70 Tomás Cohen Arazi 2020-08-25 15:01:07 UTC
Created attachment 109113 [details] [review]
Bug 23653: Add license information in about.pl

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 71 Jonathan Druart 2020-08-26 13:21:38 UTC
Created attachment 109180 [details] [review]
Bug 23653: Add license information in about.pl

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

JD Amended patch
            <p>The included <a href="https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v2.0/schema.json">api/swagger-v2-schema.json</a> file is licensed under the[-the-] <a href="https://github.com/OAI/OpenAPI-Specification/tree/master/schemas/v2.0">Apache License, Version 2.0</a>, by the <a href="https://www.openapis.org/about">OpenAPI Initiative [-(OAI)</a></p>-]{+(OAI)</a>.</p>+}
Comment 72 Jonathan Druart 2020-08-26 13:23:40 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 73 Victor Grousset/tuxayo 2020-11-13 20:37:10 UTC
(In reply to Jonathan Druart from comment #65)
> (In reply to Victor Grousset/tuxayo from comment #62)
> > Upstream answer
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496#10
> > 
[...]
> > Help welcome to try to find arguments to stop removing the schema files. The
> > duplication concern is fair but I don't think someone here has the resources
> > to push towards a common schema Debian package. Or maybe I'm wrong and it's
> > not much work? hmmmm ^^"
> 
> We don't want to rely on a remote file/server to start up correctly anyway.

The request was to include again the schema file into the Debian package to precisely not depend on a remote file.
Comment 74 Victor Grousset/tuxayo 2020-11-17 11:47:54 UTC
Should this patch be backported?
I just had a test failure on Ubuntu 18.04 due to not being able to reach the schema file at swagger.io


https://jenkins.koha-community.org/view/19.05/job/Koha_19.05_U18/320/consoleText
koha_1       | [JSON::Validator] GET http://swagger.io/v2/schema.json == Inactivity timeout at /usr/share/perl5/JSON/Validator.pm line 274.
koha_1       | # Looks like your test exited with 255 before it could output anything.
koha_1       | [10:16:53] t/db_dependent/api/v1/illrequests.t ..................................... 
koha_1       | Dubious, test returned 255 (wstat 65280, 0xff00)
koha_1       | Failed 1/1 subtests
Comment 75 Victor Grousset/tuxayo 2020-11-17 20:58:49 UTC
Two other same in this run:
https://jenkins.koha-community.org/view/19.05/job/Koha_19.05_U18/322/consoleText