I had plack working fine before this morning, but now if I have plack enabled, Koha does not respond to web requests, and apache eventually shows me a 502. These are the entries from plack-error.log. I don't know what swagger.io is - seems there is such a website, but I can't `curl` it from the webserver, at least right now. plack-error.log ```` Error while loading /etc/koha/plack.psgi: Can't load application from file "/usr/share/koha/api/v1/app.pl": [JSON::Validator] GET http://swagger.io/v2/schema.json == Inactivity timeout at /usr/share/perl5/JSON/Validator.pm line 274. Compilation failed in require at (eval 389) line 1. Child process 3159 exited with status 255 ```` The only change I made this morning was changing the server's timezone, but I doubt that's related. I did put it back to UTC, but no improvement. Disabling plack works for me, but my site is a lot slower (and it wasn't real quick to begin with).
I observed later that I was able to reach swagger.io from the server, so I tried reenabling plack and now it's working. So, this may properly be a plack issue and not a koha issue, but it seems like my library site ought to work whether or not I can connect to swagger.io, whatever that is.
Can you give us more information on your setup? The 'swagger file' is bundled in Koha and there shouldn't be any external system reached out. It would help if you additionally: $ sudo apt install pmtools and then we need the output for: $ pmvers JSON::Validator $ pmvers Mojolicious $ pmvers Mojolicious::Plugin::OpenAPI
```` rick@koha:~$ pmvers JSON::Validator 1.08 rick@koha:~$ pmvers Mojolicious 7.59 rick@koha:~$ pmvers Mojolicious::Plugin::OpenAPI 1.15 ```` Also ```` rick@koha:~$ cd / rick@koha:/$ sudo find . -name swagger ./usr/share/koha/api/v1/swagger rick@koha:/$ cd /usr/share/koha/api/v1/swagger/ rick@koha:/usr/share/koha/api/v1/swagger$ ls definitions parameters paths swagger.json definitions.json parameters.json paths.json x-primitives.json ```
I was able to reproduce this on a fresh installation of koha 20.05 from packages on debian buster. I have the webserver behind a firewall blocking all outbound traffic and could not get the koha installer to launch. After digging into the logs I saw the same connection timeouts to swagger.io As soon as I opened up our firewall the installer loaded. You can also reproduce this by dropping something bogus in your /etc/hosts file for swagger.io and restarting plack For example - 127.0.0.1 swagger.io
(In reply to Rick Conrad from comment #3) > ```` > rick@koha:~$ pmvers JSON::Validator > 1.08 > rick@koha:~$ pmvers Mojolicious > 7.59 > rick@koha:~$ pmvers Mojolicious::Plugin::OpenAPI > 1.15 > ```` hi Rick which debian version is this output from? buster?
I don't have it on-hand, but I believe it was Ubuntu 18.04.
(In reply to Rick Conrad from comment #6) > I don't have it on-hand, but I believe it was Ubuntu 18.04. for 18.04, try running these commands $ sudo echo 'deb http://debian.koha-community.org/koha 20.05 main bionic' > /etc/apt/sources.list.d/koha.list $ sudo apt-get update; apt-get -y upgrade $ sudo apt install pmtools you should have... $ pmvers JSON::Validator 3.06 $ pmvers Mojolicious 8.12 $ pmvers Mojolicious::Plugin::OpenAPI 2.12
This looks really bad if Koha is trying to contact an external site on start-up
Here is the output from my fresh buster server with koha-common 20.05.00-1 $ pmvers JSON::Validator 3.06 $ pmvers Mojolicious 8.12 $ pmvers Mojolicious::Plugin::OpenAPI 2.12 From plack-error.og Compilation failed in require at (eval 410) line 1. Child process 25034 exited with status 255 Error while loading /etc/koha/plack.psgi: Can't load application from file "/usr/share/koha/api/v1/app.pl": [JSON::Validator] GET http://swagger.io/v2/schema.json == Not Found at /usr/share/perl5/JSON/Validator.pm line 343. Compilation failed in require at (eval 410) line 1. Child process 25042 exited with status 255 Error while loading /etc/koha/plack.psgi: Can't load application from file "/usr/share/koha/api/v1/app.pl": [JSON::Validator] GET http://swagger.io/v2/schema.json == Not Found at /usr/share/perl5/JSON/Validator.pm line 343.
hmm, this does look like a legitimate bug, caused by 22522 the 'try' block always fails with... Can't locate object method "bundle" via package "JSON::Validator::OpenAPI::Mojolicious" at /kohadevbox/koha/Koha/REST/V1.pm line 76. ...and the 'catch' block calls validator->load_and_validate_schema() which does a lookup to swagger.io ./Koha/REST/V1.pm 72 73 # Try to load and merge all schemas first and validate the result just once. 74 my $spec; 75 try { 76 $spec = $validator->bundle( 77 { 78 replace => 1, 79 schema => $self->home->rel_file("api/v1/swagger/swagger.json") 80 } 81 ); 82 100 } 101 catch { 102 # Validation of the complete spec failed. Resort to validation one-by-one 103 # to catch bad ones. 104 $validator->load_and_validate_schema( 105 $self->home->rel_file("api/v1/swagger/swagger.json"), 106 { 107 allow_invalid_ref => 1, 108 } 109 );
(In reply to Mason James from comment #10) > hmm, this does look like a legitimate bug, caused by 22522 ^ oops, ignore that - i was testing on jessie :) the problem seems to be that the debian libjson-validator-perl package has the 'cache' dir omitted, which contains local copies of the various schemas Rick, a quick solution to the problem is to restore the cached schemas $ git clone https://github.com/mojolicious/json-validator $ cd json-validator $ git reset --hard 3.06 HEAD is now at 8b19ba0 Released version 3.06 $ sudo cp -a ./lib/JSON/Validator/cache /usr/share/perl5/JSON/Validator/
I followed the steps Jason provided to restore the cache folder on debian buster. Plack was able to start even when access to swagger.io was blocked. Thanks!
(In reply to Mason James from comment #11) > (In reply to Mason James from comment #10) > > hmm, this does look like a legitimate bug, caused by 22522 > > ^ oops, ignore that - i was testing on jessie :) > > > the problem seems to be that the debian libjson-validator-perl package has > the 'cache' dir omitted, which contains local copies of the various schemas > looks like the schemas were removed due to a license issue... Source: https://metacpan.org/release/JSON-Validator Upstream-Contact: Jan Henning Thorsen <jhthorsen@cpan.org> Upstream-Name: JSON-Validator Files-Excluded: lib/JSON/Validator/cache Comment: The +dfsg version is created by removing the cache files from the lib/JSON/Validator directory, where the license is unknown. https://salsa.debian.org/perl-team/modules/packages/libjson-validator-perl/-/raw/master/debian/copyright i'll attempt to build a new package with the schemas included
(In reply to Mason James from comment #13) > (In reply to Mason James from comment #11) > > (In reply to Mason James from comment #10) > > > hmm, this does look like a legitimate bug, caused by 22522 > > > > ^ oops, ignore that - i was testing on jessie :) > > > > > > the problem seems to be that the debian libjson-validator-perl package has > > the 'cache' dir omitted, which contains local copies of the various schemas > > > > looks like the schemas were removed due to a license issue... > > Source: https://metacpan.org/release/JSON-Validator > Upstream-Contact: Jan Henning Thorsen <jhthorsen@cpan.org> > Upstream-Name: JSON-Validator > Files-Excluded: lib/JSON/Validator/cache > Comment: The +dfsg version is created by removing the cache files from the > lib/JSON/Validator directory, where the license is unknown. > > https://salsa.debian.org/perl-team/modules/packages/libjson-validator-perl/-/ > raw/master/debian/copyright > > > > i'll attempt to build a new package with the schemas included That's weird. The schema is clearly licensed on the OpenAPI repository. It maybe I got it is about the process to acknowledge that on Debian side: https://github.com/OAI/OpenAPI-Specification/blob/master/LICENSE
This bug is marked as only affecting version 19.05.x, but it seem to affect 20.05 (as likely master) also, right?
== Test plan == 1. In /etc/hosts , add «127.0.0.1 swagger.io» 2. try to load a page 3. it works 4. koha-plack --restart kohadev 5. try to load a page 6. endless or timeout loading 7. remove the sabotage from /etc/hosts 8. after a few seconds, the page will load Confirmed on master. It works on 19.05 (In reply to Mason James from comment #10) > hmm, this does look like a legitimate bug, caused by 22522 Possibly. Because 19.05.x doesn't have bug 22522 patches
Hi, At about 21:00UTC today we also started to have this problem. We're running 19.05.12 on Ubuntu 18.04. The solution in comment 11 seems to work for now. % pmvers JSON::Validator 1.08 % pmvers Mojolicious 7.59 % pmvers Mojolicious::Plugin::OpenAPI 1.15 Error message in plack-error.log is: Error while loading /etc/koha/plack.psgi: Can't load application from file "/usr/share/koha/api/v1/app.pl": [JSON::Validator] GET http://swagger.io/v2/schema.json == Connect timeout at /usr/share/perl5/JSON/Validator.pm line 274. Compilation failed in require at (eval 389) line 1. Regards, Ian Gordon.
Issue reported to upstream's upstream: https://github.com/swagger-api/swagger.io/issues/281
A solution to this problem is to provide our own argument "schema" argument to the load_and_validate_schema() method. https://metacpan.org/pod/JSON::Validator#load_and_validate_schema I've looked through the code and there's a variety of ways of providing the schema, which we can source from https://raw.githubusercontent.com/mojolicious/json-validator/3.06/lib/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df I'd say we should provide it as a local file, so that we're not relying on any web servers to provide this crucial file.
The same thing happened to me, I solved it by doing >cpan > install Mojolicious::Plugin::OpenAPI
(In reply to Federico Paiz from comment #20) > The same thing happened to me, I solved it by doing > > >cpan > > install Mojolicious::Plugin::OpenAPI Yeah, the CPAN version of that module comes with the "cache" which the Debian package excludes, so you'll be fine with that. But it's a short-term solution.
I'm about to start on a patch for this. Hopefully will have something very soon.
(In reply to David Cook from comment #22) > I'm about to start on a patch for this. Hopefully will have something very > soon. We should just package the library with the correct cache
Note that this should only affect plugin-enabled instances
(In reply to Tomás Cohen Arazi from comment #23) > (In reply to David Cook from comment #22) > > I'm about to start on a patch for this. Hopefully will have something very > > soon. > > We should just package the library with the correct cache Maybe. I think there is more to this than we think. Consider ./Koha/REST/V1.pm: $validator->bundle uses a schema of $self->home->rel_file("api/v1/swagger/swagger.json"), which is a drastically reduced schema file. Calls to $validator->load_and_validate_schema (in ./Koha/REST/V1.pm and ./Koha/REST/Plugin/PluginRoutes.pm) don't use that same schema. They try to fetch http://swagger.io/v2/schema.json. The question I have is whether they should all be using $self->home->rel_file("api/v1/swagger/swagger.json"), or if *some* calls should use our local swagger.json and others should use the full swagger 2.0 schema.
Oh wait I'm misunderstanding... $validator->bundle is just loading the schema... and it doesn't validate it. Whereas $validator->load_and_validate_schema loads the schema, but then validates it against the Swagger 2.0 API schema. Wow, JSON::Validator really overloads the word "schema" in its code...
From what I can see, we're not actually validating the schema in ./Koha/REST/V1.pm when we use bundle(). We only validate it in ./Koha/REST/V1.pm when we do the catch{} block. And we always validate ./Koha/REST/Plugin/PluginRoutes.pm. Ok... yeah re-packaging libjson-validator-perl to include the "cache" probably makes the most sense. I'm going to post a code-level patch anyway (partially just because I'm already this far anyway)
(In reply to David Cook from comment #27) > I'm going to post a code-level patch anyway (partially just because I'm > already this far anyway) Plus, if I do that, then we wouldn't have to maintain another custom package dependency...
Actually, it looks like Koha::REST::Plugin::PluginRoutes doesn't receive the validator. It looks like the OpenAPI plugin must be calling a validator and that's where it's failing...
(In reply to Tomás Cohen Arazi from comment #23) > We should just package the library with the correct cache Indeed, that way, deployers only have to update the perl-deps, no patch to apply.
Created attachment 106444 [details] [review] Bug 23653: use local copy of swagger v2 schema By default, JSON::Validator::OpenAPI tries to fetch the swagger v2 schema from http://swagger.io/v2/schema.json. If you've installed from CPAN, JSON::Validator::OpenAPI will come with a cached copy, so it won't try to fetch it over HTTP. However, if you've installed from libjson-validator-perl from Debian/Ubuntu, the Debian package excludes the cached copy, so JSON::Validator::OpenAPI tries to fetch it over HTTP. Unfortunately, today and other days in the past, the file at http://swagger.io/v2/schema.json has been unavailable, and this causes Koha to crash in a perpetual loop. This patch includes a copy of the swagger v2 schema, and it loads it locally rather than fetching over HTTP. The changes to Koha/REST/Plugin/PluginRoutes.pm are not required, since the validator isn't currently called there, but I've added a patch to future proof it. To Test: 0a) Remove /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df if it exists 0b) Block external access to http://swagger.io/v2/schema.json or test during an outage when it's unavailable 0c) Do not apply patch 1) koha-plack --restart kohadev 2) Note that it crashes in a loop and is unavailable in web browser 3) Apply patch 4) koha-plack --restart kohadev 5) Note that Koha comes up and there are no errors in the Plack logs
Created attachment 106445 [details] [review] Bug 23653: use local copy of swagger v2 schema By default, JSON::Validator::OpenAPI tries to fetch the swagger v2 schema from http://swagger.io/v2/schema.json. If you've installed from CPAN, JSON::Validator::OpenAPI will come with a cached copy, so it won't try to fetch it over HTTP. However, if you've installed from libjson-validator-perl from Debian/Ubuntu, the Debian package excludes the cached copy, so JSON::Validator::OpenAPI tries to fetch it over HTTP. Unfortunately, today and other days in the past, the file at http://swagger.io/v2/schema.json has been unavailable, and this causes Koha to crash in a perpetual loop. This patch includes a copy of the swagger v2 schema, and it loads it locally rather than fetching over HTTP. The changes to Koha/REST/Plugin/PluginRoutes.pm are not required, since the validator isn't currently called there, but I've added a patch to future proof it. To Test: 0a) Remove /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df if it exists 0b) Block external access to http://swagger.io/v2/schema.json or test during an outage when it's unavailable 0c) Do not apply patch 1) koha-plack --restart kohadev 2) Note that it crashes in a loop and is unavailable in web browser 3) Apply patch 4) koha-plack --restart kohadev 5) Note that Koha comes up and there are no errors in the Plack logs Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
It works! :D (In reply to Victor Grousset/tuxayo from comment #30) > Indeed, that way, deployers only have to update the perl-deps, no patch to > apply. Should we actually expect a significant number of instances to have more difficulty applying such a simple patch (just this patch, not updating Koha) compared to updating the perl deps? The question is whether or not packaging libjson-validator-perl during some time is worth it. Not sure now. Also, if http://swagger.io/v2/schema.json would work again tomorrow that would help letting those instances count on that until their next Koha update.
(In reply to Victor Grousset/tuxayo from comment #33) > Should we actually expect a significant number of instances to have more > difficulty applying such a simple patch (just this patch, not updating Koha) > compared to updating the perl deps? > > The question is whether or not packaging libjson-validator-perl during some > time is worth it. Not sure now. > > Also, if http://swagger.io/v2/schema.json would work again tomorrow that > would help letting those instances count on that until their next Koha > update. I'm curious about this as well. While we felt this in production in Australia, I'm guessing the rest of the world might not feel it for another few hours. I don't know where Jonathan is physically located these days, but I'm guessing that we may have just missed him as well, so the Koha world is in for an interesting day...
Mason announced 30 minutes ago on koha-devel that he's updated the libjson-validator-perl package. As per his email: hi Folks i've updated the 'libjson-validator-perl' package to include the cache dir to update the package... $ sudo apt update $ sudo apt install libjson-validator-perl run the following command, and confirm the result... $ apt-cache policy libjson-validator-perl libjson-validator-perl: Installed: 3.06+dfsg-1+koha1 Candidate: 3.06+dfsg-1+koha1
I've tested the new package in koha-testing-docker and it looks good.
I'd recommend to people starting their day to look at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23653#c11 for a quick fix to get through the worst of things right now.
I think Mason found that newer versions of Ubuntu weren't installing 3.06 as they were using newer versions, but it looks like he's now released a 3.23+dfsg-1+koha1 version. I've installed that on Debian Jessie in koha-testing-docker, and it looks like it's working successfully. So folk using the Debian packages shoudl be able to just do the following: apt-get update apt-get install libjson-validator-perl
Ok that 3.23 is still in the staging repo. If you're running a Debian/Ubuntu package, you'll be vulnerable, but not if you're running a Koha Community package. So you should be fine on Debian Jessie and Debian Stretch and Ubuntu Xenial. If you're on anything newer than those, you'll need to upgrade to the Koha Community 3.06 (or 3.23 when it is available) as per the table below: __Koha Community Apt Repo__ libjson-validator-perl_0.67+dfsg-1~koha1_all.deb libjson-validator-perl_0.97-1~kohadev1_all.deb libjson-validator-perl_3.06+dfsg-1+koha1_all.deb __Debian__ Stretch 9: libjson-validator-perl (0.92+dfsg-1) Buster 10: libjson-validator-perl (3.06+dfsg-1) Bullseye 11: libjson-validator-perl (3.25+dfsg-1) __Ubuntu__ Xenial 16.04: libjson-validator-perl (0.66+dfsg-1) Bionic 18.04: libjson-validator-perl (1.08+dfsg-1) Eoan 19.10: libjson-validator-perl (3.14+dfsg-1) Focal 20.04: libjson-validator-perl (3.23+dfsg-1) Groovy 20.10: libjson-validator-perl (3.25+dfsg-1)
Here's a condensed version of the above: Good (don't worry): - Debian Jessie 8 - Debian Stretch 9 - Ubuntu Xenial 16.04 Bad (do worry): - Debian Buster 10 (and above) - Ubuntu Bionic 18.04 (and above)
For Ubuntu Debian, we at biblibre installed from CPAN JSON::Validator 4.01 and added the cache file. It works without patch on sources.
The schema hosting at swagger.io is back https://github.com/swagger-api/swagger.io/issues/282
(In reply to Victor Grousset/tuxayo from comment #42) > The schema hosting at swagger.io is back > https://github.com/swagger-api/swagger.io/issues/282 This is great! Although I hope this doesn't cause us to forget about the issue...
(In reply to David Cook from comment #35) > Mason announced 30 minutes ago on koha-devel that he's updated the > libjson-validator-perl package. > > As per his email: > > hi Folks > i've updated the 'libjson-validator-perl' package to include the cache dir > > to update the package... > $ sudo apt update > $ sudo apt install libjson-validator-perl > > run the following command, and confirm the result... > $ apt-cache policy libjson-validator-perl > libjson-validator-perl: > Installed: 3.06+dfsg-1+koha1 > Candidate: 3.06+dfsg-1+koha1 I've updated the libjson-validator-perl and libmojolicious-plugin-openapi-perl packages again, to make them work on debian/ubuntu 'unstable' $ apt-cache policy libjson-validator-perl libmojolicious-plugin-openapi-perl libjson-validator-perl: Installed: 3.25+dfsg-1+koha3 Candidate: 3.25+dfsg-1+koha3 Version table: *** 3.25+dfsg-1+koha3 500 500 http://debian.koha-community.org/koha stable/main amd64 Packages 100 /var/lib/dpkg/status .. libmojolicious-plugin-openapi-perl: Installed: 2.16-1+koha1 Candidate: 2.16-1+koha1 Version table: *** 2.16-1+koha1 500 500 http://debian.koha-community.org/koha stable/main amd64 Packages 100 /var/lib/dpkg/status
(In reply to Fridolin SOMERS from comment #41) > For Ubuntu Debian, we at biblibre installed from CPAN JSON::Validator 4.01 > and added the cache file. > It works without patch on sources. Confirming the same: just using cpanm with only one this updated all quickfixed, temporarily for libs, but now I am learning how to make this solution stable, reading all above & tracking :).
How often this schema changes ? Need to check for updates ?
From what I understand, it's a schema from a version of the OpenAPI Specification. Something like the draft 4 of the version 2. Here we can compare a version from 2014 and from now. https://web.archive.org/web/20141019005411/https://swagger.io/v2/schema.json vs https://web.archive.org/web/20200702172005/https://schema.swagger.io/v2/schema.json Here is the actual source it seems: https://github.com/OAI/OpenAPI-Specification/blob/c196cac8191069fa97028c2f1ae322d11a74f891/schemas/v2.0/schema.json vs https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v2.0/schema.json I would *guess* it will stay backward compatible.
Note by the way, these changes with dependencies brought issues with fresh installations: on Ubuntu 18.04 with those regular steps which worked everywhere usually before, (preps + "apt-get install koha-common" and voila) now fails with: install 'koha-common' failed: Unable to correct problems, you have held broken packages. Depends: libmojolicious-plugin-openapi-perl but it is not going to be installed Depends: libnet-oauth2-authorizationserver-perl but it is not going to be installed This was when: "deb http://debian.koha-community.org/koha 20.05 main" (https://wiki.koha-community.org/wiki/Koha_on_Debian#Follow_a_codename_.28version_number.29:_20.05.2C_19.11.2C_19.05) but adding mentioned above explicit release name: "deb http://debian.koha-community.org/koha 20.05 main bionic" made install seamless.
Comment on attachment 106445 [details] [review] Bug 23653: use local copy of swagger v2 schema Review of attachment 106445 [details] [review]: ----------------------------------------------------------------- ::: Koha/REST/Plugin/PluginRoutes.pm @@ +49,5 @@ > { > + my $schema = $app->home->rel_file("api/swagger-v2-schema.json"); > + if ($schema){ > + $self->{'swagger-v2-schema'} = $schema; > + } What is the test for ? rel_file returns something even if the file does not exist
The cached copy of the schema is slightly different from the one at https://schema.swagger.io/v2/schema.json Where does it come from ?
(In reply to Julian Maurice from comment #50) > The cached copy of the schema is slightly different from the one at > https://schema.swagger.io/v2/schema.json > Where does it come from ? From what seems to be the canonical source: https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v2.0/schema.json
Do we need a unit test to compare the local and remote files? (ofc it won't fail if the remove file is not reachable)
(In reply to Jonathan Druart from comment #52) > Do we need a unit test to compare the local and remote files? (ofc it won't > fail if the remove file is not reachable) I think it's not worth it. Swagger/OpenAPI 2.0 doesn't change anymore (last change was in 2017)
(In reply to Jonathan Druart from comment #52) > Do we need a unit test to compare the local and remote files? (ofc it won't > fail if the remove file is not reachable) my 2 cents: don't think it's worth it also. Because here we can see migration paths from the different draft versions: https://json-schema.org/specification.html#migrating-from-older-drafts So backward compatibility seem to be assured.
(In reply to Julian Maurice from comment #49) > What is the test for ? rel_file returns something even if the file does not > exist That's interesting. I didn't think it did for me when I tested. I'd have to look gain.
(In reply to Jonathan Druart from comment #52) > Do we need a unit test to compare the local and remote files? (ofc it won't > fail if the remove file is not reachable) I wouldn't think so. If our other API unit tests are passing, I'd say that's what would matter most.
Ticket opened upstream (Debian) to have the cache (not really cache) files included in the package. As upstream's(Debian) upstream's(json-validator) upstream (OpenAPI/Swagger) confirmed the license of these files. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496
(In reply to Victor Grousset/tuxayo from comment #57) > Ticket opened upstream (Debian) to have the cache (not really cache) files > included in the package. > As upstream's(Debian) upstream's(json-validator) upstream (OpenAPI/Swagger) > confirmed the license of these files. > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496 Nice one, Victor!
Failing QA to get an answer to comment 49
IIUC this is no longer blocker, lowering severity.
(In reply to Julian Maurice from comment #59) > Failing QA to get an answer to comment 49 Yeah that's fair. I'll put it on my TODO list.
Upstream answer https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496#10 > Indeed, the OpenAPI schema files are Apache2-licensed [3], > thus at least them could be retained. > However, copies of these schema files are already around in Debian [4], > thus I would suggest packaging OpenAPI schema files in a separate > Debian package, and symlinking them to JSON::Validator cache location. Help welcome to try to find arguments to stop removing the schema files. The duplication concern is fair but I don't think someone here has the resources to push towards a common schema Debian package. Or maybe I'm wrong and it's not much work? hmmmm ^^"
Created attachment 107984 [details] [review] Bug 23653: Remove uneeded cond test rel_file returns the path anyway
(In reply to Julian Maurice from comment #49) > Comment on attachment 106445 [details] [review] [review] > Bug 23653: use local copy of swagger v2 schema > > Review of attachment 106445 [details] [review] [review]: > ----------------------------------------------------------------- > > ::: Koha/REST/Plugin/PluginRoutes.pm > @@ +49,5 @@ > > { > > + my $schema = $app->home->rel_file("api/swagger-v2-schema.json"); > > + if ($schema){ > > + $self->{'swagger-v2-schema'} = $schema; > > + } > > What is the test for ? rel_file returns something even if the file does not > exist I removed the test. Ready for QA.
(In reply to Victor Grousset/tuxayo from comment #62) > Upstream answer > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496#10 > > > Indeed, the OpenAPI schema files are Apache2-licensed [3], > > thus at least them could be retained. > > However, copies of these schema files are already around in Debian [4], > > thus I would suggest packaging OpenAPI schema files in a separate > > Debian package, and symlinking them to JSON::Validator cache location. > > Help welcome to try to find arguments to stop removing the schema files. The > duplication concern is fair but I don't think someone here has the resources > to push towards a common schema Debian package. Or maybe I'm wrong and it's > not much work? hmmmm ^^" We don't want to rely on a remote file/server to start up correctly anyway.
What are we waiting for exactly? Can I get feedback ASAP please?
Created attachment 109095 [details] [review] Bug 23653: use local copy of swagger v2 schema By default, JSON::Validator::OpenAPI tries to fetch the swagger v2 schema from http://swagger.io/v2/schema.json. If you've installed from CPAN, JSON::Validator::OpenAPI will come with a cached copy, so it won't try to fetch it over HTTP. However, if you've installed from libjson-validator-perl from Debian/Ubuntu, the Debian package excludes the cached copy, so JSON::Validator::OpenAPI tries to fetch it over HTTP. Unfortunately, today and other days in the past, the file at http://swagger.io/v2/schema.json has been unavailable, and this causes Koha to crash in a perpetual loop. This patch includes a copy of the swagger v2 schema, and it loads it locally rather than fetching over HTTP. The changes to Koha/REST/Plugin/PluginRoutes.pm are not required, since the validator isn't currently called there, but I've added a patch to future proof it. To Test: 0a) Remove /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df if it exists 0b) Block external access to http://swagger.io/v2/schema.json or test during an outage when it's unavailable 0c) Do not apply patch 1) koha-plack --restart kohadev 2) Note that it crashes in a loop and is unavailable in web browser 3) Apply patch 4) koha-plack --restart kohadev 5) Note that Koha comes up and there are no errors in the Plack logs Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 109096 [details] [review] Bug 23653: Remove uneeded cond test rel_file returns the path anyway Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Do we need to acknowledge the file is from another project, and its license?
Created attachment 109113 [details] [review] Bug 23653: Add license information in about.pl Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 109180 [details] [review] Bug 23653: Add license information in about.pl Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> JD Amended patch <p>The included <a href="https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v2.0/schema.json">api/swagger-v2-schema.json</a> file is licensed under the[-the-] <a href="https://github.com/OAI/OpenAPI-Specification/tree/master/schemas/v2.0">Apache License, Version 2.0</a>, by the <a href="https://www.openapis.org/about">OpenAPI Initiative [-(OAI)</a></p>-]{+(OAI)</a>.</p>+}
Pushed to master for 20.11, thanks to everybody involved!
(In reply to Jonathan Druart from comment #65) > (In reply to Victor Grousset/tuxayo from comment #62) > > Upstream answer > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964496#10 > > [...] > > Help welcome to try to find arguments to stop removing the schema files. The > > duplication concern is fair but I don't think someone here has the resources > > to push towards a common schema Debian package. Or maybe I'm wrong and it's > > not much work? hmmmm ^^" > > We don't want to rely on a remote file/server to start up correctly anyway. The request was to include again the schema file into the Debian package to precisely not depend on a remote file.
Should this patch be backported? I just had a test failure on Ubuntu 18.04 due to not being able to reach the schema file at swagger.io https://jenkins.koha-community.org/view/19.05/job/Koha_19.05_U18/320/consoleText koha_1 | [JSON::Validator] GET http://swagger.io/v2/schema.json == Inactivity timeout at /usr/share/perl5/JSON/Validator.pm line 274. koha_1 | # Looks like your test exited with 255 before it could output anything. koha_1 | [10:16:53] t/db_dependent/api/v1/illrequests.t ..................................... koha_1 | Dubious, test returned 255 (wstat 65280, 0xff00) koha_1 | Failed 1/1 subtests
Two other same in this run: https://jenkins.koha-community.org/view/19.05/job/Koha_19.05_U18/322/consoleText
(In reply to Victor Grousset/tuxayo from comment #74) > Should this patch be backported? > I just had a test failure on Ubuntu 18.04 due to not being able to reach the > schema file at swagger.io hi Victor, we really need this backported, back to 19.11.x https://swagger.io/v2/schema.json was down recently and still causing breakage i'll set the ticket back to open
Backported: Pushed to 20.05.x branch for 20.05.13
This does not apply cleanly to 19.11.x please submit one formatted for 19.11.x The merge conflict in Koha/REST/V1.pm is quite large and complex so we weren't confident trying to fix it. (I did fix the merge conflict in Koha/REST/Plugin/PluginRoutes.pm okay though)
The above comment refers to the first patch: Bug 23653: use local copy of swagger v2 schema
Created attachment 122333 [details] [review] Bug 23653: [19.11] use local copy of swagger v2 schema By default, JSON::Validator::OpenAPI tries to fetch the swagger v2 schema from http://swagger.io/v2/schema.json. If you've installed from CPAN, JSON::Validator::OpenAPI will come with a cached copy, so it won't try to fetch it over HTTP. However, if you've installed from libjson-validator-perl from Debian/Ubuntu, the Debian package excludes the cached copy, so JSON::Validator::OpenAPI tries to fetch it over HTTP. Unfortunately, today and other days in the past, the file at http://swagger.io/v2/schema.json has been unavailable, and this causes Koha to crash in a perpetual loop. This patch includes a copy of the swagger v2 schema, and it loads it locally rather than fetching over HTTP. The changes to Koha/REST/Plugin/PluginRoutes.pm are not required, since the validator isn't currently called there, but I've added a patch to future proof it. To Test: 0a) Remove /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df if it exists 0b) Block external access to http://swagger.io/v2/schema.json or test during an outage when it's unavailable 0c) Do not apply patch 1) koha-plack --restart kohadev 2) Note that it crashes in a loop and is unavailable in web browser 3) Apply patch 4) koha-plack --restart kohadev 5) Note that Koha comes up and there are no errors in the Plack logs Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 122334 [details] [review] Bug 23653: [19.11] Remove uneeded cond test rel_file returns the path anyway Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 122335 [details] [review] Bug 23653: [19.11] Add license information in about.pl Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> JD Amended patch <p>The included <a href="https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v2.0/schema.json">api/swagger-v2-schema.json</a> file is licensed under the[-the-] <a href="https://github.com/OAI/OpenAPI-Specification/tree/master/schemas/v2.0">Apache License, Version 2.0</a>, by the <a href="https://www.openapis.org/about">OpenAPI Initiative [-(OAI)</a></p>-]{+(OAI)</a>.</p>+}
Wainui, I rebased the patches for 19.11.x but I'd like someone with a plugin injecting REST API route to test the patches before you push them.
(In reply to Jonathan Druart from comment #83) > Wainui, I rebased the patches for 19.11.x but I'd like someone with a plugin > injecting REST API route to test the patches before you push them. Thanks Jonathan. They applied cleanly to my branch, but I won't test and push them up until the next release (19.11.20).
(In reply to wainuiwitikapark from comment #84) > (In reply to Jonathan Druart from comment #83) > > Wainui, I rebased the patches for 19.11.x but I'd like someone with a plugin > > injecting REST API route to test the patches before you push them. > > Thanks Jonathan. They applied cleanly to my branch, but I won't test and > push them up until the next release (19.11.20). I can test in the morning.
(In reply to Tomás Cohen Arazi from comment #85) > (In reply to wainuiwitikapark from comment #84) > > (In reply to Jonathan Druart from comment #83) > > > Wainui, I rebased the patches for 19.11.x but I'd like someone with a plugin > > > injecting REST API route to test the patches before you push them. > > > > Thanks Jonathan. They applied cleanly to my branch, but I won't test and > > push them up until the next release (19.11.20). > > I can test in the morning. Hi Tomas, I'm just wondering if you have had a chance to test this yet? Thanks, Wainui :)
(In reply to wainuiwitikapark from comment #86) > (In reply to Tomás Cohen Arazi from comment #85) > > (In reply to wainuiwitikapark from comment #84) > > > (In reply to Jonathan Druart from comment #83) > > > > Wainui, I rebased the patches for 19.11.x but I'd like someone with a plugin > > > > injecting REST API route to test the patches before you push them. > > > > > > Thanks Jonathan. They applied cleanly to my branch, but I won't test and > > > push them up until the next release (19.11.20). > > > > I can test in the morning. > > Hi Tomas, > > I'm just wondering if you have had a chance to test this yet? > > Thanks, > > Wainui :) Doing it right now
Created attachment 122650 [details] [review] Bug 23653: [19.11] use local copy of swagger v2 schema By default, JSON::Validator::OpenAPI tries to fetch the swagger v2 schema from http://swagger.io/v2/schema.json. If you've installed from CPAN, JSON::Validator::OpenAPI will come with a cached copy, so it won't try to fetch it over HTTP. However, if you've installed from libjson-validator-perl from Debian/Ubuntu, the Debian package excludes the cached copy, so JSON::Validator::OpenAPI tries to fetch it over HTTP. Unfortunately, today and other days in the past, the file at http://swagger.io/v2/schema.json has been unavailable, and this causes Koha to crash in a perpetual loop. This patch includes a copy of the swagger v2 schema, and it loads it locally rather than fetching over HTTP. The changes to Koha/REST/Plugin/PluginRoutes.pm are not required, since the validator isn't currently called there, but I've added a patch to future proof it. To Test: 0a) Remove /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df if it exists 0b) Block external access to http://swagger.io/v2/schema.json or test during an outage when it's unavailable 0c) Do not apply patch 1) koha-plack --restart kohadev 2) Note that it crashes in a loop and is unavailable in web browser 3) Apply patch 4) koha-plack --restart kohadev 5) Note that Koha comes up and there are no errors in the Plack logs Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 122651 [details] [review] Bug 23653: [19.11] Remove uneeded cond test rel_file returns the path anyway Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 122652 [details] [review] Bug 23653: [19.11] Add license information in about.pl Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> JD Amended patch <p>The included <a href="https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v2.0/schema.json">api/swagger-v2-schema.json</a> file is licensed under the[-the-] <a href="https://github.com/OAI/OpenAPI-Specification/tree/master/schemas/v2.0">Apache License, Version 2.0</a>, by the <a href="https://www.openapis.org/about">OpenAPI Initiative [-(OAI)</a></p>-]{+(OAI)</a>.</p>+} Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The patches work as expected. Tests pass, and I tested the GOBI plugin which adds API routes. The routes are listed in /api/v1/.html as expected, and the plugin is functional. Thanks for the rebase, Jonathan.
Created attachment 122662 [details] [review] Bug 23653: [19.11] use local copy of swagger v2 schema By default, JSON::Validator::OpenAPI tries to fetch the swagger v2 schema from http://swagger.io/v2/schema.json. If you've installed from CPAN, JSON::Validator::OpenAPI will come with a cached copy, so it won't try to fetch it over HTTP. However, if you've installed from libjson-validator-perl from Debian/Ubuntu, the Debian package excludes the cached copy, so JSON::Validator::OpenAPI tries to fetch it over HTTP. Unfortunately, today and other days in the past, the file at http://swagger.io/v2/schema.json has been unavailable, and this causes Koha to crash in a perpetual loop. This patch includes a copy of the swagger v2 schema, and it loads it locally rather than fetching over HTTP. The changes to Koha/REST/Plugin/PluginRoutes.pm are not required, since the validator isn't currently called there, but I've added a patch to future proof it. To Test: 0a) Remove /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df if it exists 0b) Block external access to http://swagger.io/v2/schema.json or test during an outage when it's unavailable 0c) Do not apply patch 1) koha-plack --restart kohadev 2) Note that it crashes in a loop and is unavailable in web browser 3) Apply patch 4) koha-plack --restart kohadev 5) Note that Koha comes up and there are no errors in the Plack logs Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz>
Created attachment 122663 [details] [review] Bug 23653: [19.11] Remove uneeded cond test rel_file returns the path anyway Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz>
Created attachment 122664 [details] [review] Bug 23653: [19.11] Add license information in about.pl Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> JD Amended patch <p>The included <a href="https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v2.0/schema.json">api/swagger-v2-schema.json</a> file is licensed under the[-the-] <a href="https://github.com/OAI/OpenAPI-Specification/tree/master/schemas/v2.0">Apache License, Version 2.0</a>, by the <a href="https://www.openapis.org/about">OpenAPI Initiative [-(OAI)</a></p>-]{+(OAI)</a>.</p>+} Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz>
Backported to 19.11.x for 19.11.20