Bug 36722 - Koha not caching http://json-schema.org/draft-04/schema#
Summary: Koha not caching http://json-schema.org/draft-04/schema#
Status: RESOLVED DUPLICATE of bug 37361
Alias: None
Product: Koha
Classification: Unclassified
Component: Packaging (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-04-30 02:46 UTC by David Cook
Modified: 2024-08-15 02:30 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2024-04-30 02:46:39 UTC
This is arguably the same issue as bug 23653 except that it's a different website this time.

Also, Koha won't crash if json-schema.org is down, but the API will return a 404 for all endpoints.

Like bug 23653, I think that we should use a local copy of the specification.

At the moment, the way to do that is to download http://json-schema.org/draft-04/schema#, save it into our source tree, and then run the following:

my $schema = JSON::Validator::Schema::OpenAPIv2->new;
$schema->specification($saved_schema_spec);

In theory, that's it. 

I'll look at providing a patch shortly...
Comment 1 David Cook 2024-04-30 02:56:46 UTC
Note also that JSON::Validator::Schema::Draft4 used by JSON::Validator::Schema::OpenAPIv2 is set to http://json-schema.org/draft-04/schema# which redirects to the HTTPS version, so every worker does 2 web requests to it on startup. Yikes...
Comment 2 David Cook 2024-04-30 03:04:32 UTC
Note an easy way to prove this is to run the following:

Window 1:
sudo tcpdump -A -i any 'host json-schema.org'

Window 2:
sudo koha-plack --restart kohadev
Comment 3 David Cook 2024-04-30 03:49:00 UTC
Looks like JSON::Validator isn't caching files unless JSON_VALIDATOR_CACHE_PATH is defined... 

It did cache when I changed koha-plack to do this:
if JSON_VALIDATOR_CACHE_PATH=/opt/cache ${STARMAN} ${STARMANOPTS}; then
Comment 4 David Cook 2024-04-30 03:59:37 UTC
And now I'm extra confused... because JSON::Validator::Schema::OpenAPIv2 has a specification of http://swagger.io/v2/schema.json but http://json-schema.org/draft-04/schema# is what's being downloaded...

I suppose because the $schema for http://swagger.io/v2/schema.json is http://json-schema.org/draft-04/schema#
Comment 5 David Cook 2024-04-30 04:31:08 UTC
If I force JSON::Validator::Store into debug mode, it does look like it's downloading it during the resolving process...

[JSON::Validator] Resolve curr: ref=http://json-schema.org/draft-04/schema#/properties/description,base_url=http://swagger.io/v2/schema.json fragment= id=http://swagger.io/v2/schema.json root=HASH(0x55d20f974060) schema=HASH(0x55d20f9740
60) source=load

[JSON::Validator] Resolve state: base_url=http://json-schema.org/draft-04/schema fragment=/definitions/stringArray id=http://json-schema.org/draft-04/schema root=HASH(0x55d2057e4a40) schema=HASH(0x55d20f702f40) source=schema/base_url

I'm starting to think that bug 30194 actually made no difference here...

--

[JSON::Validator] Load from cache /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df

[JSON::Validator] Load from URL http://json-schema.org/draft-04/schema

And Koha can't add json-schema.org to the cache, because Koha doesn't have write access to /usr/share/perl5/JSON/Validator/cache...

36d1bd12eeed51e86c8695bd8876a9df is the cached version of the http://swagger.io/v2/schema.json file.
Comment 6 David Cook 2024-04-30 04:31:38 UTC
Note that /usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df and api/swagger-v2-schema.json are the same...
Comment 7 David Cook 2024-04-30 06:08:59 UTC
I think the shortest solution for this one is going to be to add the contents of http://json-schema.org/draft-04/schema# as /usr/share/perl5/JSON/Validator/cache/49c95b866e40f788892a7fb3c816b0e8

And that can be done via the libjson-validator-perl package that the Koha community puts together.
Comment 8 David Cook 2024-04-30 06:39:02 UTC
Changing this to a "Packaging" bug I guess. 

Technically, we could point the cache paths elsewhere and provide our own cached files but happy enough to be continuing as we are...
Comment 9 Tomás Cohen Arazi (tcohen) 2024-04-30 19:07:10 UTC
At some point I did

```shell
ktd --shell
sudo vim /usr/share/perl5/JSON/Validator/Schema/Draft4.pm
```

and replaced `http` with `https`on line 12. And it started working. But I'm not sure it is really it, as I cannot make it fail anymore :-P

Just noting in case it is useful. I think the solution is on packaging as well.
Comment 10 David Cook 2024-05-01 00:25:25 UTC
When I reported and analyzed this one, I wasn't having any errors. I just wanted to make things faster/more efficient.

The error for me was hypothetical only, so that's interesting that you did experience the error! 

To make it reliably fail, you can add the following to your /etc/hosts: 

127.0.0.1 json-schema.org

--

But yeah we should be able to fix this one by packaging (or by changing the cache paths and providing our own cached files).
Comment 11 David Cook 2024-05-02 02:46:51 UTC
(In reply to David Cook from comment #7)
> I think the shortest solution for this one is going to be to add the
> contents of http://json-schema.org/draft-04/schema# as
> /usr/share/perl5/JSON/Validator/cache/49c95b866e40f788892a7fb3c816b0e8
> 
> And that can be done via the libjson-validator-perl package that the Koha
> community puts together.

/usr/share/perl5/JSON/Validator/Store.pm removes the fragment from the URL, so you end up with the following:

echo -n "http://json-schema.org/draft-04/schema" | md5sum
49c95b866e40f788892a7fb3c816b0e8  -
Comment 12 David Cook 2024-05-02 02:52:43 UTC
Ok now I'm super curious...

My Koha 23.11 dev system on Ubuntu 22.04
libjson-validator-perl 5.99+really5.08-koha1
/usr/share/perl5/JSON/Validator/cache/089e74a6d17f64af17a9efd6d0fa0de6
/usr/share/perl5/JSON/Validator/cache/10a5eeb37fcd5d829449028f7ceb0774
/usr/share/perl5/JSON/Validator/cache/33912dbbde6e1d936140f1c82b283d01
/usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df
/usr/share/perl5/JSON/Validator/cache/3be3f46eb248daf48925640f8ef057e8
/usr/share/perl5/JSON/Validator/cache/3d35aac549d951f4cf9182ff47bff0b4
/usr/share/perl5/JSON/Validator/cache/49c95b866e40f788892a7fb3c816b0e8
/usr/share/perl5/JSON/Validator/cache/4a31fe43be9e23ca9eb8d9e9faba8892
/usr/share/perl5/JSON/Validator/cache/546acf85ddc442761c18517490215b90
/usr/share/perl5/JSON/Validator/cache/630949337805585c8e52deea27d11419
/usr/share/perl5/JSON/Validator/cache/7fe97ed1a4c3fac607dd276b2b298275
/usr/share/perl5/JSON/Validator/cache/a0f5b4b4e75ea17fc09e88ec0343d148
/usr/share/perl5/JSON/Validator/cache/a516498b60c53096b2ce2cd83ebe0abc
/usr/share/perl5/JSON/Validator/cache/c6f188eb288cf986f23db49297b25e83
/usr/share/perl5/JSON/Validator/cache/d18065ce8fb1f748e766b2737bae5200
/usr/share/perl5/JSON/Validator/cache/d8cf7ae7a0fd14accadf5d18bc84d14f
/usr/share/perl5/JSON/Validator/cache/ea34d47d4e060a1c3b12d2287aff89a7
/usr/share/perl5/JSON/Validator/cache/eaa832720f36cff0abc20c05236a9cd9

My Koha main dev system on Debian 12
libjson-validator-perl 5.99+really5.14-koha1
/usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df
/usr/share/perl5/JSON/Validator/cache/a516498b60c53096b2ce2cd83ebe0abc
Comment 13 David Cook 2024-05-02 02:58:20 UTC
So not a problem in prod for people using the "koha" apt repo.

But it is a problem in the "koha-staging" apt repo.
Comment 14 David Cook 2024-08-15 02:26:55 UTC
Ok and now it looks like it's a problem in "koha"!

dpkg -l libjson-validator-perl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                   Version               Architecture Description
+++-======================-=====================-============-=============================================
ii  libjson-validator-perl 5.99+really5.14-koha2 all          module to validate data against a JSON schema

dpkg -L libjson-validator-perl
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libjson-validator-perl
/usr/share/doc/libjson-validator-perl/NEWS.Debian.gz
/usr/share/doc/libjson-validator-perl/changelog.Debian.gz
/usr/share/doc/libjson-validator-perl/changelog.gz
/usr/share/doc/libjson-validator-perl/copyright
/usr/share/man
/usr/share/man/man3
/usr/share/man/man3/JSON::Validator.3pm.gz
/usr/share/man/man3/JSON::Validator::Error.3pm.gz
/usr/share/man/man3/JSON::Validator::Formats.3pm.gz
/usr/share/man/man3/JSON::Validator::Joi.3pm.gz
/usr/share/man/man3/JSON::Validator::Schema.3pm.gz
/usr/share/man/man3/JSON::Validator::Schema::Draft201909.3pm.gz
/usr/share/man/man3/JSON::Validator::Schema::Draft4.3pm.gz
/usr/share/man/man3/JSON::Validator::Schema::Draft6.3pm.gz
/usr/share/man/man3/JSON::Validator::Schema::Draft7.3pm.gz
/usr/share/man/man3/JSON::Validator::Schema::OpenAPIv2.3pm.gz
/usr/share/man/man3/JSON::Validator::Schema::OpenAPIv3.3pm.gz
/usr/share/man/man3/JSON::Validator::Store.3pm.gz
/usr/share/man/man3/JSON::Validator::URI.3pm.gz
/usr/share/man/man3/JSON::Validator::Util.3pm.gz
/usr/share/perl5
/usr/share/perl5/JSON
/usr/share/perl5/JSON/Validator
/usr/share/perl5/JSON/Validator/Error.pm
/usr/share/perl5/JSON/Validator/Formats.pm
/usr/share/perl5/JSON/Validator/Joi.pm
/usr/share/perl5/JSON/Validator/Schema
/usr/share/perl5/JSON/Validator/Schema/Draft201909.pm
/usr/share/perl5/JSON/Validator/Schema/Draft4.pm
/usr/share/perl5/JSON/Validator/Schema/Draft6.pm
/usr/share/perl5/JSON/Validator/Schema/Draft7.pm
/usr/share/perl5/JSON/Validator/Schema/OpenAPIv2.pm
/usr/share/perl5/JSON/Validator/Schema/OpenAPIv3.pm
/usr/share/perl5/JSON/Validator/Schema.pm
/usr/share/perl5/JSON/Validator/Store.pm
/usr/share/perl5/JSON/Validator/URI.pm
/usr/share/perl5/JSON/Validator/Util.pm
/usr/share/perl5/JSON/Validator/cache
/usr/share/perl5/JSON/Validator.pm
/usr/share/perl5/JSON/Validator/cache/36d1bd12eeed51e86c8695bd8876a9df
/usr/share/perl5/JSON/Validator/cache/a516498b60c53096b2ce2cd83ebe0abc
Comment 15 David Cook 2024-08-15 02:30:28 UTC
It looks like my warning wasn't heeded and now it's affecting at least 2 people in prod...

*** This bug has been marked as a duplicate of bug 37361 ***