Bug 26790 - Refactor Koha configuration (koha-conf.xml)
Summary: Refactor Koha configuration (koha-conf.xml)
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-10-22 22:30 UTC by David Cook
Modified: 2021-05-04 12:54 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2020-10-22 22:30:28 UTC
I propose moving /yazgfs/config out of koha-conf.xml and into koha-conf.yml (or even several different *-conf.yml files).

The majority of the /yazgfs child nodes are actually configuration values for zebrasrv (ie Zebra). We should respect /yazgfs as Zebra configuration and leave it alone.

I also propose we move from XML to YAML, as it's more human-friendly and it's easier to parse into the data structure that we need. XML::Simple, which is an obsolete module according to its creator, is currently used to parse koha-conf.xml and it's behaviour can be unpredictable. We should ideally eliminate our usage of it.
Comment 1 Julian Maurice 2020-10-23 09:55:26 UTC
+1 for moving Koha config out of zebra config file
+1 for not using XML

But -1 for using YAML. I know we already use YAML for everything in Koha, but YAML has a lot of disadvantages (for instance https://www.arp242.net/yaml-config.html, or https://hitchdev.com/strictyaml/why/implicit-typing-removed/)
One concrete problem we had in Koha recently was that some MARC field tags starting with 0 were parsed as octal numbers. (040 becomes 32 but 099 remains 99)
I think that this bug might be a good opportunity to consider alternatives (HJSON, JSON5, TOML, ...)
Comment 2 David Cook 2020-10-25 23:55:07 UTC
(In reply to Julian Maurice from comment #1)
> +1 for moving Koha config out of zebra config file
> +1 for not using XML
> 
> But -1 for using YAML. I know we already use YAML
> for everything in Koha, but YAML has a lot of disadvantages

I'm not finding those links particularly persuasive. 

Using Perl's YAML module, I wasn't able to reproduce a single issue from those links. Plus many of the proposed issues should be easily avoidable anyway. 

> One concrete problem we had in Koha recently was that some MARC field tags
> starting with 0 were parsed as octal numbers. (040 becomes 32 but 099
> remains 99)

Can you point me to a bug report for that one? That sounds very interesting. I couldn't reproduce that problem based on the description.

> I think that this bug might be a good opportunity to consider alternatives
> (HJSON, JSON5, TOML, ...)

I've never heard of HJSON or JSON5, but I have used TOML a couple times. It was easy to use, and the documentation is good.

It looks like libtoml-perl is available in the Debian repositories, so that would be OK. 

I'm not particularly wedded to YAML. I'd be fine with TOML.
Comment 3 Julian Maurice 2020-10-26 08:50:25 UTC
(In reply to David Cook from comment #2)
> (In reply to Julian Maurice from comment #1)
> > +1 for moving Koha config out of zebra config file
> > +1 for not using XML
> > 
> > But -1 for using YAML. I know we already use YAML
> > for everything in Koha, but YAML has a lot of disadvantages
> 
> I'm not finding those links particularly persuasive. 
> 
> Using Perl's YAML module, I wasn't able to reproduce a single issue from
> those links. Plus many of the proposed issues should be easily avoidable
> anyway.
> 
> > One concrete problem we had in Koha recently was that some MARC field tags
> > starting with 0 were parsed as octal numbers. (040 becomes 32 but 099
> > remains 99)
> 
> Can you point me to a bug report for that one? That sounds very interesting.
> I couldn't reproduce that problem based on the description.

Can't remember the exact details and can't reproduce either... it was about the ES mappings file, MARC fields being changed when exporting/re-importing...

But the real problem IMO is that YAML is implementation dependent.
For instance, this document:

foo: 040

can be interpreted as { "foo": "040" } or { "foo": 32 } depending on how you read it. (Try this https://onlineyamltools.com/convert-yaml-to-json)

Even in Perl where implicit typing seems to be disabled by default, it can be turned on anywhere in the code ($YAML::Syck::ImplicitTyping = 1), so you can never be sure of what will happen.
Comment 4 David Cook 2020-10-26 22:11:18 UTC
(In reply to Julian Maurice from comment #3)
> Can't remember the exact details and can't reproduce either... it was about
> the ES mappings file, MARC fields being changed when
> exporting/re-importing...
 
Hmm maybe someone else will surface it later.

> But the real problem IMO is that YAML is implementation dependent.

That's a good point.

> For instance, this document:
> foo: 040
> 
> can be interpreted as { "foo": "040" } or { "foo": 32 } depending on how you
> read it. (Try this https://onlineyamltools.com/convert-yaml-to-json)

Thanks for that link. That is interesting.

However, it seems lazy to write it as 'foo: 040'. Surely it would be safer to explicitly write it as "foo": "040", since we know that they are strings. (That being said, since Perl is weakly typed, I wonder if Perl would try to convert that internally into a number...)

> Even in Perl where implicit typing seems to be disabled by default, it can
> be turned on anywhere in the code ($YAML::Syck::ImplicitTyping = 1), so you
> can never be sure of what will happen.

In theory, we should know, but I reckon that Koha is big enough that it can be easy to lose track of how the YAML has been set up across the board.
Comment 5 David Cook 2020-10-26 22:11:48 UTC
I have other priorities/interests at the moment, but why don't we go with TOML for this refactoring? There's no harm in trying it out.
Comment 6 David Cook 2020-10-28 02:16:23 UTC
Julian, after reviewing your comments, I used TOML in a small Perl project, and found it very nice to work with.
Comment 7 Martin Renvoize 2021-05-04 12:47:26 UTC
+1 for moving Koha config out of zebra config file
+1 for not using XML

As for the TOML vs YAML debate, I don't really have a horse in the race.  The YAML support in Perl seems to have reached a suitable level of maturity so I feel it's a fairly safe bet really.. but I can understand some of the issues Julian presents.  TOML is certainly a bit simpler in scope, and we shouldn't really be using the full scope of YAML in our configs anyway really so I'd suggest we shouldn't really be falling into many of the traps it presents.. How mature is the TOML support in perl... I note David looked at the TOML module (which is pure perl) and hasn't been updated since 2016.. does that mean it's perfect or it's just not maintained?  There's also a recent TOML::XS module on cpan.. but will this get packaged and is it likely to be around for a while and maintained.

Either way.. I think the splitting out of Zebra config from the config file is a great first step.