Bug 17932

Summary: Koha::Object should provide a TO_JSON method
Product: Koha Reporter: Tomás Cohen Arazi <tomascohen>
Component: Architecture, internals, and plumbingAssignee: Tomás Cohen Arazi <tomascohen>
Status: CLOSED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: enhancement    
Priority: P5 - low CC: dcook, jonathan.druart, josef.moravec, katrin.fischer, kyle.m.hall, kyle, m.de.rooy, martin.renvoize, olli-antti.kivilahti
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 17927    
Bug Blocks: 17992, 18290, 18326, 25513    
Attachments: Bug 17932: Add a TO_JSON method to Koha::Object(s)
Bug 17932: (followup) Fix /patrons endpoint
Bug 17932: Add a TO_JSON method to Koha::Object(s)
Bug 17932: (followup) Fix /patrons endpoint
Bug 17932: Unit tests
Bug 17932: (followup) Tidy tests
Bug 17932: Add a TO_JSON method to Koha::Object(s)
Bug 17932: (followup) Fix /patrons endpoint
Bug 17932: Unit tests
Bug 17932: (followup) Tidy tests
Bug 17932: Add a TO_JSON method to Koha::Object(s)
Bug 17932: (followup) Fix /patrons endpoint
Bug 17932: Unit tests
Bug 17932: (followup) Tidy tests

Description Tomás Cohen Arazi 2017-01-18 14:25:45 UTC
The Koha::Object class should provide a TO_JSON method that prepares data for JSON output. Right now, booleans are returned as integer values, and thus invalid for JSON output as boolean values.
Comment 1 Tomás Cohen Arazi 2017-01-18 18:27:44 UTC
*** Bug 17926 has been marked as a duplicate of this bug. ***
Comment 2 Tomás Cohen Arazi 2017-01-23 15:48:14 UTC
Created attachment 59431 [details] [review]
Bug 17932: Add a TO_JSON method to Koha::Object(s)

Bug 17927 fixes data types on the current REST endpoints. If you
test those endpoints, you will notice trying to access them (for listing
or retrieving single objects yields a data type error. Notably on
booleans but also on integers.

Integers fail due to https://rt.cpan.org/Ticket/Display.html?id=119904
but it needs some global solution until there's a DBD::mysql release
backported to the supported distros. There's the option to use
http://search.cpan.org/~frew/DBIx-Class-Helpers-2.033002/lib/DBIx/Class/Helper/Row/NumifyGet.pm
to get the integer columns fixed as a workaround:

 __PACKAGE__->add_columns(
    borrowernumber => {
        data_type         => 'integer',
        is_nullable       => 0,
        is_numeric        => 1,
    }
);

I didn't find bug reports related to this (maybe because we don't use
warnings everywhere) But I don't think is worth going such a heavy
overhead.

A similar situation takes place for Boolean values. They need to be
prepared for JSON output. This could have been done using DBIx filters
as pointed out by Martin:

__PACKAGE__->filter_column(
    lost => {
       filter_to_storage => sub { $_[1] ? 1 : 0 },
       filter_from_storage =>
               sub { $_[1] ? Mojo::JSON->true :
                             Mojo::JSON->false }
       }
);

but this could have other consequences that are worth exploring on
another bug (i.e. it would mean we need to take care of every place
where this boolean data is used/set needs to handle this data types
nicely. Such would be the case if we were a Mojo-only app, but we
aren't. We use Koha::Obect(s) in the whole app. Period.

This patch adds the need to specify on the schema files, columns that
are actually boolean, because we have no way to detect them for now
(i.e. they are all tinyint, but we use tinyint for non-boolean stuff
 too).
So if this patch is accepted, we would need to specify boolean columns
like this:

__PACKAGE__->add_columns(
    '+lost' => {
        is_boolean => 1
    }
);

This patch adds a TO_JSON method for Koha::Object(s) to be used for
serializing Koha::Object-derived objects into JSON strings.

To test it (as Koha::Object(s) need to be instantiated) I provide tests
on top of the Koha::Patron(s) classes in the followup patches.

[1] Yes, we use TINYINT(1) for booleans, but from DBIC's perspective
there's no way to read the (1) in runtime.

Sponsored-by: ByWater Solutions
Comment 3 Tomás Cohen Arazi 2017-01-23 15:48:21 UTC
Created attachment 59432 [details] [review]
Bug 17932: (followup) Fix /patrons endpoint

Bug 17927 introduced data type fixes on the /patrons endpoint (integer
and boolean types got fixed). This led to the /patrons endpoint not to
work because the underlying code didn't provide the right data.

With the introduction of TO_JSON on Koha::Object(s) we now have a way to
output the proper data types.

This patch does so by:
- Adding is_boolean => 1 to the relevant columns on the Borrower.pm
  schema file.
- Tweaking the controller class for the /patrons endpoint so it doesn't
  use the $object(s)->unblessed call but just let the Mojo::JSON library
  pick out TO_JSON implementation instead on rendering the output.
- It adds a new test for booleans.

To test:
- Have 17927 applied
- Run:
  $ prove t/db_dependent/api/v1/patrons.t
=> FAIL: Tests fail [1]
- Apply this patches
- Run:
  $ prove t/db_dependent/api/v1/patrons.t
=> SUCCESS: Tests pass!
- Sign off! :-D

[1] It is self explanatory to just try the API using any of the
available tools (I use HttpRequester on Firefox)
Comment 4 Tomás Cohen Arazi 2017-01-24 13:52:06 UTC
Created attachment 59498 [details] [review]
Bug 17932: Add a TO_JSON method to Koha::Object(s)

Bug 17927 fixes data types on the current REST endpoints. If you
test those endpoints, you will notice trying to access them (for listing
or retrieving single objects yields a data type error. Notably on
booleans but also on integers.

Integers fail due to https://rt.cpan.org/Ticket/Display.html?id=119904
but it needs some global solution until there's a DBD::mysql release
backported to the supported distros. There's the option to use
http://search.cpan.org/~frew/DBIx-Class-Helpers-2.033002/lib/DBIx/Class/Helper/Row/NumifyGet.pm
to get the integer columns fixed as a workaround:

 __PACKAGE__->add_columns(
    borrowernumber => {
        data_type         => 'integer',
        is_nullable       => 0,
        is_numeric        => 1,
    }
);

I didn't find bug reports related to this (maybe because we don't use
warnings everywhere) But I don't think is worth going such a heavy
overhead.

A similar situation takes place for Boolean values. They need to be
prepared for JSON output. This could have been done using DBIx filters
as pointed out by Martin:

__PACKAGE__->filter_column(
    lost => {
       filter_to_storage => sub { $_[1] ? 1 : 0 },
       filter_from_storage =>
               sub { $_[1] ? Mojo::JSON->true :
                             Mojo::JSON->false }
       }
);

but this could have other consequences that are worth exploring on
another bug (i.e. it would mean we need to take care of every place
where this boolean data is used/set needs to handle this data types
nicely. Such would be the case if we were a Mojo-only app, but we
aren't. We use Koha::Obect(s) in the whole app. Period.

This patch adds the need to specify on the schema files, columns that
are actually boolean, because we have no way to detect them for now
(i.e. they are all tinyint, but we use tinyint for non-boolean stuff
 too).
So if this patch is accepted, we would need to specify boolean columns
like this:

__PACKAGE__->add_columns(
    '+lost' => {
        is_boolean => 1
    }
);

This patch adds a TO_JSON method for Koha::Object(s) to be used for
serializing Koha::Object-derived objects into JSON strings.

To test it (as Koha::Object(s) need to be instantiated) I provide tests
on top of the Koha::Patron(s) classes in the followup patches.

[1] Yes, we use TINYINT(1) for booleans, but from DBIC's perspective
there's no way to read the (1) in runtime.

Sponsored-by: ByWater Solutions
Comment 5 Tomás Cohen Arazi 2017-01-24 13:52:21 UTC
Created attachment 59499 [details] [review]
Bug 17932: (followup) Fix /patrons endpoint

Bug 17927 introduced data type fixes on the /patrons endpoint (integer
and boolean types got fixed). This led to the /patrons endpoint not to
work because the underlying code didn't provide the right data.

With the introduction of TO_JSON on Koha::Object(s) we now have a way to
output the proper data types.

This patch does so by:
- Adding is_boolean => 1 to the relevant columns on the Borrower.pm
  schema file.
- Tweaking the controller class for the /patrons endpoint so it doesn't
  use the $object(s)->unblessed call but just let the Mojo::JSON library
  pick out TO_JSON implementation instead on rendering the output.
- It adds a new test for booleans.

To test:
- Have 17927 applied
- Run:
  $ prove t/db_dependent/api/v1/patrons.t
=> FAIL: Tests fail [1]
- Apply this patches
- Run:
  $ prove t/db_dependent/api/v1/patrons.t
=> SUCCESS: Tests pass!
- Sign off! :-D

[1] It is self explanatory to just try the API using any of the
available tools (I use HttpRequester on Firefox)
Comment 6 Tomás Cohen Arazi 2017-01-24 13:52:32 UTC
Created attachment 59500 [details] [review]
Bug 17932: Unit tests

This patch adds unit tests for the Koha::Object::TO_JSON function.
It tests on top of Koha::Patron as Koha::Object needs to be
instantiated.

To test:
- Apply the patch
- Run:
  $ prove -v t/db_dependent/Koha/Object.t
=> SUCCESS: New tests for TO_JSON are run and return green.
- Sign off :-D
Comment 7 Tomás Cohen Arazi 2017-01-24 13:52:44 UTC
Created attachment 59501 [details] [review]
Bug 17932: (followup) Tidy tests

This patch makes the tests create its own data instead of searching the
DB for a branchcode and a categorycode. It does so on each subtest,
because there shouldn't be side effects between subtests.

I also wrapped each subtest inside a transaction, for the same reason.

To test:
- Apply this patch
- Run:
  $ prove t/db_dependent/Koha/Object.t
=> SUCCESS: Tests return green with this patch
- Sign off :-D
Comment 8 Martin Renvoize 2017-01-24 14:14:25 UTC
Looks good to me :)
Comment 9 Jonathan Druart 2017-01-25 08:17:20 UTC
(In reply to Martin Renvoize from comment #8)
> Looks good to me :)

Is it a signoff?
Comment 10 Nick Clemens 2017-01-31 18:57:13 UTC
Created attachment 59696 [details] [review]
Bug 17932: Add a TO_JSON method to Koha::Object(s)

Bug 17927 fixes data types on the current REST endpoints. If you
test those endpoints, you will notice trying to access them (for listing
or retrieving single objects yields a data type error. Notably on
booleans but also on integers.

Integers fail due to https://rt.cpan.org/Ticket/Display.html?id=119904
but it needs some global solution until there's a DBD::mysql release
backported to the supported distros. There's the option to use
http://search.cpan.org/~frew/DBIx-Class-Helpers-2.033002/lib/DBIx/Class/Helper/Row/NumifyGet.pm
to get the integer columns fixed as a workaround:

 __PACKAGE__->add_columns(
    borrowernumber => {
        data_type         => 'integer',
        is_nullable       => 0,
        is_numeric        => 1,
    }
);

I didn't find bug reports related to this (maybe because we don't use
warnings everywhere) But I don't think is worth going such a heavy
overhead.

A similar situation takes place for Boolean values. They need to be
prepared for JSON output. This could have been done using DBIx filters
as pointed out by Martin:

__PACKAGE__->filter_column(
    lost => {
       filter_to_storage => sub { $_[1] ? 1 : 0 },
       filter_from_storage =>
               sub { $_[1] ? Mojo::JSON->true :
                             Mojo::JSON->false }
       }
);

but this could have other consequences that are worth exploring on
another bug (i.e. it would mean we need to take care of every place
where this boolean data is used/set needs to handle this data types
nicely. Such would be the case if we were a Mojo-only app, but we
aren't. We use Koha::Obect(s) in the whole app. Period.

This patch adds the need to specify on the schema files, columns that
are actually boolean, because we have no way to detect them for now
(i.e. they are all tinyint, but we use tinyint for non-boolean stuff
 too).
So if this patch is accepted, we would need to specify boolean columns
like this:

__PACKAGE__->add_columns(
    '+lost' => {
        is_boolean => 1
    }
);

This patch adds a TO_JSON method for Koha::Object(s) to be used for
serializing Koha::Object-derived objects into JSON strings.

To test it (as Koha::Object(s) need to be instantiated) I provide tests
on top of the Koha::Patron(s) classes in the followup patches.

[1] Yes, we use TINYINT(1) for booleans, but from DBIC's perspective
there's no way to read the (1) in runtime.

Sponsored-by: ByWater Solutions

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 11 Nick Clemens 2017-01-31 18:57:18 UTC
Created attachment 59697 [details] [review]
Bug 17932: (followup) Fix /patrons endpoint

Bug 17927 introduced data type fixes on the /patrons endpoint (integer
and boolean types got fixed). This led to the /patrons endpoint not to
work because the underlying code didn't provide the right data.

With the introduction of TO_JSON on Koha::Object(s) we now have a way to
output the proper data types.

This patch does so by:
- Adding is_boolean => 1 to the relevant columns on the Borrower.pm
  schema file.
- Tweaking the controller class for the /patrons endpoint so it doesn't
  use the $object(s)->unblessed call but just let the Mojo::JSON library
  pick out TO_JSON implementation instead on rendering the output.
- It adds a new test for booleans.

To test:
- Have 17927 applied
- Run:
  $ prove t/db_dependent/api/v1/patrons.t
=> FAIL: Tests fail [1]
- Apply this patches
- Run:
  $ prove t/db_dependent/api/v1/patrons.t
=> SUCCESS: Tests pass!
- Sign off! :-D

[1] It is self explanatory to just try the API using any of the
available tools (I use HttpRequester on Firefox)

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 12 Nick Clemens 2017-01-31 18:57:25 UTC
Created attachment 59698 [details] [review]
Bug 17932: Unit tests

This patch adds unit tests for the Koha::Object::TO_JSON function.
It tests on top of Koha::Patron as Koha::Object needs to be
instantiated.

To test:
- Apply the patch
- Run:
  $ prove -v t/db_dependent/Koha/Object.t
=> SUCCESS: New tests for TO_JSON are run and return green.
- Sign off :-D

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 13 Nick Clemens 2017-01-31 18:57:31 UTC
Created attachment 59699 [details] [review]
Bug 17932: (followup) Tidy tests

This patch makes the tests create its own data instead of searching the
DB for a branchcode and a categorycode. It does so on each subtest,
because there shouldn't be side effects between subtests.

I also wrapped each subtest inside a transaction, for the same reason.

To test:
- Apply this patch
- Run:
  $ prove t/db_dependent/Koha/Object.t
=> SUCCESS: Tests return green with this patch
- Sign off :-D

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 14 Marcel de Rooy 2017-02-01 12:01:05 UTC
Thanks, Nick. You beat me in signing off :)
Comment 15 Tomás Cohen Arazi 2017-02-03 19:03:00 UTC
(In reply to Marcel de Rooy from comment #14)
> Thanks, Nick. You beat me in signing off :)

You can QA it ;-)
Comment 16 Marcel de Rooy 2017-02-03 20:06:51 UTC
(In reply to Tomás Cohen Arazi from comment #15)
> (In reply to Marcel de Rooy from comment #14)
> > Thanks, Nick. You beat me in signing off :)
> 
> You can QA it ;-)

Good plan. Just a dumb question btw: why do you use uppercase TO_JSON?
Is it that important :)
Note that I see everywhere get_whatever etc.
Comment 17 Tomás Cohen Arazi 2017-02-03 21:37:24 UTC
(In reply to Marcel de Rooy from comment #16)
> (In reply to Tomás Cohen Arazi from comment #15)
> > (In reply to Marcel de Rooy from comment #14)
> > > Thanks, Nick. You beat me in signing off :)
> > 
> > You can QA it ;-)
> 
> Good plan. Just a dumb question btw: why do you use uppercase TO_JSON?
> Is it that important :)
> Note that I see everywhere get_whatever etc.

I personally dislike ->get_whatever, and prefer ->whatever in most cases (->as_json would be acceptable, because it is clear we want the object in a specific serialization format).

Anyway, this time we are just sticking to what JSON libraries expect from blessed objects to accept them for rendering as JSON strings. See 
http://search.cpan.org/~makamaka/JSON-2.90/lib/JSON.pm#allow_blessed
Comment 18 Marcel de Rooy 2017-02-04 18:59:17 UTC
(In reply to Tomás Cohen Arazi from comment #17)
> Anyway, this time we are just sticking to what JSON libraries expect from
> blessed objects to accept them for rendering as JSON strings. See 
> http://search.cpan.org/~makamaka/JSON-2.90/lib/JSON.pm#allow_blessed

Good argument.
Comment 19 Marcel de Rooy 2017-02-09 13:15:28 UTC Comment hidden (obsolete)
Comment 20 Marcel de Rooy 2017-02-09 14:02:13 UTC
Created attachment 60072 [details] [review]
Bug 17932: Add a TO_JSON method to Koha::Object(s)

Bug 17927 fixes data types on the current REST endpoints. If you
test those endpoints, you will notice trying to access them (for listing
or retrieving single objects yields a data type error. Notably on
booleans but also on integers.

Integers fail due to https://rt.cpan.org/Ticket/Display.html?id=119904
but it needs some global solution until there's a DBD::mysql release
backported to the supported distros. There's the option to use
http://search.cpan.org/~frew/DBIx-Class-Helpers-2.033002/lib/DBIx/Class/Helper/Row/NumifyGet.pm
to get the integer columns fixed as a workaround:

 __PACKAGE__->add_columns(
    borrowernumber => {
        data_type         => 'integer',
        is_nullable       => 0,
        is_numeric        => 1,
    }
);

I didn't find bug reports related to this (maybe because we don't use
warnings everywhere) But I don't think is worth going such a heavy
overhead.

A similar situation takes place for Boolean values. They need to be
prepared for JSON output. This could have been done using DBIx filters
as pointed out by Martin:

__PACKAGE__->filter_column(
    lost => {
       filter_to_storage => sub { $_[1] ? 1 : 0 },
       filter_from_storage =>
               sub { $_[1] ? Mojo::JSON->true :
                             Mojo::JSON->false }
       }
);

but this could have other consequences that are worth exploring on
another bug (i.e. it would mean we need to take care of every place
where this boolean data is used/set needs to handle this data types
nicely. Such would be the case if we were a Mojo-only app, but we
aren't. We use Koha::Obect(s) in the whole app. Period.

This patch adds the need to specify on the schema files, columns that
are actually boolean, because we have no way to detect them for now
(i.e. they are all tinyint, but we use tinyint for non-boolean stuff
 too).
So if this patch is accepted, we would need to specify boolean columns
like this:

__PACKAGE__->add_columns(
    '+lost' => {
        is_boolean => 1
    }
);

This patch adds a TO_JSON method for Koha::Object(s) to be used for
serializing Koha::Object-derived objects into JSON strings.

To test it (as Koha::Object(s) need to be instantiated) I provide tests
on top of the Koha::Patron(s) classes in the followup patches.

[1] Yes, we use TINYINT(1) for booleans, but from DBIC's perspective
there's no way to read the (1) in runtime.

Sponsored-by: ByWater Solutions

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 21 Marcel de Rooy 2017-02-09 14:02:18 UTC
Created attachment 60073 [details] [review]
Bug 17932: (followup) Fix /patrons endpoint

Bug 17927 introduced data type fixes on the /patrons endpoint (integer
and boolean types got fixed). This led to the /patrons endpoint not to
work because the underlying code didn't provide the right data.

With the introduction of TO_JSON on Koha::Object(s) we now have a way to
output the proper data types.

This patch does so by:
- Adding is_boolean => 1 to the relevant columns on the Borrower.pm
  schema file.
- Tweaking the controller class for the /patrons endpoint so it doesn't
  use the $object(s)->unblessed call but just let the Mojo::JSON library
  pick out TO_JSON implementation instead on rendering the output.
- It adds a new test for booleans.

To test:
- Have 17927 applied
- Run:
  $ prove t/db_dependent/api/v1/patrons.t
=> FAIL: Tests fail [1]
- Apply this patches
- Run:
  $ prove t/db_dependent/api/v1/patrons.t
=> SUCCESS: Tests pass!
- Sign off! :-D

[1] It is self explanatory to just try the API using any of the
available tools (I use HttpRequester on Firefox)

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 22 Marcel de Rooy 2017-02-09 14:02:22 UTC
Created attachment 60074 [details] [review]
Bug 17932: Unit tests

This patch adds unit tests for the Koha::Object::TO_JSON function.
It tests on top of Koha::Patron as Koha::Object needs to be
instantiated.

To test:
- Apply the patch
- Run:
  $ prove -v t/db_dependent/Koha/Object.t
=> SUCCESS: New tests for TO_JSON are run and return green.
- Sign off :-D

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 23 Marcel de Rooy 2017-02-09 14:02:26 UTC
Created attachment 60075 [details] [review]
Bug 17932: (followup) Tidy tests

This patch makes the tests create its own data instead of searching the
DB for a branchcode and a categorycode. It does so on each subtest,
because there shouldn't be side effects between subtests.

I also wrapped each subtest inside a transaction, for the same reason.

To test:
- Apply this patch
- Run:
  $ prove t/db_dependent/Koha/Object.t
=> SUCCESS: Tests return green with this patch
- Sign off :-D

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 24 Marcel de Rooy 2017-02-09 14:03:54 UTC
Looks good to me.
One small thing: the isvstring test looks somewhat useless to me. It also passes on other borrower text columns like phonepro or B_streetnumber.
Could be handled on a follow-up.

Passed QA
Comment 25 Kyle M Hall 2017-02-17 15:33:29 UTC
Pushed to master for 17.05, thanks Tomas!
Comment 26 Katrin Fischer 2017-02-19 20:32:19 UTC
This won't get ported back to 16.11.x as it is an enhancement.
Comment 27 Olli-Antti Kivilahti 2017-03-17 06:22:06 UTC
Followup Bug 18290 fixes tests to cover a change in Mojolicious
Comment 28 David Cook 2017-03-24 01:12:47 UTC
With this patch now in master, we now have to update C4/Installer/PerlDependencies.pm to add Mojolicious as a required dependency.