Bug 15219 - Server-side processing and pagination on checkouts tables
Summary: Server-side processing and pagination on checkouts tables
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Circulation (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Julian Maurice
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on: 11703
Blocks: 15235
  Show dependency treegraph
 
Reported: 2015-11-19 15:10 UTC by Julian Maurice
Modified: 2018-06-14 21:29 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 15219: Load checkouts table incrementally (43.62 KB, patch)
2015-11-19 15:11 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Fix content of "Export" column (988 bytes, patch)
2015-11-20 09:36 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Fix column settings in table moremember/issues-table (843 bytes, patch)
2015-11-20 14:48 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Server-side processing and pagination on checkouts tables (58.24 KB, patch)
2016-02-16 10:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Server-side processing and pagination on checkouts tables (58.64 KB, patch)
2016-04-08 12:32 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Server-side processing and pagination on checkouts tables (58.46 KB, patch)
2017-02-09 13:59 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Fix date sorting when server-side processing is disabled (23.02 KB, patch)
2017-02-24 09:10 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Server-side processing and pagination on checkouts tables (57.97 KB, patch)
2017-03-16 08:58 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Fix date sorting when server-side processing is disabled (22.72 KB, patch)
2017-03-16 08:58 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Server-side processing and pagination on checkouts tables (57.94 KB, patch)
2017-03-23 14:56 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 15219: Fix date sorting when server-side processing is disabled (22.77 KB, patch)
2017-03-23 14:56 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 15219: Server-side processing and pagination on checkouts tables (61.36 KB, patch)
2018-03-15 08:31 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Fix date sorting when server-side processing is disabled (24.10 KB, patch)
2018-03-15 08:32 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Server-side processing and pagination on checkouts tables (60.51 KB, patch)
2018-04-16 09:10 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Fix date sorting when server-side processing is disabled (24.10 KB, patch)
2018-04-16 09:10 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Server-side processing and pagination on checkouts tables (62.35 KB, patch)
2018-06-08 08:55 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Fix date sorting when server-side processing is disabled (24.10 KB, patch)
2018-06-08 08:55 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 15219: Fix relatives issues table (1.39 KB, patch)
2018-06-08 08:55 UTC, Julian Maurice
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2015-11-19 15:10:08 UTC
Some libraries have patrons with more than 1000 checkouts. Loading all of them at once can be very long, and sometimes can cause timeout errors.

Following patches will prevent that by loading checkouts in the table by batches.
Comment 1 Julian Maurice 2015-11-19 15:11:22 UTC Comment hidden (obsolete)
Comment 2 Julian Maurice 2015-11-20 09:36:16 UTC Comment hidden (obsolete)
Comment 3 Julian Maurice 2015-11-20 14:48:28 UTC Comment hidden (obsolete)
Comment 4 Julian Maurice 2015-11-24 11:06:10 UTC
Changed status to ASSIGNED, the patches still need some work.
Comment 5 Julian Maurice 2016-02-16 10:12:35 UTC
Loading checkouts table incrementally takes more time to load all lines than to load them all at once, so I'll go with a classic paginated table.
Comment 6 Julian Maurice 2016-02-16 10:33:34 UTC Comment hidden (obsolete)
Comment 7 Julian Maurice 2016-04-08 12:32:12 UTC
Created attachment 50075 [details] [review]
Bug 15219: Server-side processing and pagination on checkouts tables

Some libraries have patrons with more than 1000 checkouts. Loading all
of them at once can be very long, and sometimes can cause timeout
errors.
This patch prevent that by enabling server-side processing and
pagination on checkouts tables.
This affects patron's checkouts and patron's relatives' checkouts tables
on pages circ/circulation.pl and members/moremember.pl.

As server-side processing can be useless and cumbersome with small
sets of data, a new system preference is introduced to control this
behaviour (server-side processing is disabled by default).

Additionally, this patch:
- adds a switch to turn off and on row grouping (today's checkouts vs
  previous checkouts) (on by default)
- adds "column settings" for relatives' checkouts tables
- factorize some code that was duplicated across the two DataTables
  configurations (mDataProp)

Test plan:
1. Find (or create) a patron that have at least 10 checkouts and where
   the relatives checkouts table contain at least 10 checkouts.
2. Go to the patron's detail page and check everything works fine
   in both tables (sorting, pagination, the data itself, ...)
3. Do the same on circulation page (circulation.pl)
4. Enable system preference ServerSideCheckoutsTable
5. Repeat steps 2 and 3
Comment 8 Julian Maurice 2016-04-08 12:32:57 UTC
Patch rebased on master
Comment 9 Josef Moravec 2016-11-04 10:34:57 UTC
Sorry, doesn't apply...
Comment 10 Julian Maurice 2017-02-09 13:59:17 UTC
Created attachment 60071 [details] [review]
Bug 15219: Server-side processing and pagination on checkouts tables

Some libraries have patrons with more than 1000 checkouts. Loading all
of them at once can be very long, and sometimes can cause timeout
errors.
This patch prevent that by enabling server-side processing and
pagination on checkouts tables.
This affects patron's checkouts and patron's relatives' checkouts tables
on pages circ/circulation.pl and members/moremember.pl.

As server-side processing can be useless and cumbersome with small
sets of data, a new system preference is introduced to control this
behaviour (server-side processing is disabled by default).

Additionally, this patch:
- adds a switch to turn off and on row grouping (today's checkouts vs
  previous checkouts) (on by default)
- adds "column settings" for relatives' checkouts tables
- factorize some code that was duplicated across the two DataTables
  configurations (mDataProp)

Test plan:
1. Find (or create) a patron that have at least 10 checkouts and where
   the relatives checkouts table contain at least 10 checkouts.
2. Go to the patron's detail page and check everything works fine
   in both tables (sorting, pagination, the data itself, ...)
3. Do the same on circulation page (circulation.pl)
4. Enable system preference ServerSideCheckoutsTable
5. Repeat steps 2 and 3
Comment 11 Julian Maurice 2017-02-09 13:59:43 UTC
Rebased on master
Comment 12 Jonathan Druart 2017-02-23 16:53:44 UTC
Sort by due date does not work, that was the point of the "due_date_unformatted" column.
Comment 13 Julian Maurice 2017-02-24 08:28:15 UTC
(In reply to Jonathan Druart from comment #12)
> Sort by due date does not work, that was the point of the
> "due_date_unformatted" column.

You're right, I thought mRender was only used for display and sorting was done using mData, but no, mRender is used for everything...
The columns.render options from DataTables 1.10+ (https://datatables.net/reference/option/columns.render) allow to do that, so I'll rewrite this part
Comment 14 Julian Maurice 2017-02-24 09:10:00 UTC
Created attachment 60642 [details] [review]
Bug 15219: Fix date sorting when server-side processing is disabled

Use DataTables 'columns' option instead of 'aoColumns'
This allow more control on what data is used for display, sorting,
filtering, ...

A large part of this patch is about rewording things to better match the
DataTables documentation.
The real change is that

  "mRender": renderingFunc

becomes

  "render": {
    "display": renderingFunc
  }

So the return value of renderingFunc is now only used for display (not
for filtering, sorting, ...)
Comment 15 Julian Maurice 2017-03-16 08:58:21 UTC
Created attachment 61158 [details] [review]
Bug 15219: Server-side processing and pagination on checkouts tables

Some libraries have patrons with more than 1000 checkouts. Loading all
of them at once can be very long, and sometimes can cause timeout
errors.
This patch prevent that by enabling server-side processing and
pagination on checkouts tables.
This affects patron's checkouts and patron's relatives' checkouts tables
on pages circ/circulation.pl and members/moremember.pl.

As server-side processing can be useless and cumbersome with small
sets of data, a new system preference is introduced to control this
behaviour (server-side processing is disabled by default).

Additionally, this patch:
- adds a switch to turn off and on row grouping (today's checkouts vs
  previous checkouts) (on by default)
- adds "column settings" for relatives' checkouts tables
- factorize some code that was duplicated across the two DataTables
  configurations (mDataProp)

Test plan:
1. Find (or create) a patron that have at least 10 checkouts and where
   the relatives checkouts table contain at least 10 checkouts.
2. Go to the patron's detail page and check everything works fine
   in both tables (sorting, pagination, the data itself, ...)
3. Do the same on circulation page (circulation.pl)
4. Enable system preference ServerSideCheckoutsTable
5. Repeat steps 2 and 3
Comment 16 Julian Maurice 2017-03-16 08:58:26 UTC
Created attachment 61159 [details] [review]
Bug 15219: Fix date sorting when server-side processing is disabled

Use DataTables 'columns' option instead of 'aoColumns'
This allow more control on what data is used for display, sorting,
filtering, ...

A large part of this patch is about rewording things to better match the
DataTables documentation.
The real change is that

  "mRender": renderingFunc

becomes

  "render": {
    "display": renderingFunc
  }

So the return value of renderingFunc is now only used for display (not
for filtering, sorting, ...)
Comment 17 sandboxes@biblibre.com 2017-03-23 14:56:14 UTC
Patch tested with a sandbox, by Eric Gosselin <eric.gosselin@inlibro.com>
Comment 18 sandboxes@biblibre.com 2017-03-23 14:56:49 UTC
Created attachment 61545 [details] [review]
Bug 15219: Server-side processing and pagination on checkouts tables

Some libraries have patrons with more than 1000 checkouts. Loading all
of them at once can be very long, and sometimes can cause timeout
errors.
This patch prevent that by enabling server-side processing and
pagination on checkouts tables.
This affects patron's checkouts and patron's relatives' checkouts tables
on pages circ/circulation.pl and members/moremember.pl.

As server-side processing can be useless and cumbersome with small
sets of data, a new system preference is introduced to control this
behaviour (server-side processing is disabled by default).

Additionally, this patch:
- adds a switch to turn off and on row grouping (today's checkouts vs
  previous checkouts) (on by default)
- adds "column settings" for relatives' checkouts tables
- factorize some code that was duplicated across the two DataTables
  configurations (mDataProp)

Test plan:
1. Find (or create) a patron that have at least 10 checkouts and where
   the relatives checkouts table contain at least 10 checkouts.
2. Go to the patron's detail page and check everything works fine
   in both tables (sorting, pagination, the data itself, ...)
3. Do the same on circulation page (circulation.pl)
4. Enable system preference ServerSideCheckoutsTable
5. Repeat steps 2 and 3

Signed-off-by: Eric Gosselin <eric.gosselin@inlibro.com>
Comment 19 sandboxes@biblibre.com 2017-03-23 14:56:53 UTC
Created attachment 61546 [details] [review]
Bug 15219: Fix date sorting when server-side processing is disabled

Use DataTables 'columns' option instead of 'aoColumns'
This allow more control on what data is used for display, sorting,
filtering, ...

A large part of this patch is about rewording things to better match the
DataTables documentation.
The real change is that

  "mRender": renderingFunc

becomes

  "render": {
    "display": renderingFunc
  }

So the return value of renderingFunc is now only used for display (not
for filtering, sorting, ...)

Signed-off-by: Eric Gosselin <eric.gosselin@inlibro.com>
Comment 20 Jonathan Druart 2017-03-29 19:03:22 UTC
I plan to QA it, but on top of bug 15498 and bug 14224 (also modifying checkouts.js).
Comment 21 Julian Maurice 2018-03-15 08:31:55 UTC
Created attachment 72905 [details] [review]
Bug 15219: Server-side processing and pagination on checkouts tables

Some libraries have patrons with more than 1000 checkouts. Loading all
of them at once can be very long, and sometimes can cause timeout
errors.
This patch prevent that by enabling server-side processing and
pagination on checkouts tables.
This affects patron's checkouts and patron's relatives' checkouts tables
on pages circ/circulation.pl and members/moremember.pl.

As server-side processing can be useless and cumbersome with small
sets of data, a new system preference is introduced to control this
behaviour (server-side processing is disabled by default).

Additionally, this patch:
- adds a switch to turn off and on row grouping (today's checkouts vs
  previous checkouts) (on by default)
- adds "column settings" for relatives' checkouts tables
- factorize some code that was duplicated across the two DataTables
  configurations (mDataProp)

Test plan:
1. Find (or create) a patron that have at least 10 checkouts and where
   the relatives checkouts table contain at least 10 checkouts.
2. Go to the patron's detail page and check everything works fine
   in both tables (sorting, pagination, the data itself, ...)
3. Do the same on circulation page (circulation.pl)
4. Enable system preference ServerSideCheckoutsTable
5. Repeat steps 2 and 3

Signed-off-by: Eric Gosselin <eric.gosselin@inlibro.com>
Comment 22 Julian Maurice 2018-03-15 08:32:00 UTC
Created attachment 72906 [details] [review]
Bug 15219: Fix date sorting when server-side processing is disabled

Use DataTables 'columns' option instead of 'aoColumns'
This allow more control on what data is used for display, sorting,
filtering, ...

A large part of this patch is about rewording things to better match the
DataTables documentation.
The real change is that

  "mRender": renderingFunc

becomes

  "render": {
    "display": renderingFunc
  }

So the return value of renderingFunc is now only used for display (not
for filtering, sorting, ...)

Signed-off-by: Eric Gosselin <eric.gosselin@inlibro.com>
Comment 23 Katrin Fischer 2018-04-16 06:27:56 UTC
Please rebase!

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 15219: Server-side processing and pagination on checkouts tables
Using index info to reconstruct a base tree...
M	admin/columns_settings.yml
M	installer/data/mysql/sysprefs.sql
M	koha-tmpl/intranet-tmpl/prog/en/includes/columns_settings.inc
M	koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/circulation.pref
M	koha-tmpl/intranet-tmpl/prog/en/modules/circ/circulation.tt
M	koha-tmpl/intranet-tmpl/prog/en/modules/members/moremember.tt
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/members/moremember.tt
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/members/moremember.tt
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/circ/circulation.tt
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/circ/circulation.tt
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/circulation.pref
Auto-merging koha-tmpl/intranet-tmpl/prog/en/includes/columns_settings.inc
Auto-merging installer/data/mysql/sysprefs.sql
Auto-merging admin/columns_settings.yml
Failed to merge in the changes.
Patch failed at 0001 Bug 15219: Server-side processing and pagination on checkouts tables
The copy of the patch that failed is found in:
   /home/vagrant/kohaclone/.git/rebase-apply/patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in /tmp/Bug-15219-Server-side-processing-and-pagination-on-niral1.patch
Comment 24 Julian Maurice 2018-04-16 09:10:38 UTC
Created attachment 74224 [details] [review]
Bug 15219: Server-side processing and pagination on checkouts tables

Some libraries have patrons with more than 1000 checkouts. Loading all
of them at once can be very long, and sometimes can cause timeout
errors.
This patch prevent that by enabling server-side processing and
pagination on checkouts tables.
This affects patron's checkouts and patron's relatives' checkouts tables
on pages circ/circulation.pl and members/moremember.pl.

As server-side processing can be useless and cumbersome with small
sets of data, a new system preference is introduced to control this
behaviour (server-side processing is disabled by default).

Additionally, this patch:
- adds a switch to turn off and on row grouping (today's checkouts vs
  previous checkouts) (on by default)
- adds "column settings" for relatives' checkouts tables
- factorize some code that was duplicated across the two DataTables
  configurations (mDataProp)

Test plan:
1. Find (or create) a patron that have at least 10 checkouts and where
   the relatives checkouts table contain at least 10 checkouts.
2. Go to the patron's detail page and check everything works fine
   in both tables (sorting, pagination, the data itself, ...)
3. Do the same on circulation page (circulation.pl)
4. Enable system preference ServerSideCheckoutsTable
5. Repeat steps 2 and 3

Signed-off-by: Eric Gosselin <eric.gosselin@inlibro.com>
Comment 25 Julian Maurice 2018-04-16 09:10:43 UTC
Created attachment 74225 [details] [review]
Bug 15219: Fix date sorting when server-side processing is disabled

Use DataTables 'columns' option instead of 'aoColumns'
This allow more control on what data is used for display, sorting,
filtering, ...

A large part of this patch is about rewording things to better match the
DataTables documentation.
The real change is that

  "mRender": renderingFunc

becomes

  "render": {
    "display": renderingFunc
  }

So the return value of renderingFunc is now only used for display (not
for filtering, sorting, ...)

Signed-off-by: Eric Gosselin <eric.gosselin@inlibro.com>
Comment 26 Julian Maurice 2018-04-16 09:11:13 UTC
Patches rebased on master
Comment 27 Katrin Fischer 2018-04-16 22:00:22 UTC
1) Releatives tab doesn't open. JavaScript error:
Error: Syntax error, unrecognized expression: ##relatives-issues-table

2) I think most libraries will have a mix of large accounts and smaller ones. Could it make sense to use the server side processing when the patron has a certain number of checkouts (configurable)?
Comment 28 Julian Maurice 2018-06-08 08:55:29 UTC
Created attachment 75911 [details] [review]
Bug 15219: Server-side processing and pagination on checkouts tables

Some libraries have patrons with more than 1000 checkouts. Loading all
of them at once can be very long, and sometimes can cause timeout
errors.
This patch prevent that by enabling server-side processing and
pagination on checkouts tables.
This affects patron's checkouts and patron's relatives' checkouts tables
on pages circ/circulation.pl and members/moremember.pl.

As server-side processing can be useless and cumbersome with small
sets of data, a new system preference is introduced to control this
behaviour (server-side processing is disabled by default).

Additionally, this patch:
- adds a switch to turn off and on row grouping (today's checkouts vs
  previous checkouts) (on by default)
- adds "column settings" for relatives' checkouts tables
- factorize some code that was duplicated across the two DataTables
  configurations (mDataProp)

Test plan:
1. Find (or create) a patron that have at least 10 checkouts and where
   the relatives checkouts table contain at least 10 checkouts.
2. Go to the patron's detail page and check everything works fine
   in both tables (sorting, pagination, the data itself, ...)
3. Do the same on circulation page (circulation.pl)
4. Enable system preference ServerSideCheckoutsTable
5. Repeat steps 2 and 3

Signed-off-by: Eric Gosselin <eric.gosselin@inlibro.com>
Comment 29 Julian Maurice 2018-06-08 08:55:35 UTC
Created attachment 75912 [details] [review]
Bug 15219: Fix date sorting when server-side processing is disabled

Use DataTables 'columns' option instead of 'aoColumns'
This allow more control on what data is used for display, sorting,
filtering, ...

A large part of this patch is about rewording things to better match the
DataTables documentation.
The real change is that

  "mRender": renderingFunc

becomes

  "render": {
    "display": renderingFunc
  }

So the return value of renderingFunc is now only used for display (not
for filtering, sorting, ...)

Signed-off-by: Eric Gosselin <eric.gosselin@inlibro.com>
Comment 30 Julian Maurice 2018-06-08 08:55:40 UTC
Created attachment 75913 [details] [review]
Bug 15219: Fix relatives issues table
Comment 31 Julian Maurice 2018-06-08 09:11:30 UTC
(In reply to Katrin Fischer from comment #27)
> 1) Releatives tab doesn't open. JavaScript error:
> Error: Syntax error, unrecognized expression: ##relatives-issues-table
Fixed in the last patch

> 2) I think most libraries will have a mix of large accounts and smaller
> ones. Could it make sense to use the server side processing when the patron
> has a certain number of checkouts (configurable)?

It makes sense, but I think than can go in a separate bug. Do you agree ?
Comment 32 Jonathan Druart 2018-06-14 21:20:16 UTC
Tiny ergonomic issue when the export column is displayed: https://screenshots.firefox.com/0gdGrGVyEBanCz8n/pro.kohadev.org
Comment 33 Jonathan Druart 2018-06-14 21:22:31 UTC
(In reply to Jonathan Druart from comment #32)
> Tiny ergonomic issue when the export column is displayed:
> https://screenshots.firefox.com/0gdGrGVyEBanCz8n/pro.kohadev.org

Hum, the export column should not be displayed actually, ExportCircHistory is off. But it is in the "column visibility" list
Comment 34 Jonathan Druart 2018-06-14 21:24:30 UTC
"Select all" only select visible rows. If it is expected it is confusing.