Bug 30462 - Improve the default display for the background jobs queue management page
Summary: Improve the default display for the background jobs queue management page
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: System Administration (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Martin Renvoize
QA Contact: Testopia
URL:
Keywords: release-notes-needed
Depends on: 27344
Blocks: 30982
  Show dependency treegraph
 
Reported: 2022-04-06 09:46 UTC by Jonathan Druart
Modified: 2022-11-25 07:27 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
22.11.00, 22.05.07


Attachments
B (1.25 KB, patch)
2022-06-02 06:10 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 30462: Delete the update_elastic_index jobs when done (1.31 KB, patch)
2022-06-02 12:27 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 30462: Separate Running and Complete jobs (10.53 KB, patch)
2022-06-07 10:09 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 30462: Separate Queued and Complete jobs (10.52 KB, patch)
2022-06-07 10:11 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 30462: (follow-up) Limit completed to 60 minutes (3.40 KB, patch)
2022-06-10 14:36 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 30462: Separate Queued and Complete jobs (10.82 KB, patch)
2022-06-13 09:38 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 30462: (follow-up) Limit completed to 60 minutes (3.48 KB, patch)
2022-06-13 09:38 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 30462: Remove 'Ended' column for current jobs (1.76 KB, patch)
2022-06-13 09:38 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 30462: Fix display of the complete jobs (1.29 KB, patch)
2022-06-13 09:38 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 30462: Separate Queued and Complete jobs (10.87 KB, patch)
2022-09-02 13:55 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 30462: (follow-up) Limit completed to 60 minutes (3.53 KB, patch)
2022-09-02 13:56 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 30462: Remove 'Ended' column for current jobs (1.82 KB, patch)
2022-09-02 13:56 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 30462: Fix display of the complete jobs (1.35 KB, patch)
2022-09-02 13:56 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 30462: (QA follow-up) Tab name change (1.27 KB, patch)
2022-09-08 17:50 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 30462: [22.05.x] Separate Queued and Complete jobs (12.06 KB, patch)
2022-11-01 11:36 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2022-04-06 09:46:37 UTC
We will certainly end up with lot of indexation tasks. Shouldn't we hide them by default when we list the tasks on Home > Administration > Background jobs ?
Comment 1 Tomás Cohen Arazi 2022-05-05 13:41:01 UTC
(In reply to Jonathan Druart from comment #0)
> We will certainly end up with lot of indexation tasks. Shouldn't we hide
> them by default when we list the tasks on Home > Administration > Background
> jobs ?

Yes!
Comment 2 Martin Renvoize 2022-05-05 15:02:19 UTC
Yes :)
Comment 3 Andrew Nugged 2022-05-27 11:17:51 UTC
bringing our experience from piloting production with the current master.
It's even not that "very busy day" and vacations week,
but in the "Background jobs" page we have roughly 3-10 reindexing records per minute on that page, and that's also not that biggest library too.

that makes that page unusable - it is just polluted with a lot of noise.



I'd say we should make filtering in Background Jobs and show by default:

- unfinished only + number of totals,

- then hide index updates as well or allow users to filter out later by job type.

- + maybe count stats (oh I dream we use minion with it's statistics graphs one day :) - https://raw.github.com/mojolicious/minion/main/examples/admin.png?raw=true (https://docs.mojolicious.org/Minion))
Comment 4 Andrew Nugged 2022-06-01 12:56:58 UTC
More: more days passed, more testing on production. I see huge and very uncomfortable pollution in the "background jobs" table/page - all flooded with "reindexing".

I propose even more: what if we will make configurable parameter to ERASE successful reindex jobs on finish? The rest jobs might stay. Or that parameter will be even more flexible so we can configure what jobs should be wiped-out on the "errorless finish" state.
Comment 5 Jonathan Druart 2022-06-02 06:10:05 UTC
Created attachment 135570 [details] [review]
B

Bug 30462: Delete the update_elastic_index jobs when done

One entry per reindexation will generate a lot of rows in this table and
the UI will become unusable.
We should either hide them by default or simply remove them from the DB.
This patch implements the second option.

Test plan:
Edit a bibliographic record
If you are fast enough you should see a "new" jobs in the
background_jobs DB table
When done/finished, the rows will be deleted from the DB
Comment 6 Tomás Cohen Arazi 2022-06-02 12:25:02 UTC
I think you can do better with the commit subject heh.
Comment 7 Tomás Cohen Arazi 2022-06-02 12:27:18 UTC
Created attachment 135583 [details] [review]
Bug 30462: Delete the update_elastic_index jobs when done

One entry per reindexation will generate a lot of rows in this table and
the UI will become unusable.
We should either hide them by default or simply remove them from the DB.
This patch implements the second option.

Test plan:
Edit a bibliographic record
If you are fast enough you should see a "new" jobs in the
background_jobs DB table
When done/finished, the rows will be deleted from the DB

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 8 David Cook 2022-06-03 01:53:13 UTC
+1 to hiding the indexing tasks from the general view

I don't know about deleting them immediately. On one hand, I'm in favour of this and it would be useful for some other background tasks I have in mind (like OAI-PMH harvesting of records). On the other hand, it does make it harder to track background tasks when troubleshooting.

Could we hide them from the interface and then rely on cleanup_database.pl to clean up the successful indexing tasks nightly? 

That way users don't see useless tasks, but administrators could double-check.

The other issue I see with this one is that the behaviour is encoded in Koha/BackgroundJob/UpdateElasticIndex.pm specifically rather than in the general task management. Doesn't that open up the door for inconsistent behaviours among different background jobs? 

Not opposed to the current patch. Just asking some questions.
Comment 9 Martin Renvoize 2022-06-06 15:59:53 UTC
I'm with David on this one.. I think we should be hiding 'system tasks' from the UI by default and I don't think the task should be cleaning up after itself, rather we should add cleanup to cleanup_database.pl.

I also think perhaps we could be hiding completed tasks by default on top of hiding system tasks.
Comment 10 David Cook 2022-06-07 00:35:02 UTC
(In reply to Martin Renvoize from comment #9)
> I also think perhaps we could be hiding completed tasks by default on top of
> hiding system tasks.

Oh there's a good idea too. Maybe hiding completed tasks over a certain age though? Sometimes I think you'd want to see a recently completed task?
Comment 11 Martin Renvoize 2022-06-07 06:12:40 UTC
I took a quick look at the code and realised we're not using APIs or KohaTable here yet.. that has become a stretch goal for me, but we need something quick here.  I'll have a go at doing it the old fashioned way here today as an alternative option, but if that proves too big a goal I think the patch here gets us out of a hole in the short term.
Comment 12 Katrin Fischer 2022-06-07 07:53:13 UTC
(In reply to Martin Renvoize from comment #11)
> I took a quick look at the code and realised we're not using APIs or
> KohaTable here yet.. that has become a stretch goal for me, but we need
> something quick here.  I'll have a go at doing it the old fashioned way here
> today as an alternative option, but if that proves too big a goal I think
> the patch here gets us out of a hole in the short term.

I'd say hide them all by default and maybe have an option to see them if necessary (maybe like we do for transactions on the accounting table?)
Comment 13 Martin Renvoize 2022-06-07 10:09:46 UTC
Created attachment 135749 [details] [review]
Bug 30462: Separate Running and Complete jobs

This patch updates the background jobs management page to display
running and completed jobs in their own tabs on the page.
Comment 14 Martin Renvoize 2022-06-07 10:11:45 UTC
Created attachment 135750 [details] [review]
Bug 30462: Separate Queued and Complete jobs

This patch updates the background jobs management page to display
queued and completed jobs in their own tabs on the page.
Comment 15 Martin Renvoize 2022-06-07 10:14:43 UTC
This serves as an alternative.. instead of cleaning up the tasks from inside the job I split out Queued and Completed jobs into their own tabs.

I'm tempted to go further and add a checkbox for displaying 'System initiated' jobs along with the 'User initiated' jobs and hide the system ones by default.
Comment 16 Jonathan Druart 2022-06-07 11:44:01 UTC
I am seeing this: https://snipboard.io/atirmJ.jpg
Comment 17 Jonathan Druart 2022-06-07 11:46:05 UTC
Wouldn't it better to fetch the queued jobs only by default? Thinking about perfs if there are too many entries.
Comment 18 Jonathan Druart 2022-06-07 11:47:20 UTC
Or maybe simply remove the tabs and have a "see all" link/checkbox to display all the jobs.
Comment 19 David Cook 2022-06-07 23:26:03 UTC
(In reply to Jonathan Druart from comment #17)
> Wouldn't it better to fetch the queued jobs only by default? Thinking about
> perfs if there are too many entries.

I was thinking about that as well. 

So long as there is a timestamp index, I reckon we show the queued jobs and maybe the last 60 minutes of completed jobs (no indexing jobs though), and that should be good?

A database admin or Reports user can look up indexing jobs more directly if they want to double-check that the indexing is running correctly.
Comment 20 Martin Renvoize 2022-06-10 14:36:53 UTC
Created attachment 135938 [details] [review]
Bug 30462: (follow-up) Limit completed to 60 minutes

This follow-up reduces the completed jobs list to only those jobs
completed within the last 60 minutes.
Comment 21 Martin Renvoize 2022-06-10 14:38:35 UTC
I think this should be enough to backport and get us out of the current hole we are in.. I don't think we should completely hide index jobs for now.

I think further work should be done in new bugs linked off this one to refine the whole interface and convert to API-driven tables allowing for more full-fledged filters.
Comment 22 Jonathan Druart 2022-06-13 09:38:17 UTC
Created attachment 136001 [details] [review]
Bug 30462: Separate Queued and Complete jobs

This patch updates the background jobs management page to display
queued and completed jobs in their own tabs on the page.

JD Amended patch:
Fix QA failures:
 FAIL   koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt
   FAIL   forbidden patterns
                forbidden pattern: trailing space char (line 158)
                forbidden pattern: trailing space char (line 256)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 23 Jonathan Druart 2022-06-13 09:38:22 UTC
Created attachment 136002 [details] [review]
Bug 30462: (follow-up) Limit completed to 60 minutes

This follow-up reduces the completed jobs list to only those jobs
completed within the last 60 minutes.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 24 Jonathan Druart 2022-06-13 09:38:27 UTC
Created attachment 136003 [details] [review]
Bug 30462: Remove 'Ended' column for current jobs

This column will always be empty

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 25 Jonathan Druart 2022-06-13 09:38:32 UTC
Created attachment 136004 [details] [review]
Bug 30462: Fix display of the complete jobs

We want to display the jobs terminated in the last hour, not today.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 26 Jonathan Druart 2022-06-13 09:40:40 UTC
(In reply to Martin Renvoize from comment #21)
> I think this should be enough to backport and get us out of the current hole
> we are in.. I don't think we should completely hide index jobs for now.
> 
> I think further work should be done in new bugs linked off this one to
> refine the whole interface and convert to API-driven tables allowing for
> more full-fledged filters.

We will certainly need to provide a new patch to backport anyway. Now we don't see previous jobs.

I don't think this is correct, but SO to make the discussion move forward.
Comment 27 Katrin Fischer 2022-07-25 21:31:31 UTC
Can I please have a test plan for this one? I don't know much about the background jobs yet, so I wonder what the best way would be to ensure I have a completed and a queued job?
Comment 28 Jonathan Druart 2022-07-28 07:07:39 UTC
(In reply to Katrin Fischer from comment #27)
> Can I please have a test plan for this one? I don't know much about the
> background jobs yet, so I wonder what the best way would be to ensure I have
> a completed and a queued job?

You can use one of these methods:
1. batch mod a long list of items
2. add a sleep in the ->process method of one of the Koha::BackgroundJob modules
3. modify in DB directly to simulate a running job: set status="new", ended_on=null
Comment 29 Kyle M Hall 2022-09-02 13:55:58 UTC
Created attachment 140145 [details] [review]
Bug 30462: Separate Queued and Complete jobs

This patch updates the background jobs management page to display
queued and completed jobs in their own tabs on the page.

JD Amended patch:
Fix QA failures:
 FAIL   koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt
   FAIL   forbidden patterns
                forbidden pattern: trailing space char (line 158)
                forbidden pattern: trailing space char (line 256)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 30 Kyle M Hall 2022-09-02 13:56:08 UTC
Created attachment 140146 [details] [review]
Bug 30462: (follow-up) Limit completed to 60 minutes

This follow-up reduces the completed jobs list to only those jobs
completed within the last 60 minutes.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 31 Kyle M Hall 2022-09-02 13:56:11 UTC
Created attachment 140147 [details] [review]
Bug 30462: Remove 'Ended' column for current jobs

This column will always be empty

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 32 Kyle M Hall 2022-09-02 13:56:15 UTC
Created attachment 140148 [details] [review]
Bug 30462: Fix display of the complete jobs

We want to display the jobs terminated in the last hour, not today.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 33 Tomás Cohen Arazi 2022-09-08 17:50:18 UTC
Created attachment 140353 [details] [review]
Bug 30462: (QA follow-up) Tab name change

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 34 Tomás Cohen Arazi 2022-09-08 17:53:53 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!
Comment 35 Lucas Gass 2022-10-31 20:21:43 UTC
Enhancement will not be backported for 22.05.x series
Comment 36 Nick Clemens 2022-11-01 11:36:28 UTC
Created attachment 142860 [details] [review]
Bug 30462: [22.05.x] Separate Queued and Complete jobs

This patch updates the background jobs management page to display
queued and completed jobs in their own tabs on the page.

JD Amended patch:
Fix QA failures:
 FAIL   koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt
   FAIL   forbidden patterns
                forbidden pattern: trailing space char (line 158)
                forbidden pattern: trailing space char (line 256)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Bug 30462: (follow-up) Limit completed to 60 minutes

This follow-up reduces the completed jobs list to only those jobs
completed within the last 60 minutes.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Bug 30462: Remove 'Ended' column for current jobs

This column will always be empty

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Bug 30462: Fix display of the complete jobs

We want to display the jobs terminated in the last hour, not today.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Bug 30462: (QA follow-up) Tab name change

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 37 Lucas Gass 2022-11-01 21:39:27 UTC
Backported to 22.05.x for upcoming 22.05.07 release
Comment 38 Arthur Suzuki 2022-11-14 17:05:42 UTC
depends on 27344, seems i can't apply that one on 21.11.