We will certainly end up with lot of indexation tasks. Shouldn't we hide them by default when we list the tasks on Home > Administration > Background jobs ?
(In reply to Jonathan Druart from comment #0) > We will certainly end up with lot of indexation tasks. Shouldn't we hide > them by default when we list the tasks on Home > Administration > Background > jobs ? Yes!
Yes :)
bringing our experience from piloting production with the current master. It's even not that "very busy day" and vacations week, but in the "Background jobs" page we have roughly 3-10 reindexing records per minute on that page, and that's also not that biggest library too. that makes that page unusable - it is just polluted with a lot of noise. I'd say we should make filtering in Background Jobs and show by default: - unfinished only + number of totals, - then hide index updates as well or allow users to filter out later by job type. - + maybe count stats (oh I dream we use minion with it's statistics graphs one day :) - https://raw.github.com/mojolicious/minion/main/examples/admin.png?raw=true (https://docs.mojolicious.org/Minion))
More: more days passed, more testing on production. I see huge and very uncomfortable pollution in the "background jobs" table/page - all flooded with "reindexing". I propose even more: what if we will make configurable parameter to ERASE successful reindex jobs on finish? The rest jobs might stay. Or that parameter will be even more flexible so we can configure what jobs should be wiped-out on the "errorless finish" state.
Created attachment 135570 [details] [review] B Bug 30462: Delete the update_elastic_index jobs when done One entry per reindexation will generate a lot of rows in this table and the UI will become unusable. We should either hide them by default or simply remove them from the DB. This patch implements the second option. Test plan: Edit a bibliographic record If you are fast enough you should see a "new" jobs in the background_jobs DB table When done/finished, the rows will be deleted from the DB
I think you can do better with the commit subject heh.
Created attachment 135583 [details] [review] Bug 30462: Delete the update_elastic_index jobs when done One entry per reindexation will generate a lot of rows in this table and the UI will become unusable. We should either hide them by default or simply remove them from the DB. This patch implements the second option. Test plan: Edit a bibliographic record If you are fast enough you should see a "new" jobs in the background_jobs DB table When done/finished, the rows will be deleted from the DB Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
+1 to hiding the indexing tasks from the general view I don't know about deleting them immediately. On one hand, I'm in favour of this and it would be useful for some other background tasks I have in mind (like OAI-PMH harvesting of records). On the other hand, it does make it harder to track background tasks when troubleshooting. Could we hide them from the interface and then rely on cleanup_database.pl to clean up the successful indexing tasks nightly? That way users don't see useless tasks, but administrators could double-check. The other issue I see with this one is that the behaviour is encoded in Koha/BackgroundJob/UpdateElasticIndex.pm specifically rather than in the general task management. Doesn't that open up the door for inconsistent behaviours among different background jobs? Not opposed to the current patch. Just asking some questions.
I'm with David on this one.. I think we should be hiding 'system tasks' from the UI by default and I don't think the task should be cleaning up after itself, rather we should add cleanup to cleanup_database.pl. I also think perhaps we could be hiding completed tasks by default on top of hiding system tasks.
(In reply to Martin Renvoize from comment #9) > I also think perhaps we could be hiding completed tasks by default on top of > hiding system tasks. Oh there's a good idea too. Maybe hiding completed tasks over a certain age though? Sometimes I think you'd want to see a recently completed task?
I took a quick look at the code and realised we're not using APIs or KohaTable here yet.. that has become a stretch goal for me, but we need something quick here. I'll have a go at doing it the old fashioned way here today as an alternative option, but if that proves too big a goal I think the patch here gets us out of a hole in the short term.
(In reply to Martin Renvoize from comment #11) > I took a quick look at the code and realised we're not using APIs or > KohaTable here yet.. that has become a stretch goal for me, but we need > something quick here. I'll have a go at doing it the old fashioned way here > today as an alternative option, but if that proves too big a goal I think > the patch here gets us out of a hole in the short term. I'd say hide them all by default and maybe have an option to see them if necessary (maybe like we do for transactions on the accounting table?)
Created attachment 135749 [details] [review] Bug 30462: Separate Running and Complete jobs This patch updates the background jobs management page to display running and completed jobs in their own tabs on the page.
Created attachment 135750 [details] [review] Bug 30462: Separate Queued and Complete jobs This patch updates the background jobs management page to display queued and completed jobs in their own tabs on the page.
This serves as an alternative.. instead of cleaning up the tasks from inside the job I split out Queued and Completed jobs into their own tabs. I'm tempted to go further and add a checkbox for displaying 'System initiated' jobs along with the 'User initiated' jobs and hide the system ones by default.
I am seeing this: https://snipboard.io/atirmJ.jpg
Wouldn't it better to fetch the queued jobs only by default? Thinking about perfs if there are too many entries.
Or maybe simply remove the tabs and have a "see all" link/checkbox to display all the jobs.
(In reply to Jonathan Druart from comment #17) > Wouldn't it better to fetch the queued jobs only by default? Thinking about > perfs if there are too many entries. I was thinking about that as well. So long as there is a timestamp index, I reckon we show the queued jobs and maybe the last 60 minutes of completed jobs (no indexing jobs though), and that should be good? A database admin or Reports user can look up indexing jobs more directly if they want to double-check that the indexing is running correctly.
Created attachment 135938 [details] [review] Bug 30462: (follow-up) Limit completed to 60 minutes This follow-up reduces the completed jobs list to only those jobs completed within the last 60 minutes.
I think this should be enough to backport and get us out of the current hole we are in.. I don't think we should completely hide index jobs for now. I think further work should be done in new bugs linked off this one to refine the whole interface and convert to API-driven tables allowing for more full-fledged filters.
Created attachment 136001 [details] [review] Bug 30462: Separate Queued and Complete jobs This patch updates the background jobs management page to display queued and completed jobs in their own tabs on the page. JD Amended patch: Fix QA failures: FAIL koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt FAIL forbidden patterns forbidden pattern: trailing space char (line 158) forbidden pattern: trailing space char (line 256) Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 136002 [details] [review] Bug 30462: (follow-up) Limit completed to 60 minutes This follow-up reduces the completed jobs list to only those jobs completed within the last 60 minutes. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 136003 [details] [review] Bug 30462: Remove 'Ended' column for current jobs This column will always be empty Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 136004 [details] [review] Bug 30462: Fix display of the complete jobs We want to display the jobs terminated in the last hour, not today. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(In reply to Martin Renvoize from comment #21) > I think this should be enough to backport and get us out of the current hole > we are in.. I don't think we should completely hide index jobs for now. > > I think further work should be done in new bugs linked off this one to > refine the whole interface and convert to API-driven tables allowing for > more full-fledged filters. We will certainly need to provide a new patch to backport anyway. Now we don't see previous jobs. I don't think this is correct, but SO to make the discussion move forward.
Can I please have a test plan for this one? I don't know much about the background jobs yet, so I wonder what the best way would be to ensure I have a completed and a queued job?
(In reply to Katrin Fischer from comment #27) > Can I please have a test plan for this one? I don't know much about the > background jobs yet, so I wonder what the best way would be to ensure I have > a completed and a queued job? You can use one of these methods: 1. batch mod a long list of items 2. add a sleep in the ->process method of one of the Koha::BackgroundJob modules 3. modify in DB directly to simulate a running job: set status="new", ended_on=null
Created attachment 140145 [details] [review] Bug 30462: Separate Queued and Complete jobs This patch updates the background jobs management page to display queued and completed jobs in their own tabs on the page. JD Amended patch: Fix QA failures: FAIL koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt FAIL forbidden patterns forbidden pattern: trailing space char (line 158) forbidden pattern: trailing space char (line 256) Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 140146 [details] [review] Bug 30462: (follow-up) Limit completed to 60 minutes This follow-up reduces the completed jobs list to only those jobs completed within the last 60 minutes. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 140147 [details] [review] Bug 30462: Remove 'Ended' column for current jobs This column will always be empty Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 140148 [details] [review] Bug 30462: Fix display of the complete jobs We want to display the jobs terminated in the last hour, not today. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 140353 [details] [review] Bug 30462: (QA follow-up) Tab name change Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Pushed to master for 22.11. Nice work everyone, thanks!
Enhancement will not be backported for 22.05.x series
Created attachment 142860 [details] [review] Bug 30462: [22.05.x] Separate Queued and Complete jobs This patch updates the background jobs management page to display queued and completed jobs in their own tabs on the page. JD Amended patch: Fix QA failures: FAIL koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt FAIL forbidden patterns forbidden pattern: trailing space char (line 158) forbidden pattern: trailing space char (line 256) Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Bug 30462: (follow-up) Limit completed to 60 minutes This follow-up reduces the completed jobs list to only those jobs completed within the last 60 minutes. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Bug 30462: Remove 'Ended' column for current jobs This column will always be empty Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Bug 30462: Fix display of the complete jobs We want to display the jobs terminated in the last hour, not today. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Bug 30462: (QA follow-up) Tab name change Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Backported to 22.05.x for upcoming 22.05.07 release
depends on 27344, seems i can't apply that one on 21.11.