Bug 18618 - Mana - Add reading suggestions (crontab and scripts for Koha)
Summary: Mana - Add reading suggestions (crontab and scripts for Koha)
Status: ASSIGNED
Alias: None
Product: Koha
Classification: Unclassified
Component: Web services (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Alex Arnaud
QA Contact: Testopia
URL:
Keywords:
: 18646 (view as bug list)
Depends on: 17047
Blocks:
  Show dependency treegraph
 
Reported: 2017-05-17 08:03 UTC by Baptiste
Modified: 2023-04-16 01:51 UTC (History)
14 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 18618: Mana - Add scripts to send readings (9.31 KB, patch)
2017-05-17 08:13 UTC, Baptiste
Details | Diff | Splinter Review
Bug 18618: Mana - Add reading suggestion (16.38 KB, patch)
2017-05-17 08:13 UTC, Baptiste
Details | Diff | Splinter Review
Bug 18618: Add a pagination for suggestions (11.64 KB, patch)
2017-06-22 08:55 UTC, Baptiste
Details | Diff | Splinter Review
Bug 18618: Correct Plack issues (1.03 KB, patch)
2017-07-10 13:03 UTC, Baptiste
Details | Diff | Splinter Review
Bug 18618: Mana - Add scripts to send readings (9.37 KB, patch)
2017-07-14 19:56 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 18618: Mana - Add scripts to send readings (9.37 KB, patch)
2017-07-14 19:57 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 18618: Mana - Add reading suggestion (16.44 KB, patch)
2017-07-14 19:57 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 18618: Add a pagination for suggestions (11.71 KB, patch)
2017-07-14 19:57 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 18618: Correct Plack issues (1.09 KB, patch)
2017-07-14 19:57 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 18618: Mana - Add scripts to send readings (9.37 KB, patch)
2022-02-17 10:10 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Mana - Add reading suggestion (16.29 KB, patch)
2022-02-17 10:10 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Add a pagination for suggestions (10.97 KB, patch)
2022-02-17 10:11 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Correct Plack issues (1.10 KB, patch)
2022-02-17 10:11 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Rebase modification (12.28 KB, patch)
2022-02-17 10:11 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618 : Many small refacto and improvement (14.89 KB, patch)
2022-02-17 10:11 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Add a better support for different id types (26.19 KB, patch)
2022-02-17 10:12 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Add MARC21 support (12.67 KB, patch)
2022-02-17 10:12 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618 : refacto to allow unit test and add unit test (92.06 KB, patch)
2022-02-17 10:12 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Update database (4.33 KB, patch)
2022-02-17 10:12 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Mana - Add scripts to send readings (9.30 KB, patch)
2022-02-17 13:22 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Mana - Add reading suggestion (16.22 KB, patch)
2022-02-17 13:22 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Add a pagination for suggestions (10.90 KB, patch)
2022-02-17 13:22 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Correct Plack issues (1.04 KB, patch)
2022-02-17 13:22 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Rebase modification (12.28 KB, patch)
2022-02-17 13:27 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618 : Many small refacto and improvement (14.89 KB, patch)
2022-02-17 13:27 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Add a better support for different id types (26.19 KB, patch)
2022-02-17 13:27 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Add MARC21 support (12.67 KB, patch)
2022-02-17 13:27 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618 : refacto to allow unit test and add unit test (92.06 KB, patch)
2022-02-17 13:28 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Update database (4.33 KB, patch)
2022-02-17 13:28 UTC, Marion Durand
Details | Diff | Splinter Review
Bug 18618: Mana readings pairs (96.73 KB, patch)
2022-03-31 09:35 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 18618: Update database (4.33 KB, patch)
2022-03-31 09:36 UTC, Alex Arnaud
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Baptiste 2017-05-17 08:03:51 UTC

    
Comment 1 Baptiste 2017-05-17 08:13:31 UTC
Created attachment 63496 [details] [review]
Bug 18618: Mana - Add scripts to send readings

- There are 2 differents scripts:
misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed
misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled

- Both of scripts anonymize the datas and send them to mana.
Comment 2 Baptiste 2017-05-17 08:13:40 UTC
Created attachment 63497 [details] [review]
Bug 18618: Mana - Add reading suggestion

There is now a module to get suggestions from Mana.

- It is activated by default on Mana.
- When you browse on a notice, there is now a tab called "Suggestions"
- On click on the tab, it will check in the database if it has suggestions.
- Else, it will ask to Mana for suggestions and will store in the database.

- The suggestions are stored in the NEW table reading_pairs
- Implement new dbix objets: Reading_suggestion and Reading_suggestions
Comment 3 Paul Poulain 2017-06-01 13:06:36 UTC
FYI: we've setup this cron for one of our customer (a modified version, with hardcoded token). It's harvesting pairs since more than 2 weeks now.
We'll add 10 more in the next days. Most of them having about 100 to 1000 issues/day

The mana-side code is available here : https://git.biblibre.com/biblibre/koha-mana.
It's been deployed already on mana-kb and mana-test.koha-community.org (this has no consequence on existing API. It just add new routes for managing reading suggestions)
Comment 4 Baptiste 2017-06-22 08:55:38 UTC
Created attachment 64531 [details] [review]
Bug 18618: Add a pagination for suggestions

*** Suggestions are now sent from Mana as follows

- Koha requests the first 20 suggestions
- It processes them and estimates how much it will need to get all suggestions
- It requests Mana the number of suggestions needed
- It repeats the 3 first steps until
    (1) Mana doesn't have suggestions anymore
    (2) It got 10 suggestions to display

*** Koha interface is a bit more verbose while searching (just a bit).
Comment 5 Baptiste 2017-07-10 13:03:02 UTC
Created attachment 64960 [details] [review]
Bug 18618: Correct Plack issues

Split one command line into two in order to be accepted by Plack
Comment 6 Brendan Gallagher 2017-07-14 19:56:48 UTC
Created attachment 65066 [details] [review]
Bug 18618: Mana - Add scripts to send readings

- There are 2 differents scripts:
misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed
misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled

- Both of scripts anonymize the datas and send them to mana.

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 7 Brendan Gallagher 2017-07-14 19:57:22 UTC
Created attachment 65067 [details] [review]
Bug 18618: Mana - Add scripts to send readings

- There are 2 differents scripts:
misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed
misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled

- Both of scripts anonymize the datas and send them to mana.

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 8 Brendan Gallagher 2017-07-14 19:57:31 UTC
Created attachment 65068 [details] [review]
Bug 18618: Mana - Add reading suggestion

There is now a module to get suggestions from Mana.

- It is activated by default on Mana.
- When you browse on a notice, there is now a tab called "Suggestions"
- On click on the tab, it will check in the database if it has suggestions.
- Else, it will ask to Mana for suggestions and will store in the database.

- The suggestions are stored in the NEW table reading_pairs
- Implement new dbix objets: Reading_suggestion and Reading_suggestions

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 9 Brendan Gallagher 2017-07-14 19:57:34 UTC
Created attachment 65069 [details] [review]
Bug 18618: Add a pagination for suggestions

*** Suggestions are now sent from Mana as follows

- Koha requests the first 20 suggestions
- It processes them and estimates how much it will need to get all suggestions
- It requests Mana the number of suggestions needed
- It repeats the 3 first steps until
    (1) Mana doesn't have suggestions anymore
    (2) It got 10 suggestions to display

*** Koha interface is a bit more verbose while searching (just a bit).

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 10 Brendan Gallagher 2017-07-14 19:57:37 UTC
Created attachment 65070 [details] [review]
Bug 18618: Correct Plack issues

Split one command line into two in order to be accepted by Plack

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 11 Victor Grousset/tuxayo 2018-02-08 13:29:49 UTC
Baptiste doesn't work anymore at BibLibre.
Unassigning to avoid signaling that this bug it still being supervized. Which could wrongly discourage someone to continue the work.
Comment 12 Katrin Fischer 2018-02-09 06:55:38 UTC
BLOCKED by failed base patch bug 17047
Comment 13 Alex Buckley 2018-03-28 04:05:25 UTC
*** Bug 18646 has been marked as a duplicate of this bug. ***
Comment 14 Arthur Suzuki 2019-07-26 10:43:14 UTC
Applies and test fine, not blocked anymore since Bz17047 is now pushed to master.
Shall I remove the dependancy?
Comment 15 David Nind 2021-04-10 22:14:30 UTC
Patch no longer applies 8-(..

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 18618: Mana - Add scripts to send readings
Applying: Bug 18618: Mana - Add reading suggestion
Using index info to reconstruct a base tree...
M	Koha/Biblio.pm
M	koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-detail.tt
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-detail.tt
CONFLICT (content): Merge conflict in koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-detail.tt
Auto-merging Koha/Biblio.pm
error: Failed to merge in the changes.
Patch failed at 0001 Bug 18618: Mana - Add reading suggestion
Comment 16 Marion Durand 2022-02-17 10:10:36 UTC Comment hidden (obsolete)
Comment 17 Marion Durand 2022-02-17 10:10:48 UTC Comment hidden (obsolete)
Comment 18 Marion Durand 2022-02-17 10:11:02 UTC Comment hidden (obsolete)
Comment 19 Marion Durand 2022-02-17 10:11:07 UTC Comment hidden (obsolete)
Comment 20 Marion Durand 2022-02-17 10:11:19 UTC Comment hidden (obsolete)
Comment 21 Marion Durand 2022-02-17 10:11:49 UTC Comment hidden (obsolete)
Comment 22 Marion Durand 2022-02-17 10:12:05 UTC Comment hidden (obsolete)
Comment 23 Marion Durand 2022-02-17 10:12:18 UTC Comment hidden (obsolete)
Comment 24 Marion Durand 2022-02-17 10:12:50 UTC Comment hidden (obsolete)
Comment 25 Marion Durand 2022-02-17 10:12:57 UTC Comment hidden (obsolete)
Comment 26 Marion Durand 2022-02-17 10:56:15 UTC
This feature adds reading suggestion for Koha using Mana.
It is based on pairs of document borrowed by the same patron. (If two document are often borrowed together, it is probably interesting to suggest the second while looking for suggestion linked with the first)

This feature is divided in two part : 
- One part for extracting pair to feed Mana database
- One part for getting and displaying suggestions on the OPAC

Testing is only possible with the test server of Mana because mana-side code is not integrated yet (you can find it here https://gitlab.com/koha-community/koha-mana/-/merge_requests/13)
We will let you know here when it's ready.
Comment 27 Marion Durand 2022-02-17 13:22:16 UTC
Created attachment 130735 [details] [review]
Bug 18618: Mana - Add scripts to send readings

- There are 2 differents scripts:
misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed
misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled

- Both of scripts anonymize the datas and send them to mana.
Comment 28 Marion Durand 2022-02-17 13:22:24 UTC
Created attachment 130736 [details] [review]
Bug 18618: Mana - Add reading suggestion

There is now a module to get suggestions from Mana.

- It is activated by default on Mana.
- When you browse on a notice, there is now a tab called "Suggestions"
- On click on the tab, it will check in the database if it has suggestions.
- Else, it will ask to Mana for suggestions and will store in the database.

- The suggestions are stored in the NEW table reading_pairs
- Implement new dbix objets: Reading_suggestion and Reading_suggestions
Comment 29 Marion Durand 2022-02-17 13:22:29 UTC
Created attachment 130737 [details] [review]
Bug 18618: Add a pagination for suggestions

*** Suggestions are now sent from Mana as follows

- Koha requests the first 20 suggestions
- It processes them and estimates how much it will need to get all suggestions
- It requests Mana the number of suggestions needed
- It repeats the 3 first steps until
    (1) Mana doesn't have suggestions anymore
    (2) It got 10 suggestions to display

*** Koha interface is a bit more verbose while searching (just a bit).
Comment 30 Marion Durand 2022-02-17 13:22:36 UTC
Created attachment 130738 [details] [review]
Bug 18618: Correct Plack issues

Split one command line into two in order to be accepted by Plack
Comment 31 Marion Durand 2022-02-17 13:27:36 UTC
Created attachment 130739 [details] [review]
Bug 18618: Rebase modification

Remove mana_send_pairs_without_mana.pl
  This script was here for testing purpose and is not usefull anymore
Fix name issues (some function have been renamed)
Fix a compilation issues (redeclared variable)
Remove an unused variable
Remove an extra % in an sql request
Fix display problem when getting info from mana
  When getting info from mana the biblios array contain blessed references
  but the template don't deal with that properly.
  It work fine whe koha is getting info from it's interne database because
  biblios array contain unblessed references.
Comment 32 Marion Durand 2022-02-17 13:27:47 UTC
Created attachment 130740 [details] [review]
Bug 18618 : Many small refacto and improvement

* change duration before refresh and improve the use of $temp variable
* improve SQL request for extracting pairs
     fix condition
     create pair with old_issues
     use placeholders
* improve SQL request for getting suggestion
    add the possibility to look in metadata for the id
* execute the script only if mana is activated
* optimize imbricate foreach loop
* add a max number of simultaneous issues
* remove unused option
Comment 33 Marion Durand 2022-02-17 13:27:53 UTC
Created attachment 130741 [details] [review]
Bug 18618: Add a better support for different id types

Add a better support for ISMN, ISRN, ISRC and UPC

Add normalization function for some of these ids.

When sending pairs to mana and when getting suggestions these ids are
now taken into account correctly.

Documents with multiple identifier were causing various trouble.
It was needed to add functions to normalize all ids

This patch add the possibility to:
- create pairs with documents with multiple ids (all the possible pairs will be created)
- get suggestion for a document with multiple ids
- display correctly the document when the suggested document has multiple ids
Comment 34 Marion Durand 2022-02-17 13:27:59 UTC
Created attachment 130742 [details] [review]
Bug 18618: Add MARC21 support

This functionnality is now working for both MARC21 and UNIMARC.
Comment 35 Marion Durand 2022-02-17 13:28:05 UTC
Created attachment 130743 [details] [review]
Bug 18618 : refacto to allow unit test and add unit test

Re-write pairs extraction script to use testable function
Re-write getSuggestion script to use testable function
Add unit test for normalizaation functions
Add unit test for pairs extractions function
Add unit test for getSuggestion function
Fix many small problems found while writing unit test
Comment 36 Marion Durand 2022-02-17 13:28:11 UTC
Created attachment 130744 [details] [review]
Bug 18618: Update database

create the atomicupdate file
Comment 37 Alex Arnaud 2022-03-31 09:35:49 UTC
Created attachment 132690 [details] [review]
Bug 18618: Mana readings pairs

- There are 2 differents scripts:
misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed
misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled

- Both of scripts anonymize the datas and send them to mana.

Bug 18618: Mana - Add reading suggestion

There is now a module to get suggestions from Mana.

- It is activated by default on Mana.
- When you browse on a notice, there is now a tab called "Suggestions"
- On click on the tab, it will check in the database if it has suggestions.
- Else, it will ask to Mana for suggestions and will store in the database.

- The suggestions are stored in the NEW table reading_pairs
- Implement new dbix objets: Reading_suggestion and Reading_suggestions

Bug 18618: Add a pagination for suggestions

*** Suggestions are now sent from Mana as follows

- Koha requests the first 20 suggestions
- It processes them and estimates how much it will need to get all suggestions
- It requests Mana the number of suggestions needed
- It repeats the 3 first steps until
    (1) Mana doesn't have suggestions anymore
    (2) It got 10 suggestions to display

*** Koha interface is a bit more verbose while searching (just a bit).

Bug 18618: Correct Plack issues

Split one command line into two in order to be accepted by Plack

Bug 18618: Rebase modification

Remove mana_send_pairs_without_mana.pl
  This script was here for testing purpose and is not usefull anymore
Fix name issues (some function have been renamed)
Fix a compilation issues (redeclared variable)
Remove an unused variable
Remove an extra % in an sql request
Fix display problem when getting info from mana
  When getting info from mana the biblios array contain blessed references
  but the template don't deal with that properly.
  It work fine whe koha is getting info from it's interne database because
  biblios array contain unblessed references.

Bug 18618 : Many small refacto and improvement

* change duration before refresh and improve the use of $temp variable
* improve SQL request for extracting pairs
     fix condition
     create pair with old_issues
     use placeholders
* improve SQL request for getting suggestion
    add the possibility to look in metadata for the id
* execute the script only if mana is activated
* optimize imbricate foreach loop
* add a max number of simultaneous issues
* remove unused option

Bug 18618: Add a better support for different id types

Add a better support for ISMN, ISRN, ISRC and UPC

Add normalization function for some of these ids.

When sending pairs to mana and when getting suggestions these ids are
now taken into account correctly.

Documents with multiple identifier were causing various trouble.
It was needed to add functions to normalize all ids

This patch add the possibility to:
- create pairs with documents with multiple ids (all the possible pairs will be created)
- get suggestion for a document with multiple ids
- display correctly the document when the suggested document has multiple ids

Bug 18618: Add MARC21 support

This functionnality is now working for both MARC21 and UNIMARC.

Bug 18618 : refacto to allow unit test and add unit test

Re-write pairs extraction script to use testable function
Re-write getSuggestion script to use testable function
Add unit test for normalizaation functions
Add unit test for pairs extractions function
Add unit test for getSuggestion function
Fix many small problems found while writing unit test
Comment 38 Alex Arnaud 2022-03-31 09:36:43 UTC
Created attachment 132691 [details] [review]
Bug 18618: Update database

create the atomicupdate file
Comment 39 Jonathan Druart 2022-03-31 10:25:47 UTC
After dicsussion at the hackfest

BibLibre:
* Squash + clear (perltidy) patches and changes
* Test the Koha patches (ie. fix it as it do not work)
  DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Field 'biblionumber1' doesn't have a default value at /kohadevbox/koha/Koha/Object.pm line 170
* Make sure the PR in up-to-date on gitlab
* Write release notes + doc + test plan (provide title to get relevant suggestions)
* exportemail + securitytoken => Why?
* Route shouldn't be ".json"

Joubu:
* Improve DB structure (biblionumberX)
* Remove/Adjust Koha::Utils methods + Koha::Biblio->normalize_isbn, etc.
* Remove useless tests
* Test new Koha code with prod/old server code (to make sure we won't introduce regression when upgrading the server)
Comment 40 Katrin Fischer 2023-04-16 01:51:07 UTC
Any update on this one?