Created attachment 63496 [details] [review] Bug 18618: Mana - Add scripts to send readings - There are 2 differents scripts: misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled - Both of scripts anonymize the datas and send them to mana.
Created attachment 63497 [details] [review] Bug 18618: Mana - Add reading suggestion There is now a module to get suggestions from Mana. - It is activated by default on Mana. - When you browse on a notice, there is now a tab called "Suggestions" - On click on the tab, it will check in the database if it has suggestions. - Else, it will ask to Mana for suggestions and will store in the database. - The suggestions are stored in the NEW table reading_pairs - Implement new dbix objets: Reading_suggestion and Reading_suggestions
FYI: we've setup this cron for one of our customer (a modified version, with hardcoded token). It's harvesting pairs since more than 2 weeks now. We'll add 10 more in the next days. Most of them having about 100 to 1000 issues/day The mana-side code is available here : https://git.biblibre.com/biblibre/koha-mana. It's been deployed already on mana-kb and mana-test.koha-community.org (this has no consequence on existing API. It just add new routes for managing reading suggestions)
Created attachment 64531 [details] [review] Bug 18618: Add a pagination for suggestions *** Suggestions are now sent from Mana as follows - Koha requests the first 20 suggestions - It processes them and estimates how much it will need to get all suggestions - It requests Mana the number of suggestions needed - It repeats the 3 first steps until (1) Mana doesn't have suggestions anymore (2) It got 10 suggestions to display *** Koha interface is a bit more verbose while searching (just a bit).
Created attachment 64960 [details] [review] Bug 18618: Correct Plack issues Split one command line into two in order to be accepted by Plack
Created attachment 65066 [details] [review] Bug 18618: Mana - Add scripts to send readings - There are 2 differents scripts: misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled - Both of scripts anonymize the datas and send them to mana. Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 65067 [details] [review] Bug 18618: Mana - Add scripts to send readings - There are 2 differents scripts: misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled - Both of scripts anonymize the datas and send them to mana. Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 65068 [details] [review] Bug 18618: Mana - Add reading suggestion There is now a module to get suggestions from Mana. - It is activated by default on Mana. - When you browse on a notice, there is now a tab called "Suggestions" - On click on the tab, it will check in the database if it has suggestions. - Else, it will ask to Mana for suggestions and will store in the database. - The suggestions are stored in the NEW table reading_pairs - Implement new dbix objets: Reading_suggestion and Reading_suggestions Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 65069 [details] [review] Bug 18618: Add a pagination for suggestions *** Suggestions are now sent from Mana as follows - Koha requests the first 20 suggestions - It processes them and estimates how much it will need to get all suggestions - It requests Mana the number of suggestions needed - It repeats the 3 first steps until (1) Mana doesn't have suggestions anymore (2) It got 10 suggestions to display *** Koha interface is a bit more verbose while searching (just a bit). Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 65070 [details] [review] Bug 18618: Correct Plack issues Split one command line into two in order to be accepted by Plack Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Baptiste doesn't work anymore at BibLibre. Unassigning to avoid signaling that this bug it still being supervized. Which could wrongly discourage someone to continue the work.
BLOCKED by failed base patch bug 17047
*** Bug 18646 has been marked as a duplicate of this bug. ***
Applies and test fine, not blocked anymore since Bz17047 is now pushed to master. Shall I remove the dependancy?
Patch no longer applies 8-(.. Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 18618: Mana - Add scripts to send readings Applying: Bug 18618: Mana - Add reading suggestion Using index info to reconstruct a base tree... M Koha/Biblio.pm M koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-detail.tt Falling back to patching base and 3-way merge... Auto-merging koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-detail.tt CONFLICT (content): Merge conflict in koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-detail.tt Auto-merging Koha/Biblio.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 18618: Mana - Add reading suggestion
Created attachment 130706 [details] [review] Bug 18618: Mana - Add scripts to send readings - There are 2 differents scripts: misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled - Both of scripts anonymize the datas and send them to mana.
Created attachment 130707 [details] [review] Bug 18618: Mana - Add reading suggestion There is now a module to get suggestions from Mana. - It is activated by default on Mana. - When you browse on a notice, there is now a tab called "Suggestions" - On click on the tab, it will check in the database if it has suggestions. - Else, it will ask to Mana for suggestions and will store in the database. - The suggestions are stored in the NEW table reading_pairs - Implement new dbix objets: Reading_suggestion and Reading_suggestions Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 130708 [details] [review] Bug 18618: Add a pagination for suggestions *** Suggestions are now sent from Mana as follows - Koha requests the first 20 suggestions - It processes them and estimates how much it will need to get all suggestions - It requests Mana the number of suggestions needed - It repeats the 3 first steps until (1) Mana doesn't have suggestions anymore (2) It got 10 suggestions to display *** Koha interface is a bit more verbose while searching (just a bit). Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 130709 [details] [review] Bug 18618: Correct Plack issues Split one command line into two in order to be accepted by Plack Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 130710 [details] [review] Bug 18618: Rebase modification Remove mana_send_pairs_without_mana.pl This script was here for testing purpose and is not usefull anymore Fix name issues (some function have been renamed) Fix a compilation issues (redeclared variable) Remove an unused variable Remove an extra % in an sql request Fix display problem when getting info from mana When getting info from mana the biblios array contain blessed references but the template don't deal with that properly. It work fine whe koha is getting info from it's interne database because biblios array contain unblessed references.
Created attachment 130711 [details] [review] Bug 18618 : Many small refacto and improvement * change duration before refresh and improve the use of $temp variable * improve SQL request for extracting pairs fix condition create pair with old_issues use placeholders * improve SQL request for getting suggestion add the possibility to look in metadata for the id * execute the script only if mana is activated * optimize imbricate foreach loop * add a max number of simultaneous issues * remove unused option
Created attachment 130712 [details] [review] Bug 18618: Add a better support for different id types Add a better support for ISMN, ISRN, ISRC and UPC Add normalization function for some of these ids. When sending pairs to mana and when getting suggestions these ids are now taken into account correctly. Documents with multiple identifier were causing various trouble. It was needed to add functions to normalize all ids This patch add the possibility to: - create pairs with documents with multiple ids (all the possible pairs will be created) - get suggestion for a document with multiple ids - display correctly the document when the suggested document has multiple ids
Created attachment 130713 [details] [review] Bug 18618: Add MARC21 support This functionnality is now working for both MARC21 and UNIMARC.
Created attachment 130714 [details] [review] Bug 18618 : refacto to allow unit test and add unit test Re-write pairs extraction script to use testable function Re-write getSuggestion script to use testable function Add unit test for normalizaation functions Add unit test for pairs extractions function Add unit test for getSuggestion function Fix many small problems found while writing unit test
Created attachment 130715 [details] [review] Bug 18618: Update database create the atomicupdate file
This feature adds reading suggestion for Koha using Mana. It is based on pairs of document borrowed by the same patron. (If two document are often borrowed together, it is probably interesting to suggest the second while looking for suggestion linked with the first) This feature is divided in two part : - One part for extracting pair to feed Mana database - One part for getting and displaying suggestions on the OPAC Testing is only possible with the test server of Mana because mana-side code is not integrated yet (you can find it here https://gitlab.com/koha-community/koha-mana/-/merge_requests/13) We will let you know here when it's ready.
Created attachment 130735 [details] [review] Bug 18618: Mana - Add scripts to send readings - There are 2 differents scripts: misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled - Both of scripts anonymize the datas and send them to mana.
Created attachment 130736 [details] [review] Bug 18618: Mana - Add reading suggestion There is now a module to get suggestions from Mana. - It is activated by default on Mana. - When you browse on a notice, there is now a tab called "Suggestions" - On click on the tab, it will check in the database if it has suggestions. - Else, it will ask to Mana for suggestions and will store in the database. - The suggestions are stored in the NEW table reading_pairs - Implement new dbix objets: Reading_suggestion and Reading_suggestions
Created attachment 130737 [details] [review] Bug 18618: Add a pagination for suggestions *** Suggestions are now sent from Mana as follows - Koha requests the first 20 suggestions - It processes them and estimates how much it will need to get all suggestions - It requests Mana the number of suggestions needed - It repeats the 3 first steps until (1) Mana doesn't have suggestions anymore (2) It got 10 suggestions to display *** Koha interface is a bit more verbose while searching (just a bit).
Created attachment 130738 [details] [review] Bug 18618: Correct Plack issues Split one command line into two in order to be accepted by Plack
Created attachment 130739 [details] [review] Bug 18618: Rebase modification Remove mana_send_pairs_without_mana.pl This script was here for testing purpose and is not usefull anymore Fix name issues (some function have been renamed) Fix a compilation issues (redeclared variable) Remove an unused variable Remove an extra % in an sql request Fix display problem when getting info from mana When getting info from mana the biblios array contain blessed references but the template don't deal with that properly. It work fine whe koha is getting info from it's interne database because biblios array contain unblessed references.
Created attachment 130740 [details] [review] Bug 18618 : Many small refacto and improvement * change duration before refresh and improve the use of $temp variable * improve SQL request for extracting pairs fix condition create pair with old_issues use placeholders * improve SQL request for getting suggestion add the possibility to look in metadata for the id * execute the script only if mana is activated * optimize imbricate foreach loop * add a max number of simultaneous issues * remove unused option
Created attachment 130741 [details] [review] Bug 18618: Add a better support for different id types Add a better support for ISMN, ISRN, ISRC and UPC Add normalization function for some of these ids. When sending pairs to mana and when getting suggestions these ids are now taken into account correctly. Documents with multiple identifier were causing various trouble. It was needed to add functions to normalize all ids This patch add the possibility to: - create pairs with documents with multiple ids (all the possible pairs will be created) - get suggestion for a document with multiple ids - display correctly the document when the suggested document has multiple ids
Created attachment 130742 [details] [review] Bug 18618: Add MARC21 support This functionnality is now working for both MARC21 and UNIMARC.
Created attachment 130743 [details] [review] Bug 18618 : refacto to allow unit test and add unit test Re-write pairs extraction script to use testable function Re-write getSuggestion script to use testable function Add unit test for normalizaation functions Add unit test for pairs extractions function Add unit test for getSuggestion function Fix many small problems found while writing unit test
Created attachment 130744 [details] [review] Bug 18618: Update database create the atomicupdate file
Created attachment 132690 [details] [review] Bug 18618: Mana readings pairs - There are 2 differents scripts: misc/cronjobs/mana_send_pairs.pl -> please use it if Mana is installed misc/cronjobs/mana_send_pairs_without_mana.pl -> designed to work if mana isn't enabled - Both of scripts anonymize the datas and send them to mana. Bug 18618: Mana - Add reading suggestion There is now a module to get suggestions from Mana. - It is activated by default on Mana. - When you browse on a notice, there is now a tab called "Suggestions" - On click on the tab, it will check in the database if it has suggestions. - Else, it will ask to Mana for suggestions and will store in the database. - The suggestions are stored in the NEW table reading_pairs - Implement new dbix objets: Reading_suggestion and Reading_suggestions Bug 18618: Add a pagination for suggestions *** Suggestions are now sent from Mana as follows - Koha requests the first 20 suggestions - It processes them and estimates how much it will need to get all suggestions - It requests Mana the number of suggestions needed - It repeats the 3 first steps until (1) Mana doesn't have suggestions anymore (2) It got 10 suggestions to display *** Koha interface is a bit more verbose while searching (just a bit). Bug 18618: Correct Plack issues Split one command line into two in order to be accepted by Plack Bug 18618: Rebase modification Remove mana_send_pairs_without_mana.pl This script was here for testing purpose and is not usefull anymore Fix name issues (some function have been renamed) Fix a compilation issues (redeclared variable) Remove an unused variable Remove an extra % in an sql request Fix display problem when getting info from mana When getting info from mana the biblios array contain blessed references but the template don't deal with that properly. It work fine whe koha is getting info from it's interne database because biblios array contain unblessed references. Bug 18618 : Many small refacto and improvement * change duration before refresh and improve the use of $temp variable * improve SQL request for extracting pairs fix condition create pair with old_issues use placeholders * improve SQL request for getting suggestion add the possibility to look in metadata for the id * execute the script only if mana is activated * optimize imbricate foreach loop * add a max number of simultaneous issues * remove unused option Bug 18618: Add a better support for different id types Add a better support for ISMN, ISRN, ISRC and UPC Add normalization function for some of these ids. When sending pairs to mana and when getting suggestions these ids are now taken into account correctly. Documents with multiple identifier were causing various trouble. It was needed to add functions to normalize all ids This patch add the possibility to: - create pairs with documents with multiple ids (all the possible pairs will be created) - get suggestion for a document with multiple ids - display correctly the document when the suggested document has multiple ids Bug 18618: Add MARC21 support This functionnality is now working for both MARC21 and UNIMARC. Bug 18618 : refacto to allow unit test and add unit test Re-write pairs extraction script to use testable function Re-write getSuggestion script to use testable function Add unit test for normalizaation functions Add unit test for pairs extractions function Add unit test for getSuggestion function Fix many small problems found while writing unit test
Created attachment 132691 [details] [review] Bug 18618: Update database create the atomicupdate file
After dicsussion at the hackfest BibLibre: * Squash + clear (perltidy) patches and changes * Test the Koha patches (ie. fix it as it do not work) DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Field 'biblionumber1' doesn't have a default value at /kohadevbox/koha/Koha/Object.pm line 170 * Make sure the PR in up-to-date on gitlab * Write release notes + doc + test plan (provide title to get relevant suggestions) * exportemail + securitytoken => Why? * Route shouldn't be ".json" Joubu: * Improve DB structure (biblionumberX) * Remove/Adjust Koha::Utils methods + Koha::Biblio->normalize_isbn, etc. * Remove useless tests * Test new Koha code with prod/old server code (to make sure we won't introduce regression when upgrading the server)
Any update on this one?