Bug 6772 - Implementation of a recommendations engine
Summary: Implementation of a recommendations engine
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: OPAC (show other bugs)
Version: master
Hardware: All All
: P3 enhancement (vote)
Assignee: Bugs List
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-08-25 01:51 UTC by Robin Sheat
Modified: 2019-05-27 12:45 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 6772 - recommendation system (24.43 KB, patch)
2011-08-25 06:38 UTC, Robin Sheat
Details | Diff | Splinter Review
Signed-off patch (24.57 KB, patch)
2011-09-01 12:02 UTC, Frédéric Demians
Details | Diff | Splinter Review
Fixing array reference bug in recommendations.pm (796 bytes, patch)
2013-04-16 07:03 UTC, Chris Hall
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Robin Sheat 2011-08-25 01:51:55 UTC
We'd like the ability to have the OPAC say "other people who read this, also read that."

We have some very old code that does that, so I'm going to make it work with 3.4 and generally tidy it up.
Comment 1 Robin Sheat 2011-08-25 06:38:10 UTC Comment hidden (obsolete)
Comment 2 Robin Sheat 2011-08-25 06:48:39 UTC
Testing notes: you'll need a database with entries in old_issues, turn on the syspref (in Opac->Features->ShowRecommendations) and then run the cron job (misc/cronjobs/recommendations.pl.) This will take a fair while to run, and when it's done, viewing a record in the OPAC that's ever been borrowed will give a list of recommendations.

Some points:
* This will conflict with anonymising history.
* It's slow to build the index.
* I'd like to have it update the database as items get returned (perhaps as a queue) to allow it to not have to spend ages running a terribly long cron job. But that'll require some thought on how to approach it.
Comment 3 Frédéric Demians 2011-09-01 12:02:38 UTC
Created attachment 5250 [details] [review]
Signed-off patch
Comment 4 Frédéric Demians 2011-09-01 12:03:25 UTC
It works as announced. 

I confirm 'recommendations' table generation is insanly slow. The result
is a huge table. For a smallish 25,000 items DB, I get a recommendation
table with 3,728,251 records! If 'recommendations' has to be truncated
and re-created frequently, coupled with MySQL issue not releasing innodb
freespace, that could conduct to disk out-of-space.

Another issue is on the OPAC Recommendations tab. It shows the 100 first
related books. It would be nice to have a syspref to reduce the list.

I sign-off this patch, since it's disabled by default.
Comment 5 Robin Sheat 2011-09-01 23:36:15 UTC
Yep, I agree with all your points :) I'm thinking that this is a suitable starting point, there are a lot of improvements that can be made. 

In particular, I'd like to build a queue of returns that have occurred, and at the end of the day just merge that with the existing index. This would make it terribly faster.

Also, I'd somehow like to reduce the number of rows it puts into the database (I already halved it from the original implementation :) but I'm not quite sure how.

Another thing that would be fun to do is allow it to integrate with star ratings, so 1 star counts as a negative weight, and 5 as a really positive one.
Comment 6 Paul Poulain 2011-10-06 14:44:48 UTC
This is not as assistant QA manager that I write this. I've been thinking for long to this feature.
With anonymization that is mandatory after 3 months in France, such a feature, that rebuild from scratch all associations, will mostly be useless.
The only solution I see would be to have the recommendations table being filled incrementally : we run it every week, and deal only with issues of the week (linked to all readings still in old_issues/issues table)
That would not be perfect (you won't connect a book issued today to a book issued 6 months ago, as it has been anonymized), but that would be much better than the actual behaviour.

That would also solve the speed problem noticed by you & Frédéric

Note : to avoid the DB space, the solution is to create the table with engine=myISAM
As there are no constraints, that won't do any difference

Marking "failed QA" as the feature as it is today is unusable. Too bad, it's something i'm dreaming of !
Comment 7 Robin Sheat 2011-10-06 22:53:29 UTC
I'm fine with that, I'll just have to find the time to make it work incrementally.
Comment 8 Chris Hall 2013-04-16 07:03:10 UTC
Created attachment 17483 [details] [review]
Fixing array reference bug in recommendations.pm

Attached patch fixes a small reference bug in Recommendations.pm (included in the previous patch) that only occurs when we have more recommendations than the limit (default value of 100)

$res = \@{ @{$res}[0..$limit] } if (@$res > $limit);

will cause an error as it tries to interpret the items within $res as arrayrefs when they are hashrefs. What we really want to do is shorten our results down to never be longer than limit:

$res = [ @{$res}[0..$limit] ] if (@$res > $limit);
Comment 9 Katrin Fischer 2013-04-16 07:59:43 UTC
Hi Robin and Chris,
as this adds a new module I think we will need unit tests for it. Maybe also worth moving it to the Koha namespace?
Comment 10 Robin Sheat 2013-04-16 22:24:02 UTC
(In reply to comment #9)
> Hi Robin and Chris,
> as this adds a new module I think we will need unit tests for it. Maybe also
> worth moving it to the Koha namespace?

It would need a lot more really. We're mostly leaving it here because we use it, and other people might want it too. For eligibility to Koha, it'd need a lot of work to address Frédéric and Paul's points as well. But yes, new namespace and unit tests would also be good things.
Comment 11 Jonathan Druart 2015-02-20 11:59:49 UTC
(In reply to Robin Sheat from comment #10)
> It would need a lot more really. We're mostly leaving it here because we use
> it, and other people might want it too. For eligibility to Koha, it'd need a
> lot of work to address Frédéric and Paul's points as well. But yes, new
> namespace and unit tests would also be good things.

Does it mean you don't plan to work on it anymore?
If so, then could you remove your name from the assigned to field?
Comment 12 Arthur Suzuki 2016-09-19 13:48:48 UTC
Hello everyone,
I'll put my things here since this bug seems to be about what I want to implement.

I wanted to write a plugin to demonstrate the feature and bring some modularity to Koha.

The code is available for review here :
https://github.com/Liliputech/KohaReadSuggestion

Some improvement since I posted this on the koha-devel list, it is now compatible with both MARC21 and UNIMARC and also support CSV output.
Best regards,
Arthur Suzuki
Comment 13 Arthur Suzuki 2019-05-27 12:44:47 UTC
Just checking out this bug again...
What shall we do about it?
My plugin actually works the same way and it doesn't need rebase now.
It also doesn't need the creation of an index.

There is also this Mana patch (just added in the "See also").
Is it still worth it to keep that one opened? Is any one willing to take that one?
Arthur