Bug 11926

Summary: HEA - Report usage statistics to Koha community
Product: Koha Reporter: claire.hernandez <claire.hernandez>
Component: Websites, Mailing Lists, etcAssignee: Jonathan Druart <jonathan.druart+koha>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: new feature    
Priority: P5 - low CC: claire.hernandez, fridolin.somers, indradg, jonathan.druart+koha, julian.maurice, katrin.fischer, magnus, mtj, tomascohen
Version: master   
Hardware: All   
OS: All   
URL: http://hea.koha-community.org
Change sponsored?: --- Patch complexity: Medium patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on:    
Bug Blocks: 13149, 13585, 13966, 14608, 14707, 17802, 17803, 18066    
Attachments: work in progress
Package updates: * POD doc * hardcoding hea.koha-community.org as the URL where the stats must be sent * fixing some typos * adding all sysprefs we want to upload
Bug 11926: URI-encode data sent to statistics server
Bug 11926: Add a cronjob
Bug 11926: Add new sysprefs and fix encoding issues
Bug 11926: Add the items table
Bug 11926: Render community koha statistic usages
Bug 11926: Package updates
Bug 11926: URI-encode data sent to statistics server
Bug 11926: Add a cronjob
Bug 11926: Add new sysprefs and fix encoding issues
Bug 11926: Add the items table
Bug 11926: DB changes
Bug 11926: Render community koha statistic usages
Bug 11926: Package updates
Bug 11926: URI-encode data sent to statistics server
Bug 11926: Add a cronjob
Bug 11926: Add new sysprefs and fix encoding issues
Bug 11926: Add the items table
Bug 11926: DB changes
Bug 11926: Render community koha statistic usages
Bug 11926: Package updates
Bug 11926: URI-encode data sent to statistics server
Bug 11926: Add a cronjob
Bug 11926: Add new sysprefs and fix encoding issues
Bug 11926: Add the items table
Bug 11926: DB changes
Bug 11926: Follow-up - remove SearchEngine pref / fix POD
Bug 11926: Follow-up - Fix new cronjob's file permissions
[PASSED QA] Bug 11926: Render community koha statistic usages
[PASSED QA] Bug 11926: Package updates
[PASSED QA] Bug 11926: URI-encode data sent to statistics server
[PASSED QA] Bug 11926: Add a cronjob
[PASSED QA] Bug 11926: Add new sysprefs and fix encoding issues
[PASSED QA] Bug 11926: Add the items table
[PASSED QA] Bug 11926: DB changes
Bug 11926: Follow-up - remove SearchEngine pref / fix POD
Bug 11926: Follow-up - Fix new cronjob's file permissions
Bug 11926 [Follow-up] HEA - Render community koha statistic usages

Description claire.hernandez@biblibre.com 2014-03-12 09:32:04 UTC
http://wiki.koha-community.org/wiki/KohaUsageStat_RFC
Comment 1 claire.hernandez@biblibre.com 2014-03-12 16:40:07 UTC Comment hidden (obsolete)
Comment 2 Magnus Enger 2014-03-12 22:18:51 UTC
*** Bug 6293 has been marked as a duplicate of this bug. ***
Comment 3 Paul Poulain 2014-03-14 11:40:18 UTC Comment hidden (obsolete)
Comment 4 Julian Maurice 2014-03-14 15:05:11 UTC Comment hidden (obsolete)
Comment 5 Jonathan Druart 2014-09-04 13:09:34 UTC Comment hidden (obsolete)
Comment 6 Jonathan Druart 2014-09-04 13:09:39 UTC Comment hidden (obsolete)
Comment 7 Jonathan Druart 2014-09-04 14:18:59 UTC Comment hidden (obsolete)
Comment 8 Jonathan Druart 2014-10-06 21:44:26 UTC Comment hidden (obsolete)
Comment 9 Jonathan Druart 2014-10-06 21:44:34 UTC Comment hidden (obsolete)
Comment 10 Jonathan Druart 2014-10-06 21:44:39 UTC Comment hidden (obsolete)
Comment 11 Jonathan Druart 2014-10-06 21:44:45 UTC Comment hidden (obsolete)
Comment 12 Jonathan Druart 2014-10-06 21:44:51 UTC Comment hidden (obsolete)
Comment 13 Jonathan Druart 2014-10-06 21:44:56 UTC Comment hidden (obsolete)
Comment 14 Jonathan Druart 2014-10-06 21:45:01 UTC Comment hidden (obsolete)
Comment 15 Jonathan Druart 2014-10-06 21:48:00 UTC
I added the changes to updatedb.pl and syspref.sql into a specific patch to make easier the installation.
Comment 16 Jonathan Druart 2014-10-11 18:56:20 UTC
This cans be signed off.
Comment 17 Chris Cormack 2014-10-11 19:39:47 UTC Comment hidden (obsolete)
Comment 18 Chris Cormack 2014-10-11 19:40:04 UTC Comment hidden (obsolete)
Comment 19 Chris Cormack 2014-10-11 19:40:20 UTC Comment hidden (obsolete)
Comment 20 Chris Cormack 2014-10-11 19:40:40 UTC Comment hidden (obsolete)
Comment 21 Chris Cormack 2014-10-11 19:40:56 UTC Comment hidden (obsolete)
Comment 22 Chris Cormack 2014-10-11 19:41:11 UTC Comment hidden (obsolete)
Comment 23 Chris Cormack 2014-10-11 19:41:32 UTC Comment hidden (obsolete)
Comment 24 Fridolin Somers 2014-10-13 08:32:02 UTC
The patch needs something :
 Bug 11926: Add a cronjob
It adds the cronjob script with permissions 644 instead of 755
Comment 25 Katrin Fischer 2014-10-16 21:54:01 UTC
Some ideas for improvements:
- Country is a free text field, so it's not really clear what to enter there. Should we maybe point to a list of codes to use (ISO?) so that we get cleaner data?
- Add information about data privacy on the HEA website. Are logs (IP addresses) regularly deleted? How long is data kept? Which data is stored exactly, etc.
- Add a bit more documentation - are LibraryName and LibraryURL mandatory? If not, I think it would be nice to mark them as optional in the pref description.
- Mention the cronjob in the pref descriptions?

I did a few follow-ups - addressing Fridolin's last comment and some other things found by the QA scrpt.
Comment 26 Katrin Fischer 2014-10-16 21:54:44 UTC Comment hidden (obsolete)
Comment 27 Katrin Fischer 2014-10-16 21:54:50 UTC Comment hidden (obsolete)
Comment 28 Katrin Fischer 2014-10-16 21:54:58 UTC Comment hidden (obsolete)
Comment 29 Katrin Fischer 2014-10-16 21:55:03 UTC Comment hidden (obsolete)
Comment 30 Katrin Fischer 2014-10-16 21:55:08 UTC Comment hidden (obsolete)
Comment 31 Katrin Fischer 2014-10-16 21:55:12 UTC Comment hidden (obsolete)
Comment 32 Katrin Fischer 2014-10-16 21:55:16 UTC Comment hidden (obsolete)
Comment 33 Katrin Fischer 2014-10-16 21:55:21 UTC Comment hidden (obsolete)
Comment 34 Katrin Fischer 2014-10-16 21:55:24 UTC Comment hidden (obsolete)
Comment 35 Katrin Fischer 2014-10-16 21:59:59 UTC
Created attachment 32464 [details] [review]
[PASSED QA] Bug 11926: Render community koha statistic usages

Goals:
- Collecting Koha usage statistics
- Rendering stats on a community website
- Having a big bicture of how koha is used

3 parts in the project:
- this patch in koha
- hea-ws which collects data
- hea-app which renders data

Installation:
1/ Fill systempreferences:
UsageStatsLastUpdateTime UsageStatsID UsageStatsShare UsageStatsLibraryName
2/ Setup a cron in your crontab (ex: at 3:00 every first of the month):
0 3 1 * * export KOHA_CONF=/home/koha/etc/koha-conf.xml; export PERL5LIB=/home/koha/src; perl /home/koha/src/C4/UsageStats.pm

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 36 Katrin Fischer 2014-10-16 22:00:04 UTC
Created attachment 32465 [details] [review]
[PASSED QA] Bug 11926: Package updates

* POD doc
* hardcoding hea.koha-community.org as the URL where the stats must
be sent
* fixing some typos
* adding all sysprefs we want to upload.

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 37 Katrin Fischer 2014-10-16 22:00:09 UTC
Created attachment 32466 [details] [review]
[PASSED QA] Bug 11926: URI-encode data sent to statistics server

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 38 Katrin Fischer 2014-10-16 22:00:13 UTC
Created attachment 32467 [details] [review]
[PASSED QA] Bug 11926: Add a cronjob

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 39 Katrin Fischer 2014-10-16 22:00:17 UTC
Created attachment 32468 [details] [review]
[PASSED QA] Bug 11926: Add new sysprefs and fix encoding issues

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 40 Katrin Fischer 2014-10-16 22:00:22 UTC
Created attachment 32469 [details] [review]
[PASSED QA] Bug 11926: Add the items table

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 41 Katrin Fischer 2014-10-16 22:00:26 UTC
Created attachment 32470 [details] [review]
[PASSED QA] Bug 11926: DB changes

This patch adds the syspref in the database.

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 42 Katrin Fischer 2014-10-16 22:00:30 UTC
Created attachment 32471 [details] [review]
Bug 11926: Follow-up - remove SearchEngine pref / fix POD

Patch set reintroduced the formerly removed system preference,
just removing it again.

Also fixes complaints of the QA script about POD in the new module.
Comment 43 Katrin Fischer 2014-10-16 22:00:36 UTC
Created attachment 32472 [details] [review]
Bug 11926: Follow-up - Fix new cronjob's file permissions
Comment 44 Owen Leonard 2014-10-22 15:34:39 UTC
Created attachment 32582 [details] [review]
Bug 11926 [Follow-up] HEA - Render community koha statistic usages

This follow-up makes some language and spelling corrections to the
system preference descriptions. I have updated the preferences heading
to read "Share anonymous usage statistics" in order to emphasize the
anonymous aspect.

I have also updated the main preference name from "UsageStatsShare" to
"UsageStats" so that it is alphabetized first in the list of
"UsageStats*" preferences. I think this will make it clearer to the Koha
administrator what feature they are configuring.
Comment 45 Tomás Cohen Arazi 2014-10-22 18:26:04 UTC
Patches pushed to master.

Nice feature!
Thanks everyone
Comment 46 Jonathan Druart 2015-01-14 12:10:20 UTC
Hi Mason,
The patch set applies cleanly on 3.16.x (only 2 conflicts on installer/data/mysql/updatedatabase.pl and koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref, very easy to fix).
You can backport it on 3.16.x if you like.