Bug 11926 - HEA - Report usage statistics to Koha community
Summary: HEA - Report usage statistics to Koha community
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Websites, Mailing Lists, etc (show other bugs)
Version: master
Hardware: All All
: P5 - low new feature (vote)
Assignee: Jonathan Druart
QA Contact: Testopia
URL: http://hea.koha-community.org
Keywords:
: 6293 (view as bug list)
Depends on:
Blocks: 13149 13585 13966 14608 14707 17802 17803 18066
  Show dependency treegraph
 
Reported: 2014-03-12 09:32 UTC by claire.hernandez@biblibre.com
Modified: 2017-02-06 16:58 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
work in progress (7.46 KB, patch)
2014-03-12 16:40 UTC, claire.hernandez@biblibre.com
Details | Diff | Splinter Review
Package updates: * POD doc * hardcoding hea.koha-community.org as the URL where the stats must be sent * fixing some typos * adding all sysprefs we want to upload (8.79 KB, patch)
2014-03-14 11:40 UTC, Paul Poulain
Details | Diff | Splinter Review
Bug 11926: URI-encode data sent to statistics server (932 bytes, patch)
2014-03-14 15:05 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 11926: Add a cronjob (25.96 KB, patch)
2014-09-04 13:09 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: Add new sysprefs and fix encoding issues (11.65 KB, patch)
2014-09-04 13:09 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: Add the items table (751 bytes, patch)
2014-09-04 14:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: Render community koha statistic usages (4.79 KB, patch)
2014-10-06 21:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: Package updates (8.75 KB, patch)
2014-10-06 21:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: URI-encode data sent to statistics server (926 bytes, patch)
2014-10-06 21:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: Add a cronjob (22.31 KB, patch)
2014-10-06 21:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: Add new sysprefs and fix encoding issues (7.65 KB, patch)
2014-10-06 21:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: Add the items table (751 bytes, patch)
2014-10-06 21:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: DB changes (3.69 KB, patch)
2014-10-06 21:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11926: Render community koha statistic usages (4.85 KB, patch)
2014-10-11 19:39 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11926: Package updates (8.80 KB, patch)
2014-10-11 19:40 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11926: URI-encode data sent to statistics server (982 bytes, patch)
2014-10-11 19:40 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11926: Add a cronjob (22.36 KB, patch)
2014-10-11 19:40 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11926: Add new sysprefs and fix encoding issues (7.71 KB, patch)
2014-10-11 19:40 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11926: Add the items table (807 bytes, patch)
2014-10-11 19:41 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11926: DB changes (3.71 KB, patch)
2014-10-11 19:41 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11926: Render community koha statistic usages (5.07 KB, patch)
2014-10-16 21:54 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: Package updates (8.80 KB, patch)
2014-10-16 21:54 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: URI-encode data sent to statistics server (982 bytes, patch)
2014-10-16 21:54 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: Add a cronjob (22.35 KB, patch)
2014-10-16 21:55 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: Add new sysprefs and fix encoding issues (7.71 KB, patch)
2014-10-16 21:55 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: Add the items table (807 bytes, patch)
2014-10-16 21:55 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: DB changes (3.71 KB, patch)
2014-10-16 21:55 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: Follow-up - remove SearchEngine pref / fix POD (1.62 KB, patch)
2014-10-16 21:55 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: Follow-up - Fix new cronjob's file permissions (574 bytes, patch)
2014-10-16 21:55 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 11926: Render community koha statistic usages (5.13 KB, patch)
2014-10-16 21:59 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 11926: Package updates (8.87 KB, patch)
2014-10-16 22:00 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 11926: URI-encode data sent to statistics server (1.03 KB, patch)
2014-10-16 22:00 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 11926: Add a cronjob (22.42 KB, patch)
2014-10-16 22:00 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 11926: Add new sysprefs and fix encoding issues (7.78 KB, patch)
2014-10-16 22:00 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 11926: Add the items table (877 bytes, patch)
2014-10-16 22:00 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 11926: DB changes (3.78 KB, patch)
2014-10-16 22:00 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: Follow-up - remove SearchEngine pref / fix POD (1.62 KB, patch)
2014-10-16 22:00 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926: Follow-up - Fix new cronjob's file permissions (574 bytes, patch)
2014-10-16 22:00 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 11926 [Follow-up] HEA - Render community koha statistic usages (11.88 KB, patch)
2014-10-22 15:34 UTC, Owen Leonard
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description claire.hernandez@biblibre.com 2014-03-12 09:32:04 UTC
http://wiki.koha-community.org/wiki/KohaUsageStat_RFC
Comment 1 claire.hernandez@biblibre.com 2014-03-12 16:40:07 UTC Comment hidden (obsolete)
Comment 2 Magnus Enger 2014-03-12 22:18:51 UTC
*** Bug 6293 has been marked as a duplicate of this bug. ***
Comment 3 Paul Poulain 2014-03-14 11:40:18 UTC Comment hidden (obsolete)
Comment 4 Julian Maurice 2014-03-14 15:05:11 UTC Comment hidden (obsolete)
Comment 5 Jonathan Druart 2014-09-04 13:09:34 UTC Comment hidden (obsolete)
Comment 6 Jonathan Druart 2014-09-04 13:09:39 UTC Comment hidden (obsolete)
Comment 7 Jonathan Druart 2014-09-04 14:18:59 UTC Comment hidden (obsolete)
Comment 8 Jonathan Druart 2014-10-06 21:44:26 UTC Comment hidden (obsolete)
Comment 9 Jonathan Druart 2014-10-06 21:44:34 UTC Comment hidden (obsolete)
Comment 10 Jonathan Druart 2014-10-06 21:44:39 UTC Comment hidden (obsolete)
Comment 11 Jonathan Druart 2014-10-06 21:44:45 UTC Comment hidden (obsolete)
Comment 12 Jonathan Druart 2014-10-06 21:44:51 UTC Comment hidden (obsolete)
Comment 13 Jonathan Druart 2014-10-06 21:44:56 UTC Comment hidden (obsolete)
Comment 14 Jonathan Druart 2014-10-06 21:45:01 UTC Comment hidden (obsolete)
Comment 15 Jonathan Druart 2014-10-06 21:48:00 UTC
I added the changes to updatedb.pl and syspref.sql into a specific patch to make easier the installation.
Comment 16 Jonathan Druart 2014-10-11 18:56:20 UTC
This cans be signed off.
Comment 17 Chris Cormack 2014-10-11 19:39:47 UTC Comment hidden (obsolete)
Comment 18 Chris Cormack 2014-10-11 19:40:04 UTC Comment hidden (obsolete)
Comment 19 Chris Cormack 2014-10-11 19:40:20 UTC Comment hidden (obsolete)
Comment 20 Chris Cormack 2014-10-11 19:40:40 UTC Comment hidden (obsolete)
Comment 21 Chris Cormack 2014-10-11 19:40:56 UTC Comment hidden (obsolete)
Comment 22 Chris Cormack 2014-10-11 19:41:11 UTC Comment hidden (obsolete)
Comment 23 Chris Cormack 2014-10-11 19:41:32 UTC Comment hidden (obsolete)
Comment 24 Fridolin Somers 2014-10-13 08:32:02 UTC
The patch needs something :
 Bug 11926: Add a cronjob
It adds the cronjob script with permissions 644 instead of 755
Comment 25 Katrin Fischer 2014-10-16 21:54:01 UTC
Some ideas for improvements:
- Country is a free text field, so it's not really clear what to enter there. Should we maybe point to a list of codes to use (ISO?) so that we get cleaner data?
- Add information about data privacy on the HEA website. Are logs (IP addresses) regularly deleted? How long is data kept? Which data is stored exactly, etc.
- Add a bit more documentation - are LibraryName and LibraryURL mandatory? If not, I think it would be nice to mark them as optional in the pref description.
- Mention the cronjob in the pref descriptions?

I did a few follow-ups - addressing Fridolin's last comment and some other things found by the QA scrpt.
Comment 26 Katrin Fischer 2014-10-16 21:54:44 UTC Comment hidden (obsolete)
Comment 27 Katrin Fischer 2014-10-16 21:54:50 UTC Comment hidden (obsolete)
Comment 28 Katrin Fischer 2014-10-16 21:54:58 UTC Comment hidden (obsolete)
Comment 29 Katrin Fischer 2014-10-16 21:55:03 UTC Comment hidden (obsolete)
Comment 30 Katrin Fischer 2014-10-16 21:55:08 UTC Comment hidden (obsolete)
Comment 31 Katrin Fischer 2014-10-16 21:55:12 UTC Comment hidden (obsolete)
Comment 32 Katrin Fischer 2014-10-16 21:55:16 UTC Comment hidden (obsolete)
Comment 33 Katrin Fischer 2014-10-16 21:55:21 UTC Comment hidden (obsolete)
Comment 34 Katrin Fischer 2014-10-16 21:55:24 UTC Comment hidden (obsolete)
Comment 35 Katrin Fischer 2014-10-16 21:59:59 UTC
Created attachment 32464 [details] [review]
[PASSED QA] Bug 11926: Render community koha statistic usages

Goals:
- Collecting Koha usage statistics
- Rendering stats on a community website
- Having a big bicture of how koha is used

3 parts in the project:
- this patch in koha
- hea-ws which collects data
- hea-app which renders data

Installation:
1/ Fill systempreferences:
UsageStatsLastUpdateTime UsageStatsID UsageStatsShare UsageStatsLibraryName
2/ Setup a cron in your crontab (ex: at 3:00 every first of the month):
0 3 1 * * export KOHA_CONF=/home/koha/etc/koha-conf.xml; export PERL5LIB=/home/koha/src; perl /home/koha/src/C4/UsageStats.pm

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 36 Katrin Fischer 2014-10-16 22:00:04 UTC
Created attachment 32465 [details] [review]
[PASSED QA] Bug 11926: Package updates

* POD doc
* hardcoding hea.koha-community.org as the URL where the stats must
be sent
* fixing some typos
* adding all sysprefs we want to upload.

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 37 Katrin Fischer 2014-10-16 22:00:09 UTC
Created attachment 32466 [details] [review]
[PASSED QA] Bug 11926: URI-encode data sent to statistics server

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 38 Katrin Fischer 2014-10-16 22:00:13 UTC
Created attachment 32467 [details] [review]
[PASSED QA] Bug 11926: Add a cronjob

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 39 Katrin Fischer 2014-10-16 22:00:17 UTC
Created attachment 32468 [details] [review]
[PASSED QA] Bug 11926: Add new sysprefs and fix encoding issues

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 40 Katrin Fischer 2014-10-16 22:00:22 UTC
Created attachment 32469 [details] [review]
[PASSED QA] Bug 11926: Add the items table

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 41 Katrin Fischer 2014-10-16 22:00:26 UTC
Created attachment 32470 [details] [review]
[PASSED QA] Bug 11926: DB changes

This patch adds the syspref in the database.

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 42 Katrin Fischer 2014-10-16 22:00:30 UTC
Created attachment 32471 [details] [review]
Bug 11926: Follow-up - remove SearchEngine pref / fix POD

Patch set reintroduced the formerly removed system preference,
just removing it again.

Also fixes complaints of the QA script about POD in the new module.
Comment 43 Katrin Fischer 2014-10-16 22:00:36 UTC
Created attachment 32472 [details] [review]
Bug 11926: Follow-up - Fix new cronjob's file permissions
Comment 44 Owen Leonard 2014-10-22 15:34:39 UTC
Created attachment 32582 [details] [review]
Bug 11926 [Follow-up] HEA - Render community koha statistic usages

This follow-up makes some language and spelling corrections to the
system preference descriptions. I have updated the preferences heading
to read "Share anonymous usage statistics" in order to emphasize the
anonymous aspect.

I have also updated the main preference name from "UsageStatsShare" to
"UsageStats" so that it is alphabetized first in the list of
"UsageStats*" preferences. I think this will make it clearer to the Koha
administrator what feature they are configuring.
Comment 45 Tomás Cohen Arazi 2014-10-22 18:26:04 UTC
Patches pushed to master.

Nice feature!
Thanks everyone
Comment 46 Jonathan Druart 2015-01-14 12:10:20 UTC
Hi Mason,
The patch set applies cleanly on 3.16.x (only 2 conflicts on installer/data/mysql/updatedatabase.pl and koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref, very easy to fix).
You can backport it on 3.16.x if you like.