Bug 18066 - Hea - Version 2
Summary: Hea - Version 2
Status: Pushed to Master
Alias: None
Product: Koha
Classification: Unclassified
Component: System Administration (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low new feature (vote)
Assignee: Jonathan Druart
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
: 17802 17803 (view as bug list)
Depends on: 11926 14608
Blocks: 16497
  Show dependency treegraph
 
Reported: 2017-02-06 16:58 UTC by Jonathan Druart
Modified: 2017-05-24 02:03 UTC (History)
11 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:
Hea is a service to collect usage data from libraries using Koha. With this development Hea can collect the geolocations of the libraries in your installation and create a map. A new configuration page allows to configure easily what information is shared with the Koha community. Hea statistics can been seen on https://hea.koha-community.org/


Attachments
Bug 18066: Hea V2 (30.09 KB, patch)
2017-02-07 09:56 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 18066: Add the leaflet library (1.01 MB, patch)
2017-02-07 09:57 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 18066: Do not use token for OSM tiles (1.55 KB, patch)
2017-02-28 11:17 UTC, Jonathan Druart
Details | Diff | Splinter Review
[DO NOT PUSH] Use hea test server instead of prod (1.83 KB, patch)
2017-02-28 11:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 18066: Use https for ws (750 bytes, patch)
2017-02-28 17:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 18066: Hea V2 (30.14 KB, patch)
2017-02-28 21:06 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 18066: Add the leaflet library (1021.94 KB, patch)
2017-02-28 21:06 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 18066: Do not use token for OSM tiles (1.60 KB, patch)
2017-02-28 21:07 UTC, Chris Cormack
Details | Diff | Splinter Review
Use hea test server instead of prod (1.82 KB, patch)
2017-02-28 21:07 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 18066: Use https for ws (804 bytes, patch)
2017-02-28 21:07 UTC, Chris Cormack
Details | Diff | Splinter Review
Bad position of the map when you reload the page when not on top of the page (84.57 KB, image/png)
2017-03-21 10:05 UTC, Julian Maurice
Details
Bad position of the map when you scroll down (398.51 KB, image/png)
2017-03-21 10:05 UTC, Julian Maurice
Details
Bug 18066: Fix tests (4.56 KB, patch)
2017-03-21 10:22 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 18066: Fix map positioning in admin/usage_statistics.pl (2.38 KB, patch)
2017-03-21 11:02 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Fix map positioning in admin/usage_statistics.pl (2.41 KB, patch)
2017-03-21 11:11 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Hea V2 (30.20 KB, patch)
2017-03-21 14:48 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Add the leaflet library (1021.96 KB, patch)
2017-03-21 14:48 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Do not use token for OSM tiles (1.61 KB, patch)
2017-03-21 14:49 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Use https for ws (1.80 KB, patch)
2017-03-21 14:49 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Fix tests (4.57 KB, patch)
2017-03-21 14:49 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Fix map positioning in admin/usage_statistics.pl (2.41 KB, patch)
2017-03-21 14:49 UTC, Julian Maurice
Details | Diff | Splinter Review
[DO NOT PUSH] Bug 18066: Use hea test server instead of prod (1.81 KB, patch)
2017-03-21 14:49 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Hea V2 (30.26 KB, patch)
2017-03-22 08:27 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Add the leaflet library (1022.02 KB, patch)
2017-03-22 08:28 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Do not use token for OSM tiles (1.67 KB, patch)
2017-03-22 08:28 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Use https for ws (1.86 KB, patch)
2017-03-22 08:28 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Fix tests (4.62 KB, patch)
2017-03-22 08:28 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Fix map positioning in admin/usage_statistics.pl (2.62 KB, patch)
2017-03-22 08:28 UTC, Julian Maurice
Details | Diff | Splinter Review
[DO NOT PUSH] Bug 18066: Use hea test server instead of prod (1.81 KB, patch)
2017-03-22 08:28 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 18066: Add the new geolocation column to kohastructure.sql (1016 bytes, patch)
2017-03-23 16:31 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 18066: Remove warnings from translate script (1.30 KB, patch)
2017-04-24 19:30 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2017-02-06 16:58:01 UTC
I plan to submit a big improvement for Hea (https://wiki.koha-community.org/wiki/KohaUsageStat_RFC).

The different goals are:
1/ Add more information on how Hea is collecting the data and what we do with them
2/ Collect geolocation to display the Koha installations on a map
3/ Collect data per library instead of per Koha installation
Comment 1 Jonathan Druart 2017-02-06 18:05:37 UTC
Code is available at:

Koha    - https://github.com/joubu/Koha/commits/hea_v2
hea-ws  - https://github.com/joubu/hea-ws/commits/v2
hea-app - https://github.com/joubu/hea-app/commits/v2 (contains the different from the last 6 months).
Comment 2 Jonathan Druart 2017-02-06 18:13:10 UTC
Screenshots:

Koha side:
New page to configure the info to share:
  https://pageshot.net/opGCLAZFt0s1HVzp/pro.kohadev.vm (with libraries info)
  https://pageshot.net/SKTFZgr40EKg6fge/pro.kohadev.vm (without libraries info)

Hea side:
New layout for the homepage:
  https://pageshot.net/GGKbBZlZwxpXuCoi/localhost
View "see all libraries" (existing one):
  https://pageshot.net/2XplbZXFFr9rLNJS/localhost
View "see libraries by country":
  https://pageshot.net/G5czRcqh47WFIfRD/localhost
View "see libraries on a map":
  https://pageshot.net/Xa4qW3Vc4MclrEhO/localhost
View "public view for a library" (to show libraries what is public):
  https://pageshot.net/7whCONhkPuJUUxDP/localhost

There is a popup to display the library's name on hover the markers.
Comment 3 Katrin Fischer 2017-02-06 18:46:33 UTC
Pretty! +100
Comment 4 Chris Cormack 2017-02-06 20:59:36 UTC
Jonathan, 

I have pushed a hea repo to git.koha-community.org

You should have push rights to that, if you want to push up the hea specific code to there as well.

I will sign off 14608 so this can go forward, it is looking great
Comment 5 Jonathan Druart 2017-02-07 07:44:41 UTC
*** Bug 17802 has been marked as a duplicate of this bug. ***
Comment 6 Jonathan Druart 2017-02-07 07:45:02 UTC
*** Bug 17803 has been marked as a duplicate of this bug. ***
Comment 7 Jonathan Druart 2017-02-07 09:56:59 UTC
Created attachment 59975 [details] [review]
Bug 18066: Hea V2

This patch is the Koha part of the Hea v2 project.
You can find the (testing) code for the server at
  hea-ws  - https://github.com/joubu/hea-ws/commits/v2
  hea-app - https://github.com/joubu/hea-app/commits/v2
They contain the different pull requests made over the last 6 months.

More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC
The goal of this commit message is to provide an overview of what could
be a new version of Hea.

Prior to these changes, the Hea database was filled with 1 line per Koha
installation. System preferences were filled by the libraries and a
cronjob (share_usage_with_koha_community.pl) collected these values to send
them to a webservice (hea-ws/upload.pl).

With the need to collect more data we would want to collect data at the library
level (branch) and not at the installation level.
For instance the geolocation, the url or the country can be different from one
library to another, even if managed from the same Koha installation.
The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql).

The hidden goal of this patch is to make Hea sexier and explain
better to libraries how it can be useful to share their information
with the Koha community. I guess the main problem is the lack of
communication and explanations about what we are doing we these data.
To fill this gap I'd like to (TODO)
  1. Communicate on the ML about this new version of Hea (once it got
pushed and backported)
  2. Link the Privacy_Policy.md from the Hea interface
  3. Get help from a native English speaker to add
popup/help/info/whatever on "Home › Administration › Usage statistics",
to clearly explain what happens (and what will not happen!) when an option or
another is set.

You can find screenshot of this whole enhancement on bug 18066, comment 2.

What this patch does:
- Create a new branches.geolocation DB field
- Add 3 new sysprefs:
  * UsageStatsGeolocation
  * UsageStatsLibrariesInfo
  * UsageStatsPublicID
- Integrate the Leaflet JS library to get a fancy map to pick
geolocations

How does it works:
On the new administration page where statistics to share are configured,
there are several new things. It is now possible to share information either
per Koha installation or libraries. If UsageStatsLibrariesInfo is set,
the info at library level (url, name, country, geolocation) will be
sent to the Hea webservice. If it is not set, you can decide to fill
UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry,
UsageStatsGeolocation to share these information. Note that even if the
data are retrieved at installation level, it's better to fill the prefs
as well: On the Hea website the different libraries defined for a given
Koha installation could be displayed on the same page.
This page is a public page which will be attributed to every
installation (with the pref UsageStatsPublicID). On this page all the
info available publicly will be displayed.

TODO later:
- Add a button on the administration page to delete the info shared
publicly. It will be easy to show that the info are no longer displayed
on the public page.
- Add an icon per Koha installation to get a better "public page"
- Any suggestions?

Test plan:
We will need to test hea-ws, hea-app and the Koha-side code to test the
whole enhancement.
1/ To start, clone the hea-ws and hea-app project and checkout the
'master' branch (*not* 'v2')
2/ Create the hea database and user
  CREATE DATABASE hea
  CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea';
  GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost';
  FLUSH PRIVILEGES;
3/ Fill the DB with some data
  mysql hea < hea-app/sql/schema.sql
  mysql hea < hea-app/sql/sql/mock-data.sql
4/ Checkout the 'v2' branch for both hea-ws and hea-app
5/ Execute the upgrade DB script
  % cd hea-app
  % perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info
  % perl sql/upgrade.pl
Now the DB is using the v2 structure. That means we have 1 installation
row per library previously defined. 1 library row has also been created.
5/ Configure hea-ws
% echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts
<VirtualHost *:80>
  DocumentRoot "/path/to/hea-ws"
  ServerName "hea.koha-community.org"
  <Directory "/">
    Options +ExecCGI
    Require all granted
    AddHandler cgi-script .pl
  </Directory>
</VirtualHost>

And enable it with a2ensite, then restart apache.
The copy the database.yml.sample to database.yml and edit it to fill the
DB info.

6/ Launch the hea-app
  % cd hea-app
  % edit README.md # to install the missing modules
  % cp environments/config.yml environments/development.yml
  % edit environments/development.yml # to fill the DB info
  % perl bin/app.pl
Then hit localhost:3000
You should see a local version of Hea with sample data

7/ Back to Koha side
A. We will test that the webservice still works with previous version of Koha (without v2)
a. Do not configure Hea
  % perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
Then hit localhost:3000
=> Nothing added
b. Configure Hea on admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> New library added
c. Modify the Hea configuration
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Info are modified

B. Not we will test that it works with the new version (much more fun ;))
% git checkout hea-v2 # koha
a. Configure Hea using /admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Check the result on localhost:3000
b. Share libraries's info
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
c. Continue to play a bit and share the info.
Comment 8 Jonathan Druart 2017-02-07 09:57:05 UTC
Created attachment 59976 [details] [review]
Bug 18066: Add the leaflet library
Comment 9 Jonathan Druart 2017-02-28 11:17:57 UTC
Created attachment 60729 [details] [review]
Bug 18066: Do not use token for OSM tiles
Comment 10 Jonathan Druart 2017-02-28 11:20:55 UTC
Created attachment 60730 [details] [review]
[DO NOT PUSH] Use hea test server instead of prod
Comment 11 Jonathan Druart 2017-02-28 11:26:09 UTC
Test plan using the Hea test server:
BibLibre provided me a test server (Merci Laurent!) to test easily this patch. You will not need to set up hea-ws and hea-app.

The test server is at http://community-hea.preprod.biblibre.eu

To test Koha-side you just will need these patches (as well as those from bug 14608), it's easier to use the hea_v2 branch from my github repo.
With the "[DO NOT PUSH]" patch applied you will be able to do all what you want to stress the Hea app.

Follow the "7/ Back to Koha side" part of the original test plan.
Comment 12 Jonathan Druart 2017-02-28 17:13:31 UTC
Created attachment 60734 [details] [review]
Bug 18066: Use https for ws
Comment 13 Chris Cormack 2017-02-28 21:06:25 UTC
Created attachment 60742 [details] [review]
Bug 18066: Hea V2

This patch is the Koha part of the Hea v2 project.
You can find the (testing) code for the server at
  hea-ws  - https://github.com/joubu/hea-ws/commits/v2
  hea-app - https://github.com/joubu/hea-app/commits/v2
They contain the different pull requests made over the last 6 months.

More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC
The goal of this commit message is to provide an overview of what could
be a new version of Hea.

Prior to these changes, the Hea database was filled with 1 line per Koha
installation. System preferences were filled by the libraries and a
cronjob (share_usage_with_koha_community.pl) collected these values to send
them to a webservice (hea-ws/upload.pl).

With the need to collect more data we would want to collect data at the library
level (branch) and not at the installation level.
For instance the geolocation, the url or the country can be different from one
library to another, even if managed from the same Koha installation.
The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql).

The hidden goal of this patch is to make Hea sexier and explain
better to libraries how it can be useful to share their information
with the Koha community. I guess the main problem is the lack of
communication and explanations about what we are doing we these data.
To fill this gap I'd like to (TODO)
  1. Communicate on the ML about this new version of Hea (once it got
pushed and backported)
  2. Link the Privacy_Policy.md from the Hea interface
  3. Get help from a native English speaker to add
popup/help/info/whatever on "Home › Administration › Usage statistics",
to clearly explain what happens (and what will not happen!) when an option or
another is set.

You can find screenshot of this whole enhancement on bug 18066, comment 2.

What this patch does:
- Create a new branches.geolocation DB field
- Add 3 new sysprefs:
  * UsageStatsGeolocation
  * UsageStatsLibrariesInfo
  * UsageStatsPublicID
- Integrate the Leaflet JS library to get a fancy map to pick
geolocations

How does it works:
On the new administration page where statistics to share are configured,
there are several new things. It is now possible to share information either
per Koha installation or libraries. If UsageStatsLibrariesInfo is set,
the info at library level (url, name, country, geolocation) will be
sent to the Hea webservice. If it is not set, you can decide to fill
UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry,
UsageStatsGeolocation to share these information. Note that even if the
data are retrieved at installation level, it's better to fill the prefs
as well: On the Hea website the different libraries defined for a given
Koha installation could be displayed on the same page.
This page is a public page which will be attributed to every
installation (with the pref UsageStatsPublicID). On this page all the
info available publicly will be displayed.

TODO later:
- Add a button on the administration page to delete the info shared
publicly. It will be easy to show that the info are no longer displayed
on the public page.
- Add an icon per Koha installation to get a better "public page"
- Any suggestions?

Test plan:
We will need to test hea-ws, hea-app and the Koha-side code to test the
whole enhancement.
1/ To start, clone the hea-ws and hea-app project and checkout the
'master' branch (*not* 'v2')
2/ Create the hea database and user
  CREATE DATABASE hea
  CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea';
  GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost';
  FLUSH PRIVILEGES;
3/ Fill the DB with some data
  mysql hea < hea-app/sql/schema.sql
  mysql hea < hea-app/sql/sql/mock-data.sql
4/ Checkout the 'v2' branch for both hea-ws and hea-app
5/ Execute the upgrade DB script
  % cd hea-app
  % perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info
  % perl sql/upgrade.pl
Now the DB is using the v2 structure. That means we have 1 installation
row per library previously defined. 1 library row has also been created.
5/ Configure hea-ws
% echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts
<VirtualHost *:80>
  DocumentRoot "/path/to/hea-ws"
  ServerName "hea.koha-community.org"
  <Directory "/">
    Options +ExecCGI
    Require all granted
    AddHandler cgi-script .pl
  </Directory>
</VirtualHost>

And enable it with a2ensite, then restart apache.
The copy the database.yml.sample to database.yml and edit it to fill the
DB info.

6/ Launch the hea-app
  % cd hea-app
  % edit README.md # to install the missing modules
  % cp environments/config.yml environments/development.yml
  % edit environments/development.yml # to fill the DB info
  % perl bin/app.pl
Then hit localhost:3000
You should see a local version of Hea with sample data

7/ Back to Koha side
A. We will test that the webservice still works with previous version of Koha (without v2)
a. Do not configure Hea
  % perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
Then hit localhost:3000
=> Nothing added
b. Configure Hea on admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> New library added
c. Modify the Hea configuration
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Info are modified

B. Not we will test that it works with the new version (much more fun ;))
% git checkout hea-v2 # koha
a. Configure Hea using /admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Check the result on localhost:3000
b. Share libraries's info
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
c. Continue to play a bit and share the info.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 14 Chris Cormack 2017-02-28 21:06:45 UTC
Created attachment 60743 [details] [review]
Bug 18066: Add the leaflet library

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 15 Chris Cormack 2017-02-28 21:07:01 UTC
Created attachment 60744 [details] [review]
Bug 18066: Do not use token for OSM tiles

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 16 Chris Cormack 2017-02-28 21:07:31 UTC
Created attachment 60745 [details] [review]
Use hea test server instead of prod
Comment 17 Chris Cormack 2017-02-28 21:07:53 UTC
Created attachment 60746 [details] [review]
Bug 18066: Use https for ws

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 18 Julian Maurice 2017-03-21 09:58:57 UTC
Some tests are failing:

t/db_dependent/UsageStats.t .. 1/59
#   Failed test ''$report->{library} is a HASH' isa 'HASH''
#   at t/db_dependent/UsageStats.t line 91.
#     '$report->{library} is a HASH' isn't defined

#   Failed test 'There are 6 fields in HASH(0x84c72f0)'
#   at t/db_dependent/UsageStats.t line 92.
#          got: '0'
#     expected: '6'

#   Failed test 'UsageStatsID           is good'
#   at t/db_dependent/UsageStats.t line 93.
#          got: undef
#     expected: '0'

#   Failed test 'UsageStatsLibraryName  is good'
#   at t/db_dependent/UsageStats.t line 94.
#          got: undef
#     expected: ''

#   Failed test 'UsageStatsLibraryUrl   is good'
#   at t/db_dependent/UsageStats.t line 95.
#          got: undef
#     expected: ''

#   Failed test 'UsageStatsLibraryType  is good'
#   at t/db_dependent/UsageStats.t line 96.
#          got: undef
#     expected: ''

#   Failed test 'UsageStatsCountry      is good'
#   at t/db_dependent/UsageStats.t line 97.
#          got: undef
#     expected: ''

#   Failed test ''$report->{library} is a HASH' isa 'HASH''
#   at t/db_dependent/UsageStats.t line 111.
#     '$report->{library} is a HASH' isn't defined

#   Failed test 'There are 6 fields in HASH(0x84554f8)'
#   at t/db_dependent/UsageStats.t line 112.
#          got: '0'
#     expected: '6'

#   Failed test 'UsageStatsID            is good'
#   at t/db_dependent/UsageStats.t line 113.
#          got: undef
#     expected: '1'

#   Failed test 'UsageStatsLibraryName   is good'
#   at t/db_dependent/UsageStats.t line 114.
#          got: undef
#     expected: 'NAME'

#   Failed test 'UsageStatsLibraryUrl    is good'
#   at t/db_dependent/UsageStats.t line 115.
#          got: undef
#     expected: 'URL'

#   Failed test 'UsageStatsLibraryType   is good'
#   at t/db_dependent/UsageStats.t line 116.
#          got: undef
#     expected: 'TYPE'

#   Failed test 'UsageStatsCountry       is good'
#   at t/db_dependent/UsageStats.t line 117.
#          got: undef
#     expected: 'COUNTRY'

#   Failed test 'UsageStatsLibrariesInfo is good'
#   at t/db_dependent/UsageStats.t line 119.
#          got: undef
#     expected: '33'
# Looks like you failed 15 tests of 59.
t/db_dependent/UsageStats.t .. Dubious, test returned 15 (wstat 3840, 0xf00)
Failed 15/59 subtests

Test Summary Report
-------------------
t/db_dependent/UsageStats.t (Wstat: 3840 Tests: 59 Failed: 15)
  Failed tests:  12-18, 21-28


Additionally, the map in usage_statistics.pl can be positioned outside the borders of its container. I'll attach screenshots to show what I mean
Comment 19 Julian Maurice 2017-03-21 10:05:06 UTC
Created attachment 61355 [details]
Bad position of the map when you reload the page when not on top of the page
Comment 20 Julian Maurice 2017-03-21 10:05:40 UTC
Created attachment 61356 [details]
Bad position of the map when you scroll down
Comment 21 Julian Maurice 2017-03-21 10:06:37 UTC
Also, when the screen width is too small (1280px for instance) the map goes completely outside its container.
Comment 22 Jonathan Druart 2017-03-21 10:08:21 UTC
(In reply to Julian Maurice from comment #18)
> Additionally, the map in usage_statistics.pl can be positioned outside the
> borders of its container. I'll attach screenshots to show what I mean

Yes I know, I did not manage to fix it. Any ideas?
Comment 23 Jonathan Druart 2017-03-21 10:22:04 UTC
Created attachment 61358 [details] [review]
Bug 18066: Fix tests
Comment 24 Julian Maurice 2017-03-21 11:02:12 UTC
Created attachment 61361 [details] [review]
Bug 18066: Fix map positioning in admin/usage_statistics.pl
Comment 25 Julian Maurice 2017-03-21 11:11:50 UTC
Created attachment 61362 [details] [review]
Bug 18066: Fix map positioning in admin/usage_statistics.pl
Comment 26 Julian Maurice 2017-03-21 13:13:09 UTC
Ok tests pass now.

But what should we do about the patches that change the URL of Hea ? They shouldn't be pushed, right ? But if we don't push them, the URL is http instead of https. I feel that something is missing here.
Should we move that URL in a syspref or in $KOHA_CONF for easier testing ?
Comment 27 Julian Maurice 2017-03-21 14:48:48 UTC
Created attachment 61392 [details] [review]
Bug 18066: Hea V2

This patch is the Koha part of the Hea v2 project.
You can find the (testing) code for the server at
  hea-ws  - https://github.com/joubu/hea-ws/commits/v2
  hea-app - https://github.com/joubu/hea-app/commits/v2
They contain the different pull requests made over the last 6 months.

More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC
The goal of this commit message is to provide an overview of what could
be a new version of Hea.

Prior to these changes, the Hea database was filled with 1 line per Koha
installation. System preferences were filled by the libraries and a
cronjob (share_usage_with_koha_community.pl) collected these values to send
them to a webservice (hea-ws/upload.pl).

With the need to collect more data we would want to collect data at the library
level (branch) and not at the installation level.
For instance the geolocation, the url or the country can be different from one
library to another, even if managed from the same Koha installation.
The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql).

The hidden goal of this patch is to make Hea sexier and explain
better to libraries how it can be useful to share their information
with the Koha community. I guess the main problem is the lack of
communication and explanations about what we are doing we these data.
To fill this gap I'd like to (TODO)
  1. Communicate on the ML about this new version of Hea (once it got
pushed and backported)
  2. Link the Privacy_Policy.md from the Hea interface
  3. Get help from a native English speaker to add
popup/help/info/whatever on "Home › Administration › Usage statistics",
to clearly explain what happens (and what will not happen!) when an option or
another is set.

You can find screenshot of this whole enhancement on bug 18066, comment 2.

What this patch does:
- Create a new branches.geolocation DB field
- Add 3 new sysprefs:
  * UsageStatsGeolocation
  * UsageStatsLibrariesInfo
  * UsageStatsPublicID
- Integrate the Leaflet JS library to get a fancy map to pick
geolocations

How does it works:
On the new administration page where statistics to share are configured,
there are several new things. It is now possible to share information either
per Koha installation or libraries. If UsageStatsLibrariesInfo is set,
the info at library level (url, name, country, geolocation) will be
sent to the Hea webservice. If it is not set, you can decide to fill
UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry,
UsageStatsGeolocation to share these information. Note that even if the
data are retrieved at installation level, it's better to fill the prefs
as well: On the Hea website the different libraries defined for a given
Koha installation could be displayed on the same page.
This page is a public page which will be attributed to every
installation (with the pref UsageStatsPublicID). On this page all the
info available publicly will be displayed.

TODO later:
- Add a button on the administration page to delete the info shared
publicly. It will be easy to show that the info are no longer displayed
on the public page.
- Add an icon per Koha installation to get a better "public page"
- Any suggestions?

Test plan:
We will need to test hea-ws, hea-app and the Koha-side code to test the
whole enhancement.
1/ To start, clone the hea-ws and hea-app project and checkout the
'master' branch (*not* 'v2')
2/ Create the hea database and user
  CREATE DATABASE hea
  CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea';
  GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost';
  FLUSH PRIVILEGES;
3/ Fill the DB with some data
  mysql hea < hea-app/sql/schema.sql
  mysql hea < hea-app/sql/sql/mock-data.sql
4/ Checkout the 'v2' branch for both hea-ws and hea-app
5/ Execute the upgrade DB script
  % cd hea-app
  % perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info
  % perl sql/upgrade.pl
Now the DB is using the v2 structure. That means we have 1 installation
row per library previously defined. 1 library row has also been created.
5/ Configure hea-ws
% echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts
<VirtualHost *:80>
  DocumentRoot "/path/to/hea-ws"
  ServerName "hea.koha-community.org"
  <Directory "/">
    Options +ExecCGI
    Require all granted
    AddHandler cgi-script .pl
  </Directory>
</VirtualHost>

And enable it with a2ensite, then restart apache.
The copy the database.yml.sample to database.yml and edit it to fill the
DB info.

6/ Launch the hea-app
  % cd hea-app
  % edit README.md # to install the missing modules
  % cp environments/config.yml environments/development.yml
  % edit environments/development.yml # to fill the DB info
  % perl bin/app.pl
Then hit localhost:3000
You should see a local version of Hea with sample data

7/ Back to Koha side
A. We will test that the webservice still works with previous version of Koha (without v2)
a. Do not configure Hea
  % perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
Then hit localhost:3000
=> Nothing added
b. Configure Hea on admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> New library added
c. Modify the Hea configuration
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Info are modified

B. Not we will test that it works with the new version (much more fun ;))
% git checkout hea-v2 # koha
a. Configure Hea using /admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Check the result on localhost:3000
b. Share libraries's info
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
c. Continue to play a bit and share the info.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 28 Julian Maurice 2017-03-21 14:48:59 UTC
Created attachment 61393 [details] [review]
Bug 18066: Add the leaflet library

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 29 Julian Maurice 2017-03-21 14:49:05 UTC
Created attachment 61394 [details] [review]
Bug 18066: Do not use token for OSM tiles

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 30 Julian Maurice 2017-03-21 14:49:12 UTC
Created attachment 61395 [details] [review]
Bug 18066: Use https for ws

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 31 Julian Maurice 2017-03-21 14:49:21 UTC
Created attachment 61396 [details] [review]
Bug 18066: Fix tests
Comment 32 Julian Maurice 2017-03-21 14:49:28 UTC
Created attachment 61397 [details] [review]
Bug 18066: Fix map positioning in admin/usage_statistics.pl
Comment 33 Julian Maurice 2017-03-21 14:49:40 UTC
Created attachment 61398 [details] [review]
[DO NOT PUSH] Bug 18066: Use hea test server instead of prod
Comment 34 Jonathan Druart 2017-03-21 18:46:32 UTC
Julian, was it a PQA?
Comment 35 Julian Maurice 2017-03-22 08:27:58 UTC
Created attachment 61427 [details] [review]
Bug 18066: Hea V2

This patch is the Koha part of the Hea v2 project.
You can find the (testing) code for the server at
  hea-ws  - https://github.com/joubu/hea-ws/commits/v2
  hea-app - https://github.com/joubu/hea-app/commits/v2
They contain the different pull requests made over the last 6 months.

More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC
The goal of this commit message is to provide an overview of what could
be a new version of Hea.

Prior to these changes, the Hea database was filled with 1 line per Koha
installation. System preferences were filled by the libraries and a
cronjob (share_usage_with_koha_community.pl) collected these values to send
them to a webservice (hea-ws/upload.pl).

With the need to collect more data we would want to collect data at the library
level (branch) and not at the installation level.
For instance the geolocation, the url or the country can be different from one
library to another, even if managed from the same Koha installation.
The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql).

The hidden goal of this patch is to make Hea sexier and explain
better to libraries how it can be useful to share their information
with the Koha community. I guess the main problem is the lack of
communication and explanations about what we are doing we these data.
To fill this gap I'd like to (TODO)
  1. Communicate on the ML about this new version of Hea (once it got
pushed and backported)
  2. Link the Privacy_Policy.md from the Hea interface
  3. Get help from a native English speaker to add
popup/help/info/whatever on "Home › Administration › Usage statistics",
to clearly explain what happens (and what will not happen!) when an option or
another is set.

You can find screenshot of this whole enhancement on bug 18066, comment 2.

What this patch does:
- Create a new branches.geolocation DB field
- Add 3 new sysprefs:
  * UsageStatsGeolocation
  * UsageStatsLibrariesInfo
  * UsageStatsPublicID
- Integrate the Leaflet JS library to get a fancy map to pick
geolocations

How does it works:
On the new administration page where statistics to share are configured,
there are several new things. It is now possible to share information either
per Koha installation or libraries. If UsageStatsLibrariesInfo is set,
the info at library level (url, name, country, geolocation) will be
sent to the Hea webservice. If it is not set, you can decide to fill
UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry,
UsageStatsGeolocation to share these information. Note that even if the
data are retrieved at installation level, it's better to fill the prefs
as well: On the Hea website the different libraries defined for a given
Koha installation could be displayed on the same page.
This page is a public page which will be attributed to every
installation (with the pref UsageStatsPublicID). On this page all the
info available publicly will be displayed.

TODO later:
- Add a button on the administration page to delete the info shared
publicly. It will be easy to show that the info are no longer displayed
on the public page.
- Add an icon per Koha installation to get a better "public page"
- Any suggestions?

Test plan:
We will need to test hea-ws, hea-app and the Koha-side code to test the
whole enhancement.
1/ To start, clone the hea-ws and hea-app project and checkout the
'master' branch (*not* 'v2')
2/ Create the hea database and user
  CREATE DATABASE hea
  CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea';
  GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost';
  FLUSH PRIVILEGES;
3/ Fill the DB with some data
  mysql hea < hea-app/sql/schema.sql
  mysql hea < hea-app/sql/sql/mock-data.sql
4/ Checkout the 'v2' branch for both hea-ws and hea-app
5/ Execute the upgrade DB script
  % cd hea-app
  % perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info
  % perl sql/upgrade.pl
Now the DB is using the v2 structure. That means we have 1 installation
row per library previously defined. 1 library row has also been created.
5/ Configure hea-ws
% echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts
<VirtualHost *:80>
  DocumentRoot "/path/to/hea-ws"
  ServerName "hea.koha-community.org"
  <Directory "/">
    Options +ExecCGI
    Require all granted
    AddHandler cgi-script .pl
  </Directory>
</VirtualHost>

And enable it with a2ensite, then restart apache.
The copy the database.yml.sample to database.yml and edit it to fill the
DB info.

6/ Launch the hea-app
  % cd hea-app
  % edit README.md # to install the missing modules
  % cp environments/config.yml environments/development.yml
  % edit environments/development.yml # to fill the DB info
  % perl bin/app.pl
Then hit localhost:3000
You should see a local version of Hea with sample data

7/ Back to Koha side
A. We will test that the webservice still works with previous version of Koha (without v2)
a. Do not configure Hea
  % perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
Then hit localhost:3000
=> Nothing added
b. Configure Hea on admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> New library added
c. Modify the Hea configuration
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Info are modified

B. Not we will test that it works with the new version (much more fun ;))
% git checkout hea-v2 # koha
a. Configure Hea using /admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Check the result on localhost:3000
b. Share libraries's info
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
c. Continue to play a bit and share the info.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 36 Julian Maurice 2017-03-22 08:28:04 UTC
Created attachment 61428 [details] [review]
Bug 18066: Add the leaflet library

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 37 Julian Maurice 2017-03-22 08:28:11 UTC
Created attachment 61429 [details] [review]
Bug 18066: Do not use token for OSM tiles

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 38 Julian Maurice 2017-03-22 08:28:15 UTC
Created attachment 61430 [details] [review]
Bug 18066: Use https for ws

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 39 Julian Maurice 2017-03-22 08:28:20 UTC
Created attachment 61431 [details] [review]
Bug 18066: Fix tests

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 40 Julian Maurice 2017-03-22 08:28:24 UTC
Created attachment 61432 [details] [review]
Bug 18066: Fix map positioning in admin/usage_statistics.pl

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 41 Julian Maurice 2017-03-22 08:28:28 UTC
Created attachment 61433 [details] [review]
[DO NOT PUSH] Bug 18066: Use hea test server instead of prod
Comment 42 Julian Maurice 2017-03-22 08:29:13 UTC
(In reply to Jonathan Druart from comment #34)
> Julian, was it a PQA?

It wasn't, but now it is ;)
Comment 43 Brendan Gallagher 2017-03-23 15:22:29 UTC
Pushed to Master - Should be in the 17.05 release.  Thanks!
Comment 44 Jonathan Druart 2017-03-23 16:31:50 UTC
Created attachment 61582 [details] [review]
Bug 18066: Add the new geolocation column to kohastructure.sql
Comment 45 Jonathan Druart 2017-03-23 16:34:12 UTC
Last patch pushed to master and DBIx::Class schema regenerated.
Comment 46 Katrin Fischer 2017-03-26 20:56:53 UTC
Not pushing this to 16.11.x yet, might reconsider for the next bugfix release.
Comment 47 Katrin Fischer 2017-04-02 15:47:00 UTC
Jonathan, is it ok to push this for 16.11.07?
Will be my first Schema change patch, so want to push it early if possible.
Comment 48 Katrin Fischer 2017-04-16 13:01:33 UTC
Is this ready for stable?
Comment 49 Jonathan Druart 2017-04-17 12:23:44 UTC
(In reply to Katrin Fischer from comment #48)
> Is this ready for stable?

I did not get the confirmation yet that the test server has been moved to production.
Comment 50 Katrin Fischer 2017-04-17 13:13:25 UTC
We are in string freeze now - so will check again for 16.11.08
Comment 51 Jonathan Druart 2017-04-18 15:29:07 UTC
The production server has been updated, this is ready to be backported.
Comment 52 Jonathan Druart 2017-04-24 19:30:06 UTC
Created attachment 62646 [details] [review]
Bug 18066: Remove warnings from translate script

This patch removes the following warnings:
Use of uninitialized value $pref_name in concatenation (.) or string at
LangInstaller.pm
Comment 53 Jonathan Druart 2017-04-24 19:30:26 UTC
(In reply to Jonathan Druart from comment #52)
> Created attachment 62646 [details] [review] [review]
> Bug 18066: Remove warnings from translate script
> 
> This patch removes the following warnings:
> Use of uninitialized value $pref_name in concatenation (.) or string at
> LangInstaller.pm

This patch has been pushed to master!
Comment 54 Julian Maurice 2017-05-10 14:36:45 UTC
Pushed to 3.22.x for 3.22.21
Comment 55 Mason James 2017-05-24 02:03:13 UTC
Pushed to 16.05.x, for 16.05.13 release