Bug 13691 - Add some selenium scripts
Summary: Add some selenium scripts
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Test Suite (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Jonathan Druart
QA Contact: Tomás Cohen Arazi
URL:
Keywords:
Depends on:
Blocks: 19384 18804
  Show dependency treegraph
 
Reported: 2015-02-10 15:02 UTC by Jonathan Druart
Modified: 2018-06-04 20:10 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 13691: Add basic selenium script (8.50 KB, patch)
2015-02-10 15:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13691: Remove existing selenium scripts (12.24 KB, patch)
2015-02-10 15:27 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13691 - PageObjectPattern implementation. (9.62 KB, patch)
2015-07-07 14:13 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 13691: Add changes for 3.22 (939 bytes, patch)
2016-03-25 17:04 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13691: Add basic selenium script (8.50 KB, patch)
2016-03-25 17:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13691: Add changes for 3.22 (939 bytes, patch)
2016-03-25 17:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13691: Remove existing selenium scripts (12.30 KB, patch)
2016-06-03 09:19 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 13691: Add basic selenium script (8.75 KB, patch)
2016-06-03 09:20 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 13691: Add changes for 3.22 (1.11 KB, patch)
2016-06-03 09:20 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 13691: Remove existing selenium scripts (12.35 KB, patch)
2016-06-13 13:57 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 13691: Add basic selenium script (8.80 KB, patch)
2016-06-13 13:57 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 13691: Add changes for 3.22 (1.16 KB, patch)
2016-06-13 13:57 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2015-02-10 15:02:21 UTC
To catch performance regressions it would be great to have some selenium scripts.
Comment 1 Jonathan Druart 2015-02-10 15:26:14 UTC Comment hidden (obsolete)
Comment 2 Jonathan Druart 2015-02-10 15:26:59 UTC
This is a POC, feedback welcome :)
Comment 3 Jonathan Druart 2015-02-10 15:27:20 UTC Comment hidden (obsolete)
Comment 4 Jonathan Druart 2015-02-11 10:08:50 UTC
If the community thinks that these tests are relevant and useful, I can rewrite them to make them pushable.
Comment 5 Tomás Cohen Arazi 2015-02-17 04:05:18 UTC
Current integration tests rely on Test::WWW::Mechanize. This option seems more interesting, but we should try to think "what" we want to actually test first.
Comment 6 Olli-Antti Kivilahti 2015-03-06 16:44:10 UTC
Check out the Perl Cucumber integration to use as a wrapper for feature documentation and acceptance/integration testing framework.
It looks very promising and could be a good candidate as a mandatory testing framework for the new REST-API.

https://metacpan.org/pod/Test::BDD::Cucumber::Manual::Integration
https://metacpan.org/pod/App::pherkin

Oslo is using Ruby, but we could try how the Perl implementation works.

However we should consider some test driven mechanism as a mandatory part of getting REST API patches pushed to master.
Comment 7 koha 2015-03-09 17:35:48 UTC
I have a feeling that Koha could benefit from having a few tests that exercise the most critical functionality from a user perspective, both as tools to measure performance changes and to automate test of critical user scenarios.

These tests should optimally be in the same repo and be versioned along with the Koha code. That way developers could get rapid feedback on the changes they make, and also we would require the tests to pass along with production code changes. 

However, if this turns out to be to much work for the individual developer, these tests can be in a separate repo and perhaps be maintained by a separate team – as they in theory not break unless we are changing existing user functionality. In reality, however, they will from time to time break from changes the users cannot see, and they will have a certain cost to maintain. 

They will also need installations of Koha running the versions to test against. My feeling is that one could spend a little effort setting up an infrastructure for this and do some experiments on how this works, with a small number of scenarios. 

(Oslo Public Library has some experience in using automated browser based testing on Koha, using Cucumber and step definitions in Ruby.)

A couple of comments:

*** Browser-based testing ***

This is about testing a running web application:

Examples of tools: 

* Mechanize: (which already is used a little)
- simple 
- only parses html (it has no understanding of JavaScript, which Koha uses more and more)

* Webdriver (in Selenium 2): (which is what Jonathan has created a patch for)
- drives actual browsers with JavaScript and all (can be "headless" browser)
- you can programatically interact with visible elements like a user
- there is currently no *direct* binding in perl, you have to run "selenium server" (which needs java) which in turn drives the browsers (in ruby you do not need this intermediary server)  https://github.com/gempesaw/Selenium-Remote-Driver/issues/189

*** Feature based testing / behaviour driven testing / example driven testing ***

This is about how your test looks, and in addition to using normal unit test tools, there are tools like Fit/-nesse and Cucumber that lets you write automated tests in a way that is easier for users to understand. (Olli-Antti touched on this.) These tests may or may not be browser-based, but in a web-based application it is more likely that they do than that they don't. 

*** Acceptance tests ***

These are typically tests for a user story / scenario and confirms that a piece of functionality is in place and works. They may be expressed in a tool that is feature-based like the ones mentioned above, and will often drive a browser interface in a web application like Koha.
Comment 8 Olli-Antti Kivilahti 2015-05-04 07:53:54 UTC
FYI: Just started working on this.
https://metacpan.org/pod/Selenium::Remote::Driver
no longer needs the stand-alone server !! YAY YAY!!!
Comment 9 Olli-Antti Kivilahti 2015-05-04 08:38:06 UTC
Cannot proceed because generating test users with test permissions has no API in Koha:: or C4::
Looking into the Koha REST API to create one.
Comment 10 Olli-Antti Kivilahti 2015-07-06 15:31:12 UTC
Hi there!
I just added bug 14495 to easily generate all kinds of WebDrivers/Test::Mojo things to do integration tests. This basically creates Selenium::Remote::Drivers based on the given parameters.

Looking into implementing a PageObject to test password authentication, but it might be a bit difficult regarding how Koha deals with the authentication.
This is to test the Authentication Rewriting at Bug 7174
Comment 11 Olli-Antti Kivilahti 2015-07-06 16:43:28 UTC
Hi! just drew a schematic to implement a PageObjectPattern-system on.
There is a nice code example to make robust UI tests.
https://docs.google.com/drawings/d/1k_y1YZi72UFxlVEDpqINT68TGQBEy-f2ZkTGLpMcTJU/edit?usp=sharing
Comment 12 Olli-Antti Kivilahti 2015-07-07 14:13:47 UTC Comment hidden (obsolete)
Comment 13 Olli-Antti Kivilahti 2015-07-16 12:19:01 UTC
Comment on attachment 40819 [details] [review]
Bug 13691 - PageObjectPattern implementation.

Created a separate bug for the PageObject Pattern.
Comment 14 Jonathan Druart 2015-09-22 10:31:46 UTC
Sending to the Needs Signoff queue.
Comment 15 Jonathan Druart 2016-03-25 17:04:36 UTC Comment hidden (obsolete)
Comment 16 Jonathan Druart 2016-03-25 17:26:40 UTC Comment hidden (obsolete)
Comment 17 Jonathan Druart 2016-03-25 17:26:44 UTC Comment hidden (obsolete)
Comment 18 Jonathan Druart 2016-03-25 17:28:43 UTC
This is still needed! We need it to add more tests and graph performances over the versions.
I have updated the test plan to wget the last version.
Please get involved!
Comment 19 Chris Cormack 2016-06-03 09:19:55 UTC Comment hidden (obsolete)
Comment 20 Chris Cormack 2016-06-03 09:20:17 UTC Comment hidden (obsolete)
Comment 21 Chris Cormack 2016-06-03 09:20:38 UTC Comment hidden (obsolete)
Comment 22 Chris Cormack 2016-06-03 10:26:20 UTC
The tests need to be updated for master, and also need to add a user so that the selenium can connect as that user. But it does add all the structure to allow us to write selenium tests, so I think we should do the updated selenium tests in other patches.

Having this pushed makes writing selenium tests quite easy.
Comment 23 Tomás Cohen Arazi 2016-06-13 13:57:17 UTC
Created attachment 52317 [details] [review]
Bug 13691: Remove existing selenium scripts

They are not usable.

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 24 Tomás Cohen Arazi 2016-06-13 13:57:26 UTC
Created attachment 52318 [details] [review]
Bug 13691: Add basic selenium script

This script has been used to compare 3.16.x and 3.18.x performances on
bug 13690.

What it does:
- Go on the mainpage and process a log in
- Create a patron category
- Create a patron
- Add 3 items
- check the 3 items out to the patron
- check the 3 items in

How to use it?

$ wget https://selenium-release.storage.googleapis.com/2.53/selenium-server-standalone-2.53.0.jar

$ vim /etc/apt/sources.list.d/firefox.list
  deb http://packages.linuxmint.com debian import
$ apt-get update
$ apt-get install firefox
$ sudo apt-get install xvfb

$ SELENIUM_PATH=/home/koha/tools/selenium-server-standalone-2.53.0.jar
$ Xvfb :1 -screen 0 1024x768x24 2>&1 >/dev/null &
$ DISPLAY=:1 java -jar $SELENIUM_PATH

perl t/db_dependent/selenium/basic_workflow.t

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Note the pages have changed so the tests will not all pass.
Also you need Selenium::Remote::Driver installed and staffClientBaseURL
must be set

I will change the tests in a follow up patch.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 25 Tomás Cohen Arazi 2016-06-13 13:57:38 UTC
Created attachment 52319 [details] [review]
Bug 13691: Add changes for 3.22

To test:
Run the tests
t/db_dependent/selenium/basic_workflow.t

After you have set up selenium following the tests in the previous patch

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 26 Brendan Gallagher 2016-07-08 20:39:07 UTC
Pushed to Master - Should be in the November 2016 release.  thanks!