To catch performance regressions it would be great to have some selenium scripts.
Created attachment 35797 [details] [review] Bug 13691: Add basic selenium script This script has been used to compare 3.16.x and 3.18.x performances on bug 13690. What it does: - Go on the mainpage and process a log in - Create a patron category - Create a patron - Add 3 items - check the 3 items out to the patron - check the 3 items in How to use it? $ wget http://selenium-release.storage.googleapis.com/2.44/selenium-server-standalone-2.44.0.jar $ vim /etc/apt/sources.list.d/firefox.list deb http://packages.linuxmint.com debian import $ apt-get update $ apt-get install firefox $ sudo apt-get install xvfb $ SELENIUM_PATH=/home/koha/tools/selenium-server-standalone-2.44.0.jar $ Xvfb :1 -screen 0 1024x768x24 2>&1 >/dev/null & $ DISPLAY=:1 java -jar $SELENIUM_PATH perl t/db_dependent/selenium/basic_workflow.t
This is a POC, feedback welcome :)
Created attachment 35798 [details] [review] Bug 13691: Remove existing selenium scripts They are not usable.
If the community thinks that these tests are relevant and useful, I can rewrite them to make them pushable.
Current integration tests rely on Test::WWW::Mechanize. This option seems more interesting, but we should try to think "what" we want to actually test first.
Check out the Perl Cucumber integration to use as a wrapper for feature documentation and acceptance/integration testing framework. It looks very promising and could be a good candidate as a mandatory testing framework for the new REST-API. https://metacpan.org/pod/Test::BDD::Cucumber::Manual::Integration https://metacpan.org/pod/App::pherkin Oslo is using Ruby, but we could try how the Perl implementation works. However we should consider some test driven mechanism as a mandatory part of getting REST API patches pushed to master.
I have a feeling that Koha could benefit from having a few tests that exercise the most critical functionality from a user perspective, both as tools to measure performance changes and to automate test of critical user scenarios. These tests should optimally be in the same repo and be versioned along with the Koha code. That way developers could get rapid feedback on the changes they make, and also we would require the tests to pass along with production code changes. However, if this turns out to be to much work for the individual developer, these tests can be in a separate repo and perhaps be maintained by a separate team – as they in theory not break unless we are changing existing user functionality. In reality, however, they will from time to time break from changes the users cannot see, and they will have a certain cost to maintain. They will also need installations of Koha running the versions to test against. My feeling is that one could spend a little effort setting up an infrastructure for this and do some experiments on how this works, with a small number of scenarios. (Oslo Public Library has some experience in using automated browser based testing on Koha, using Cucumber and step definitions in Ruby.) A couple of comments: *** Browser-based testing *** This is about testing a running web application: Examples of tools: * Mechanize: (which already is used a little) - simple - only parses html (it has no understanding of JavaScript, which Koha uses more and more) * Webdriver (in Selenium 2): (which is what Jonathan has created a patch for) - drives actual browsers with JavaScript and all (can be "headless" browser) - you can programatically interact with visible elements like a user - there is currently no *direct* binding in perl, you have to run "selenium server" (which needs java) which in turn drives the browsers (in ruby you do not need this intermediary server) https://github.com/gempesaw/Selenium-Remote-Driver/issues/189 *** Feature based testing / behaviour driven testing / example driven testing *** This is about how your test looks, and in addition to using normal unit test tools, there are tools like Fit/-nesse and Cucumber that lets you write automated tests in a way that is easier for users to understand. (Olli-Antti touched on this.) These tests may or may not be browser-based, but in a web-based application it is more likely that they do than that they don't. *** Acceptance tests *** These are typically tests for a user story / scenario and confirms that a piece of functionality is in place and works. They may be expressed in a tool that is feature-based like the ones mentioned above, and will often drive a browser interface in a web application like Koha.
FYI: Just started working on this. https://metacpan.org/pod/Selenium::Remote::Driver no longer needs the stand-alone server !! YAY YAY!!!
Cannot proceed because generating test users with test permissions has no API in Koha:: or C4:: Looking into the Koha REST API to create one.
Hi there! I just added bug 14495 to easily generate all kinds of WebDrivers/Test::Mojo things to do integration tests. This basically creates Selenium::Remote::Drivers based on the given parameters. Looking into implementing a PageObject to test password authentication, but it might be a bit difficult regarding how Koha deals with the authentication. This is to test the Authentication Rewriting at Bug 7174
Hi! just drew a schematic to implement a PageObjectPattern-system on. There is a nice code example to make robust UI tests. https://docs.google.com/drawings/d/1k_y1YZi72UFxlVEDpqINT68TGQBEy-f2ZkTGLpMcTJU/edit?usp=sharing
Created attachment 40819 [details] [review] Bug 13691 - PageObjectPattern implementation.
Comment on attachment 40819 [details] [review] Bug 13691 - PageObjectPattern implementation. Created a separate bug for the PageObject Pattern.
Sending to the Needs Signoff queue.
Created attachment 49592 [details] [review] Bug 13691: Add changes for 3.22
Created attachment 49593 [details] [review] Bug 13691: Add basic selenium script This script has been used to compare 3.16.x and 3.18.x performances on bug 13690. What it does: - Go on the mainpage and process a log in - Create a patron category - Create a patron - Add 3 items - check the 3 items out to the patron - check the 3 items in How to use it? $ wget https://selenium-release.storage.googleapis.com/2.53/selenium-server-standalone-2.53.0.jar $ vim /etc/apt/sources.list.d/firefox.list deb http://packages.linuxmint.com debian import $ apt-get update $ apt-get install firefox $ sudo apt-get install xvfb $ SELENIUM_PATH=/home/koha/tools/selenium-server-standalone-2.53.0.jar $ Xvfb :1 -screen 0 1024x768x24 2>&1 >/dev/null & $ DISPLAY=:1 java -jar $SELENIUM_PATH perl t/db_dependent/selenium/basic_workflow.t
Created attachment 49594 [details] [review] Bug 13691: Add changes for 3.22
This is still needed! We need it to add more tests and graph performances over the versions. I have updated the test plan to wget the last version. Please get involved!
Created attachment 51974 [details] [review] Bug 13691: Remove existing selenium scripts They are not usable. Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Created attachment 51975 [details] [review] Bug 13691: Add basic selenium script This script has been used to compare 3.16.x and 3.18.x performances on bug 13690. What it does: - Go on the mainpage and process a log in - Create a patron category - Create a patron - Add 3 items - check the 3 items out to the patron - check the 3 items in How to use it? $ wget https://selenium-release.storage.googleapis.com/2.53/selenium-server-standalone-2.53.0.jar $ vim /etc/apt/sources.list.d/firefox.list deb http://packages.linuxmint.com debian import $ apt-get update $ apt-get install firefox $ sudo apt-get install xvfb $ SELENIUM_PATH=/home/koha/tools/selenium-server-standalone-2.53.0.jar $ Xvfb :1 -screen 0 1024x768x24 2>&1 >/dev/null & $ DISPLAY=:1 java -jar $SELENIUM_PATH perl t/db_dependent/selenium/basic_workflow.t Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> Note the pages have changed so the tests will not all pass. Also you need Selenium::Remote::Driver installed and staffClientBaseURL must be set I will change the tests in a follow up patch.
Created attachment 51976 [details] [review] Bug 13691: Add changes for 3.22 To test: Run the tests t/db_dependent/selenium/basic_workflow.t After you have set up selenium following the tests in the previous patch Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
The tests need to be updated for master, and also need to add a user so that the selenium can connect as that user. But it does add all the structure to allow us to write selenium tests, so I think we should do the updated selenium tests in other patches. Having this pushed makes writing selenium tests quite easy.
Created attachment 52317 [details] [review] Bug 13691: Remove existing selenium scripts They are not usable. Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 52318 [details] [review] Bug 13691: Add basic selenium script This script has been used to compare 3.16.x and 3.18.x performances on bug 13690. What it does: - Go on the mainpage and process a log in - Create a patron category - Create a patron - Add 3 items - check the 3 items out to the patron - check the 3 items in How to use it? $ wget https://selenium-release.storage.googleapis.com/2.53/selenium-server-standalone-2.53.0.jar $ vim /etc/apt/sources.list.d/firefox.list deb http://packages.linuxmint.com debian import $ apt-get update $ apt-get install firefox $ sudo apt-get install xvfb $ SELENIUM_PATH=/home/koha/tools/selenium-server-standalone-2.53.0.jar $ Xvfb :1 -screen 0 1024x768x24 2>&1 >/dev/null & $ DISPLAY=:1 java -jar $SELENIUM_PATH perl t/db_dependent/selenium/basic_workflow.t Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> Note the pages have changed so the tests will not all pass. Also you need Selenium::Remote::Driver installed and staffClientBaseURL must be set I will change the tests in a follow up patch. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 52319 [details] [review] Bug 13691: Add changes for 3.22 To test: Run the tests t/db_dependent/selenium/basic_workflow.t After you have set up selenium following the tests in the previous patch Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Pushed to Master - Should be in the November 2016 release. thanks!