Bug 19185 - Web installer and onboarding tool selenium test
Summary: Web installer and onboarding tool selenium test
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Test Suite (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Jonathan Druart
QA Contact: Tomás Cohen Arazi
URL:
Keywords:
Depends on: 19821 19243
Blocks: 19384
  Show dependency treegraph
 
Reported: 2017-08-27 04:18 UTC by Alex Buckley
Modified: 2019-10-15 09:02 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 19185 - Selenium test for the Koha web installer and onboarding tool (11.92 KB, patch)
2017-08-27 04:26 UTC, Alex Buckley
Details | Diff | Splinter Review
Bug 19185: Change to use ENV, fix login and passwords (2.39 KB, patch)
2017-10-30 04:27 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 19185: Use submit button if only one exists on the page (1.10 KB, patch)
2018-05-04 17:53 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 19185: Ease elements selection (2.61 KB, patch)
2018-05-04 17:53 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 19185: Add selenium tests for installation and onboarding process (5.14 KB, patch)
2018-05-04 17:53 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 19185: Use submit button if only one exists on the page (1.17 KB, patch)
2018-11-02 16:27 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 19185: Ease elements selection (2.73 KB, patch)
2018-11-02 16:27 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 19185: Add selenium tests for installation and onboarding process (5.19 KB, patch)
2018-11-02 16:27 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Buckley 2017-08-27 04:18:12 UTC
This Selenium test will go through the workflow for installing Koha including going through the web installer and onboarding tool
Comment 1 Alex Buckley 2017-08-27 04:26:19 UTC
Created attachment 66508 [details] [review]
Bug 19185 - Selenium test for the Koha web installer and onboarding tool

This Selenium test will go through the web installer and onboarding tool
testing the Koha installation workflow.

Test plan (this test plan includes how to install Selenium as well as how to run this test, for the benefit of people who have not got
Selenium installed on their machines):

1. wget https://selenium-release.storage.googleapis.com/2.53/selenium-server-standalone-2.53.0.jar

2. vim /etc/apt/sources.list.d/firefox.list

3. Paste into the file:
deb http://packages.linuxmint.com debian import

4. sudo apt-get update

5. sudo apt-get install firefox

6. sudo apt-get install xvfb

7. Set the SELENIUM_PATH vartiable:
SELENIUM_PATH=/home/vagrant/kohaclone/selenium-server-standalone-2.53.0.jar

8. Xvfb :1 -screen 0 1024x768x24 2>&1 >/dev/null &

9. DISPLAY=:1 java -jar $SELENIUM_PATH

Note: This will start up the selenium server. Everytime you want to
shutdown this terminal window and want to restart Selenium just run the step 9 command to restart the Selenium server

10. Open a new terminal window and write in:
git clone https://github.com/gempesaw/Selenium-Remote-Driver --branch build/master --single-branch --depth 1

11. cd Selenium-Remote-Driver

12. perl Makefile.PL

13. make

14. make test

15. sudo make install

16. Now everything is installed and you can et up for running the
selenium tests

17. In a new terminal restart memcached and drop and recreate the Koha database

18. sudo koha-shell kohadev

19. perl t/db_dependent/selenium/installkoha.t

20. Notice the db errors saying that database tables do not exist

21. Notice the CP test output messages stating what the test is doing

Note: warns exist in this selenium test because using Selenium to test
the web installer and onboarding tool has proved to be flaky. The use of warning out the title of the currently loaded page and implementing pauses has helped get the test to pass after many hours of troubleshooting. If anyone can suggest alternatives please do so.

Sponsored-By: Catalyst IT
Comment 2 Alex Buckley 2017-08-27 04:27:09 UTC
This patch was originally attached to 18974 however I have placed it in its own bug report to make testing more straightforward
Comment 3 M. Tompsett 2017-10-30 04:27:02 UTC
Created attachment 68830 [details] [review]
Bug 19185: Change to use ENV, fix login and passwords

The kohadevbox has admin/admin set, so for testing when
creating the initial superuser, if you wish to minimize
effort, use admin/admin. Otherwise, make sure to:
export KOHA_USER={whatever your user is}
export KOHA_PASS={whatever your password is}
before attempting to run the test.

Also, this test was failing because 'sub auth' was using
the ugliest of xpath's to find the submit button.
I've noticed the other selenium tests have similar functions.
It might be an idea to build a t/lib/Selenium.pm so as to
reduce code duplication and older versions differing from
newer versions.
Comment 4 M. Tompsett 2017-10-30 04:27:54 UTC
Even with my attached patch, I hit problems.
Comment 5 Jonathan Druart 2017-10-30 13:36:24 UTC
Blocked by bug 19243. We need to find a clean way to continue.
Comment 6 Jonathan Druart 2018-05-04 14:31:19 UTC
I am working on this.
Comment 7 Jonathan Druart 2018-05-04 17:53:43 UTC
Created attachment 75088 [details] [review]
Bug 19185: Use submit button if only one exists on the page

During the installation process we do not have fieldset.action, in order
to not complicate changes we are just going to use the submit button if
only one exists.
Comment 8 Jonathan Druart 2018-05-04 17:53:47 UTC
Created attachment 75089 [details] [review]
Bug 19185: Ease elements selection

- The marcflavour select needs an id to be selected easily
- "Continue to the next step" should be a button (like others)
Comment 9 Jonathan Druart 2018-05-04 17:53:50 UTC
Created attachment 75090 [details] [review]
Bug 19185: Add selenium tests for installation and onboarding process
Comment 10 Jonathan Druart 2018-05-04 17:55:40 UTC
Maybe not ready yet for integration but need feedbacks.
Comment 11 Martin Renvoize 2018-08-03 21:30:43 UTC
Comment on attachment 75090 [details] [review]
Bug 19185: Add selenium tests for installation and onboarding process

Review of attachment 75090 [details] [review]:
-----------------------------------------------------------------

::: t/db_dependent/selenium/00-installation.t
@@ +52,5 @@
> +    my $db_pass = C4::Context->config('pass');
> +
> +    $driver->get($base_url."mainpage.pl");
> +
> +    my $lang = "en"; # The idea here is to loop on all languages

Question: Why would we want to loop for all languages? Are you still intending on implementing this?
Comment 12 Martin Renvoize 2018-08-03 21:46:36 UTC
This generally looks pretty solid to me.. certainly a good first step.. i'm sure we could test the negative cases a bit more (for example using bad db credentials and checking the right errors are present), but generally speaking this looks good to me.  Of course, these tests can only be run on a brand new DB and as such make most sense when used alongside the dependency bug 19821 which given the right config will create a brand new clean database for each test run.
Comment 13 Jonathan Druart 2018-08-07 13:42:28 UTC
(In reply to Martin Renvoize from comment #11)
> Comment on attachment 75090 [details] [review] [review]
> Bug 19185: Add selenium tests for installation and onboarding process
> 
> Review of attachment 75090 [details] [review] [review]:
> -----------------------------------------------------------------
> 
> ::: t/db_dependent/selenium/00-installation.t
> @@ +52,5 @@
> > +    my $db_pass = C4::Context->config('pass');
> > +
> > +    $driver->get($base_url."mainpage.pl");
> > +
> > +    my $lang = "en"; # The idea here is to loop on all languages
> 
> Question: Why would we want to loop for all languages? Are you still
> intending on implementing this?

We should test the installer process for different languages, in the script I have listed the ones we have for the manual. The idea would be to run it for each of them.
We should also test with and without sample data (i.e. without and with onboarding tool), see commented tests.

But it's for the next steps (if it does not take too long to be part of the codebase).
Comment 14 Martin Renvoize 2018-11-02 16:27:23 UTC
Created attachment 81913 [details] [review]
Bug 19185: Use submit button if only one exists on the page

During the installation process we do not have fieldset.action, in order
to not complicate changes we are just going to use the submit button if
only one exists.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 15 Martin Renvoize 2018-11-02 16:27:26 UTC
Created attachment 81914 [details] [review]
Bug 19185: Ease elements selection

- The marcflavour select needs an id to be selected easily
- "Continue to the next step" should be a button (like others)

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 16 Martin Renvoize 2018-11-02 16:27:29 UTC
Created attachment 81915 [details] [review]
Bug 19185: Add selenium tests for installation and onboarding process

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 17 Martin Renvoize 2018-11-02 16:27:56 UTC
With a bit of perseverance, I got it to work. Signing Off
Comment 18 Martin Renvoize 2018-11-02 16:28:54 UTC
As CI maintainer, I've set Tomas to QA this one.. I think he's best placed to give it the final go ahead.
Comment 19 Katrin Fischer 2019-02-15 07:11:49 UTC
BLOCKED by FQA on the dependent bug for the moment.
Comment 20 Jonathan Druart 2019-02-17 13:52:40 UTC
Back to SO.
Comment 21 Katrin Fischer 2019-05-11 19:23:37 UTC
I've applied both bugs, bug 19821 and this one.

With Joubu's help (thx!) my Selenium tests run in general now, but these sadly still fail:

t/db_dependent/selenium/00-installation.t .. 
STRACE:	/usr/share/perl5/Try/Tiny.pm:124 in Selenium::Remote::Driver::catch {...} 
	/usr/local/share/perl/5.24.1/Selenium/Remote/Driver.pm:348 in Try::Tiny::try
	(eval 330):1 in Selenium::Remote::Driver::__ANON__
	(eval 332):2 in Selenium::Remote::Driver::__ANON__
	/usr/local/share/perl/5.24.1/Selenium/Remote/Driver.pm:1230 in Selenium::Remote::Driver::_execute_command
	/usr/local/share/perl/5.24.1/Selenium/Remote/Driver.pm:1230 in (eval)
	/home/vagrant/kohaclone/t/lib/Selenium.pm:120 in Selenium::Remote::Driver::find_element
	t/db_dependent/selenium/00-installation.t:63 in t::lib::Selenium::fill_form

sh: 1: curl: not found
malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm line 171.
# Looks like your test exited with 127 before it could output anything.
t/db_dependent/selenium/00-installation.t .. Dubious, test returned 127 (wstat 32512, 0x7f00)
Failed 2/2 subtests 

Test Summary Report
-------------------
t/db_dependent/selenium/00-installation.t (Wstat: 32512 Tests: 0 Failed: 0)
  Non-zero exit status: 127
  Parse errors: Bad plan.  You planned 2 tests but ran 0.
Comment 22 Katrin Fischer 2019-05-11 19:27:33 UTC
Hm, looks like curl was missing, installed and tried again.

It looks like it failed at logging in, did I miss a preparation step?

kohadev-koha@kohadevbox:/home/vagrant/kohaclone$ prove t/db_dependent/selenium/00-installation.t
t/db_dependent/selenium/00-installation.t .. 
STRACE:	/usr/share/perl5/Try/Tiny.pm:124 in Selenium::Remote::Driver::catch {...} 
	/usr/local/share/perl/5.24.1/Selenium/Remote/Driver.pm:348 in Try::Tiny::try
	(eval 330):1 in Selenium::Remote::Driver::__ANON__
	(eval 332):2 in Selenium::Remote::Driver::__ANON__
	/usr/local/share/perl/5.24.1/Selenium/Remote/Driver.pm:1230 in Selenium::Remote::Driver::_execute_command
	/usr/local/share/perl/5.24.1/Selenium/Remote/Driver.pm:1230 in (eval)
	/home/vagrant/kohaclone/t/lib/Selenium.pm:120 in Selenium::Remote::Driver::find_element
	t/db_dependent/selenium/00-installation.t:63 in t::lib::Selenium::fill_form


SCREENSHOT: https://framapic.org/lOB1mxx6MxFE/TSvv7dPkJkCy
An element could not be located on the page using the given search parameters: //*[@id="language"],xpath at /home/vagrant/kohaclone/t/lib/Selenium.pm line 120.
# Looks like your test exited with 255 before it could output anything.
t/db_dependent/selenium/00-installation.t .. Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 2/2 subtests 

Test Summary Report
-------------------
t/db_dependent/selenium/00-installation.t (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 2 tests but ran 0.
Comment 23 Jonathan Druart 2019-05-12 17:24:10 UTC
Test plan (for bug 19185 and bug 19821):
% prove -Mt::lib::Bootstrap t/db_dependent/selenium/00-installation.t
Config entry 'database_test' does not exist at /home/vagrant/kohaclone/t/lib/Bootstrap.pm line 18

# Edit $KOHA_CONF, add a database_test entry identical to database (then restart_all):
% prove -Mt::lib::Bootstrap t/db_dependent/selenium/00-installation.t
Entries 'database_test' and 'database' have the same value in your config at /home/vagrant/kohaclone/t/lib/Bootstrap.pm line 20.

# Edit $KOHA_CONF, edit database_test with koha_kohadev_test (then restart_all)
% prove -Mt::lib::Bootstrap t/db_dependent/selenium/00-installation.t
Access denied for user 'koha_kohadev'@'localhost' to database 'koha_kohadev_test'

# Login as the admin mysql user
MariaDB [(none)]> GRANT ALL PRIVILEGES ON `koha_kohadev_test`.* TO 'koha_kohadev'@'localhost';

% prove -Mt::lib::Bootstrap t/db_dependent/selenium/00-installation.t
t/db_dependent/selenium/00-installation.t .. ok 
All tests successful.
Files=1, Tests=2, 25 wallclock secs ( 0.02 usr  0.01 sys +  1.34 cusr  0.24 csys =  1.61 CPU)
Result: PASS
Comment 24 Katrin Fischer 2019-10-08 11:47:21 UTC
We've discussed this and the other patch set on bug 19812 during the hackfest and decided to wait until Jonathan's return to pick it up again. IIRC we were not sure if bug 19821 is really needed for this one to proceed and how they fit together. There was also a question on permissions etc. for the second database needed and how this would work for the various installation scenarios we currently support.

Jonathan, could you get in touch with me once around again? Maybe also something we could clear up in a dev meeting.
Comment 25 Jonathan Druart 2019-10-15 09:02:26 UTC
(In reply to Katrin Fischer from comment #24)
> We've discussed this and the other patch set on bug 19812 during the
> hackfest and decided to wait until Jonathan's return to pick it up again.
> IIRC we were not sure if bug 19821 is really needed for this one to proceed
> and how they fit together. There was also a question on permissions etc. for
> the second database needed and how this would work for the various
> installation scenarios we currently support.
> 
> Jonathan, could you get in touch with me once around again? Maybe also
> something we could clear up in a dev meeting.

Hi Katrin,
We definitely need bug 19821 to make those tests work correctly. Did you write the questions down?