Bug 27055

Summary: Update Firefox version used in Selenium GUI tests
Product: Koha Reporter: Victor Grousset/tuxayo <victor>
Component: Test SuiteAssignee: Victor Grousset/tuxayo <victor>
Status: CLOSED FIXED QA Contact: Tomás Cohen Arazi <tomascohen>
Severity: major    
Priority: P5 - low CC: andrewfh, caroline.cyr-la-rose, dcook, fridolin.somers, jonathan.druart, mtj, tomascohen
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
21.05.00,20.11.01,20.05.07,19.11.14
Bug Depends on:    
Bug Blocks: 19384, 29285    
Attachments: WIP Bug 27055: Fix compatibility with newer Selenium+Firefox version
WIP Bug 27055: Fix compatibility with newer Selenium+Firefox version
Bug 27055: Fix compatibility with newer Firefox+Selenium version
Bug 27055: Fix compatibility with newer Firefox+Selenium version
Bug 27055: Fix compatibility with newer Firefox+Selenium version

Description Victor Grousset/tuxayo 2020-11-19 11:56:11 UTC
It's currently 47, if it stays like that, we will have to drop these tests at some point.

Followup of https://gitlab.com/koha-community/koha-testing-docker/-/issues/203

Past update attempt failed due to like incompatibility/bug between recent Selenium version and our Perl Selenium library (Selenium::Remote::Driver)
Comment 1 Victor Grousset/tuxayo 2020-11-19 13:03:40 UTC Comment hidden (obsolete)
Comment 2 Victor Grousset/tuxayo 2020-11-23 13:15:10 UTC Comment hidden (obsolete)
Comment 3 Victor Grousset/tuxayo 2020-11-24 15:17:29 UTC
Got a problem of /dev/shm getting full in the container after a few runs of tests.

Tried to reproduce and found another issue instead (which might affect us anyway)
"quit() hangs while Firefox is at 100% CPU"
https://github.com/SeleniumHQ/docker-selenium/issues/1146
Comment 4 Victor Grousset/tuxayo 2020-11-24 16:57:54 UTC
> Got a problem of /dev/shm getting full in the container after a few runs of tests.

Ok it's when we don't call quit() a firefox stays there and in recent versions, it uses /dev/shm.

TODO follow up to ensure all test files call quit() in the end.
Comment 5 Victor Grousset/tuxayo 2020-11-26 12:19:11 UTC
Created attachment 114005 [details] [review]
Bug 27055: Fix compatibility with newer Firefox+Selenium version

Fix "submit is not a function error"
A submit button should not be named "submit", in this case, it's id.
https://stackoverflow.com/questions/833032/submit-is-not-a-function-error-in-javascript

Fix some uses of get_attribute()

Fix a fail by setting a global implicit_wait_timeout, default value is 0
in our lib. Other libs set it higher which helps to not have to manually
deal with part of the timing issues.

Fix: remove usage of click_when_visible() because it doesn't work with
elements not in the top of the page. Because they are off screen.

Fix: use $driver->quit() in error_handler to not forget an open Firefox.
With the current version, it fills /dev/shm and fails with around 5
Firefox opened.
Also use quit() it at the end of every script.

Fix: filling item fields, to fill only the displayed one (not those
with display:none)

== Test plan ==
1. Update selenium/standalone-firefox to the latest version [1]
2. prove t/db_dependent/selenium/authentication.t
3. It fails with: arguments[0].form.submit is not a function
4. Apply patch
5. Retest
6. Success

[1] In koha-testing-docker you can do it with
docker-compose.yml:
     selenium:
-        image: selenium/standalone-firefox:2.53.1-americium
+        image: selenium/standalone-firefox
Comment 6 Victor Grousset/tuxayo 2020-11-26 12:36:31 UTC
All Selenium tests were run more than 120 times and no flakiness so far
Comment 7 Victor Grousset/tuxayo 2020-11-26 13:00:58 UTC
Steps after push to master:

koha-testing-docker:
docker-compose.yml:

     selenium:
-        image: selenium/standalone-firefox:2.53.1-americium
+        image: selenium/standalone-firefox

KohaDevBox:
Update Selenium URL
https://gitlab.com/koha-community/kohadevbox/-/blob/5e9b27d1cb8249431c46215b183c95272239e371/vars/defaults.yml#L30
Comment 8 Jonathan Druart 2020-12-01 09:19:43 UTC
Victor, I don't understand some of the changes, can you explain them?

-    $self->click_when_visible( $default_submit_selector );
+    $self->driver->find_element($default_submit_selector)->click
and
-    $self->click_when_visible( $xpath_selector );
+    $self->driver->find_element($xpath_selector)->click

From the commit message:
    Fix: remove usage of click_when_visible() because it doesn't work with
    elements not in the top of the page. Because they are off screen.

We are certainly going to reintroduce some random issues we tried to fix during the last cycle.

What are the elements at the top, and why are they out of the screen?
Comment 9 Jonathan Druart 2020-12-01 09:23:34 UTC
Tests are passing but I see lot of occurrences of this error:

selenium_1   | [Child 3211, Main Thread] WARNING: fallocate failed to set shm size: No space left on device: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/shared_memory_posix.cc:184
Comment 10 Victor Grousset/tuxayo 2020-12-01 15:17:29 UTC
>     Fix: remove usage of click_when_visible() because it doesn't work with
>    elements not in the top of the page. Because they are off screen.

click_when_visible() uses wait_for_element_visible() which now use the current viewport for it's definition of visibility.
Thus with elements out of the screen - lower in the page - wait_for_element_visible() waits forever.

> We are certainly going to reintroduce some random issues we tried to fix during the last cycle.

Before, the implicit_wait_timeout was 0, now it's 5sec, hopefully that should catch the the bulk of flaky tests. The whole suite was stable during more than 150 runs locally.

> fallocate failed to set shm size: No space left on device

This is due to a small /dev/shm and multiple Firefox staying open. Which shouldn't happen as close() is called at the end of each file. And in the error_handler.

Remaining Firefox instances can be checked with:
docker exec -it koha_selenium_1 bash
df -h
top

Did you interrupt (ctrl-c) the runs several times? Or was it on a freshly started env on the first runs of the tests? That's only way I know for now about how to hit this issue with the new selenium container.

-------


Good news, it's possible to change size of /dev/shm :D

docker-compose.yml
     selenium:
         image: selenium/standalone-firefox
+        shm_size: 1gb

This seems to be a recurring problem for running browsers in docker. As the default is only 64MiB. Testing shows that it doesn't increase the RAM usage, it might work like tmpfs thus only using more RAM as it's filled up.

References for the size change:
https://stackoverflow.com/questions/58952478/changing-shared-memory-size-in-docker-compose
https://docs.docker.com/compose/compose-file/#shm_size
Comment 11 Victor Grousset/tuxayo 2020-12-01 19:06:28 UTC
Merge requests to update koha-testing-docker and KohaDevBox
https://gitlab.com/koha-community/koha-testing-docker/-/merge_requests/184
https://gitlab.com/koha-community/kohadevbox/-/merge_requests/285
Comment 12 Mason James 2020-12-22 05:28:08 UTC
Created attachment 114584 [details] [review]
Bug 27055: Fix compatibility with newer Firefox+Selenium version

Fix "submit is not a function error"
A submit button should not be named "submit", in this case, it's id.
https://stackoverflow.com/questions/833032/submit-is-not-a-function-error-in-javascript

Fix some uses of get_attribute()

Fix a fail by setting a global implicit_wait_timeout, default value is 0
in our lib. Other libs set it higher which helps to not have to manually
deal with part of the timing issues.

Fix: remove usage of click_when_visible() because it doesn't work with
elements not in the top of the page. Because they are off screen.

Fix: use $driver->quit() in error_handler to not forget an open Firefox.
With the current version, it fills /dev/shm and fails with around 5
Firefox opened.
Also use quit() it at the end of every script.

Fix: filling item fields, to fill only the displayed one (not those
with display:none)

== Test plan ==
1. Update selenium/standalone-firefox to the latest version [1]
2. prove t/db_dependent/selenium/authentication.t
3. It fails with: arguments[0].form.submit is not a function
4. Apply patch
5. Retest
6. Success

[1] In koha-testing-docker you can do it with
docker-compose.yml:
     selenium:
-        image: selenium/standalone-firefox:2.53.1-americium
+        image: selenium/standalone-firefox

Signed-off-by: Mason James <mtj@kohaaloha.com>
Comment 13 Mason James 2020-12-22 05:32:14 UTC
this patch is working well for me with the following patch applied...
 https://gitlab.com/koha-community/koha-testing-docker/-/merge_requests/184

so signing off :)
Comment 14 Tomás Cohen Arazi 2020-12-29 15:18:45 UTC
Created attachment 114725 [details] [review]
Bug 27055: Fix compatibility with newer Firefox+Selenium version

Fix "submit is not a function error"
A submit button should not be named "submit", in this case, it's id.
https://stackoverflow.com/questions/833032/submit-is-not-a-function-error-in-javascript

Fix some uses of get_attribute()

Fix a fail by setting a global implicit_wait_timeout, default value is 0
in our lib. Other libs set it higher which helps to not have to manually
deal with part of the timing issues.

Fix: remove usage of click_when_visible() because it doesn't work with
elements not in the top of the page. Because they are off screen.

Fix: use $driver->quit() in error_handler to not forget an open Firefox.
With the current version, it fills /dev/shm and fails with around 5
Firefox opened.
Also use quit() it at the end of every script.

Fix: filling item fields, to fill only the displayed one (not those
with display:none)

== Test plan ==
1. Update selenium/standalone-firefox to the latest version [1]
2. prove t/db_dependent/selenium/authentication.t
3. It fails with: arguments[0].form.submit is not a function
4. Apply patch
5. Retest
6. Success

[1] In koha-testing-docker you can do it with
docker-compose.yml:
     selenium:
-        image: selenium/standalone-firefox:2.53.1-americium
+        image: selenium/standalone-firefox

Signed-off-by: Mason James <mtj@kohaaloha.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 15 Tomás Cohen Arazi 2020-12-29 15:20:18 UTC
I tested this with the latest selenium-firefox image.
It works after the patch is applied.

The changes are sensible. Specially the submit id change. And how things got adjusted makes sense.

QA script is happy as well.

Well done!
Comment 16 Tomás Cohen Arazi 2020-12-30 15:06:37 UTC
Pushed, to end the year with happy Jenkins.

Thanks everyone
Comment 17 Fridolin Somers 2021-01-05 07:51:11 UTC
Pushed to 20.11.x for 20.11.01
Comment 18 Andrew Fuerste-Henry 2021-01-05 13:51:12 UTC
Pushed to 20.05.x for 20.05.07
Comment 19 Victor Grousset/tuxayo 2021-01-14 18:25:39 UTC
Backported: Pushed to 19.11.x branch for 19.11.14