It's currently 47, if it stays like that, we will have to drop these tests at some point. Followup of https://gitlab.com/koha-community/koha-testing-docker/-/issues/203 Past update attempt failed due to like incompatibility/bug between recent Selenium version and our Perl Selenium library (Selenium::Remote::Driver)
Created attachment 113810 [details] [review] WIP Bug 27055: Fix compatibility with newer Selenium+Firefox version Fix "submit is not a function error" A submit button should be named "submit", in this case, it's id. https://stackoverflow.com/questions/833032/submit-is-not-a-function-error-in-javascript WIP: 1. Fix SCSS which used the id=submit 2. Fix other tests, there are major issues like crash at the beginning or endless duration == Test plan == 1. Update selenium/standalone-firefox to the latest version [1] 2. prove t/db_dependent/selenium/authentication.t 3. It fails with: arguments[0].form.submit is not a function 4. Apply patch 5. Retest 6. Success [1] In koha-testing-docker you can do it with docker-compose.yml selenium: - image: selenium/standalone-firefox:2.53.1-americium + image: selenium/standalone-firefox
Created attachment 113922 [details] [review] WIP Bug 27055: Fix compatibility with newer Selenium+Firefox version Fix "submit is not a function error" A submit button should be named "submit", in this case, it's id. https://stackoverflow.com/questions/833032/submit-is-not-a-function-error-in-javascript Fix some uses of get_attribute() Fix with wait_for_element_visible() WIP: 1. Fix SCSS which used the id=submit 2. Fix other tests, there are major issues like crash at the beginning or endless duration == Test plan == 1. Update selenium/standalone-firefox to the latest version [1] 2. prove t/db_dependent/selenium/authentication.t 3. It fails with: arguments[0].form.submit is not a function 4. Apply patch 5. Retest 6. Success [1] In koha-testing-docker you can do it with docker-compose.yml selenium: - image: selenium/standalone-firefox:2.53.1-americium + image: selenium/standalone-firefox
Got a problem of /dev/shm getting full in the container after a few runs of tests. Tried to reproduce and found another issue instead (which might affect us anyway) "quit() hangs while Firefox is at 100% CPU" https://github.com/SeleniumHQ/docker-selenium/issues/1146
> Got a problem of /dev/shm getting full in the container after a few runs of tests. Ok it's when we don't call quit() a firefox stays there and in recent versions, it uses /dev/shm. TODO follow up to ensure all test files call quit() in the end.
Created attachment 114005 [details] [review] Bug 27055: Fix compatibility with newer Firefox+Selenium version Fix "submit is not a function error" A submit button should not be named "submit", in this case, it's id. https://stackoverflow.com/questions/833032/submit-is-not-a-function-error-in-javascript Fix some uses of get_attribute() Fix a fail by setting a global implicit_wait_timeout, default value is 0 in our lib. Other libs set it higher which helps to not have to manually deal with part of the timing issues. Fix: remove usage of click_when_visible() because it doesn't work with elements not in the top of the page. Because they are off screen. Fix: use $driver->quit() in error_handler to not forget an open Firefox. With the current version, it fills /dev/shm and fails with around 5 Firefox opened. Also use quit() it at the end of every script. Fix: filling item fields, to fill only the displayed one (not those with display:none) == Test plan == 1. Update selenium/standalone-firefox to the latest version [1] 2. prove t/db_dependent/selenium/authentication.t 3. It fails with: arguments[0].form.submit is not a function 4. Apply patch 5. Retest 6. Success [1] In koha-testing-docker you can do it with docker-compose.yml: selenium: - image: selenium/standalone-firefox:2.53.1-americium + image: selenium/standalone-firefox
All Selenium tests were run more than 120 times and no flakiness so far
Steps after push to master: koha-testing-docker: docker-compose.yml: selenium: - image: selenium/standalone-firefox:2.53.1-americium + image: selenium/standalone-firefox KohaDevBox: Update Selenium URL https://gitlab.com/koha-community/kohadevbox/-/blob/5e9b27d1cb8249431c46215b183c95272239e371/vars/defaults.yml#L30
Victor, I don't understand some of the changes, can you explain them? - $self->click_when_visible( $default_submit_selector ); + $self->driver->find_element($default_submit_selector)->click and - $self->click_when_visible( $xpath_selector ); + $self->driver->find_element($xpath_selector)->click From the commit message: Fix: remove usage of click_when_visible() because it doesn't work with elements not in the top of the page. Because they are off screen. We are certainly going to reintroduce some random issues we tried to fix during the last cycle. What are the elements at the top, and why are they out of the screen?
Tests are passing but I see lot of occurrences of this error: selenium_1 | [Child 3211, Main Thread] WARNING: fallocate failed to set shm size: No space left on device: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/shared_memory_posix.cc:184
> Fix: remove usage of click_when_visible() because it doesn't work with > elements not in the top of the page. Because they are off screen. click_when_visible() uses wait_for_element_visible() which now use the current viewport for it's definition of visibility. Thus with elements out of the screen - lower in the page - wait_for_element_visible() waits forever. > We are certainly going to reintroduce some random issues we tried to fix during the last cycle. Before, the implicit_wait_timeout was 0, now it's 5sec, hopefully that should catch the the bulk of flaky tests. The whole suite was stable during more than 150 runs locally. > fallocate failed to set shm size: No space left on device This is due to a small /dev/shm and multiple Firefox staying open. Which shouldn't happen as close() is called at the end of each file. And in the error_handler. Remaining Firefox instances can be checked with: docker exec -it koha_selenium_1 bash df -h top Did you interrupt (ctrl-c) the runs several times? Or was it on a freshly started env on the first runs of the tests? That's only way I know for now about how to hit this issue with the new selenium container. ------- Good news, it's possible to change size of /dev/shm :D docker-compose.yml selenium: image: selenium/standalone-firefox + shm_size: 1gb This seems to be a recurring problem for running browsers in docker. As the default is only 64MiB. Testing shows that it doesn't increase the RAM usage, it might work like tmpfs thus only using more RAM as it's filled up. References for the size change: https://stackoverflow.com/questions/58952478/changing-shared-memory-size-in-docker-compose https://docs.docker.com/compose/compose-file/#shm_size
Merge requests to update koha-testing-docker and KohaDevBox https://gitlab.com/koha-community/koha-testing-docker/-/merge_requests/184 https://gitlab.com/koha-community/kohadevbox/-/merge_requests/285
Created attachment 114584 [details] [review] Bug 27055: Fix compatibility with newer Firefox+Selenium version Fix "submit is not a function error" A submit button should not be named "submit", in this case, it's id. https://stackoverflow.com/questions/833032/submit-is-not-a-function-error-in-javascript Fix some uses of get_attribute() Fix a fail by setting a global implicit_wait_timeout, default value is 0 in our lib. Other libs set it higher which helps to not have to manually deal with part of the timing issues. Fix: remove usage of click_when_visible() because it doesn't work with elements not in the top of the page. Because they are off screen. Fix: use $driver->quit() in error_handler to not forget an open Firefox. With the current version, it fills /dev/shm and fails with around 5 Firefox opened. Also use quit() it at the end of every script. Fix: filling item fields, to fill only the displayed one (not those with display:none) == Test plan == 1. Update selenium/standalone-firefox to the latest version [1] 2. prove t/db_dependent/selenium/authentication.t 3. It fails with: arguments[0].form.submit is not a function 4. Apply patch 5. Retest 6. Success [1] In koha-testing-docker you can do it with docker-compose.yml: selenium: - image: selenium/standalone-firefox:2.53.1-americium + image: selenium/standalone-firefox Signed-off-by: Mason James <mtj@kohaaloha.com>
this patch is working well for me with the following patch applied... https://gitlab.com/koha-community/koha-testing-docker/-/merge_requests/184 so signing off :)
Created attachment 114725 [details] [review] Bug 27055: Fix compatibility with newer Firefox+Selenium version Fix "submit is not a function error" A submit button should not be named "submit", in this case, it's id. https://stackoverflow.com/questions/833032/submit-is-not-a-function-error-in-javascript Fix some uses of get_attribute() Fix a fail by setting a global implicit_wait_timeout, default value is 0 in our lib. Other libs set it higher which helps to not have to manually deal with part of the timing issues. Fix: remove usage of click_when_visible() because it doesn't work with elements not in the top of the page. Because they are off screen. Fix: use $driver->quit() in error_handler to not forget an open Firefox. With the current version, it fills /dev/shm and fails with around 5 Firefox opened. Also use quit() it at the end of every script. Fix: filling item fields, to fill only the displayed one (not those with display:none) == Test plan == 1. Update selenium/standalone-firefox to the latest version [1] 2. prove t/db_dependent/selenium/authentication.t 3. It fails with: arguments[0].form.submit is not a function 4. Apply patch 5. Retest 6. Success [1] In koha-testing-docker you can do it with docker-compose.yml: selenium: - image: selenium/standalone-firefox:2.53.1-americium + image: selenium/standalone-firefox Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I tested this with the latest selenium-firefox image. It works after the patch is applied. The changes are sensible. Specially the submit id change. And how things got adjusted makes sense. QA script is happy as well. Well done!
Pushed, to end the year with happy Jenkins. Thanks everyone
Pushed to 20.11.x for 20.11.01
Pushed to 20.05.x for 20.05.07
Backported: Pushed to 19.11.x branch for 19.11.14