Bug 13849 - Introduce acceptance tests with cucumber
Summary: Introduce acceptance tests with cucumber
Status: RESOLVED WONTFIX
Alias: None
Product: Koha
Classification: Unclassified
Component: Test Suite (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Chris Cormack
QA Contact: Galen Charlton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-03-17 09:41 UTC by Jonathan Druart
Modified: 2023-06-21 12:06 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 13849: POC - my first Cucumber contact (13.22 KB, patch)
2015-03-17 09:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
scenarioNumberForTableDrivenScenarios. Keep track of the Scenario data array location to get data as HASH instead of ARRAY. (1.52 KB, text/plain)
2015-03-17 10:14 UTC, Olli-Antti Kivilahti
Details
Example Cucumber tests for the overdues rewrite (18.26 KB, application/octet-stream)
2015-04-09 14:17 UTC, Olli-Antti Kivilahti
Details
Trigger script for Cucumber tests suite (3.25 KB, text/plain)
2015-04-11 13:58 UTC, Olli-Antti Kivilahti
Details
Bug 13849 : Adding Cucumber tests to create a new biblio (6.13 KB, patch)
2015-06-08 08:05 UTC, Julian FIOL
Details | Diff | Splinter Review
Cucumber test suite Vaara-style (25.13 KB, application/octet-stream)
2015-06-10 13:53 UTC, Olli-Antti Kivilahti
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2015-03-17 09:41:07 UTC
During the hackfest 15 in Marseille, akafred present us how he integrated acceptance tests in Koha using Cucumber.
I would like to get developer's opinions on the need of this kind of tests into Koha.
Comment 1 Jonathan Druart 2015-03-17 09:43:09 UTC
Created attachment 36940 [details] [review]
Bug 13849: POC - my first Cucumber contact

During the hackfest 15 in Marseille, akafred present us how he
integrated acceptance tests in Koha using Cucumber.

With this patch, I would like to get other developer opinions on the
usefulness of this kind of tests for the Koha software.

To test:
0/
* Install the selenium server (have a look at bug 13691)
* Install libtest-bdd-cucumber-perl (pkg) and App::pherkin (cpan)
* Edit t/db_dependent/cucumber/features/step_definitions/basic_steps.pl
and fill the login and password of a superlibrarian user.
Maybe you should also change the address of the selenium server (search
for 'remote_server_addr').

1/ Start the selenium server and launch the tests:
$ cd t/db_dependent/cucumber
$ pherkin
  or
$ prove tests.t

Even if the second method is recommended in
Test::BDD::Cucumber::Manual::Integration, I prefer the first one
Indeed, the tests will pass if a step is not defined (maybe something is
wrong somewhere...).
Comment 2 Olli-Antti Kivilahti 2015-03-17 10:13:10 UTC
Hi there!
Awesome that somebody else is in flames about Cucumber!

I have been writing the Cucumber tests into another directory structure, and I have encountered some issues with it.

My file hierarchy:

kohaclone/t/Cucumber/runCucumberTests.pl #Binds *::Cucumber::Manual::Integration
                    #And runs all Cucumber tests.
kohaclone/t/Cucumber/Overdues/features/*.feature   #Feature definitions
kohaclone/t/Cucumber/Overdues/features/step_definitions/*_steps.pl #Step handlers
kohaclone/t/Cucumber/TestObjects/Koha/Overdues/*.pm #Packages to actually
                    #implement the test code. I cannot set debugger breakpoints
                    #into the step-files, because of how the Test::BDD-modules
                    #load them as code closures, so I implement the business logic
                    #in the TestObjects. Return the comparable values and run the
                    #Test::More function on the returned values, eg two objects or
                    #an error code.


Note regarding passing test data arrays in the Scenario description:
    Examples:
      | branchCode | borrowerCategory | letterNumber |
      |            | S                | 1            |
      |            | S                | 2            |
      |            | S                | 3            |
      |            | PT               | 1            |
      |            | PT               | 2            |
      |            | PT               | 3            |
      | CPL        | PT               | 1            |
      | CPL        | PT               | 2            |
      | FTL        | YA               | 1            |
      | FTL        | YA               | 2            |
      | FTL        | YA               | 3            |

Instead of getting the array values from the step-definition regexp, you can get them from the StepContext-object:
See my attachment scenarioNumberForTableDriveScenarios.txt
Comment 3 Olli-Antti Kivilahti 2015-03-17 10:14:27 UTC Comment hidden (obsolete)
Comment 4 Olli-Antti Kivilahti 2015-04-09 14:17:30 UTC
Created attachment 37622 [details]
Example Cucumber tests for the overdues rewrite

Having written a new overdue gathering and sending module and using Cucumber tests, here are the test features I used.
Comment 5 Olli-Antti Kivilahti 2015-04-11 13:58:39 UTC
Created attachment 37678 [details]
Trigger script for Cucumber tests suite
Comment 6 Olli-Antti Kivilahti 2015-05-26 15:29:26 UTC
Some tutorial on how to do Cukes in Koha!
http://pastebin.com/3Tya8kwR
Comment 7 Julian FIOL 2015-05-27 12:11:30 UTC
To test:
0/
* Install the selenium server (have a look at bug 13691)
* Install libtest-bdd-cucumber-perl (pkg) and App::pherkin (cpan)
* Edit t/db_dependent/cucumber/features/step_definitions/basic_steps.pl
and fill the login and password of a superlibrarian user.
Maybe you should also change the address of the selenium server (search
for 'remote_server_addr').

1/ Start the selenium server and launch the tests:
$ cd t/db_dependent/cucumber
$ pherkin
  or
$ prove tests.t

Hi Jonathan, I tried your patch using pherkin and prove, and I have the same issue at both times :

  Create a Category is OK
  [...]

  But create a patron crash after submit :

  Create a patron
    As a librarian
    I want to create a new patron

    Scenario: Create a patron
      Given I am logged in as a library admin
      Given I have gone to the new patron page
      When I have filled the field surname with "test_patron_surname"
      And I have filled the field cardnumber with "4242424242"
      And I have filled the field userid with "test_username"
      And I have filled the field password with "password"
      And I have filled the field password2 with "password"
      And I have clicked on the submit button
      Then The patron "test_username" should have been created
        step defined at features/1-patron.feature line 22.
        ok 1 - Starting to execute step: The patron "test_username" should have been created
        not ok 2 - The patron test_username should exist
        #   Failed test 'The patron test_username should exist'
        #   at features/step_definitions/1-patron_steps.pl line 12.
        #          got: '0'
        #     expected: '1'
        1..2
Comment 8 Jonathan Druart 2015-05-27 14:18:04 UTC
All works perfectly here.
Make sure you have a superlibrarian user "koha" with pwd "koha" or edit features/step_definitions/basic_steps.pl
Comment 9 Julian FIOL 2015-06-01 08:12:16 UTC
Works fine now
My problem was the 'BorrowerMandatoryField' syspref which was waiting more fields filled than tests did.
Comment 10 Julian FIOL 2015-06-08 08:05:59 UTC
Created attachment 39983 [details] [review]
Bug 13849 : Adding Cucumber tests to create a new biblio

To test:
 0/
    * Install the selenium server (have a look at bug 13691)
    * Install libtest-bdd-cucumber-perl (pkg) and App::pherkin (cpan)
    * Edit t/db_dependent/cucumber/features/step_definitions/basic_steps.pl
    and fill the login and password of a superlibrarian user.
    Maybe you should also change the address of the selenium server (search
    for 'remote_server_addr').

 1/ Start the selenium server and launch the tests:
    $ cd t/db_dependent/cucumber
    $ pherkin
      or
    $ prove tests.t

This patch add a new scenario creating a new biblio
Comment 11 Olli-Antti Kivilahti 2015-06-10 13:53:13 UTC
Created attachment 40070 [details]
Cucumber test suite Vaara-style

I discussed with Oslo about how to best organize these Cucumber tests.
They hinted that we should use Page object pattern to do these Cucumber/Selenium tests.
Which is a jolly good idea!
I haven't had time to do any of those Page Object Pattern-driven objects for hiding ugly implementation specifics for web-crawling.

I also have some new features in production utilizing Cucumber tests and I started to encounter some test structuring issues.
So I cooked up a better way to organize the test components.
See testsStructuring.tutorial in the .tar.gz
This is 100% compatible with the runCucumberTests.t-script and IMHO makes for a much more beautiful code structuring than the way of doing things in the Perl documentation for Cucumber.
Comment 12 David Cook 2021-04-14 00:23:44 UTC
I don't see any obvious advantage to including Cucumber tests but happy to be convinced otherwise. 

Is the idea that the test reports are prettier and the scenarios more verbose?
Comment 13 Jonathan Druart 2021-04-14 08:23:16 UTC
It's no longer relevant (to me).

The idea was to let anybody write tests, even without technical skills.
Another goal was automatically generate (internationalized) screenshots for the manual
https://lists.katipo.co.nz/pipermail/koha/2018-January/049807.html
Comment 14 Jonathan Druart 2023-06-21 12:05:35 UTC Comment hidden (obsolete)
Comment 15 Jonathan Druart 2023-06-21 12:06:21 UTC
Abandoned. See bug 34076.