Bug 13849 - Introduce acceptance tests with cucumber
Summary: Introduce acceptance tests with cucumber
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: Test Suite (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Jonathan Druart
QA Contact: Galen Charlton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-03-17 09:41 UTC by Jonathan Druart
Modified: 2015-06-10 13:53 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 13849: POC - my first Cucumber contact (13.22 KB, patch)
2015-03-17 09:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
scenarioNumberForTableDrivenScenarios. Keep track of the Scenario data array location to get data as HASH instead of ARRAY. (1.52 KB, text/plain)
2015-03-17 10:14 UTC, Olli-Antti Kivilahti
Details
Example Cucumber tests for the overdues rewrite (18.26 KB, application/octet-stream)
2015-04-09 14:17 UTC, Olli-Antti Kivilahti
Details
Trigger script for Cucumber tests suite (3.25 KB, text/plain)
2015-04-11 13:58 UTC, Olli-Antti Kivilahti
Details
Bug 13849 : Adding Cucumber tests to create a new biblio (6.13 KB, patch)
2015-06-08 08:05 UTC, Julian FIOL
Details | Diff | Splinter Review
Cucumber test suite Vaara-style (25.13 KB, application/octet-stream)
2015-06-10 13:53 UTC, Olli-Antti Kivilahti
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2015-03-17 09:41:07 UTC
During the hackfest 15 in Marseille, akafred present us how he integrated acceptance tests in Koha using Cucumber.
I would like to get developer's opinions on the need of this kind of tests into Koha.
Comment 1 Jonathan Druart 2015-03-17 09:43:09 UTC
Created attachment 36940 [details] [review]
Bug 13849: POC - my first Cucumber contact

During the hackfest 15 in Marseille, akafred present us how he
integrated acceptance tests in Koha using Cucumber.

With this patch, I would like to get other developer opinions on the
usefulness of this kind of tests for the Koha software.

To test:
0/
* Install the selenium server (have a look at bug 13691)
* Install libtest-bdd-cucumber-perl (pkg) and App::pherkin (cpan)
* Edit t/db_dependent/cucumber/features/step_definitions/basic_steps.pl
and fill the login and password of a superlibrarian user.
Maybe you should also change the address of the selenium server (search
for 'remote_server_addr').

1/ Start the selenium server and launch the tests:
$ cd t/db_dependent/cucumber
$ pherkin
  or
$ prove tests.t

Even if the second method is recommended in
Test::BDD::Cucumber::Manual::Integration, I prefer the first one
Indeed, the tests will pass if a step is not defined (maybe something is
wrong somewhere...).
Comment 2 Olli-Antti Kivilahti 2015-03-17 10:13:10 UTC
Hi there!
Awesome that somebody else is in flames about Cucumber!

I have been writing the Cucumber tests into another directory structure, and I have encountered some issues with it.

My file hierarchy:

kohaclone/t/Cucumber/runCucumberTests.pl #Binds *::Cucumber::Manual::Integration
                    #And runs all Cucumber tests.
kohaclone/t/Cucumber/Overdues/features/*.feature   #Feature definitions
kohaclone/t/Cucumber/Overdues/features/step_definitions/*_steps.pl #Step handlers
kohaclone/t/Cucumber/TestObjects/Koha/Overdues/*.pm #Packages to actually
                    #implement the test code. I cannot set debugger breakpoints
                    #into the step-files, because of how the Test::BDD-modules
                    #load them as code closures, so I implement the business logic
                    #in the TestObjects. Return the comparable values and run the
                    #Test::More function on the returned values, eg two objects or
                    #an error code.


Note regarding passing test data arrays in the Scenario description:
    Examples:
      | branchCode | borrowerCategory | letterNumber |
      |            | S                | 1            |
      |            | S                | 2            |
      |            | S                | 3            |
      |            | PT               | 1            |
      |            | PT               | 2            |
      |            | PT               | 3            |
      | CPL        | PT               | 1            |
      | CPL        | PT               | 2            |
      | FTL        | YA               | 1            |
      | FTL        | YA               | 2            |
      | FTL        | YA               | 3            |

Instead of getting the array values from the step-definition regexp, you can get them from the StepContext-object:
See my attachment scenarioNumberForTableDriveScenarios.txt
Comment 3 Olli-Antti Kivilahti 2015-03-17 10:14:27 UTC Comment hidden (obsolete)
Comment 4 Olli-Antti Kivilahti 2015-04-09 14:17:30 UTC
Created attachment 37622 [details]
Example Cucumber tests for the overdues rewrite

Having written a new overdue gathering and sending module and using Cucumber tests, here are the test features I used.
Comment 5 Olli-Antti Kivilahti 2015-04-11 13:58:39 UTC
Created attachment 37678 [details]
Trigger script for Cucumber tests suite
Comment 6 Olli-Antti Kivilahti 2015-05-26 15:29:26 UTC
Some tutorial on how to do Cukes in Koha!
http://pastebin.com/3Tya8kwR
Comment 7 Julian FIOL 2015-05-27 12:11:30 UTC
To test:
0/
* Install the selenium server (have a look at bug 13691)
* Install libtest-bdd-cucumber-perl (pkg) and App::pherkin (cpan)
* Edit t/db_dependent/cucumber/features/step_definitions/basic_steps.pl
and fill the login and password of a superlibrarian user.
Maybe you should also change the address of the selenium server (search
for 'remote_server_addr').

1/ Start the selenium server and launch the tests:
$ cd t/db_dependent/cucumber
$ pherkin
  or
$ prove tests.t

Hi Jonathan, I tried your patch using pherkin and prove, and I have the same issue at both times :

  Create a Category is OK
  [...]

  But create a patron crash after submit :

  Create a patron
    As a librarian
    I want to create a new patron

    Scenario: Create a patron
      Given I am logged in as a library admin
      Given I have gone to the new patron page
      When I have filled the field surname with "test_patron_surname"
      And I have filled the field cardnumber with "4242424242"
      And I have filled the field userid with "test_username"
      And I have filled the field password with "password"
      And I have filled the field password2 with "password"
      And I have clicked on the submit button
      Then The patron "test_username" should have been created
        step defined at features/1-patron.feature line 22.
        ok 1 - Starting to execute step: The patron "test_username" should have been created
        not ok 2 - The patron test_username should exist
        #   Failed test 'The patron test_username should exist'
        #   at features/step_definitions/1-patron_steps.pl line 12.
        #          got: '0'
        #     expected: '1'
        1..2
Comment 8 Jonathan Druart 2015-05-27 14:18:04 UTC
All works perfectly here.
Make sure you have a superlibrarian user "koha" with pwd "koha" or edit features/step_definitions/basic_steps.pl
Comment 9 Julian FIOL 2015-06-01 08:12:16 UTC
Works fine now
My problem was the 'BorrowerMandatoryField' syspref which was waiting more fields filled than tests did.
Comment 10 Julian FIOL 2015-06-08 08:05:59 UTC
Created attachment 39983 [details] [review]
Bug 13849 : Adding Cucumber tests to create a new biblio

To test:
 0/
    * Install the selenium server (have a look at bug 13691)
    * Install libtest-bdd-cucumber-perl (pkg) and App::pherkin (cpan)
    * Edit t/db_dependent/cucumber/features/step_definitions/basic_steps.pl
    and fill the login and password of a superlibrarian user.
    Maybe you should also change the address of the selenium server (search
    for 'remote_server_addr').

 1/ Start the selenium server and launch the tests:
    $ cd t/db_dependent/cucumber
    $ pherkin
      or
    $ prove tests.t

This patch add a new scenario creating a new biblio
Comment 11 Olli-Antti Kivilahti 2015-06-10 13:53:13 UTC
Created attachment 40070 [details]
Cucumber test suite Vaara-style

I discussed with Oslo about how to best organize these Cucumber tests.
They hinted that we should use Page object pattern to do these Cucumber/Selenium tests.
Which is a jolly good idea!
I haven't had time to do any of those Page Object Pattern-driven objects for hiding ugly implementation specifics for web-crawling.

I also have some new features in production utilizing Cucumber tests and I started to encounter some test structuring issues.
So I cooked up a better way to organize the test components.
See testsStructuring.tutorial in the .tar.gz
This is 100% compatible with the runCucumberTests.t-script and IMHO makes for a much more beautiful code structuring than the way of doing things in the Perl documentation for Cucumber.