Description
Mason James
2017-02-05 01:44:42 UTC
Created attachment 59900 [details] [review] Bug 18055 - Speed up '00-strict.t' test, with Parallel::ForkManager to test... 1/ run 00-strict.t test, (16.5 mins on a 4xcpu system) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 994 wallclock secs ( 0.19 usr 0.04 sys + 873.40 cusr 116.20 csys = 989.83 CPU) Result: PASS real 16m34.104s 2/ apply patch 3/ install Parallel::ForkManager package $ sudo apt-get install libparallel-forkmanager-perl 4/ run 00-strict.t test again, (now 6 mins.. much faster) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 364 wallclock secs ( 0.07 usr 0.01 sys + 1159.20 cusr 153.41 csys = 1312.69 CPU) Result: PASS real 6m4.355s Created attachment 59901 [details] [review] Add Sys::CPU package, too Created attachment 59902 [details] [review] Bug 18055 - Speed up '00-strict.t' test, with Parallel::ForkManager to test... 1/ run 00-strict.t test, (16.5 mins on a 4xcpu system) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 994 wallclock secs ( 0.19 usr 0.04 sys + 873.40 cusr 116.20 csys = 989.83 CPU) Result: PASS real 16m34.104s 2/ apply patch 3/ install Parallel::ForkManager package $ sudo apt-get install libparallel-forkmanager-perl libsys-cpu-perl 4/ run 00-strict.t test again, (now 6 mins.. much faster) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 364 wallclock secs ( 0.07 usr 0.01 sys + 1159.20 cusr 153.41 csys = 1312.69 CPU) Result: PASS real 6m4.355s > $ sudo apt-get install libparallel-forkmanager-perl libsys-cpu-perl
Shouldn't this be a PerlDependencies tweak too? Obviously not required, but it is good to install for testing. Kind of like Test::DBIx::Class. :)
Also, acqui which runs first has 42 tests, and sequentially you would expect 43, but admin restarts numbering at 1. This confuses the prove. I get a test failure. (In reply to M. Tompsett from comment #5) > Also, acqui which runs first has 42 tests, and sequentially you would expect > 43, but admin restarts numbering at 1. This confuses the prove. I get a test > failure. works for me on the latest kohadevbox ------------- vagrant@kohadevbox:~/kohaclone$ git log --oneline | head -3 b35842c Bug 18055 - Speed up '00-strict.t' test, with Parallel::ForkManager 6abf904 Bug 17960 - DBRev 16.12.00.007 2e9ae12 Bug 17960: Replace missing occurrences vagrant@kohadevbox:~/kohaclone$ sudo koha-shell -c 'time prove ./t/db_dependent/00-strict.t ' kohadev ./t/db_dependent/00-strict.t .. ok All tests successful. Files=1, Tests=661, 537 wallclock secs ( 0.06 usr 0.00 sys + 943.76 cusr 61.74 csys = 1005.56 CPU) Result: PASS real 8m57.167s user 15m43.892s sys 1m1.752s ------------- (In reply to M. Tompsett from comment #4) > > $ sudo apt-get install libparallel-forkmanager-perl libsys-cpu-perl > > Shouldn't this be a PerlDependencies tweak too? yes, the patch includes this already... ------------- diff --git a/C4/Installer/PerlDependencies.pm b/C4/Installer/PerlDependencies.pm index 4e1f75f..1ed5943 100644 --- a/C4/Installer/PerlDependencies.pm +++ b/C4/Installer/PerlDependencies.pm @@ -852,6 +852,16 @@ our $PERL_DEPS = { required => 1, min_ver => '0.28', }, + 'Parallel::ForkManager' => { + usage => 'Core', + required => 0, + min_ver => '0.75', + }, + 'Sys::CPU' => { + usage => 'Core', + required => 0, + min_ver => '0.52', + }, }; ------------- (In reply to Mason James from comment #7) > (In reply to M. Tompsett from comment #4) > > > $ sudo apt-get install libparallel-forkmanager-perl libsys-cpu-perl > > Shouldn't this be a PerlDependencies tweak too? > yes, the patch includes this already... Yep, my bad. But I was running it on a non-kohadevbox git install. Fails horribly after running everything successfully, because of the test numbering. (In reply to M. Tompsett from comment #8) > (In reply to Mason James from comment #7) > > (In reply to M. Tompsett from comment #4) > > > > $ sudo apt-get install libparallel-forkmanager-perl libsys-cpu-perl > > > Shouldn't this be a PerlDependencies tweak too? > > yes, the patch includes this already... > > Yep, my bad. But I was running it on a non-kohadevbox git install. > Fails horribly after running everything successfully, because of the test > numbering. fyi, it should make no difference please provide a detailed log of your error (In reply to Mason James from comment #9) > (In reply to M. Tompsett from comment #8) > > (In reply to Mason James from comment #7) > > > (In reply to M. Tompsett from comment #4) > > > > > $ sudo apt-get install libparallel-forkmanager-perl libsys-cpu-perl > > > > Shouldn't this be a PerlDependencies tweak too? > > > yes, the patch includes this already... > > > > Yep, my bad. But I was running it on a non-kohadevbox git install. > > Fails horribly after running everything successfully, because of the test > > numbering. > i assume your 00-strict.t fails, without the patch too? (In reply to Mason James from comment #6) > (In reply to M. Tompsett from comment #5) > > Also, acqui which runs first has 42 tests, and sequentially you would expect > > 43, but admin restarts numbering at 1. This confuses the prove. I get a test > > failure. > > > works for me on the latest kohadevbox > > ------------- here's a more verbose run... vagrant@kohadevbox:kohaclone(1)$ git log --oneline | head -3 b35842c Bug 18055 - Speed up '00-strict.t' test, with Parallel::ForkManager 6abf904 Bug 17960 - DBRev 16.12.00.007 2e9ae12 Bug 17960: Replace missing occurrences vagrant@kohadevbox:kohaclone(1)$ sudo koha-shell -c 'time prove -v ./t/db_dependent/00-strict.t ' kohadev ./t/db_dependent/00-strict.t .. ok 1 - Syntax check acqui/modordernotes.pl ok 2 - Syntax check acqui/edifactmsgs.pl ok 3 - Syntax check acqui/check_uniqueness.pl ok 4 - Syntax check acqui/transferorder.pl ok 5 - Syntax check acqui/invoice-files.pl ok 6 - Syntax check acqui/check_budget_total.pl ok 7 - Syntax check acqui/invoices.pl ok 8 - Syntax check acqui/booksellers.pl ok 9 - Syntax check acqui/newordersuggestion.pl ok 10 - Syntax check acqui/lateorders-export.pl ok 11 - Syntax check acqui/edimsg.pl ok 12 - Syntax check acqui/basketgroup.pl ok 13 - Syntax check acqui/parcels.pl ok 14 - Syntax check acqui/edi_ean.pl ok 15 - Syntax check acqui/add_user_search.pl ok 16 - Syntax check acqui/check_duplicate_barcode_ajax.pl ok 17 - Syntax check acqui/histsearch.pl ok 18 - Syntax check acqui/neworderbiblio.pl ok 19 - Syntax check acqui/newordersubscription.pl ok 20 - Syntax check acqui/addorderiso2709.pl ok 21 - Syntax check acqui/uncertainprice.pl ok 22 - Syntax check acqui/acqui-home.pl ok 23 - Syntax check acqui/ordered.pl ok 24 - Syntax check acqui/finishreceive.pl ok 25 - Syntax check acqui/cancelorder.pl ok 26 - Syntax check acqui/z3950_search.pl ok 27 - Syntax check acqui/spent.pl ok 28 - Syntax check acqui/basketheader.pl ok 29 - Syntax check acqui/neworderempty.pl ok 30 - Syntax check acqui/updatesupplier.pl ok 31 - Syntax check acqui/lateorders.pl ok 32 - Syntax check acqui/orderreceive.pl ok 33 - Syntax check acqui/basket.pl ok 34 - Syntax check acqui/supplier.pl ok 35 - Syntax check acqui/addorder.pl ok 36 - Syntax check acqui/invoice.pl ok 37 - Syntax check acqui/parcel.pl ok 38 - Syntax check acqui/ajax-getauthvaluedropbox.pl ok 39 - Syntax check acqui/pdfformat/layout2pagesde.pm ok 40 - Syntax check acqui/pdfformat/layout2pages.pm ok 41 - Syntax check acqui/pdfformat/layout3pages.pm ok 42 - Syntax check acqui/pdfformat/layout3pagesfr.pm ok 43 - Syntax check admin/branches.pl ok 44 - Syntax check admin/audio_alerts.pl ok 45 - Syntax check admin/cities.pl ok 46 - Syntax check admin/transport-cost-matrix.pl ok 47 - Syntax check admin/sru_modmapping.pl ok 48 - Syntax check admin/koha2marclinks.pl ok 49 - Syntax check admin/aqbudgets.pl ok 50 - Syntax check admin/smart-rules.pl ok 51 - Syntax check admin/biblio_framework.pl ok 52 - Syntax check admin/fieldmapping.pl ok 53 - Syntax check admin/admin-home.pl ok 54 - Syntax check admin/check_parent_total.pl ok 55 - Syntax check admin/checkmarc.pl ok 56 - Syntax check admin/patron-attr-types.pl ok 57 - Syntax check admin/add_user_search.pl ok 58 - Syntax check admin/env_tz_test.pl ok 59 - Syntax check admin/aqbudgetperiods.pl ok 60 - Syntax check admin/preferences.pl ok 61 - Syntax check admin/classsources.pl ok 62 - Syntax check admin/import_export_framework.pl ok 63 - Syntax check admin/systempreferences.pl ok 64 - Syntax check admin/printers.pl ok 65 - Syntax check admin/branch_transfer_limits.pl ok 66 - Syntax check admin/marctagstructure.pl ok 67 - Syntax check admin/localization.pl ok 68 - Syntax check admin/items_search_field.pl ok 69 - Syntax check admin/oai_set_mappings.pl ok 70 - Syntax check admin/oai_sets.pl ok 71 - Syntax check admin/currency.pl ok 72 - Syntax check admin/item_circulation_alerts.pl ok 73 - Syntax check admin/matching-rules.pl ok 74 - Syntax check admin/clone-rules.pl ok 75 - Syntax check admin/auth_tag_structure.pl ok 76 - Syntax check admin/didyoumean.pl ok 77 - Syntax check admin/z3950servers.pl ok 78 - Syntax check admin/marc_subfields_structure.pl ok 79 - Syntax check admin/check_budget_parent.pl ok 80 - Syntax check admin/auth_subfields_structure.pl ok 81 - Syntax check admin/authorised_values.pl ok 82 - Syntax check admin/edi_accounts.pl ok 83 - Syntax check admin/aqplan.pl ok 84 - Syntax check admin/itemtypes.pl ok 85 - Syntax check admin/authtypes.pl ok 86 - Syntax check admin/aqcontract.pl ok 87 - Syntax check admin/edi_ean_accounts.pl ok 88 - Syntax check admin/categories.pl ok 89 - Syntax check admin/columns_settings.pl ok 90 - Syntax check admin/items_search_fields.pl ok 91 - Syntax check admin/sms_providers.pl ok 92 - Syntax check admin/searchengine/elasticsearch/mappings.pl ok 93 - Syntax check authorities/merge.pl ok 94 - Syntax check authorities/blinddetail-biblio-search.pl ok 95 - Syntax check authorities/authorities-home.pl ok 96 - Syntax check authorities/ysearch.pl ok 97 - Syntax check authorities/detail-biblio-search.pl ok 98 - Syntax check authorities/export.pl ok 99 - Syntax check authorities/authorities.pl ok 100 - Syntax check authorities/auth_finder.pl ok 101 - Syntax check authorities/merge_ajax.pl ok 102 - Syntax check authorities/authorities-list.pl ok 103 - Syntax check authorities/detail.pl ok 104 - Syntax check basket/sendbasket.pl ok 105 - Syntax check basket/downloadcart.pl ok 106 - Syntax check basket/basket.pl ok 107 - Syntax check catalogue/labeledMARCdetail.pl ok 108 - Syntax check catalogue/image.pl ok 109 - Syntax check catalogue/MARCdetail.pl ok 110 - Syntax check catalogue/search.pl ok 111 - Syntax check catalogue/ISBDdetail.pl ok 112 - Syntax check catalogue/issuehistory.pl ok 113 - Syntax check catalogue/itemsearch.pl ok 114 - Syntax check catalogue/export.pl ok 115 - Syntax check catalogue/getitem-ajax.pl ok 116 - Syntax check catalogue/search-history.pl ok 117 - Syntax check catalogue/imageviewer.pl ok 118 - Syntax check catalogue/showmarc.pl ok 119 - Syntax check catalogue/moredetail.pl ok 120 - Syntax check catalogue/updateitem.pl ok 121 - Syntax check catalogue/detail.pl ok 122 - Syntax check cataloguing/editor.pl ok 123 - Syntax check cataloguing/plugin_launcher.pl ok 124 - Syntax check cataloguing/z3950_auth_search.pl ok 125 - Syntax check cataloguing/merge.pl ok 126 - Syntax check cataloguing/addbiblio.pl ok 127 - Syntax check cataloguing/ysearch.pl ok 128 - Syntax check cataloguing/linkitem.pl ok 129 - Syntax check cataloguing/additem.pl ok 130 - Syntax check cataloguing/addbooks.pl ok 131 - Syntax check cataloguing/z3950_search.pl ok 132 - Syntax check cataloguing/merge_ajax.pl ok 133 - Syntax check cataloguing/moveitem.pl ok 134 - Syntax check cataloguing/value_builder/dateaccessioned.pl ok 135 - Syntax check cataloguing/value_builder/unimarc_field_122.pl ok 136 - Syntax check cataloguing/value_builder/unimarc_field_105.pl ok 137 - Syntax check cataloguing/value_builder/unimarc_field_700-4.pl ok 138 - Syntax check cataloguing/value_builder/marc21_leader.pl ok 139 - Syntax check cataloguing/value_builder/unimarc_field_140.pl ok 140 - Syntax check cataloguing/value_builder/barcode.pl ok 141 - Syntax check cataloguing/value_builder/marc21_leader_authorities.pl ok 142 - Syntax check cataloguing/value_builder/unimarc_field_121a.pl ok 143 - Syntax check cataloguing/value_builder/stocknumber.pl ok 144 - Syntax check cataloguing/value_builder/unimarc_field_4XX.pl ok 145 - Syntax check cataloguing/value_builder/unimarc_field_121b.pl ok 146 - Syntax check cataloguing/value_builder/unimarc_field_124g.pl ok 147 - Syntax check cataloguing/value_builder/unimarc_field_124e.pl ok 148 - Syntax check cataloguing/value_builder/unimarc_field_115a.pl ok 149 - Syntax check cataloguing/value_builder/unimarc_field_124f.pl ok 150 - Syntax check cataloguing/value_builder/unimarc_field_127.pl ok 151 - Syntax check cataloguing/value_builder/marc21_field_008_authorities.pl ok 152 - Syntax check cataloguing/value_builder/stocknumberam123.pl ok 153 - Syntax check cataloguing/value_builder/unimarc_field_124.pl ok 154 - Syntax check cataloguing/value_builder/unimarc_field_135a.pl ok 155 - Syntax check cataloguing/value_builder/marc21_linking_section.pl ok 156 - Syntax check cataloguing/value_builder/normarc_leader.pl ok 157 - Syntax check cataloguing/value_builder/unimarc_field_126b.pl ok 158 - Syntax check cataloguing/value_builder/unimarc_field_123d.pl ok 159 - Syntax check cataloguing/value_builder/unimarc_field_141.pl ok 160 - Syntax check cataloguing/value_builder/unimarc_field_686a.pl ok 161 - Syntax check cataloguing/value_builder/marc21_field_245h.pl ok 162 - Syntax check cataloguing/value_builder/unimarc_field_117.pl ok 163 - Syntax check cataloguing/value_builder/unimarc_field_120.pl ok 164 - Syntax check cataloguing/value_builder/callnumber-KU.pl ok 165 - Syntax check cataloguing/value_builder/unimarc_field_125a.pl ok 166 - Syntax check cataloguing/value_builder/marc21_field_006.pl ok 167 - Syntax check cataloguing/value_builder/normarc_field_007.pl ok 168 - Syntax check cataloguing/value_builder/unimarc_field_110.pl ok 169 - Syntax check cataloguing/value_builder/unimarc_field_123i.pl ok 170 - Syntax check cataloguing/value_builder/unimarc_field_225a_bis.pl ok 171 - Syntax check cataloguing/value_builder/unimarc_field_225a.pl ok 172 - Syntax check cataloguing/value_builder/macles.pl ok 173 - Syntax check cataloguing/value_builder/unimarc_field_010.pl ok 174 - Syntax check cataloguing/value_builder/cn_browser.pl ok 175 - Syntax check cataloguing/value_builder/stocknumberAV.pl ok 176 - Syntax check cataloguing/value_builder/unimarc_field_125b.pl ok 177 - Syntax check cataloguing/value_builder/unimarc_field_128b.pl ok 178 - Syntax check cataloguing/value_builder/unimarc_field_123f.pl ok 179 - Syntax check cataloguing/value_builder/unimarc_field_126.pl ok 180 - Syntax check cataloguing/value_builder/unimarc_field_100_authorities.pl ok 181 - Syntax check cataloguing/value_builder/unimarc_field_125.pl ok 182 - Syntax check cataloguing/value_builder/marc21_field_008.pl ok 183 - Syntax check cataloguing/value_builder/unimarc_field_123g.pl ok 184 - Syntax check cataloguing/value_builder/barcode_manual.pl ok 185 - Syntax check cataloguing/value_builder/unimarc_field_124d.pl ok 186 - Syntax check cataloguing/value_builder/unimarc_leader.pl ok 187 - Syntax check cataloguing/value_builder/normarc_field_008.pl ok 188 - Syntax check cataloguing/value_builder/upload.pl ok 189 - Syntax check cataloguing/value_builder/unimarc_field_124c.pl ok 190 - Syntax check cataloguing/value_builder/EXAMPLE.pl ok 191 - Syntax check cataloguing/value_builder/unimarc_field_116.pl ok 192 - Syntax check cataloguing/value_builder/marc21_field_007.pl ok 193 - Syntax check cataloguing/value_builder/unimarc_field_124a.pl ok 194 - Syntax check cataloguing/value_builder/unimarc_field_126a.pl ok 195 - Syntax check cataloguing/value_builder/unimarc_field_130.pl ok 196 - Syntax check cataloguing/value_builder/unimarc_field_100.pl ok 197 - Syntax check cataloguing/value_builder/unimarc_field_210c.pl ok 198 - Syntax check cataloguing/value_builder/unimarc_field_128c.pl ok 199 - Syntax check cataloguing/value_builder/unimarc_field_115b.pl ok 200 - Syntax check cataloguing/value_builder/unimarc_field_123a.pl ok 201 - Syntax check cataloguing/value_builder/unimarc_field_210c_bis.pl ok 202 - Syntax check cataloguing/value_builder/callnumber.pl ok 203 - Syntax check cataloguing/value_builder/unimarc_field_123j.pl ok 204 - Syntax check cataloguing/value_builder/unimarc_field_106.pl ok 205 - Syntax check cataloguing/value_builder/marc21_field_005.pl ok 206 - Syntax check cataloguing/value_builder/unimarc_field_123e.pl ok 207 - Syntax check cataloguing/value_builder/marc21_orgcode.pl ok 208 - Syntax check cataloguing/value_builder/unimarc_field_124b.pl ok 209 - Syntax check cataloguing/value_builder/unimarc_field_128a.pl ok 210 - Syntax check changelanguage.pl ok 211 - Syntax check circ/ypattrodue-attr-search-authvalue.pl ok 212 - Syntax check circ/request-article.pl ok 213 - Syntax check circ/pendingreserves.pl ok 214 - Syntax check circ/selectbranchprinter.pl ok 215 - Syntax check circ/article-requests.pl ok 216 - Syntax check circ/offline.pl ok 217 - Syntax check circ/renew.pl ok 218 - Syntax check circ/circulation-home.pl ok 219 - Syntax check circ/circulation.pl ok 220 - Syntax check circ/offline-mf.pl ok 221 - Syntax check circ/ysearch.pl ok 222 - Syntax check circ/reserveratios.pl ok 223 - Syntax check circ/waitingreserves.pl ok 224 - Syntax check circ/view_holdsqueue.pl ok 225 - Syntax check circ/bookcount.pl ok 226 - Syntax check circ/branchoverdues.pl ok 227 - Syntax check circ/on-site_checkouts.pl ok 228 - Syntax check circ/del_message.pl ok 229 - Syntax check circ/overdue.pl ok 230 - Syntax check circ/article-request-slip.pl ok 231 - Syntax check circ/returns.pl ok 232 - Syntax check circ/add_message.pl ok 233 - Syntax check circ/transfer-slip.pl ok 234 - Syntax check circ/branchtransfers.pl ok 235 - Syntax check circ/hold-transfer-slip.pl ok 236 - Syntax check circ/transferstoreceive.pl ok 237 - Syntax check debian/bd-to-depends ok 238 - Syntax check debian/build-git-snapshot ok 239 - Syntax check debian/list-deps ok 240 - Syntax check debian/scripts/koha-shell ok 241 - Syntax check debian/templates/plack.psgi ok 242 - Syntax check docs/CAS/CASProxy/examples/proxy_cas_data.pl ok 243 - Syntax check docs/CAS/CASProxy/examples/proxy_cas_callback.pl ok 244 - Syntax check docs/CAS/CASProxy/examples/koha_webservice.pl ok 245 - Syntax check docs/CAS/CASProxy/examples/proxy_cas.pl ok 246 - Syntax check edithelp.pl ok 247 - Syntax check errors/400.pl ok 248 - Syntax check errors/403.pl ok 249 - Syntax check errors/401.pl ok 250 - Syntax check errors/404.pl ok 251 - Syntax check errors/500.pl ok 252 - Syntax check errors/402.pl ok 253 - Syntax check fix-perl-path.PL ok 254 - Syntax check help.pl ok 255 - Syntax check installer/html-template-to-template-toolkit.pl ok 256 - Syntax check installer/install.pl ok 257 - Syntax check installer/externalmodules.pl ok 258 - Syntax check installer/data/mysql/update22to30.pl ok 259 - Syntax check installer/data/mysql/patroncards_upgrade.pl ok 260 - Syntax check installer/data/mysql/updatedatabase.pl ok 261 - Syntax check installer/data/mysql/fix_unclosed_nonaccruing_fines_bug17135.pl ok 262 - Syntax check installer/data/mysql/labels_upgrade.pl ok 263 - Syntax check installer/data/mysql/backfill_statistics.pl ok 264 - Syntax check koha_perl_deps.pl ok 265 - Syntax check kohaversion.pl ok 266 - Syntax check labels/label-create-pdf.pl ok 267 - Syntax check labels/spinelabel-home.pl ok 268 - Syntax check labels/label-edit-template.pl ok 269 - Syntax check labels/label-print.pl ok 270 - Syntax check labels/label-home.pl ok 271 - Syntax check labels/label-create-csv.pl ok 272 - Syntax check labels/label-manage.pl ok 273 - Syntax check labels/label-edit-batch.pl ok 274 - Syntax check labels/label-edit-profile.pl ok 275 - Syntax check labels/spinelabel-print.pl ok 276 - Syntax check labels/label-create-xml.pl ok 277 - Syntax check labels/label-item-search.pl ok 278 - Syntax check labels/label-edit-layout.pl ok 279 - Syntax check mainpage.pl ok 280 - Syntax check Makefile.PL ok 281 - Syntax check members/print_overdues.pl ok 282 - Syntax check members/readingrec.pl ok 283 - Syntax check members/routing-lists.pl ok 284 - Syntax check members/nl-search.pl ok 285 - Syntax check members/files.pl ok 286 - Syntax check members/housebound.pl ok 287 - Syntax check members/guarantor_search.pl ok 288 - Syntax check members/printinvoice.pl ok 289 - Syntax check members/printfeercpt.pl ok 290 - Syntax check members/memberentry.pl ok 291 - Syntax check members/discharges.pl ok 292 - Syntax check members/notices.pl ok 293 - Syntax check members/members-home.pl ok 294 - Syntax check members/moremember.pl ok 295 - Syntax check members/printslip.pl ok 296 - Syntax check members/default_messageprefs.pl ok 297 - Syntax check members/purchase-suggestions.pl ok 298 - Syntax check members/paycollect.pl ok 299 - Syntax check members/members-update.pl ok 300 - Syntax check members/pay.pl ok 301 - Syntax check members/discharge.pl ok 302 - Syntax check members/statistics.pl ok 303 - Syntax check members/boraccount.pl ok 304 - Syntax check members/member-flags.pl ok 305 - Syntax check members/member-password.pl ok 306 - Syntax check members/member.pl ok 307 - Syntax check members/mancredit.pl ok 308 - Syntax check members/maninvoice.pl ok 309 - Syntax check members/update-child.pl ok 310 - Syntax check members/summary-print.pl ok 311 - Syntax check members/setstatus.pl ok 312 - Syntax check members/mod_debarment.pl ok 313 - Syntax check members/patronimage.pl ok 314 - Syntax check members/deletemem.pl ok 315 - Syntax check members/members-update-do.pl ok 316 - Syntax check misc/stage_file.pl ok 317 - Syntax check misc/perlmodule_ls.pl ok 318 - Syntax check misc/sax_parser_test.pl ok 319 - Syntax check misc/batchCompareMARCvsFrameworks.pl ok 320 - Syntax check misc/commit_file.pl ok 321 - Syntax check misc/export_borrowers.pl ok 322 - Syntax check misc/batchdeletebiblios.pl ok 323 - Syntax check misc/perlmodule_rm.pl ok 324 - Syntax check misc/exportauth.pl ok 325 - Syntax check misc/check_sysprefs.pl ok 326 - Syntax check misc/export_records.pl ok 327 - Syntax check misc/batchRepairMissingBiblionumbers.pl ok 328 - Syntax check misc/sip_cli_emulator.pl ok 329 - Syntax check misc/mod_zebraqueue.pl ok 330 - Syntax check misc/batchImportMARCWithBiblionumbers.pl ok 331 - Syntax check misc/recreateIssueStatistics.pl ok 332 - Syntax check misc/link_bibs_to_authorities.pl ok 333 - Syntax check misc/batchDeleteUnusedSubfields.pl ok 334 - Syntax check misc/batchRebuildBiblioTables.pl ok 335 - Syntax check misc/batchRebuildItemsTables.pl ok 336 - Syntax check misc/sax_parser_print.pl ok 337 - Syntax check misc/translator/text-extract2.pl ok 338 - Syntax check misc/translator/translate ok 339 - Syntax check misc/translator/text-extract.pl ok 340 - Syntax check misc/translator/xgettext.pl ok 341 - Syntax check misc/translator/TmplTokenizer.pm ok 342 - Syntax check misc/translator/tmpl_process3.pl ok 343 - Syntax check misc/translator/VerboseWarnings.pm ok 344 - Syntax check misc/translator/LangInstaller.pm ok 345 - Syntax check misc/migration_tools/import_lexile.pl ok 346 - Syntax check misc/migration_tools/remove_unused_authorities.pl ok 347 - Syntax check misc/migration_tools/rebuild_zebra.pl ok 348 - Syntax check misc/migration_tools/build_oai_sets.pl ok 349 - Syntax check misc/migration_tools/merge_authority.pl ok 350 - Syntax check misc/migration_tools/checkNonIndexedBiblios.pl ok 351 - Syntax check misc/migration_tools/fix_onloan.pl ok 352 - Syntax check misc/migration_tools/bulkmarcimport.pl ok 353 - Syntax check misc/migration_tools/upgradeitems.pl ok 354 - Syntax check misc/migration_tools/koha-svc.pl ok 355 - Syntax check misc/migration_tools/switch_marc21_series_info.pl ok 356 - Syntax check misc/migration_tools/buildEDITORS.pl ok 357 - Syntax check misc/migration_tools/buildLANG.pl ok 358 - Syntax check misc/migration_tools/create_analytical_rel.pl ok 359 - Syntax check misc/migration_tools/buildCOUNTRY.pl ok 360 - Syntax check misc/migration_tools/22_to_30/export_Authorities.pl ok 361 - Syntax check misc/migration_tools/22_to_30/move_marc_to_authheader.pl ok 362 - Syntax check misc/migration_tools/22_to_30/convert_to_utf8.pl ok 363 - Syntax check misc/migration_tools/22_to_30/rebuild_unimarc_100.pl ok 364 - Syntax check misc/migration_tools/22_to_30/missing090field.pl ok 365 - Syntax check misc/migration_tools/22_to_30/move_marc_to_biblioitems.pl ok 366 - Syntax check misc/migration_tools/22_to_30/export_Authorities_xml.pl ok 367 - Syntax check misc/migration_tools/22_to_30/rebuild_leader.pl ok 368 - Syntax check misc/search_tools/rebuild_elastic_search.pl ok 369 - Syntax check misc/maintenance/fix_accountlines_rmdupfines_bug8253.pl ok 370 - Syntax check misc/maintenance/process_record_through_filter.pl ok 371 - Syntax check misc/maintenance/touch_all_biblios.pl ok 372 - Syntax check misc/maintenance/sanitize_records.pl ok 373 - Syntax check misc/maintenance/UNIMARC_fix_collectiontitle.pl ok 374 - Syntax check misc/maintenance/fix_accountlines_date.pl ok 375 - Syntax check misc/maintenance/UNIMARC_sync_date_created_with_marc_biblio.pl ok 376 - Syntax check misc/maintenance/fix_mysql_constraints.pl ok 377 - Syntax check misc/maintenance/borrowers-force-messaging-defaults ok 378 - Syntax check misc/maintenance/touch_all_items.pl ok 379 - Syntax check misc/maintenance/cmp_sysprefs.pl ok 380 - Syntax check misc/maintenance/remove_items_from_biblioitems.pl ok 381 - Syntax check misc/maintenance/make_zebra_dom_cfg_from_record_abs ok 382 - Syntax check misc/admin/koha-preferences ok 383 - Syntax check misc/cronjobs/gather_print_notices.pl ok 384 - Syntax check misc/cronjobs/membership_expiry.pl ok 385 - Syntax check misc/cronjobs/purge_suggestions.pl ok 386 - Syntax check misc/cronjobs/delete_items.pl ok 387 - Syntax check misc/cronjobs/create_koc_db.pl ok 388 - Syntax check misc/cronjobs/share_usage_with_koha_community.pl ok 389 - Syntax check misc/cronjobs/cloud-kw.pl ok 390 - Syntax check misc/cronjobs/edi_cron.pl ok 391 - Syntax check misc/cronjobs/nl-sync-from-koha.pl ok 392 - Syntax check misc/cronjobs/nl-sync-to-koha.pl ok 393 - Syntax check misc/cronjobs/import_webservice_batch.pl ok 394 - Syntax check misc/cronjobs/sitemap.pl ok 395 - Syntax check misc/cronjobs/notice_unprocessed_suggestions.pl ok 396 - Syntax check misc/cronjobs/services_throttle.pl ok 397 - Syntax check misc/cronjobs/overdue_notices.pl ok 398 - Syntax check misc/cronjobs/advance_notices.pl ok 399 - Syntax check misc/cronjobs/runreport.pl ok 400 - Syntax check misc/cronjobs/batch_anonymise.pl ok 401 - Syntax check misc/cronjobs/check-url.pl ok 402 - Syntax check misc/cronjobs/update_totalissues.pl ok 403 - Syntax check misc/cronjobs/serialsUpdate.pl ok 404 - Syntax check misc/cronjobs/build_browser_and_cloud.pl ok 405 - Syntax check misc/cronjobs/check-url-quick.pl ok 406 - Syntax check misc/cronjobs/longoverdue.pl ok 407 - Syntax check misc/cronjobs/fines.pl ok 408 - Syntax check misc/cronjobs/cart_to_shelf.pl ok 409 - Syntax check misc/cronjobs/delete_unverified_opac_registrations.pl ok 410 - Syntax check misc/cronjobs/delete_expired_opac_registrations.pl ok 411 - Syntax check misc/cronjobs/process_message_queue.pl ok 412 - Syntax check misc/cronjobs/staticfines.pl ok 413 - Syntax check misc/cronjobs/cleanup_database.pl ok 414 - Syntax check misc/cronjobs/delete_patrons.pl ok 415 - Syntax check misc/cronjobs/automatic_item_modification_by_age.pl ok 416 - Syntax check misc/cronjobs/remove_temporary_edifiles.pl ok 417 - Syntax check misc/cronjobs/automatic_renewals.pl ok 418 - Syntax check misc/cronjobs/j2a.pl ok 419 - Syntax check misc/cronjobs/delete_records_via_leader.pl ok 420 - Syntax check misc/cronjobs/rss/rss.pl ok 421 - Syntax check misc/cronjobs/thirdparty/TalkingTech_itiva_inbound.pl ok 422 - Syntax check misc/cronjobs/thirdparty/TalkingTech_itiva_outbound.pl ok 423 - Syntax check misc/cronjobs/social_data/update_social_data.pl ok 424 - Syntax check misc/cronjobs/social_data/get_report_social_data.pl ok 425 - Syntax check misc/cronjobs/holds/auto_unsuspend_holds.pl ok 426 - Syntax check misc/cronjobs/holds/build_holds_queue.pl ok 427 - Syntax check misc/cronjobs/holds/cancel_expired_holds.pl ok 428 - Syntax check misc/load_testing/benchmark_circulation.pl ok 429 - Syntax check misc/load_testing/benchmark_staff.pl ok 430 - Syntax check misc/load_testing/benchmark_webservices.pl ok 431 - Syntax check misc/devel/populate_db.pl ok 432 - Syntax check misc/devel/update_dbix_class_files.pl ok 433 - Syntax check misc/devel/coverage.pl ok 434 - Syntax check misc/devel/create_superlibrarian.pl ok 435 - Syntax check misc/bin/connexion_import_daemon.pl ok 436 - Syntax check offline_circ/process.pl ok 437 - Syntax check offline_circ/service.pl ok 438 - Syntax check offline_circ/list.pl ok 439 - Syntax check offline_circ/enqueue_koc.pl ok 440 - Syntax check offline_circ/download.pl ok 441 - Syntax check offline_circ/process_koc.pl ok 442 - Syntax check opac/opac-addbybiblionumber.pl ok 443 - Syntax check opac/opac-changelanguage.pl ok 444 - Syntax check opac/opac-main.pl ok 445 - Syntax check opac/ilsdi.pl ok 446 - Syntax check opac/opac-shareshelf.pl ok 447 - Syntax check opac/opac-request-article.pl ok 448 - Syntax check opac/opac-ratings-ajax.pl ok 449 - Syntax check opac/opac-review.pl ok 450 - Syntax check opac/opac-course-details.pl ok 451 - Syntax check opac/opac-showreviews.pl ok 452 - Syntax check opac/unapi ok 453 - Syntax check opac/opac-suggestions.pl ok 454 - Syntax check opac/opac-modrequest-suspend.pl ok 455 - Syntax check opac/oai.pl ok 456 - Syntax check opac/opac-authorities-home.pl ok 457 - Syntax check opac/opac-browser.pl ok 458 - Syntax check opac/opac-retrieve-file.pl ok 459 - Syntax check opac/opac-idref.pl ok 460 - Syntax check opac/opac-ics.pl ok 461 - Syntax check opac/opac-modrequest.pl ok 462 - Syntax check opac/opac-shelves.pl ok 463 - Syntax check opac/opac-memberentry.pl ok 464 - Syntax check opac/opac-privacy.pl ok 465 - Syntax check opac/opac-sendbasket.pl ok 466 - Syntax check opac/opac-blocked.pl ok 467 - Syntax check opac/opac-article-request-cancel.pl ok 468 - Syntax check opac/opac-reserve.pl ok 469 - Syntax check opac/opac-serial-issues.pl ok 470 - Syntax check opac/opac-readingrecord.pl ok 471 - Syntax check opac/opac-authoritiesdetail.pl ok 472 - Syntax check opac/opac-discharge.pl ok 473 - Syntax check opac/opac-MARCdetail.pl ok 474 - Syntax check opac/opac-imageviewer.pl ok 475 - Syntax check opac/opac-ratings.pl ok 476 - Syntax check opac/opac-tags.pl ok 477 - Syntax check opac/opac-passwd.pl ok 478 - Syntax check opac/opac-sendshelf.pl ok 479 - Syntax check opac/opac-showmarc.pl ok 480 - Syntax check opac/opac-search-history.pl ok 481 - Syntax check opac/tracklinks.pl ok 482 - Syntax check opac/opac-downloadcart.pl ok 483 - Syntax check opac/opac-downloadshelf.pl ok 484 - Syntax check opac/opac-basket.pl ok 485 - Syntax check opac/opac-mymessages.pl ok 486 - Syntax check opac/opac-messaging.pl ok 487 - Syntax check opac/opac-news-rss.pl ok 488 - Syntax check opac/opac-ISBDdetail.pl ok 489 - Syntax check opac/opac-account-pay-paypal-return.pl ok 490 - Syntax check opac/opac-detail.pl ok 491 - Syntax check opac/opac-search.pl ok 492 - Syntax check opac/opac-course-reserves.pl ok 493 - Syntax check opac/opac-overdrive-search.pl ok 494 - Syntax check opac/maintenance.pl ok 495 - Syntax check opac/opac-password-recovery.pl ok 496 - Syntax check opac/opac-export.pl ok 497 - Syntax check opac/opac-alert-subscribe.pl ok 498 - Syntax check opac/opac-restrictedpage.pl ok 499 - Syntax check opac/opac-renew.pl ok 500 - Syntax check opac/opac-account-pay.pl ok 501 - Syntax check opac/opac-patron-image.pl ok 502 - Syntax check opac/opac-registration-verify.pl ok 503 - Syntax check opac/opac-tags_subject.pl ok 504 - Syntax check opac/opac-account.pl ok 505 - Syntax check opac/opac-user.pl ok 506 - Syntax check opac/opac-image.pl ok 507 - Syntax check opac/opac-topissues.pl ok 508 - Syntax check opac/svc/overdrive_proxy ok 509 - Syntax check opac/svc/report ok 510 - Syntax check opac/svc/shelfbrowser.pl ok 511 - Syntax check opac/svc/suggestion ok 512 - Syntax check opac/svc/patron/show_checkouts_to_relatives ok 513 - Syntax check opac/svc/auth/googleopenidconnect ok 514 - Syntax check opac/sco/sco-main.pl ok 515 - Syntax check opac/sco/sco-patron-image.pl ok 516 - Syntax check opac/sco/printslip.pl ok 517 - Syntax check opac/sco/help.pl ok 518 - Syntax check opac/errors/400.pl ok 519 - Syntax check opac/errors/403.pl ok 520 - Syntax check opac/errors/401.pl ok 521 - Syntax check opac/errors/404.pl ok 522 - Syntax check opac/errors/500.pl ok 523 - Syntax check opac/errors/402.pl ok 524 - Syntax check patroncards/print.pl ok 525 - Syntax check patroncards/manage.pl ok 526 - Syntax check patroncards/edit-profile.pl ok 527 - Syntax check patroncards/add_user_search.pl ok 528 - Syntax check patroncards/edit-batch.pl ok 529 - Syntax check patroncards/edit-template.pl ok 530 - Syntax check patroncards/edit-layout.pl ok 531 - Syntax check patroncards/home.pl ok 532 - Syntax check patroncards/image-manage.pl ok 533 - Syntax check patroncards/create-pdf.pl ok 534 - Syntax check reports/bor_issues_top.pl ok 535 - Syntax check reports/issues_avg_stats.pl ok 536 - Syntax check reports/itemslost.pl ok 537 - Syntax check reports/serials_stats.pl ok 538 - Syntax check reports/borrowers_stats.pl ok 539 - Syntax check reports/itemtypes.plugin ok 540 - Syntax check reports/manager.pl ok 541 - Syntax check reports/cat_issues_top.pl ok 542 - Syntax check reports/issues_stats.pl ok 543 - Syntax check reports/catalogue_out.pl ok 544 - Syntax check reports/dictionary.pl ok 545 - Syntax check reports/orders_by_fund.pl ok 546 - Syntax check reports/reserves_stats.pl ok 547 - Syntax check reports/issues_by_borrower_category.plugin ok 548 - Syntax check reports/cash_register_stats.pl ok 549 - Syntax check reports/catalogue_stats.pl ok 550 - Syntax check reports/reports-home.pl ok 551 - Syntax check reports/stats.screen.pl ok 552 - Syntax check reports/guided_reports.pl ok 553 - Syntax check reports/borrowers_out.pl ok 554 - Syntax check reports/stats.print.pl ok 555 - Syntax check reports/acquisitions_stats.pl ok 556 - Syntax check reserve/placerequest.pl ok 557 - Syntax check reserve/modrequest.pl ok 558 - Syntax check reserve/modrequest_suspendall.pl ok 559 - Syntax check reserve/request.pl ok 560 - Syntax check reviews/reviewswaiting.pl ok 561 - Syntax check rewrite-config.PL ok 562 - Syntax check rotating_collections/editCollections.pl ok 563 - Syntax check rotating_collections/addItems.pl ok 564 - Syntax check rotating_collections/transferCollection.pl ok 565 - Syntax check rotating_collections/rotatingCollections.pl ok 566 - Syntax check serials/subscription-frequencies.pl ok 567 - Syntax check serials/create-numberpattern.pl ok 568 - Syntax check serials/serials-search.pl ok 569 - Syntax check serials/serials-collection.pl ok 570 - Syntax check serials/routing.pl ok 571 - Syntax check serials/serials-edit.pl ok 572 - Syntax check serials/serials-home.pl ok 573 - Syntax check serials/subscription-numberpatterns.pl ok 574 - Syntax check serials/subscription-bib-search.pl ok 575 - Syntax check serials/add_user_search.pl ok 576 - Syntax check serials/claims.pl ok 577 - Syntax check serials/subscription-frequency.pl ok 578 - Syntax check serials/serial-issues.pl ok 579 - Syntax check serials/viewalerts.pl ok 580 - Syntax check serials/subscription-add.pl ok 581 - Syntax check serials/routing-preview.pl ok 582 - Syntax check serials/subscription-detail.pl ok 583 - Syntax check serials/acqui-search-result.pl ok 584 - Syntax check serials/showpredictionpattern.pl ok 585 - Syntax check serials/lateissues-export.pl ok 586 - Syntax check serials/checkexpiration.pl ok 587 - Syntax check serials/subscription-numberpattern.pl ok 588 - Syntax check serials/acqui-search.pl ok 589 - Syntax check serials/reorder_members.pl ok 590 - Syntax check serials/subscription-history.pl ok 591 - Syntax check serials/add_fields.pl ok 592 - Syntax check serials/subscription-renew.pl ok 593 - Syntax check services/itemrecorddisplay.pl ok 594 - Syntax check sms/sms.pl ok 595 - Syntax check sms/sms_listen.pl ok 596 - Syntax check suggestion/suggestion.pl ok 597 - Syntax check svc/article_request ok 598 - Syntax check svc/import_bib ok 599 - Syntax check svc/report ok 600 - Syntax check svc/renew ok 601 - Syntax check svc/holds ok 602 - Syntax check svc/checkin ok 603 - Syntax check svc/bib ok 604 - Syntax check svc/localization ok 605 - Syntax check svc/new_bib ok 606 - Syntax check svc/bib_profile ok 607 - Syntax check svc/barcode ok 608 - Syntax check svc/checkouts ok 609 - Syntax check svc/letters ok 610 - Syntax check svc/authentication ok 611 - Syntax check svc/cover_images ok 612 - Syntax check svc/hold/suspend ok 613 - Syntax check svc/hold/resume ok 614 - Syntax check svc/virtualshelves/search ok 615 - Syntax check svc/members/add_to_list ok 616 - Syntax check svc/members/search ok 617 - Syntax check svc/config/systempreferences ok 618 - Syntax check svc/cataloguing/framework ok 619 - Syntax check svc/cataloguing/metasearch ok 620 - Syntax check svc/records/preview ok 621 - Syntax check tags/review.pl ok 622 - Syntax check tags/list.pl ok 623 - Syntax check tools/marc_modification_templates.pl ok 624 - Syntax check tools/overduerules.pl ok 625 - Syntax check tools/csv-profiles.pl ok 626 - Syntax check tools/batchMod.pl ok 627 - Syntax check tools/batch_record_modification.pl ok 628 - Syntax check tools/modborrowers.pl ok 629 - Syntax check tools/upload-cover-image.pl ok 630 - Syntax check tools/batch_records_ajax.pl ok 631 - Syntax check tools/ajax-inventory.pl ok 632 - Syntax check tools/scheduler.pl ok 633 - Syntax check tools/quotes-upload.pl ok 634 - Syntax check tools/newHolidays.pl ok 635 - Syntax check tools/copy-holidays.pl ok 636 - Syntax check tools/manage-marc-import.pl ok 637 - Syntax check tools/inventory.pl ok 638 - Syntax check tools/holidays.pl ok 639 - Syntax check tools/export.pl ok 640 - Syntax check tools/batch_delete_records.pl ok 641 - Syntax check tools/letter.pl ok 642 - Syntax check tools/picture-upload.pl ok 643 - Syntax check tools/upload-file.pl ok 644 - Syntax check tools/showdiffmarc.pl ok 645 - Syntax check tools/stage-marc-import.pl ok 646 - Syntax check tools/upload.pl ok 647 - Syntax check tools/cleanborrowers.pl ok 648 - Syntax check tools/exceptionHolidays.pl ok 649 - Syntax check tools/background-job-progress.pl ok 650 - Syntax check tools/quotes.pl ok 651 - Syntax check tools/tools-home.pl ok 652 - Syntax check tools/viewlog.pl ok 653 - Syntax check tools/automatic_item_modification_by_age.pl ok 654 - Syntax check tools/koha-news.pl ok 655 - Syntax check tools/import_borrowers.pl ok 656 - Syntax check tools/quotes/quotes_ajax.pl ok 657 - Syntax check tools/quotes/quotes-upload_ajax.pl ok 658 - Syntax check virtualshelves/shelves.pl ok 659 - Syntax check virtualshelves/addbybiblionumber.pl ok 660 - Syntax check virtualshelves/downloadshelf.pl ok 661 - Syntax check virtualshelves/sendshelf.pl 1..661 ok All tests successful. Files=1, Tests=661, 492 wallclock secs ( 0.06 usr 0.03 sys + 869.45 cusr 51.84 csys = 921.38 CPU) Result: PASS real 8m12.122s user 14m29.584s sys 0m51.876s vagrant@kohadevbox:kohaclone(1)$ I have a simplified counter-patch coming. Created attachment 59966 [details] [review] [COUNTER PATCH] Bug 18055: Speed up t/db_dependent/00-strict.t TEST PLAN --------- 1/ run 00-strict.t test, (16.5 mins on a 4xcpu system) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 994 wallclock secs ( 0.19 usr 0.04 sys + 873.40 cusr 116.20 csys = 989.83 CPU) Result: PASS real 16m34.104s 2/ apply patch 3/ run 00-strict.t test again, (now 6 mins.. much faster) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 364 wallclock secs ( 0.07 usr 0.01 sys + 1159.20 cusr 153.41 csys = 1312.69 CPU) Result: PASS real 6m4.355s 4/ run koha qa test tools. > ok 1 - Syntax check acqui/modordernotes.pl > ok 2 - Syntax check acqui/edifactmsgs.pl > ok 3 - Syntax check acqui/check_uniqueness.pl > ok 4 - Syntax check acqui/transferorder.pl > ok 5 - Syntax check acqui/invoice-files.pl > ok 6 - Syntax check acqui/check_budget_total.pl > ok 7 - Syntax check acqui/invoices.pl > ok 8 - Syntax check acqui/booksellers.pl > ok 9 - Syntax check acqui/newordersuggestion.pl > ok 10 - Syntax check acqui/lateorders-export.pl ... > ok 657 - Syntax check tools/quotes/quotes-upload_ajax.pl > ok 658 - Syntax check virtualshelves/shelves.pl > ok 659 - Syntax check virtualshelves/addbybiblionumber.pl > ok 660 - Syntax check virtualshelves/downloadshelf.pl > ok 661 - Syntax check virtualshelves/sendshelf.pl > 1..661 > ok > All tests successful. > Files=1, Tests=661, 492 wallclock secs ( 0.06 usr 0.03 sys + 869.45 cusr > 51.84 csys = 921.38 CPU) > Result: PASS Counter patch isn't as pretty order-wise as this, but it is simpler. And, I hope runs faster. I couldn't get the first patch to work for me, so I threw up a counter patch. Test them both. Decide your preference. Sign it off, obsolete the other. :) (In reply to M. Tompsett from comment #15) > I couldn't get the first patch to work for me, so I threw up a counter patch. > Test them both. Decide your preference. Sign it off, obsolete the other. :) hmm, your patch OOMs my server, then fails... vagrant@kohadevbox:kohaclone(1)$ sudo koha-shell -c 'time prove -v ./t/db_dependent/00-strict.t ' kohadev ./t/db_dependent/00-strict.t .. No subtests run Test Summary Report ------------------- ./t/db_dependent/00-strict.t (Wstat: 9 Tests: 0 Failed: 0) Non-zero wait status: 9 Parse errors: No plan found in TAP output Files=1, Tests=0, 37 wallclock secs ( 0.05 usr 0.02 sys + 3.03 cusr 0.58 csys = 3.68 CPU) Result: FAIL real 0m37.559s user 0m3.192s sys 0m0.620s vagrant@kohadevbox:~$ dmesg -T | grep -i oom-kil [Tue Feb 7 05:07:08 2017] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [Tue Feb 7 05:07:09 2017] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [Tue Feb 7 05:07:09 2017] perl invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [Tue Feb 7 05:07:09 2017] perl invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [Tue Feb 7 05:07:15 2017] perl invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [Tue Feb 7 05:07:15 2017] perl invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [Tue Feb 7 05:07:15 2017] sshd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [Tue Feb 7 05:07:15 2017] sshd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [Tue Feb 7 05:07:22 2017] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [Tue Feb 7 05:07:25 2017] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 (In reply to Mason James from comment #16) > (In reply to M. Tompsett from comment #15) > > I couldn't get the first patch to work for me, so I threw up a counter patch. > > Test them both. Decide your preference. Sign it off, obsolete the other. :) > > > hmm, your patch OOMs my server, then fails... > I'm happy to refine my initial patch, but thanks for your help :) (In reply to Mason James from comment #17) > hmm, your patch OOMs my server, then fails... So it does for kohadevbox. :( I tested it in a 8GB RAM VM. Hi! Pretty good improvement. What concerns me is how well does this module detect the available CPU cores? For example on LXC all the cores are visible to the test runner. There can be resource limits, but we might get a situation where we fork 32 test runners because the LXC container is ran on a server with 32 cores. However resource limits prevent utilization of more than 4 on this LXC-container. This is not good. Maybe a better solution would be to accelerate the whole test running infrastructure. I am not seeing anything in t or $KOHA_PATH which would allow to easily run all Koha's tests, but I have been working on something like Bug 18238. Parallelizing the test runs makes sense in this bigger context, but needs multiple parallel test databases and KOHA_CONF's and is generally a big hassle and prone to all kinds of strange race condition errors. Maybe it is better to parallellize tests from infrastructure using your DevOps (TM) build pipeline and run tests on multiple Koha-intances? Based on the last two comments I think this should be Failed QA. (In reply to Olli-Antti Kivilahti from comment #19) > Hi! > > Pretty good improvement. > > What concerns me is how well does this module detect the available CPU cores? > > For example on LXC all the cores are visible to the test runner. > > There can be resource limits, but we might get a situation where we fork 32 > test runners because the LXC container is ran on a server with 32 cores. > However resource limits prevent utilization of more than 4 on this > LXC-container. > > This is not good. Olli, We can add an enviroment variable to manually limit the number of CPUS, to handle your situation I can add a patch for that, no problem (In reply to Owen Leonard from comment #20) > Based on the last two comments I think this should be Failed QA. hi Owen, Mark's comment refers to an obsoleted patch it's not relevant (In reply to Mason James from comment #22) > (In reply to Owen Leonard from comment #20) > > Based on the last two comments I think this should be Failed QA. > > hi Owen, > Mark's comment refers to an obsoleted patch > > it's not relevant Correct, though this patch does NOT run correctly for me. The tests don't test in order properly for me, which triggers a failure. As noted in comment #8. If others are able to replicate comment #11 output, then I have no issue with this moving forward. Created attachment 63341 [details] [review] [SIGNED-OFF] Bug 18055 - Speed up '00-strict.t' test, with Parallel::ForkManager to test... 1/ run 00-strict.t test, (16.5 mins on a 4xcpu system) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 994 wallclock secs ( 0.19 usr 0.04 sys + 873.40 cusr 116.20 csys = 989.83 CPU) Result: PASS real 16m34.104s 2/ apply patch 3/ install Parallel::ForkManager package $ sudo apt-get install libparallel-forkmanager-perl libsys-cpu-perl 4/ run 00-strict.t test again, (now 6 mins.. much faster) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 364 wallclock secs ( 0.07 usr 0.01 sys + 1159.20 cusr 153.41 csys = 1312.69 CPU) Result: PASS real 6m4.355s Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Marking this as an enhancement rather than a bugfix. Created attachment 64322 [details] [review] Bug 18055 - Speed up '00-strict.t' test, with Parallel::ForkManager add KOHA_PROVE_CPUS envar to manually set number of cpus to test.. 1/ set number of cpus to 1 $ export KOHA_PROVE_CPUS=1 2/ observe 1 cpus are used $ time prove -v t/db_dependent/00-strict.t t/db_dependent/00-strict.t .. Using 1 CPUs... 3/ unset number of cpus $ export KOHA_PROVE_CPUS= 4/ observe all available cpus are used $ time prove -v t/db_dependent/00-strict.t t/db_dependent/00-strict.t .. Using 32 CPUs... (In reply to Mason James from comment #21) > (In reply to Olli-Antti Kivilahti from comment #19) > > Hi! > > > > Pretty good improvement. > > > > What concerns me is how well does this module detect the available CPU cores? > > > > For example on LXC all the cores are visible to the test runner. > > > > There can be resource limits, but we might get a situation where we fork 32 > > test runners because the LXC container is ran on a server with 32 cores. > > However resource limits prevent utilization of more than 4 on this > > LXC-container. > > > > This is not good. > > Olli, We can add an enviroment variable to manually limit the number of > CPUS, to handle your situation > > I can add a patch for that, no problem I've added this patch Could we use the same method for t/00-testcritic.t, t/00-merge-conflict-markers.t, xt/author/valid-templates.t and xt/author/translatable-templates.t as well? Any volunteers? Something is wrong for me: Inside the a kohadevbox, I have only 1 CPU, so no speed up noticed. From the host, 4 CPUs (I launched the tests 00-strict.t with only the admin and acqui dirs): time perl t/db_dependent/00-strict.t Using 4 CPUs... ok 1 - Syntax check admin/systempreferences.pl ok 1 - Syntax check acqui/z3950_search.pl ok 2 - Syntax check admin/cities.pl ok 2 - Syntax check acqui/neworderbiblio.pl ok 3 - Syntax check admin/edi_accounts.pl ok 3 - Syntax check acqui/edi_ean.pl [SKIP] ok 48 - Syntax check admin/auth_subfields_structure.pl ok 49 - Syntax check admin/currency.pl ok 50 - Syntax check admin/searchengine/elasticsearch/mappings.pl 1..0 # No tests run! real 1m0.178s user 1m44.540s sys 0m6.592s => This is ok But, with prove: $ prove t/db_dependent/00-strict.t t/db_dependent/00-strict.t .. 50/? # No tests run! t/db_dependent/00-strict.t .. skipped: (no reason given) Test Summary Report ------------------- t/db_dependent/00-strict.t (Wstat: 65280 Tests: 92 Failed: 0) Non-zero exit status: 255 Parse errors: Tests out of sequence. Found (1) but expected (2) Tests out of sequence. Found (2) but expected (3) Tests out of sequence. Found (2) but expected (4) Tests out of sequence. Found (3) but expected (5) Tests out of sequence. Found (3) but expected (6) Displayed the first 5 of 92 TAP syntax errors. Re-run prove with the -p option to see them all. Files=1, Tests=92, 63 wallclock secs ( 0.04 usr 0.02 sys + 110.22 cusr 6.87 csys = 117.15 CPU) Result: FAIL (In reply to Jonathan Druart from comment #29) Solved! I upgraded Test::Simple from 1.001014 to 1.302075 and the tests now pass! Master, with 4 dirs acqui, admin, authorities, basket: real 2m17.136s user 2m8.156s sys 0m8.184s Master + these patches, same dirs: real 1m10.204s user 2m28.904s sys 0m8.720s (In reply to Jonathan Druart from comment #28) > Could we use the same method for > t/00-testcritic.t, => Not needed, from the POD: all_critic_ok() is also optimized to run tests in parallel over multiple cores (if you have them) so it is usually better to call this function than calling critic_ok() directly. > t/00-merge-conflict-markers.t, => Not efficient. > xt/author/valid-templates.t and => See bug 18979 > xt/author/translatable-templates.t => Fork already Created attachment 66769 [details] [review] Bug 18055: Speed up '00-strict.t' test, with Parallel::ForkManager to test... 1/ run 00-strict.t test, (16.5 mins on a 4xcpu system) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 994 wallclock secs ( 0.19 usr 0.04 sys + 873.40 cusr 116.20 csys = 989.83 CPU) Result: PASS real 16m34.104s 2/ apply patch 3/ install Parallel::ForkManager package $ sudo apt-get install libparallel-forkmanager-perl libsys-cpu-perl 4/ run 00-strict.t test again, (now 6 mins.. much faster) $ time prove t/db_dependent/00-strict.t ... Files=1, Tests=654, 364 wallclock secs ( 0.07 usr 0.01 sys + 1159.20 cusr 153.41 csys = 1312.69 CPU) Result: PASS real 6m4.355s Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Edit: removed debian/control changes as the file is generated Created attachment 66770 [details] [review] Bug 18055: Speed up '00-strict.t' test, with Parallel::ForkManager add KOHA_PROVE_CPUS envar to manually set number of cpus to test.. 1/ set number of cpus to 1 $ export KOHA_PROVE_CPUS=1 2/ observe 1 cpus are used $ time prove -v t/db_dependent/00-strict.t t/db_dependent/00-strict.t .. Using 1 CPUs... 3/ unset number of cpus $ export KOHA_PROVE_CPUS= 4/ observe all available cpus are used $ time prove -v t/db_dependent/00-strict.t t/db_dependent/00-strict.t .. Using 32 CPUs... Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Created attachment 66771 [details] [review] Bug 18055: (QA followup) Only print output if DEBUG Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> I removed the changes to debian/control from the first patch. That file is actually generated and we shuold leave that to Mirko. Baptiste doesn't work anymore at BibLibre. Unassigning to avoid signaling that this bug is still being supervized. Which could wrongly discourage someone to continue the work. (In reply to Jonathan Druart from comment #30) > (In reply to Jonathan Druart from comment #29) > > Solved! > I upgraded Test::Simple from 1.001014 to 1.302075 and the tests now pass! Hi Jonathan Mirko has packaged libtest-simple-perl in BZ-18993 (resolved) ..therefore this bug can be switched back to PASSED-QA :) Created attachment 73403 [details] [review] Bug 18055: The 2 new deps are used for testing purpose Pushed to master for 18.05, thanks to everybody involved! Enhancement, not backported to 17.11.x |