Bug 37824

Summary: Replace webpack with rspack for fun and profit
Product: Koha Reporter: Paul Derscheid <paul.derscheid>
Component: Architecture, internals, and plumbingAssignee: Paul Derscheid <paul.derscheid>
Status: Pushed to main --- QA Contact: Martin Renvoize (ashimema) <martin.renvoize>
Severity: critical    
Priority: P1 - high CC: jonathan.druart, martin.renvoize, philippe.blouin
Version: MainKeywords: additional_work_needed
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.11.00
Circulation function:
Bug Depends on:    
Bug Blocks: 37911, 38167    
Attachments: Bug 37824: Replace webpack with rspack for fun and profit
Bug 37824: Replace webpack with rspack for fun and profit
Bug 37824: Replace webpack with rspack for fun and profit
Bug 37824: (follow-up) Add rspack.config.js to Makefile.PL
Bug 37824: (QA follow-up) Fix improper handling of jQuery and DataTables libraries by marking them as externals
Bug 37824: remove references to non-existent files in Makefile.PL
Bug 37824: remove references to non-existent files in Makefile.PL

Description Paul Derscheid 2024-09-03 15:32:46 UTC
Webpack is slow and nobody likes it. It gives us a lot of flexibility, though (for example in comparison to rollup, which is pretty great).

Let's replace it with what's just reached v1 and is advertised as more or less a drop-in-replacement. 

Full Disclosure: rspack is a ByteDance project (MIT licensed, though).
Comment 1 Paul Derscheid 2024-09-03 15:41:33 UTC
Created attachment 170992 [details] [review]
Bug 37824: Replace webpack with rspack for fun and profit

I left the webpack.config.js as well as the devDependencies in place for now.
We can remove them in a follow-up patch on this bug after testing it out.

To test:
1) Run js:build and js:build:prod
2) Note the build time
3) Apply patch
4) Run yarn install
5) Run js:build and js:build:prod again
6) Note the much faster build time
7) Extra credit: take a look at the ERM or preservations module and make sure everything works as expected.
8) Extra credit: run the cypress tests.
9) Sign off or give your opinion
Comment 2 Marcel de Rooy 2024-09-06 07:50:48 UTC
Created attachment 171126 [details] [review]
Bug 37824: Replace webpack with rspack for fun and profit

I left the webpack.config.js as well as the devDependencies in place for now.
We can remove them in a follow-up patch on this bug after testing it out.

To test:
1) Run js:build and js:build:prod
2) Note the build time
3) Apply patch
4) Run yarn install
5) Run js:build and js:build:prod again
6) Note the much faster build time
7) Extra credit: take a look at the ERM or preservations module and make sure everything works as expected.
8) Extra credit: run the cypress tests.
9) Sign off or give your opinion

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Looks faster to me. Where webpack failed for me (Killed, error 137), this passed.
Comment 3 Paul Derscheid 2024-09-12 13:46:10 UTC
Concerning the production readiness of rspack, I'd just like to say that ByteDance themselves already use it in production. And I guess what works for a billion $ company should be stable enough for us?
Comment 4 Jonathan Druart 2024-09-12 13:55:43 UTC
I don't have the time to test it right now but if it's faster and considered stable we should switch!
Comment 5 Paul Derscheid 2024-09-12 14:08:25 UTC
There's a solid argument joubu made against mine:

ByteDance as well as the big testemonials:
- Microsoft
- Amazon
- Intuit
- Discord

have time and resources to adapt their codebases to changes in the build tool and we don't.

However: they aim to reach as close to 100% compatibility as possible with webpack, which to me means that:
- they can't do crazy stuff
- we can always fallback onto webpack, it's not going anywhere.

Therefore I think we can be a bit bleeding edge here ;)
Comment 6 Martin Renvoize (ashimema) 2024-09-13 06:11:48 UTC
I'll be taking this for a spin today. Now that I've researched the current state of these sorts of tools I'm happy for us to take the leap too. There's a clear back step should we need to take it, but the project itself does feel well regarded and supported so I don't predict we'll need to take such measures.

Assuming I get the same results as prior testers, and I can't see why I wouldn't, 8 think this is well worthwhile and likely to pass today.
Comment 7 Paul Derscheid 2024-09-13 07:19:17 UTC
Thanks for testing this Martin :)
Comment 8 Martin Renvoize (ashimema) 2024-09-13 07:59:50 UTC
Created attachment 171438 [details] [review]
Bug 37824: Replace webpack with rspack for fun and profit

I left the webpack.config.js as well as the devDependencies in place for now.
We can remove them in a follow-up patch on this bug after testing it out.

To test:
1) Run js:build and js:build:prod
2) Note the build time
3) Apply patch
4) Run yarn install
5) Run js:build and js:build:prod again
6) Note the much faster build time
7) Extra credit: take a look at the ERM or preservations module and make sure everything works as expected.
8) Extra credit: run the cypress tests.
9) Sign off or give your opinion

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Looks faster to me. Where webpack failed for me (Killed, error 137), this passed.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 9 Martin Renvoize (ashimema) 2024-09-13 08:00:25 UTC
I'm happy this is a solid improvement and worthwhile. All worked in testing very seamlessly.

Passing QA

Nice work Paul
Comment 10 Katrin Fischer 2024-09-16 08:58:13 UTC
Just wondering: do we need to do anything to make sure the yarn build step is done on other ktd, Jenkins, sandboxes?
Comment 11 Paul Derscheid 2024-09-16 09:00:46 UTC
Only of they directly call webpack which I hope is not the case.

If everything is just yarn *:* and the node_modules is up-to-date, it will just work (and better than before imo).
Comment 12 Katrin Fischer 2024-09-16 11:49:11 UTC
Pushed for 24.11!

Well done everyone, thank you!
Comment 13 Jonathan Druart 2024-09-16 12:39:08 UTC
Failure at ktd startup

koha-1       | $ rspack build --mode development
koha-1       | [rspack-cli] [Error:   × Rspack FS Error: Error: EACCES: permission denied, open '/kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/js/vue/dist/erm.js.map'
koha-1       |   │ 
koha-1       |   ╰─▶ Error: EACCES: permission denied, open '/kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/js/vue/dist/erm.js.map'
Comment 14 Jonathan Druart 2024-09-16 12:43:02 UTC
(In reply to Jonathan Druart from comment #13)
> Failure at ktd startup
> 
> koha-1       | $ rspack build --mode development
> koha-1       | [rspack-cli] [Error:   × Rspack FS Error: Error: EACCES:
> permission denied, open
> '/kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/js/vue/dist/erm.js.map'
> koha-1       |   │ 
> koha-1       |   ╰─▶ Error: EACCES: permission denied, open
> '/kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/js/vue/dist/erm.js.map'

I chowned my whole repo and now ktd is starting correctly.
Comment 15 Jonathan Druart 2024-09-16 12:48:09 UTC
Cypress tests failing since this change:

% yarn cypress run --spec t/cypress/integration/ERM/Agreements_spec.ts 

  1) Agreement CRUD operations                                                                            
       Delete agreement:                                                                                  
     Error: The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection.
                                                                                                          
  > Cannot extend unknown button type: excelHtml5                                                         

When Cypress detects uncaught errors originating from your application it will automatically fail the current test.
                                                                                                          
This behavior is configurable, and you can choose to turn this off by listening to the `uncaught:exception` event.    
                                                     
https://on.cypress.io/uncaught-exception-from-application
Comment 16 Paul Derscheid 2024-09-16 12:51:01 UTC
Taking a look!
Comment 17 Katrin Fischer 2024-09-16 14:36:43 UTC
We have another failing test here: t/Makefile.t
I think we need to put the new config file in this list:
https://git.koha-community.org/Koha-community/Koha/src/branch/main/Makefile.PL#L298
Comment 18 Katrin Fischer 2024-09-16 14:42:47 UTC
Created attachment 171552 [details] [review]
Bug 37824: (follow-up) Add rspack.config.js to Makefile.PL

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Comment 19 Katrin Fischer 2024-09-16 14:44:02 UTC
Follow-up for Makefile.PL pushed to main.
Comment 20 Paul Derscheid 2024-09-16 16:30:17 UTC
Created attachment 171568 [details] [review]
Bug 37824: (QA follow-up) Fix improper handling of jQuery and DataTables libraries by marking them as externals

This commit addresses an issue where jQuery and DataTables libraries (datatables.net and related extensions) were not functioning correctly when bundled with rspack. These libraries expect to be loaded in the global scope rather than as bundled modules, leading to initialization issues.

To resolve this, jQuery and all related DataTables libraries (datatables.net, datatables.net-buttons, datatables.net-buttons/js/buttons.html5, etc.) are marked as externals in the rspack configuration. This ensures they are treated as global dependencies, preventing conflicts and allowing proper initialization.
Comment 21 Mason James 2024-09-24 01:14:04 UTC
Created attachment 171909 [details] [review]
Bug 37824: remove references to non-existent files in Makefile.PL

to test...

0/ install debhelper tool

1/ run 'DEB_BUILD_OPTIONS=nocheck dh build', observe failure
    make: *** No rule to make target 'koha-tmpl/intranet-tmpl/prog/js/vue/dist/erm.js.LICENSE.txt', needed by 'pm_to_blib'.  Stop.
    dh_auto_build: error: make -j1 returned exit code 2

2/ apply patch

3/ run 'DEB_BUILD_OPTIONS=nocheck dh build', observe success
Comment 22 Blou 2024-09-27 13:47:40 UTC
Created attachment 172113 [details] [review]
Bug 37824: remove references to non-existent files in Makefile.PL

to test...

0/ install debhelper tool

1/ run 'DEB_BUILD_OPTIONS=nocheck dh build', observe failure
    make: *** No rule to make target 'koha-tmpl/intranet-tmpl/prog/js/vue/dist/erm.js.LICENSE.txt', needed by 'pm_to_blib'.  Stop.
    dh_auto_build: error: make -j1 returned exit code 2

2/ apply patch

3/ run 'DEB_BUILD_OPTIONS=nocheck dh build', observe success

Signed-off-by: Blou <blou@inlibro.com>
Comment 23 Blou 2024-09-27 13:54:59 UTC
Tested Mason's patch on the main, and it resolved the issue.

This otherwise prevents all usage of main (including signoffs, etc..)
Comment 24 Katrin Fischer 2024-09-27 14:41:29 UTC
(In reply to Blou from comment #23)
> Tested Mason's patch on the main, and it resolved the issue.
> 
> This otherwise prevents all usage of main (including signoffs, etc..)

Pushed the follow-up to main.