Explore ESM modules instead of regular WP scripts #34140

youknowriad · 2021-08-18T14:21:41Z

A playground for the future (#36716).

This is an experiment: The basic idea is that registering and enqueing scripts like WP does was very good for the JS of 10 years ago but today, we want more flexibility, we want modules, lazy-loading... The PR is trying to explore what could a script loader for the next 10 years look like?

To do so it leverages, import maps and es modules. Each WP script becomes a module and WP generates an import map to be able to load these modules synchronously or asynchronously.

The other aspect I tried to test in this PR is the backward compatibility. How can we introduce such system without breaking existing third-party usage and extensibility using the wp.* globals. Since <script type="module"> is asynchronous, we can't have a synchronous inline script or external script relying on a wp global generated from a module. So this PR includes a hack that makes all scripts of type "module" diminishing the breaking changes.

youknowriad · 2021-08-18T14:29:41Z

bin/esm/build.js

+/**
+ * External dependencies
+ */
+const esbuild = require( 'esbuild' );


I've used esbuild because I didn't manage to use webpack to generate esmodules output, apparently the support is there as experimental but I failed to make it work. Also esbuild is very very very fast.

youknowriad · 2021-08-18T14:30:03Z

bin/esm/build.js

+
+esbuild
+	.build( {
+		entryPoints: packages.map( ( pkg ) => `./packages/${ pkg }` ),


For each package, I generate an esm bundle in build/esm/package.js

youknowriad · 2021-08-18T14:30:44Z

bin/esm/build.js

+			'moment',
+			'lodash',
+			'moment-timezone/moment-timezone',
+			'moment-timezone/moment-timezone-utils',


These are "vendors" that are shared across package, so all of these should be modules as well, right now I'm registering them in import-map.php using skypack esm cdn.

youknowriad · 2021-08-18T14:31:27Z

lib/import-map.php

+	';
+}
+
+add_action( 'admin_enqueue_scripts', 'gutenberg_inject_import_map' );


The import map is injected at the top of all WP pages (admin for now, but could be added to frontend too)

youknowriad · 2021-08-18T14:31:41Z

lib/import-map.php

+gutenberg_register_import_map_entry( 'react-dom', 'https://cdn.skypack.dev/[email protected]' );
+gutenberg_register_import_map_entry( 'moment', 'https://cdn.skypack.dev/moment/moment' );
+gutenberg_register_import_map_entry( 'moment-timezone/moment-timezone', 'https://cdn.skypack.dev/moment-timezone/moment-timezone' );
+gutenberg_register_import_map_entry( 'moment-timezone/moment-timezone-utils', 'https://cdn.skypack.dev/moment-timezone/moment-timezone-utils' );


This allows us to register the vendors in the import map.

youknowriad · 2021-08-18T14:32:35Z

lib/import-map.php

+
+foreach ( glob( gutenberg_dir_path() . 'build/esm/*.js' ) as $path ) {
+	$package_name = substr( basename( $path ), 0, -3 );
+	gutenberg_register_import_map_entry( '@wordpress/' . $package_name, plugins_url( 'build/esm', __DIR__ ) . '/' . $package_name . '.js' );


Here we define the import map entries for all the wp packages. I wasn't able to use the "folder" approach of import maps because the .js extension is ignored in imports (maybe I should generate extension less files).

What do you mean by "the "folder" approach"?

Also, have you considered not shipping a bundle for every package (#34140 (comment)), but re-exporting those from the combined bundle?

The POC of the idea is at #36716 (comment).

That way we can map a package_name to an inlined module that re-exports only the given package from the one and only bundle.js. This way we could also re-export packages from 3rd party vendors instead of using CDN.

youknowriad · 2021-08-18T14:33:26Z

lib/import-map.php

+		$tag          = str_replace(
+			sprintf( "<script type=\"module\" src='%s' id='%s-js'></script>", $src, esc_attr( $handle ) ),
+			sprintf(
+				'<script id="script-%s" type="module">import %s from "@wordpress/%s"; window.wp = window.wp || {}; window.wp.%s = mod;</script>',


backward compatibility for the wp.* global, basically I put in the global the default or all the exports of the module (depends on the module).

Can we use an import map for WP packages, too? I assume, we would have to pass the local path instead and it should work like for vendors. Unless it has some negative impact on the build process.

That's what I'm doing here #34140 (comment)

I missed that the code calls str_replace. I understood that both entries are printed 😅

youknowriad · 2021-08-18T14:34:16Z

lib/import-map.php

+		$package_name = substr( $handle, 3 );
+		$import       = in_array( $package_name, $export_default_packages, true ) ? 'mod' : '* as mod';
+		// Replace all scripts with a module script to defer them.
+		$tag = str_replace( '<script ', '<script type="module" ', $tag );


script_loader_tag is only called once and includes before, after and the script itself:

for the before/after I just mark the script as "type module" to defer it, but for the script source itself, I replace it with the module import.

youknowriad · 2021-08-18T14:34:58Z

lib/import-map.php

+		);
+	} elseif ( in_array( $handle, $vendors, true ) ) {
+		$tag = sprintf(
+			'<script id="script-%s" type="module">import * as mod from "%s"; window.%s = mod;</script>',


This is buggy and might need to be done package per package because the globals for the vendors differ and don't follow any naming guideline.

Could this be retrieved via a custom esbuild/webpack plugin similar to the dependency extraction one?

The number of vendors in WordPress requiring this is pretty small, so we could in theory just hard code these. I don't expect the number of vendor scripts like that to grow much in Core.

youknowriad · 2021-08-18T14:36:03Z

lib/import-map.php

+			$handle,
+			gutenberg_to_camel_case( $handle )
+		);
+	} elseif ( false === strpos( $handle, 'thickbox' ) && false === strpos( $handle, 'tinymce' ) && false === strpos( $handle, 'clipboard' ) && false === strpos( $handle, 'jquery' ) && false === strpos( $handle, 'hoverintent' ) && false === strpos( $handle, 'moxie' ) ) {


There are scripts across WP code base that inject <script> tags without relying on script_loader_tag filter, meaning if they have dependencies (like tinymce), that dependency shouldn't be a script either, that's why I added all these exceptions. There's probably a better way here.

youknowriad · 2021-08-18T14:36:29Z

packages/block-editor/src/components/block-list/block-html.js

@@ -1,7 +1,7 @@
 /**
 * External dependencies
 */
-import TextareaAutosize from 'react-autosize-textarea';
+//import TextareaAutosize from 'react-autosize-textarea/src';


This package doesn't expose an esm version so I had issues when bundling.

youknowriad · 2021-08-18T14:36:39Z

packages/block-editor/src/components/list-view/leaf.js

@@ -1,7 +1,7 @@
 /**
 * External dependencies
 */
-import { animated } from 'react-spring/web.cjs';
+import { animated } from 'react-spring';


Moved to the esm version.

youknowriad · 2021-08-18T14:37:04Z

packages/components/src/date-time/date.js

@@ -6,7 +6,7 @@ import classnames from 'classnames';

 // react-dates doesn't tree-shake correctly, so we import from the individual
 // component here, to avoid including too much of the library
-import DayPickerSingleDateController from 'react-dates/lib/components/DayPickerSingleDateController';
+// import DayPickerSingleDateController from 'react-dates/lib/components/DayPickerSingleDateController';


react-dates does have an esm version but I wasn't able to use it so I just commented it for now.

youknowriad · 2021-08-18T14:37:38Z

packages/date/src/index.js

-import 'moment-timezone/moment-timezone';
-import 'moment-timezone/moment-timezone-utils';
+//import 'moment-timezone/moment-timezone';
+//import 'moment-timezone/moment-timezone-utils';


I didn't manage to rely on the modules for moment-timezone, I had an error that I didn't understand. It will require a deeper look.

gziolo · 2021-08-19T08:15:10Z

The specification (not a W3C Standard) for Import Maps can be found at https://wicg.github.io/import-maps/.

It might turn out that we will be using this feature extensively 😄

swissspidy · 2021-08-19T09:52:08Z

lib/import-map.php

+}
+
+/**
+ * Hichjack the scripts initialization to use the esmodules instead.


Suggested change

* Hichjack the scripts initialization to use the esmodules instead.

* Hijack the scripts initialization to use the esmodules instead.

youknowriad · 2021-08-19T10:11:07Z

Just wanted to update folks following here:

At the moment, I'm thinking the following: Updating core scripts to be esm modules right way is going to be a big breaking change regardless of what we do, so my current thinking is to follow a scaled down but safer approach, like so:

1- Add the APIs to inject insert map and esmodules
2- Use it for new packages/scripts exclusively (and allow third-party plugins to use it)
3- Explore one or two use-case in Core itself for lazy loading (I'm thinking of starting with syntax highlighting in code editor as an example)

Now, for the other existing scripts, I'm thinking of exploring this approach:

1- Ship ESM modules for wp-* scripts that are just "proxies" relying on the wp.* globals.
2- Soft deprecate the direct use of the wp globals to give time for third-party plugins to switch to module dependencies instead. (Would likely require at least 2 or 3 years)
3- Remove the globals and the proxies and ship ESM modules for Core scripts.

youknowriad · 2021-08-19T12:18:09Z

1- Add the APIs to inject insert map and esmodules
2- Use it for new packages/scripts exclusively (and allow third-party plugins to use it)

I've opened an alternative and smaller PR here to achieve these two steps #34172

kevin940726 · 2021-08-24T09:13:47Z

I just found this externalsType option in webpack that seems to be doing some similar things that we've already been doing. Not sure if you're aware of it? I haven't played with this option though so not sure if it's helpful at all.

Explore ESM modules instead of regular WP scripts

0c8bc3c

youknowriad commented Aug 18, 2021

View reviewed changes

Restore webpack config

c69cd3c

youknowriad requested review from gziolo and sgomes August 18, 2021 14:57

gziolo added [Type] Technical Prototype Offers a technical exploration into an idea as an example of what's possible [Type] Build Tooling Issues or PRs related to build tooling labels Aug 19, 2021

gziolo mentioned this pull request Aug 19, 2021

Remove unnecessary build steps from dev environments #25077

Open

swissspidy reviewed Aug 19, 2021

View reviewed changes

youknowriad mentioned this pull request Aug 19, 2021

Add an example esm-example package and use it in edit-post #34172

Closed

3 tasks

This was referenced Nov 22, 2021

Plugin: Start using JavaScript Modules and Import Maps #36715

Closed

Start using JavaScript Modules and Import Maps #36716

Closed

youknowriad closed this Nov 23, 2023

youknowriad deleted the poc/esm-modules branch November 23, 2023 20:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Explore ESM modules instead of regular WP scripts #34140

Explore ESM modules instead of regular WP scripts #34140

youknowriad commented Aug 18, 2021 •

edited by gziolo

Loading

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

tomalec Nov 23, 2021

youknowriad Aug 18, 2021

gziolo Aug 19, 2021

youknowriad Aug 19, 2021

gziolo Aug 19, 2021

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

swissspidy Aug 19, 2021

youknowriad Aug 19, 2021

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

youknowriad Aug 18, 2021

gziolo commented Aug 19, 2021 •

edited

Loading

swissspidy Aug 19, 2021

youknowriad commented Aug 19, 2021

youknowriad commented Aug 19, 2021

kevin940726 commented Aug 24, 2021

	* Hichjack the scripts initialization to use the esmodules instead.
	* Hijack the scripts initialization to use the esmodules instead.

Explore ESM modules instead of regular WP scripts #34140

Explore ESM modules instead of regular WP scripts #34140

Conversation

youknowriad commented Aug 18, 2021 • edited by gziolo Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gziolo commented Aug 19, 2021 • edited Loading

Choose a reason for hiding this comment

youknowriad commented Aug 19, 2021

youknowriad commented Aug 19, 2021

kevin940726 commented Aug 24, 2021

youknowriad commented Aug 18, 2021 •

edited by gziolo

Loading

gziolo commented Aug 19, 2021 •

edited

Loading