How to transform AST in Rspack?

[ahabhgk]

Currently, there are JavaScript AST transform options located in the builtins options (such as builtins.react, builtins.presetEnv, builtins.decorator, builtins.emotion, and builtins.relay). Typically, these options should be configured by babel-loader plugins in Webpack. However, in Rspack, we do not recommend using babel-loader in Rspack because it would negatively impact performance. We use SWC to parse, generate, and transform JavaScript AST to implement features like JSX and decorator. As our use cases grow, we need to implement presetEnv, emotion, and relay. Since not all projects need these features, we have made them configurable within the builtins options. However, there is a problem with this approach. For example, a project may use both emotion and styled-components. Files under src/emotion may use emotion, while files under src/styled may use styled-components. Similarly, a project may use React, Solid, and Vue simultaneously, which requires different JSX transformation logic depending on the file path. Therefore, the proper location for JavaScript AST transformation should be under module.rule[].

However, there are two remaining questions:

1. Should we use a loader to perform AST transformations, like using babel-loader in Webpack?

   • Technically, babel-loader is not performing AST transformations in Webpack. It is still a string transform because babel-loader will parse the source code, transform the AST, generate the source code from the transformed AST, and then return the generated source code. Thus, if we perform AST transformations in a loader, we actually extend the ability of the loader, which is not only to transform the source code but also to transform the AST.
   • In Webpack, module.rule[].parse and module.rule[].generate can be used to configure parse and generate options. A specific module.rule[].transform for Rspack might be a good alternative way to perform AST transformations in a loader.

2. Should we keep loader-runner in Rust or move it to JavaScript?

   • The reason we have a loader-runner in Rust is that we want to perform some source code transformations in Rust using the built-in loader (currently only builtin:sass-loader, and we haven't expose this on our document for some reason). However, the scenarios that require a built-in loader are very few (sass-embedded-host-rust, mdxjs-rs, currently hard to find the thrid one), and there may be other ways to implement them (e.g., using type: 'mdx' | 'sass', module.rule[].loader: { mdx: true }).
   • In Webpack, a module will collect all the loaders it needs to run during factorize and then use a standalone library, loader-runner, to run the collected loaders during doBuild. Thus, running all the loaders can be treated as a separate part from Webpack. Collecting all the loaders in Rust and then running them in JavaScript might be a sound design for Rspack and may make it easier to implement further features like inline-loaders, matchResource, pitching-loaders, etc.

PS:

• Rspack will not drop builtins.react, builtins.presetEnv and all other AST transform related options placed in builtins before v1.0.0, we will compatible with them to avoid breaking changes.
• The current implementation of loader is in the following files:
  • Rust version loader-runner
  • JS loaders will be composed to avoid Rust <-> JS overhead
  • Convert composed JS loader to JsLoaderAdapter
  • Collect JS loader and builtin loader
  • Collect loaders which a module needs to run in factorize
  • Run loaders of a module in build

[ahabhgk]

Plan 1

Place all the AST transform in parser/transformer/generator options and use the loader only for string transform, leaving all the work to the JS-side loader using js version loader-runner.

rules: [
  {
    test: /\.mdx$/,
    use: ["@mdx-js/loader"], // js loader, do string transform
    transformer: { presetEnv: 'es5', emotion: true }, // or `parser`/`generator`, do AST transform
  },
]

• Cons:

  • We need to get rid of the built-in loader. There is little demand for string transform on the Rust side other than for sass and mdxjs-rs, and it is currently difficult to find the third one, and it can be implemented in other ways.
  • Some non-language-specific AST transforms may not be suitable for placement in parser/transform/generate options, and it is more appropriate to allow users to specify them in the loader, such as emotion and relay.
  • Users cannot specify the order of AST transforms, they can only choose whether to enable them. If users have ordering requirements or need to obtain the results of AST transform, they need to fall back to a JS loader (such as babel-loader) to handle it, which leads to performance fragmentation.

• Pros:

  • Easier to implement pitching-loader, enforce, and inline-loader, and performance may be better.
    • JS version loader-runner has already implemented the call to pitching-loader.
    • Enforce and inline-loader need to be distinguished into four types of loaders: preLoader, normalLoader, inlineLoader, and postLoader. They are collected in order during the factorize phase and called through JS side loader-runner in the build phase. it is easy to implement one-shot loader calls using napi-rs in Rspack.

[ahabhgk]

Plan 2

Introducing meta as the third returned value of a loader, which's type is Meta { ast: Option<RspackAST> }, the ast field can be used to reuse AST for reducing one parse and generate overhead between loaders, and reducing one parse overhead between loader and parser (swc determines the span and generates the source map during generate, so generate overhead is only reduced when cheap or false source map are specified). And introduce builtin:swc-loader to align with babel-loader in webpack. Of course, the Rust version loader-runner need to keep for implementing builtin loaders.

rules: [
  {
    test: /\.mdx$/,
    use: [
      "@mdx-js/loader", // js loader, but we can intergrate `mdxjs-rs` to implement `builtin:mdx-loader`
      {
        loader: "builtin:swc-loader",
        options: {
          plugins: [
            ["presetEnv", { targets: 'es5' }],
            "emotion",
          ],
        },
      },
    ],
  },
]

• Pros:

  • The third returned value of loader is already exsited in webpack loader, postcss-loader and css-loader is using it to avoid one parse overhead. So we are not creating new concepts, but reusing concepts that already exist in webpack.
  • builtin:swc-loader plugins is a more appropriate way for non-language-specific AST transforms.
  • Users can specify the order of AST transforms by the order of builtin:swc-loader plugins.
  • Better performence since parse can reuse AST parsed by builtin:swc-loader in most cases.
  • Easier to migrate from webpack bebel-loader.

• Cons:

  • For obtain the results of AST transform, we might need to write something like this:

    rules: [
      { loader: "builtin:swc-loader", options: { plugin: ["typescript"] } },
      { loader: "custom-loader" },
      { loader: "builtin:swc-loader", options: { plugin: ["preset-env"] } },
    ]

  • The builtin:swc-loader can implement typescript and jsx through plugins, so it may not be necessary to have type: "ts" and type: "jsx", introduce more breaking changes in v1.0.0.