RFC-010: Runtime ( Module Loader & Generation)

[liuliudada-w]

Summary

This RFC contains two parts

• Contents of rspack Runtime code and module loader
• Generation strategy and configuration of Rspack Runtime code

Motivation

Runtime is necessary for many web build tools, because js module system is complex, build tools need an independent module loader to unify different module behaviors. Since the Runtime module of webpack has been tested for a long time, the Runtime module of rspack will be based on webpack, and on this basis, it will be optimized for the special situation of rspack and some actual business conditions.

Detailed design

Runtime Code Content

Core Structure and Data Structure

After the Runtime code is executed, a Runtime instance will be injected into the top-level scope (web is usually window), and different Runtime instances will be isolated, so as to ensure that the compilation products of multiple projects can run at the same time (micro front-end), all module information , chunk information, functional functions and more are all mounted on the runtime instance. Runtime will support module loading and execution, asynchronous Chunk loading and execution, hmr client, Lazy Compile client and other functions.

type Runtime = {
  installedModules: Record<ModuleID, Module>;
  installedChunks: Record<ChunkID, ChunkStatus>;
  installedCssChunks: Record<ChunkID, ChunkStatus>;
  moduleCache: Record<ModuleID, ModuleExports>;
  chunkHashData: {
    js: Record<ChunkID, String>;
    css: Record<ChunkID, String>;
  }
  publicPath: String;
}

// The code loading function will replace the import & require in the source code
interface __rspack_require__ {
  (moduleId: ModuleID): ModuleExports;
}

// chunk registration function
interface __rspack_register__ {
  (chunkIds: ChunkID, modules: Module[], callback: Function): void;
}

// Dynamic import function, which will replace import() in the code
interface __rspack_dynamic_require__ {
  (chunkIds: ChunkID): Array<Promise<Module>>;
}

// Load dynamic js functions, responsible for producing corresponding tags and registering promises
interface __rspack_load_dynamic_js__ {
  (chunkIds: ChunkID, promises: Array<Promise<void>>): void;
}

// Load dynamic css functions, responsible for producing corresponding tags and registering promises
interface __rspack_load_dynamic_css__ {
  (chunkIds: ChunkID, promises: Array<Promise<void>>): void;
}

// Generate chunk URL from data
interface __rspack_get_dynamic_chunk_url__ {
  (chunkIds: ChunkID, type: ChunkType): string;
}

Process of executed

• Initial load
  When the page is loaded for the first time, the runtime module code will be loaded first to ensure that the initialization of the runtime instance is completed. Then load each JS file, each JS file will execute the rspack_register function in chunks, and executing the rspack_register function will register the module in the Chunk into the installedModule. The entry module will provide a callback function to be executed after the registration is completed. The callback function will execute rspack_require on the entry Module, and then execute the corresponding rspack_require according to the code execution order.

• Dynamic Chunk Loading
  Dynamic Chunk will execute the rspack_dynamic_require function when the code is executed to the corresponding part, and pass in the requested ChunkID. The function will find the corresponding hash in chunkHashData according to the incoming ChunkID and determine the URL of the Chunk, and then execute the corresponding loading function according to the type of Chunk, and generate the corresponding HTML tag to complete the loading. After the Chunk is loaded, the Promise.then operation will be executed at the original code execution place, and rspack_require will be executed to request the required Module.

• HMR
  HMR is supported by the dev server. After the developer modifies the code, the dev server will monitor file changes and trigger compilation, and return the updated result to the client through websocket. The client will re-execute the code and replace installedModules and moduleCache module information, and then refresh according to different hmr strategies(react refresh or full reload)
• Lazy Compile
  Lazy Compile will take effect on the dynamic import module in dev mode, will generate a virtual module for the module that only contains the loading function, and add it to the Chunk as a sync module. When the code is executed to the corresponding module, the loading function in the virtual Module will be executed first. The loading function will request the dev server and return the Module information. The dev server will trigger the compilation process of the actual code of the Module according to the information. After the compilation is completed, the result of this compilation will be returned, and the corresponding Module information will be updated by hmr, and the virtual Module will be replaced by the real Module.

Runtime Generate

Runtime code is mainly generated on the Rust side. The main process is similar to webpack. Use rspack-sources to assemble the corresponding Runtime code segment according to the user configuration and compilation mode, and finally generate the Runtime module and insert it before the code entry for execution.
The general runtime code will converge into rspack_runtime_plugin as a whole. For other special types of runtime code, each plugin needs to implement its own runtime hook, and rspack will accept the return value of the runtime hook as the final generated content of the runtime.

pub trait Plugin: Debug + Send + Sync {
  fn name(&self) -> &'static str {
    "unknown"
  }

  fn generate_runtime(&self, _ctx: PluginContext<&mut NormalModuleFactoryContext>, source: &Option<ConcatSource>) -> PluginGenerateRuntimeOuput {
    let concat_source = ConcatSource::new(vec![
      &mut source_map_source_rollup,
      &mut source_map_source_minify,
    ]);
    Ok(concat_source)
  }
}

Prior Art

Basic runtime
First, load all modules to webpack_modules object, then the process will execute the entry to load and execute the actual modules

__webpack_modules__: object, contains all modules except the entry
__webpack_module_cache__: object, contains the module cache
__webpack_require__: the require function
__webpack_require__.d: function, webpack/runtime/define property getters
__webpack_require__.o: function, webpack/runtime/hasOwnProperty
__webpack_require__.r: function, define __esModule on exports

Async load, the following runtime will be added

Difference from webpack

Although generally similar to Webpack, rspack has been adjusted according to the actual situation, so the following points are different from the default behavior of webpack

• Runtime code will become an independent module by default and is inserted before the entry execution (for web, it is added to the head of HTML), and in webpack this capability needs to be manually enabled
• Some data information of async chunks will be extracted as independent modules, and data registration will be performed in the entry js, and a strategy of extracting chunk information into independent files is also provided.
• The runtime data is uniformly mounted on the corresponding Runtime instance, instead of storing a large amount of data in closures like webpack. Based on this, we can expand more after runtime and integrate with some runtime frameworks
• Split require & dynamic require functions
• For some runtime build types that webpack does not natively support, rspack will provide default builds (eg css) because rspack will treat them as first class citizens

Existing questions and discussions

• Whether to treat the runtime generation specially, there is no hook specially added for the runtime code in webpack
• Whether to split the runtime generation into many plugins like webpack

[h-a-n-a]

Good job, but I have a few questions about the technical design:

1. Could you shed more light on dealing with the dependency relationship of different runtimes? Like Runtime Code A -- depend on -- Runtime Code B and how are we going to implement it under the circumstances of the architecture of Rspack.
2. How are we going to track the usage of the runtime?
3. Is it guaranteed that CSS is preloaded than the JavaScript in the same chunk or it will cause the FOUC and how are we going to implement this under the current architecture?
4. I don't know if the architecture of the RFC is enough for us to implement features like module federation later, or is it possible for us to use the idea of custom ensurers like webpack had been doing? Haven't compared which one is a better solution. cc/ @underfin
5. IMHO, whether we are going to split into a single runtime is controlled by the user end by providing an option like webpack's optimization.runtimeChunk but the default value can be adjusted according to the best practice.

It will be nice to shed more light on the existing implementation of wepack like what the splitting of runtime plugins try to do and how they are going to solve the problem for wepack other than the different of the Rspack after all as I don't quite get much of the context of wepack's implementation.

[hardfist]

should runtime depends on promise, otherwise it's must loaded after promise polyfill loaded