Browser freezes when highlighting a lot of compressed code

[keaiwkaw]

The browser crashed while I was writing 60k compressed js code into the monaco-editor. When I commented out the highlighted logic, the compressed code appeared normally

[xueerli]

@keaiwkaw Hi, have you solved this problem?

[kachurun]

@xueerli
@keaiwkaw

I solved it by patching method wireTmGrammars from here.
I just don't allow the tokenizer to tokenize lines that are longer than 300 symbols.

Easy peasy:

/**
 * Wires up monaco-editor with monaco-textmate
 */
export function wireTmGrammars(monaco: MonacoEditor, registry: Registry, languages: Map<string, string>, editor?: MonacoEditorInstance) {
    const maxLineLength = 300;

    return Promise.all(
        Array.from(languages.keys())
            .map(async(languageId) => {
                const grammar = await registry.loadGrammar(languages.get(languageId));
                monaco.languages.setTokensProvider(languageId, {
                    getInitialState: () => new TokenizerState(INITIAL),
                    tokenize: (line: string, state: TokenizerState) => {
                        // Return plaintext for long lines
                        if (line.length > maxLineLength) {
                            return {
                                endState: state,
                                tokens: [{ startIndex: 0, scopes: 'plaintext' }]
                            };
                        }

                        const res = grammar.tokenizeLine(line, state.ruleStack);

                        editor ??= monaco.editor.getEditors()[0] as MonacoEditorInstance;

                        return {
                            endState: new TokenizerState(res.ruleStack),
                            tokens: res.tokens.map(token => ({
                                ...token,
                                scopes: editor ? TMToMonacoToken(editor, token.scopes) : token.scopes.at(-1)
                            })),
                        };
                    }
                });
            })
    );
}

[XiayidanAlimu]

@xueerli @keaiwkaw

I solved it by patching method wireTmGrammars from here. I just don't allow the tokenizer to tokenize lines that are longer than 300 symbols.

Easy peasy:

/**
 * Wires up monaco-editor with monaco-textmate
 */
export function wireTmGrammars(monaco: MonacoEditor, registry: Registry, languages: Map<string, string>, editor?: MonacoEditorInstance) {
    const maxLineLength = 300;

    return Promise.all(
        Array.from(languages.keys())
            .map(async(languageId) => {
                const grammar = await registry.loadGrammar(languages.get(languageId));
                monaco.languages.setTokensProvider(languageId, {
                    getInitialState: () => new TokenizerState(INITIAL),
                    tokenize: (line: string, state: TokenizerState) => {
                        // Return plaintext for long lines
                        if (line.length > maxLineLength) {
                            return {
                                endState: state,
                                tokens: [{ startIndex: 0, scopes: 'plaintext' }]
                            };
                        }

                        const res = grammar.tokenizeLine(line, state.ruleStack);

                        editor ??= monaco.editor.getEditors()[0] as MonacoEditorInstance;

                        return {
                            endState: new TokenizerState(res.ruleStack),
                            tokens: res.tokens.map(token => ({
                                ...token,
                                scopes: editor ? TMToMonacoToken(editor, token.scopes) : token.scopes.at(-1)
                            })),
                        };
                    }
                });
            })
    );
}

I am inspired by your solution, but I don't know where should I wrote this code in my project, as the function wireTmGrammars that you overwrite is in monaco-editor-textmate repository. can you help me ?

[kachurun]

@XiayidanAlimu This repository contains only two files. Just add it to your project and patch the function, or use patch-package for this.

[XiayidanAlimu]

Thank you for your kindly response. patch-package is a useful tool, which I haven't used before. At present, I have encountered some dependencies issues with my project folder node_modules, but I am hopefully expecting that your solution could work. have a nice day