Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory limit exceeded during ghost install process #1898

Open
4 tasks done
danieldinter opened this issue Nov 11, 2024 · 9 comments · May be fixed by #1900
Open
4 tasks done

Memory limit exceeded during ghost install process #1898

danieldinter opened this issue Nov 11, 2024 · 9 comments · May be fixed by #1900

Comments

@danieldinter
Copy link

danieldinter commented Nov 11, 2024 •
edited
Loading

Summary

My Ghost blog is running on a virtual machine provided by an internet hosting service. They limit single processes to 1.5 GB memory usage. Processes exceeding 1.5 GB memory are automatically killed.

When I try to upgrade Ghost from v5.86.2 to the recent version (i.e. v5.100.1) using the Ghost CLI, the process is killed due to memory usage during the step that says
Installing dependencies > [5/5] Building fresh packages...

Steps to Reproduce & Log File

I switched away from trying to upgrade my blog (as it fails) to setting up a new blog on a green field on the same machine. That makes testing easier.

  1. Run the install command (notice the "Killed" at the end):
$ ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

✔ Checking system Node.js version - found v18.20.4
✔ Checking current folder permissions
✔ Checking memory availability
✔ Checking free space
✔ Checking for latest Ghost version
✔ Setting up install directory
☲ Downloading and installing Ghost v5.100.1 > Installing dependencies > [5/5] Building fresh packages...Killed
  1. Switch to current version directory: $ cd versions/5.100.1/
  2. Try to build dependencies manually (also gets killed):
$ yarn install
yarn install v1.22.22
[1/5] Validating package.json...
warning [email protected]: The engine "cli" appears to be invalid.
[2/5] Resolving packages...
warning Resolution field "[email protected]" is incompatible with requested version "moment@^2.29.1"
warning Resolution field "[email protected]" is incompatible with requested version "moment@^2.29.4"
warning Resolution field "[email protected]" is incompatible with requested version "[email protected]"
warning Resolution field "[email protected]" is incompatible with requested version "moment@^2.27.0"
warning Resolution field "[email protected]" is incompatible with requested version "[email protected]"
warning Resolution field "[email protected]" is incompatible with requested version "[email protected]"
warning Resolution field "[email protected]" is incompatible with requested version "[email protected]"
warning Resolution field "[email protected]" is incompatible with requested version "jackspeak@^3.1.2"
warning Resolution field "@tryghost/[email protected]" is incompatible with requested version "@tryghost/[email protected]"
warning Resolution field "@tryghost/[email protected]" is incompatible with requested version "@tryghost/[email protected]"
warning Resolution field "[email protected]" is incompatible with requested version "[email protected]"
warning Resolution field "[email protected]" is incompatible with requested version "[email protected]"
warning Resolution field "@tryghost/[email protected]" is incompatible with requested version "@tryghost/errors@^1.3.6"
warning Resolution field "@tryghost/[email protected]" is incompatible with requested version "@tryghost/errors@^1.3.6"
warning Resolution field "@tryghost/[email protected]" is incompatible with requested version "@tryghost/errors@^1.3.6"
warning Resolution field "[email protected]" is incompatible with requested version "[email protected]"
warning Resolution field "@tryghost/[email protected]" is incompatible with requested version "@tryghost/[email protected]"
warning Resolution field "@tryghost/[email protected]" is incompatible with requested version "@tryghost/[email protected]"
warning Resolution field "[email protected]" is incompatible with requested version "moment@^2.29.1"
[3/5] Fetching packages...
warning [email protected]: The engine "bun" appears to be invalid.
warning [email protected]: The engine "deno" appears to be invalid.
[4/5] Linking dependencies...
warning " > [email protected]" has incorrect peer dependency "knex@>=0.15.0 <0.22.0".
[5/5] Building fresh packages...
[-/6] ⠄ waiting...
[-/6] ⠄ waiting...
[-/6] ⠄ waiting...
Killed⠄ re2

The messages showing before the kill were:

[5/5] Building fresh packages...
[-/6] ⠂ waiting...
[-/6] ⠂ waiting...
[-/6] ⠂ waiting...
[4/6] ⠄ re2
[6/6] ⠄ @sentry/profiling-node

As it became apparent that memory could be a problem, I tried to set the NODE_OPTIONS parameter --max-old-space-size=1024:

  1. I tried setting it in ~/.npmrc: node-options=--max-old-space-size=1024
  2. I tried setting an environment variable:
$ export NODE_OPTIONS="--max-old-space-size=1024"
$ ghost install …

However, the parameter is not passed by node to yarn. The builds failed again. Although yarn lists the parameter:

$ yarn config list
yarn config v1.22.22
info yarn config
{
 ...
}
info npm config
{
  'node-options': '--max-old-space-size=1024',
  ...
}
Done in 0.07s.

The only way that works is using cross-env:

$ npm i -g cross-env
$ cross-env NODE_OPTIONS=--max-old-space-size=1024 yarn install

That way, the yarn install command finishes regularly. However, I cannot continue the Ghost CLI install process from this point, as it tells me the directory is non-empty:

$ ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable
Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

A SystemError occurred.

Message: Current directory is not empty, Ghost cannot be installed here.

Debug Information:
    OS: CentOS Linux, v7
    Node Version: v18.20.4
    Ghost-CLI Version: 1.26.1
    Environment: production
    Command: 'ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.

Solutions

If I may I like to propose two solutions:

  1. Support cross-env to pass node parameters to yarn.
  2. Allow manual build of dependencies, e.g. by adding a switch to Ghost CLI that ignores non-empty directories.

Technical details

This is automatically output by Ghost-CLI if an error occurs, please copy & paste:

  • OS: CentOS Linux 7 (Core)
  • Node Version: v18.20.4
  • Ghost-CLI Version: 1.26.1
  • Environment: production
  • Command: install

Bug submission checklist

Please fill out this checklist to acknowledge that you followed the requirements to submit a bug report.

  • Tried to find help in the forum & docs
  • Checked for existing issues
  • Attached log file
  • Provided technical details incl. operating system
@acburdine
Copy link
Member

so - adding cross-env wouldn't fix the issue here I think, but I do think we can pass the NODE_OPTIONS env var through to the underlying yarn install command.

acburdine added a commit to acburdine/Ghost-CLI that referenced this issue Nov 12, 2024
@acburdine
fix(yarn): passthrough NODE_OPTIONS env var to yarn install
b87dba4 
closes TryGhost#1898
- passthrough NODE_OPTIONS env var to yarn install if set
@acburdine acburdine linked a pull request Nov 12, 2024 that will close this issue
Draft
@acburdine
Copy link
Member

@danieldinter if you have the ability - would you be able to test the change in #1900? I wasn't able to replicate the behavior you're seeing locally, but the change in that PR should allow you to set NODE_OPTIONS='--max-old-space-size=1024' and have it be respected by yarn install.

@danieldinter
Copy link
Author

danieldinter commented Nov 12, 2024 •
edited
Loading

@acburdine thanks for your effort. I tested the PR and the process still gets killed. Looks like the parameter is not passed. What I did:

$ git clone https://github.com/TryGhost/Ghost-CLI.git
$ cd Ghost-CLI/
$ git fetch origin pull/1900/head:memoryfix
$ git checkout memoryfix
$ yarn install

$ yarn link
yarn link v1.22.22
warning There's already a linked binary called "ghost" in your global Yarn bin. Could not link this package's "ghost" bin entry.
success Registered "ghost-cli".
info You can now run `yarn link "ghost-cli"` in the projects where you want to use this package and it will be used instead.
Done in 0.14s.

$ cd ~/ghost-new/
$ yarn link "ghost-cli"
yarn link v1.22.22
success Using linked package for "ghost-cli".
Done in 0.10s.

$ rm -R content/ node_modules/ versions/ yarn.lock

$ export NODE_OPTIONS="--max-old-space-size=1024"
$ echo $NODE_OPTIONS
--max-old-space-size=1024

$ ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

✔ Checking system Node.js version - found v18.20.4
✔ Checking current folder permissions
✔ Checking memory availability
✔ Checking free space
✔ Checking for latest Ghost version
✔ Setting up install directory
☱ Downloading and installing Ghost v5.100.1 > Installing dependencies > [5/5] Building fresh packages...Killed

@vikaspotluri123
Copy link
Member

@dani I see a fetch but I don't see a checkout - can you confirm if you're running on main or this PR?

@danieldinter
Copy link
Author

@vikaspotluri123 yes, indeed, I did a checkout:

$ git fetch origin pull/1900/head:memoryfix
$ git checkout memoryfix

Sorry for the confusion. I also see the changes made in Ghost-CLI/lib/tasks/yarn-install.js on the file system.

@danieldinter
Copy link
Author

A problem might be that I first yarn linked the package and then ran rm because the first creates a symlink in node_modules/ which has then been deleted. I tested again:

$ cd ~/ghost-new/
$ yarn link "ghost-cli"
yarn link v1.22.22
success Using linked package for "ghost-cli".
Done in 0.12s.
$ ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

A SystemError occurred.

Message: Current directory is not empty, Ghost cannot be installed here.

Debug Information:
    OS: CentOS Linux, v7
    Node Version: v18.20.4
    Ghost-CLI Version: 1.26.1
    Environment: production
    Command: 'ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.
$ ls -la node_modules/
.. ghost-cli -> ../../.config/yarn/link/ghost-cli

I tried to install it globally:

$ npm uninstall -g ghost-cli
removed 596 packages in 6s
$ yarn global add file:/home/…/Ghost-CLI/

Then ran the ghost install command but again it was killed.

@danieldinter
Copy link
Author

I edited the file Ghost-CLI/lib/tasks/yarn-install.js to be:

            // #1898: pass through NODE_OPTIONS so max_old_space_size can be configured
            // in memory-constrained environments
            console.log("process.env.NODE_OPTIONS: " + process.env.NODE_OPTIONS);
            if (process.env.NODE_OPTIONS) {
                env.NODE_OPTIONS = process.env.NODE_OPTIONS;
                console.log("env.NODE_OPTIONS: " + env.NODE_OPTIONS);
            }

Running yarn global add file:/home/…/Ghost-CLI/ and ghost install ... leads to the output

✔ Checking for latest Ghost version
✔ Setting up install directory
☱ Downloading and installing Ghost v5.100.1 > Downloadingprocess.env.NODE_OPTIONS: --max-old-space-size=1024
env.NODE_OPTIONS: --max-old-space-size=1024
☱ Downloading and installing Ghost v5.100.1 > Installing dependencies > [5/5] Building fresh packages...Killed

So the variable is set but it is ignored by yarn. The same behaviour occurs when I manually run yarn install on the command line - it ignores NODE_OPTIONS. Usage of cross-env fixes this.

@acburdine
Copy link
Member

hmm - thanks for the additional tests/examples @danieldinter

I'll look more into what cross-env is doing that makes it work correctly with NODE_OPTIONS. If I can figure out what they're doing under the hood to make the node options accepted by yarn then I'll replicate that in the CLI 👍🏻

@acburdine
Copy link
Member

so I looked a bit further and we're also using cross-spawn under the hood (which cross-env is also using), though likely a rather old version of it.

I'll try to replicate the conditions you're experiencing to see if I can get a memory killed error, though I'm having a hard time finding a working copy of a centos7 iso that boots in a vm on my mac 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(yarn): passthrough NODE_OPTIONS env var to yarn install acburdine/Ghost-CLI
3 participants
@danieldinter @vikaspotluri123 @acburdine