Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build fails without @aws-sdk/client-s3 package in package.json #330

Open
MaksimDedov opened this issue Jul 23, 2024 · 15 comments · May be fixed by #331
Open

Build fails without @aws-sdk/client-s3 package in package.json #330

MaksimDedov opened this issue Jul 23, 2024 · 15 comments · May be fixed by #331

Comments

@MaksimDedov
Copy link

Hey!
When I install the unzipper npm package and don't install separately @aws-sdk/client-s3 I get the build exception like below.
Please move @aws-sdk/client-s3 to required dependencies for automatic installation, or fix it usage.

Thanks in advance

ERROR in ./node_modules/unzipper/lib/Open/index.js 97:52-81
Module not found: Error: Can't resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
  Parsed request is a module
  using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./lib/Open)
    resolve as module
      E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open\node_modules doesn't exist or is not a directory
      E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\node_modules doesn't exist or is not a directory
      looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules
        single file module
          using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
            no extension
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
            .ts
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.ts doesn't exist
            .js
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.js doesn't exist
            .json
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.json doesn't exist
            .node
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.node doesn't exist
        E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
      E:\Work\finsemble-mono\packages\electron-adapter\node_modules\node_modules doesn't exist or is not a directory
      looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules  
        single file module
          using description file: E:\Work\finsemble-mono\packages\electron-adapter\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
            no extension
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
            .ts
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.ts doesn't exist
            .js
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.js doesn't exist
            .json
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.json doesn't exist
            .node
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.node doesn't exist
        E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
      E:\Work\finsemble-mono\packages\node_modules doesn't exist or is not a directory      
      looking for modules in E:\Work\finsemble-mono\node_modules
        single file module
          using description file: E:\Work\finsemble-mono\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
            no extension
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
            .ts
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.ts doesn't exist       
            .js
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.js doesn't exist       
            .json
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.json doesn't exist     
            .node
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.node doesn't exist     
        E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
      E:\Work\node_modules doesn't exist or is not a directory
      E:\node_modules doesn't exist or is not a directory
 @ ./node_modules/unzipper/unzip.js 5:0-36
 @ ./src/main/ExternalApplicationManager.ts 4:0-32 70:27-43
 @ ./src/main/Main.ts 4:0-74 120:43-69
 @ ./src/app.ts 12:0-35 100:17-21

ERROR in E:\Work\finsemble-mono\packages\electron-adapter\src\main\ExternalApplicationManager.ts
33:20-40
[tsl] ERROR in E:\Work\finsemble-mono\packages\electron-adapter\src\main\ExternalApplicationManager.ts(33,21)
      TS2307: Cannot find module '@aws-sdk/client-s3' or its corresponding type declarations.
ts-loader-default_04bb2ec108ef140d

webpack 5.91.0 compiled with 2 errors and 2 warnings in 18167 ms
@aashimawadhwaa
Copy link

@MaksimDedov plus one facing the same issue.

@revolunet
Copy link

same here

@zachstence zachstence linked a pull request Jul 28, 2024 that will close this issue
Open
@ZJONSSON
Copy link
Owner

To keep node-unzipper super small, a decision was made to not include optional third party sdks as a part of the library itself. unzipper has plenty of users that do not require the s3 features and it would be very inefficient to force them to do so (example: https://www.npmjs.com/package/exceljs)

If you want to use the aws_s3 you will have to ensure that the sdk is installed, i.e. npm install unzipper @aws-sdk/client-s3, perhaps the right thing to do is clarify this in the README

@ZJONSSON
Copy link
Owner

@MaksimDedov can you confirm the build error only occurs because you are actually use the aws_v3 in the build process?

@MaksimDedov
Copy link
Author

@ZJONSSON In my code I use unzipper in the next way and only once:

const unzipperStream = unzipper.Extract({ path: downloadAsset.assetFolder });
fileStream.pipe(unzipperStream);

I don't call any other functions. The downloadAsset.assetFolder is the path to local folder on machine, fileStream is Node.js transform stream. As I understood the Extract uses aws_v3 somewhere under the hood in ./node_modules/unzipper/lib/Open/index.js 97:52-81

The build error occurs if I don't add the aws_v3 in my package.json near unzipper

@ZJONSSON
Copy link
Owner

Thank that is very helpful. This seems to be a typescript/ts-loader compilation error not a runtime error. Seems like the appropriate fix here is a simple @ts-ignore.

@ZJONSSON
Copy link
Owner

@MaksimDedov can you try with [email protected] that was just published?

@MaksimDedov
Copy link
Author

unfortunately I see the same issue for [email protected]:

ERROR in ./node_modules/unzipper/lib/Open/index.js 98:52-81
Module not found: Error: Can't resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
  Parsed request is a module
  using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./lib/Open)
    resolve as module
      E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open\node_modules doesn't exist or is not a directory
      E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\node_modules doesn't exist or is not a directory
      looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules
        single file module
          using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
            no extension
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
            .ts
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.ts doesn't exist
            .js
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.js doesn't exist
            .json
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.json doesn't exist
            .node
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.node doesn't exist
        E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
      E:\Work\finsemble-mono\packages\electron-adapter\node_modules\node_modules doesn't exist or is not a directory
      looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules
        single file module
          using description file: E:\Work\finsemble-mono\packages\electron-adapter\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
            no extension
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
            .ts
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.ts doesn't exist
            .js
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.js doesn't exist
            .json
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.json doesn't exist
            .node
              E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.node doesn't exist
        E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
      E:\Work\finsemble-mono\packages\node_modules doesn't exist or is not a directory
      looking for modules in E:\Work\finsemble-mono\node_modules
        single file module
          using description file: E:\Work\finsemble-mono\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
            no extension
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
            .ts
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.ts doesn't exist
            .js
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.js doesn't exist
            .json
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.json doesn't exist
            .node
              E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.node doesn't exist
        E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
      E:\Work\node_modules doesn't exist or is not a directory
      E:\node_modules doesn't exist or is not a directory
 @ ./node_modules/unzipper/unzip.js 5:0-36
 @ ./src/main/ExternalApplicationManager.ts 4:0-32 70:27-43
 @ ./src/main/Main.ts 4:0-74 120:43-69
 @ ./src/exports.ts 31:24-46

webpack 5.91.0 compiled with 1 error and 2 warnings in 19696 ms

@Juknum
Copy link

Juknum commented Aug 7, 2024 •
edited
Loading

Same error on my side ^^"

"unzipper": "^0.12.3"

Stack trace from Next:

Module not found: Can't resolve '@aws-sdk/client-s3'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/unzipper/unzip.js
./src/server/actions/files.ts
./src/server/data/mods.ts
./node_modules/next/dist/build/webpack/loaders/next-flight-action-entry-loader.js?actions=%5B%5B%22E%3A%5C%5CGitHub%5C%5Cfaithful%5C%5Cwebsite%5C%5Csrc%5C%5Cserver%5C%5Cdata%5C%5Cmods.ts%22%2C%5B%22getModsWithVersions%22%2C%22deleteMod%22%2C%22updateModPicture%22%2C%22voidMods%22%2C%22getMods%22%2C%22getModsFromIds%22%2C%22getModWithModVersions%22%2C%22updateMod%22%2C%22createMod%22%2C%22modHasUnknownVersion%22%5D%5D%2C%5B%22E%3A%5C%5CGitHub%5C%5Cfaithful%5C%5Cwebsite%5C%5Csrc%5C%5Cserver%5C%5Cdata%5C%5Cmods-version.ts%22%2C%5B%22getModVersionProgression%22%2C%22createModVersion%22%2C%22getSupportedMinecraftVersions%22%2C%22removeModpackFromModVersion%22%2C%22getModsVersionsFromResources%22%2C%22updateModVersion%22%2C%22deleteModVersion%22%2C%22getModVersionsWithModpacks%22%2C%22getNumberOfTextureFromModVersion%22%2C%22getModVersions%22%2C%22addModVersionsFromJAR%22%5D%5D%5D&__client_imported__=true!
 GET / 500 in 1049ms
 GET /_next/static/webpack/a1548000c16fc3ef.webpack.hot-update.json 500 in 1028ms
 ⚠ Fast Refresh had to perform a full reload due to a runtime error.

And the only thing I'm doing with unzipper is the following:

const bytes   = await jar.arrayBuffer(); // where jar is a File
const buffer  = Buffer.from(bytes);
const archive = await unzipper.Open.buffer(buffer);

@cordant-jd
Copy link

cordant-jd commented Aug 8, 2024 •
edited
Loading

Similar issue when using "unzipper": "^0.12.3" and NextJS

> next build

  ▲ Next.js 14.2.5
  - Environments: .env

   Creating an optimized production build ...
Failed to compile.

./node_modules/unzipper/lib/Open/index.js
Module not found: Can't resolve '@aws-sdk/client-s3'

https://nextjs.org/docs/messages/module-not-found

Temporary workaround: Add client-s3 to devDependencies

npm i -D @aws-sdk/client-s3@latest

@Juknum Juknum mentioned this issue Aug 19, 2024
Open
@RCVarley
Copy link

RCVarley commented Sep 17, 2024 •
edited
Loading

I'm getting this error in my vue app

My Code
This is the only instance where I'm using the library, and it builds fine without it.

const directory = await Unzipper.Open.buffer(newFile)

Terminal Error

13:29:14 [vite] hmr update /src/pages/media/add/media.vue, /src/components/media/AddFiles.vue (x2)
✘ [ERROR] Could not resolve "@aws-sdk/client-s3"

    node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/Open/index.js:98:60:
      98 │     const { GetObjectCommand, HeadObjectCommand } = require('@aws-sdk/client-s3');
         ╵                                                             ~~~~~~~~~~~~~~~~~~~~

  You can mark the path "@aws-sdk/client-s3" as external to exclude it from the bundle, which will
  remove this error and leave the unresolved path in the bundle. You can also surround this
  "require" call with a try/catch block to handle this failure at run-time instead of bundle-time.

13:29:17 [vite] error while updating dependencies:
Error: Build failed with 1 error:
node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/Open/index.js:98:60: ERROR: Could not resolve "@aws-sdk/client-s3"
    at failureErrorWithLog (C:\Users\ryanvarley\Code\cms.frontend.ui\node_modules\.pnpm\[email protected]\node_modules\esbuild\lib\main.js:1651:15)
    at C:\Users\ryanvarley\Code\cms.frontend.ui\node_modules\.pnpm\[email protected]\node_modules\esbuild\lib\main.js:1059:25
    at C:\Users\ryanvarley\Code\cms.frontend.ui\node_modules\.pnpm\[email protected]\node_modules\esbuild\lib\main.js:1527:9
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

UPDATE 1
I tried adding in the error handling as described by the message in my terminal and it got me past the error. Unfortunately I'm getting some other errors now:

TypeError: util.inherits is not a function
    at node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/PullStream.js (PullStream.js:18:6)
    at __require2 (chunk-GFT2G5UO.js?v=9e3c5976:18:50)
    at node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/parse.js (parse.js:4:20)
    at __require2 (chunk-GFT2G5UO.js?v=9e3c5976:18:50)
    at node_modules/.pnpm/[email protected]/node_modules/unzipper/unzip.js (unzip.js:2:17)
    at __require2 (chunk-GFT2G5UO.js?v=9e3c5976:18:50)
    at unzip.js:5:16

Is Unzipper not designed to work within a non-server environment or something? If it's any help, I'm using Node v20.14.0 and I'm compiling to ESNext

UPDATE 2
util is a node specific module that is not available in the browser. A step towards solving this is to polyfill those modules. As my project uses vite, this is best achieved with a package like https://github.com/davidmyersdev/vite-plugin-node-polyfills#vite-plugin-node-polyfills. That said, this still didn't quite work for me, so I'm going to try a browser first package like jszip.

This package does look really nice though!

@minawalphonce
Copy link

@ZJONSSON
it is a good idea to keep the package small. but since you are using require then it failed to bundle or build in the absence of the optional packages
i recommend to use something like "optional-require" so it wont fail and you can check the dependency you need before using it

@rnbrady
Copy link

rnbrady commented Dec 2, 2024

This is a bundling issue. Bundler can't tell at build time whether the dependency will be used or not, so tries to bundle it but it's not installed.

For those using Nextjs (or a webpack based framework), something like the following will work as an alternative to installing the dependency:

const nextConfig: NextConfig = {
  webpack: (config) => {
    config.resolve.alias = {
      ...config.resolve.alias,
      "@aws-sdk/client-s3": false,
    };
  }

@rnbrady
Copy link

rnbrady commented Dec 2, 2024

@ZJONSSON how about having different entry points for this? Something like:

import { Open } from `unzipper/s3`

I think you can do this within the one package by splitting out the code and modifying your package.json:

"name": "unzipper",
  "exports": {
    ".": "./unzip.js",
    "./s3": "./lib/Open/s3.js",
  }
}

Alternatively listing @aws-sdk/client-s3 as a peerDependency might generate more helpful logs.

PS: great package, thanks for shipping! ❤️

@axel7083
Copy link

axel7083 commented Dec 4, 2024 •
edited
Loading

I am also facing the problem Error: Cannot find module '@aws-sdk/client-s3' with vite and version 0.12.3

"@aws-sdk/client-s3" is imported by "@aws-sdk/client-s3?commonjs-external", but could not be resolved – treating it as an external dependency.

⚠️ rollback to version 0.11.6 fixed the problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: move @aws-sdk/client-s3 out of devDependencies zachstence/node-unzipper
10 participants
@revolunet @rnbrady @ZJONSSON @minawalphonce @RCVarley @axel7083 @Juknum @MaksimDedov @cordant-jd @aashimawadhwaa