push of locally built snap fails on arm64 device #5132

kenvandine · 2024-10-24T13:51:29Z

Bug Description

When attempting to push a locally built snap to the store with my qualcomm based laptop I get the following stderr:

FATAL ERROR: Data queue size is too large

To Reproduce

snapcraft push SOMELOCALLYBUILT.snap

Environment

snapcraft: 12827

uname: Linux xps13-qc 6.11.0-35-qcom-x1e #35-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 23 15:18:06 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

snapcraft.yaml

NA

Relevant log output

2024-10-22 16:18:30.743 Starting snapcraft, version 8.4.3
2024-10-22 16:18:30.743 Log verbosity level set to BRIEF
2024-10-22 16:18:30.743 Preparing application...
2024-10-22 16:18:30.744 Configuring application...
2024-10-22 16:18:30.744 Setting up ConfigService
2024-10-22 16:18:30.758 Build plan: platform=None, build_for=None
2024-10-22 16:18:30.758 Running snapcraft push on host
2024-10-22 16:18:30.819 snapcraft internal error: SnapDataExtractionError('thunderbird_128.3.3esr-1_arm64.snap')
2024-10-22 16:18:30.829 Traceback (most recent call last):
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/snapcraft_legacy/_store.py", line 58, in get_data_from_snap_file
2024-10-22 16:18:30.829     output = subprocess.check_output(
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/usr/lib/python3.10/subprocess.py", line 421, in check_output
2024-10-22 16:18:30.829     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/usr/lib/python3.10/subprocess.py", line 526, in run
2024-10-22 16:18:30.829     raise CalledProcessError(retcode, process.args,
2024-10-22 16:18:30.829 subprocess.CalledProcessError: Command '['/snap/snapcraft/12827/usr/bin/unsquashfs', '-d', '/tmp/tmpoh95i6tt/squashfs-root', PosixPath('thunderbird_128.3.3esr-1_arm64.snap'), 'meta/snap.yaml', 'snap/manifest.yaml']' returned non-zero exit status 1.
2024-10-22 16:18:30.829 
2024-10-22 16:18:30.829 During handling of the above exception, another exception occurred:
2024-10-22 16:18:30.829 Traceback (most recent call last):
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/craft_application/application.py", line 568, in run
2024-10-22 16:18:30.829     return_code = self._run_inner()
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/snapcraft/application.py", line 215, in _run_inner
2024-10-22 16:18:30.829     return_code = super()._run_inner()
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/craft_application/application.py", line 549, in _run_inner
2024-10-22 16:18:30.829     return_code = dispatcher.run() or os.EX_OK
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 487, in run
2024-10-22 16:18:30.829     return self._loaded_command.run(self._parsed_command_args)
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/snapcraft/commands/upload.py", line 116, in run
2024-10-22 16:18:30.829     snap_yaml, manifest_yaml = get_data_from_snap_file(snap_file)
2024-10-22 16:18:30.829   File "/snap/snapcraft/12827/lib/python3.10/site-packages/snapcraft_legacy/_store.py", line 70, in get_data_from_snap_file
2024-10-22 16:18:30.829     raise SnapDataExtractionError(os.path.basename(snap_path))
2024-10-22 16:18:30.829 snapcraft_legacy.internal.errors.SnapDataExtractionError: Cannot read data from snap 'thunderbird_128.3.3esr-1_arm64.snap'. The file may be corrupted.



### Additional context

_No response_

The text was updated successfully, but these errors were encountered:

syncronize-issues-to-jira · 2024-10-24T13:52:49Z

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-3614.

This message was autogenerated

mr-cal · 2024-10-24T14:20:57Z

I was unable to reproduce this on an M1.

@kenvandine was unable to reproduce this when running the same subprocess call manually /snap/snapcraft/current/usr/bin/unsquashfs -d /tmp/tmpoh95i6tt/squashfs-root SOMELOCALLYBUILT.snap meta/snap.yaml snap/manifest.yaml on the same machine. It could be useful to try snap run --shell snapcraft and then try unsquashing. Perhaps it's something in snapd's environment.

The snap/manifest.yaml is a red herring - unsquashfs should not fail if the file does not exist.

The FATAL ERROR: Data queue size is too large error comes from unsquashfs: https://github.com/plougher/squashfs-tools/blob/0d1e6b0944f3938962238d886d9d1a8bfdf4dfcc/squashfs-tools/unsquashfs.c#L4607-L4608

Snapcraft is also using the version of squashfs-tools that shipped with Jammy. There have been many bugfixes since then, so it would be instructive to build and test with a copy of snapcraft that uses a newer version.

dariuszd21 · 2024-11-07T20:59:57Z

What's the size of your tmpfs ?
As I can see you're on XPS with X1 chip, so you're probably using 24.10.
May it be related with the oracular change that /tmp is tmpfs mount ?
Out of curiosity, could you check the size of your /tmp dir?

df -h /tmp

https://discourse.ubuntu.com/t/oracular-oriole-release-notes/44878

kenvandine · 2024-11-08T18:21:15Z

What's the size of your tmpfs ? As I can see you're on XPS with X1 chip, so you're probably using 24.10. May it be related with the oracular change that /tmp is tmpfs mount ? Out of curiosity, could you check the size of your /tmp dir?
df -h /tmp
https://discourse.ubuntu.com/t/oracular-oriole-release-notes/44878

You are correct, I am on 24.10 and it is using tmpfs for /tmp

df reports /tmp is 15G with 15G available

dariuszd21 · 2024-11-14T21:37:52Z

Out of curiosity what's the ulimit -n result on your laptop, I found an interesting issue that may be causing this kind of problems

kenvandine · 2024-11-14T22:37:27Z

Out of curiosity what's the ulimit -n result on your laptop, I found an interesting issue that may be causing this kind of problems

ulimit is 122376

sergio-costas · 2024-11-18T09:36:44Z

I had a similar error too: https://forum.snapcraft.io/t/fatal-error-data-queue-size-is-too-large/43875

I did a test with this little program:

#!/usr/bin/env python3

import subprocess
import sys
import os
from pathlib import Path

temp_dir = "."
snap_path = sys.argv[1]

output = subprocess.check_output(
    [
        "/snap/snapcraft/current/usr/bin/unsquashfs",
        "-d",
        os.path.join(temp_dir, "squashfs-root"),
        snap_path,
        # cygwin unsquashfs on windows uses unix paths.
        Path("meta", "snap.yaml").as_posix(),
        Path("snap", "manifest.yaml").as_posix(),
    ]
)
print(output)

and it fails. But if I replace /snap/snapcraft/current/usr/bin/unsquashfs with /usr/bin/unsquashfs, it does work.

The snap has unsquashfs version 4.5, while my system has version 4.6.1.

EDIT: my test was done in an AMD x86_64 laptop.

sergio-costas · 2024-11-18T09:54:30Z

BTW: maybe the size of the snap has something to do. Mine is 1,7GB (yes, yes, I know...)

When trying to push a big snap, snapcraft fails with the error FATAL ERROR: Data queue size is too large This is a bug in unsquashfs tool 4.5. But it has been fixed in 4.6.1, the current stable. This patch builds squashfs-tools version 4.6.1 from the GIT repository, thus fixing this problem. Fix canonical#5132

sergio-costas · 2024-11-18T12:45:27Z

Hey @kenvandine , can you test this patch when you have some spare time, please? It should fix your problem: #5147

It fixes my problem, at least...

kenvandine · 2024-11-18T16:26:28Z

Hey @kenvandine , can you test this patch when you have some spare time, please? It should fix your problem: #5147

It fixes my problem, at least...

I can confirm it does fix the problem I ran into on my arm64 device.

sergio-costas · 2024-11-18T16:32:55Z

@kenvandine Thanks!!!

When trying to push a big snap, snapcraft fails with the error FATAL ERROR: Data queue size is too large This is a bug in unsquashfs tool 4.5. But it has been fixed in 4.6.1, the current stable. This patch builds squashfs-tools version 4.6.1 from the GIT repository, thus fixing this problem. Fix canonical#5132

mr-cal added bug Actual bad behavior that don't fall into maintenance or documentation triaged labels Oct 24, 2024

sergio-costas mentioned this issue Nov 18, 2024

fix(unsquashfs): fix Fatal Error when using unsquashfs #5147

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

push of locally built snap fails on arm64 device #5132

push of locally built snap fails on arm64 device #5132

kenvandine commented Oct 24, 2024

syncronize-issues-to-jira bot commented Oct 24, 2024

mr-cal commented Oct 24, 2024 •

edited

Loading

dariuszd21 commented Nov 7, 2024

kenvandine commented Nov 8, 2024

dariuszd21 commented Nov 14, 2024

kenvandine commented Nov 14, 2024

sergio-costas commented Nov 18, 2024 •

edited

Loading

sergio-costas commented Nov 18, 2024

sergio-costas commented Nov 18, 2024 •

edited

Loading

kenvandine commented Nov 18, 2024

sergio-costas commented Nov 18, 2024

push of locally built snap fails on arm64 device #5132

push of locally built snap fails on arm64 device #5132

Comments

kenvandine commented Oct 24, 2024

Bug Description

To Reproduce

Environment

snapcraft.yaml

Relevant log output

syncronize-issues-to-jira bot commented Oct 24, 2024

mr-cal commented Oct 24, 2024 • edited Loading

dariuszd21 commented Nov 7, 2024

kenvandine commented Nov 8, 2024

dariuszd21 commented Nov 14, 2024

kenvandine commented Nov 14, 2024

sergio-costas commented Nov 18, 2024 • edited Loading

sergio-costas commented Nov 18, 2024

sergio-costas commented Nov 18, 2024 • edited Loading

kenvandine commented Nov 18, 2024

sergio-costas commented Nov 18, 2024

mr-cal commented Oct 24, 2024 •

edited

Loading

sergio-costas commented Nov 18, 2024 •

edited

Loading

sergio-costas commented Nov 18, 2024 •

edited

Loading