poetry build: crash on encountering a symlink #3589

brechtm · 2021-01-20T14:08:48Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
OS version and name: macOS High Sierra 10.13.6
Poetry version: 1.1.4

Issue

I'm converting my project from a setuptools to Poetry. For building the sdist, Poetry already makes things easier since it excludes files listed in .gitignore files, where this information needs to be duplicated in the MANIFEST.in file when using setuptools.

However, I wanted to go one step further and just include all of my repository's files instead of listing them all, counting on the .gitignore files to exclude unwanted files. To this effect, I am using this configuration in pyproject.toml:

include = [{ path = "**/*", format = "sdist" }]

This should work well, since I typically want all files under source control to be included in the sdist, and this will not require adjustments to pyproject.toml when new files are added to the repository. I might want to exclude certain files such as CI service configuration files, however.

Unfortuntely, Poetry crashes when encountering a symlink in a .tox virtualenv directory with when building the sdist:

  ValueError

  '/Users/brechtm/.pyenv/versions/3.7.7/bin/python3.7' does not start with '/Users/brechtm/Documents/Code/rinohtype'

Since the parent .tox directory is excluded by .gitignore, Poetry shouldn't bother checking any of the files under it.

Full output of: poetry build -vvv -f sdist

Using virtualenv: /Users/brechtm/Documents/Code/rinohtype/.venv
Building rinohtype (0.5.0-dev.0)
  - Building sdist
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/__init__.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/__main__.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/annotation.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/attribute.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/color.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/csl_formatter.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/dimension.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/document.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/draw.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/element.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/flowable.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/glossary.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/highlight.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/hyphenator.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/image.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/index.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/inline.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/layout.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/number.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/paper.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/paragraph.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/reference.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/resource.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/strings.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/structure.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/style.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/styleds.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/styles.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/table.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/template.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/text.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/util.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/src/rinoh/warnings.py
  - Adding: /Users/brechtm/Documents/Code/rinohtype/.DS_Store
  - Adding: /Users/brechtm/Documents/Code/rinohtype/.bumpversion.cfg
  - Adding: /Users/brechtm/Documents/Code/rinohtype/.coverage
  - Adding: /Users/brechtm/Documents/Code/rinohtype/.coveragerc
  - Adding: /Users/brechtm/Documents/Code/rinohtype/.envrc
  - Adding: /Users/brechtm/Documents/Code/rinohtype/.gitattributes
  - Adding: /Users/brechtm/Documents/Code/rinohtype/.gitignore
  - Adding: /Users/brechtm/Documents/Code/rinohtype/.gitmodules
  - Adding: /Users/brechtm/Documents/Code/rinohtype/.python-version

  Stack trace:

  10  ~/.poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py:131 in run
       129│             parsed_args = resolved_command.args
       130│
     → 131│             status_code = command.handle(parsed_args, io)
       132│         except KeyboardInterrupt:
       133│             status_code = 1

   9  ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:120 in handle
       118│     def handle(self, args, io):  # type: (Args, IO) -> int
       119│         try:
     → 120│             status_code = self._do_handle(args, io)
       121│         except KeyboardInterrupt:
       122│             if io.is_debug():

   8  ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:171 in _do_handle
       169│         handler_method = self._config.handler_method
       170│
     → 171│         return getattr(handler, handler_method)(args, io, self)
       172│
       173│     def __repr__(self):  # type: () -> str

   7  ~/.poetry/lib/poetry/_vendor/py3.8/cleo/commands/command.py:92 in wrap_handle
        90│         self._command = command
        91│
     →  92│         return self.handle()
        93│
        94│     def handle(self):  # type: () -> Optional[int]

   6  ~/.poetry/lib/poetry/console/commands/build.py:36 in handle
       34│
       35│         builder = Builder(self.poetry)
     → 36│         builder.build(fmt, executable=self.env.python)
       37│

   5  ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/masonry/builder.py:30 in build
       28│
       29│         for builder in builders:
     → 30│             builder(self._poetry, executable=executable).build()
       31│

   4  ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/masonry/builders/sdist.py:77 in build
        75│             tar_dir = "{}-{}".format(self._package.pretty_name, self._meta.version)
        76│
     →  77│             files_to_add = self.find_files_to_add(exclude_build=False)
        78│
        79│             for file in sorted(files_to_add, key=lambda x: x.relative_to_source_root()):

   3  ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/masonry/builders/sdist.py:307 in find_files_to_add
       305│         self, exclude_build=False
       306│     ):  # type: (bool) -> Set[BuildIncludeFile]
     → 307│         to_add = super(SdistBuilder, self).find_files_to_add(exclude_build)
       308│
       309│         # add any additional files, starting with all LICENSE files

   2  ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/masonry/builders/builder.py:166 in find_files_to_add
       164│
       165│                             if not current_file.is_dir() and not self.is_excluded(
     → 166│                                 include_file.relative_to_source_root()
       167│                             ):
       168│                                 to_add.add(include_file)

   1  ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/masonry/builders/builder.py:362 in relative_to_source_root
       360│     def relative_to_source_root(self):  # type(): -> Path
       361│         if self.source_root is not None:
     → 362│             return self.path.relative_to(self.source_root)
       363│         return self.path
       364│

  ValueError

  '/Users/brechtm/.pyenv/versions/3.7.7/bin/python3.7' does not start with '/Users/brechtm/Documents/Code/rinohtype'

  at ~/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py:904 in relative_to
       900│         n = len(to_abs_parts)
       901│         cf = self._flavour.casefold_parts
       902│         if (root or drv) if n == 0 else cf(abs_parts[:n]) != cf(to_abs_parts):
       903│             formatted = self._format_parsed_parts(to_drv, to_root, to_parts)
    →  904│             raise ValueError("{!r} does not start with {!r}"
       905│                              .format(str(self), str(formatted)))
       906│         return self._from_parsed_parts('', root if n == 1 else '',
       907│                                        abs_parts[n:])
       908│

The text was updated successfully, but these errors were encountered:

finswimmer · 2021-01-20T20:00:45Z

Hello @brechtm,

as you said, poetry automatically excludes all file listed in .gitignore. Unlike setuptools poetry includes any file it can found in the package by default. So poetry should give you your desired result out-of-the box, and the include directive in the pyproject.toml should not be necessary.

fin swimmer

brechtm · 2021-01-20T21:35:22Z

as you said, poetry automatically excludes all file listed in .gitignore. Unlike setuptools poetry includes any file it can found in the package by default. So poetry should give you your desired result out-of-the box, and the include directive in the pyproject.toml should not be necessary.

I was not aware of that. In that case, the fact that my Python package is located in the src directory might be an important factor. This is of course configured in pyproject.toml_:

packages = [{ include = "rinoh", from = "src" }]

brechtm · 2021-01-20T21:41:07Z

Forget that. That doesn't make a difference.

I now understand the confusion however. I want to include all files in my repository in the sdist, not just the the Python packages. So that includes tests and configuration files for tox and other tools. I have updated my first message to hopefully make this clearer.

sinoroc · 2021-01-21T17:54:02Z

I tried to sum up the situation in that ticket: #3380. Maybe that can somehow help you...

brechtm · 2021-01-22T09:08:50Z

Thanks for pointer, @sinoroc. The problem is more complex than I thought. I ran into some strange things in the code that might explain some of these issues when working on my PR. I think the best approach would be to list all requirements (collecting information for related tickets) and design a solution that can handle them.

albireox · 2021-01-22T21:27:47Z

I'm having an issue that may seem related. My library coordio lives in python/coordio in my repository. I have a build.py that compiles an extension whose .so gets build into python.coordio.libsofa. My pyproject.toml includes

packages = [
    { include = "coordio", from = "python" }
]
include = [
    {path = "python/coordio/etc/*"},
    {path = "cextern/**/*", format='sdist'},
    {path = "LICENSE.md"},
    {path = "python/coordio/*.so",  format = "wheel"}
]

[tool.poetry.build]
script = "build.py"
generate-setup-file = false

This used to work fine a few months ago, I think, but now when I run pip install . the shared library is copied to site-packages/python/coordio/libsofa.so instead to site-packages/coordio/libsofa.so with the other files, and the installed library cannot import it. It also copies the LICENSE.md file to the root of site-packages.

If I move the library to the root of the repository (remove the python/ level) and modify the build.py and pyproject.toml accordingly, everything seems to work as expected.

christopherpickering · 2022-02-14T20:03:23Z

As a note to other people finding this thread... I had the same issue, or at least error message, but it was the result of having symlinks in a subdirectory of my project. Removing the symlink (temp files generated by my code) resolved the problem.

brechtm added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Jan 20, 2021

brechtm linked a pull request Jan 21, 2021 that will close this issue

Conceptual fix for issue #3589 python-poetry/poetry-core#126

Draft

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

poetry build: crash on encountering a symlink #3589

poetry build: crash on encountering a symlink #3589

brechtm commented Jan 20, 2021 •

edited

Loading

finswimmer commented Jan 20, 2021

brechtm commented Jan 20, 2021

brechtm commented Jan 20, 2021 •

edited

Loading

sinoroc commented Jan 21, 2021

brechtm commented Jan 22, 2021

albireox commented Jan 22, 2021

christopherpickering commented Feb 14, 2022

poetry build: crash on encountering a symlink #3589

poetry build: crash on encountering a symlink #3589

Comments

brechtm commented Jan 20, 2021 • edited Loading

Issue

finswimmer commented Jan 20, 2021

brechtm commented Jan 20, 2021

brechtm commented Jan 20, 2021 • edited Loading

sinoroc commented Jan 21, 2021

brechtm commented Jan 22, 2021

albireox commented Jan 22, 2021

christopherpickering commented Feb 14, 2022

brechtm commented Jan 20, 2021 •

edited

Loading

brechtm commented Jan 20, 2021 •

edited

Loading