pycurl: Add curl as a depedency

[bzadm]

Adds support for pycurl via override. Tested on GNU/Linux with NixOS. Fixes #1283

Test

pyproject.toml:

[tool.poetry]
name = "pycurl-test"
version = "0.1.0"
description = ""
authors = ["nhr-zib <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.10"
pycurl = "7.45.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

flake.nix:

{
  description = "pycurl test";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
    poetry2nix = {
      url = "github:bzadm/poetry2nix/master";
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.flake-utils.follows = "flake-utils";
    };
  };

  outputs = { self, poetry2nix, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let pkgs = nixpkgs.legacyPackages.${system};
      in {
        devShells.default = pkgs.mkShell {
          buildInputs = [
            (poetry2nix.legacyPackages.${system}.mkPoetryEnv {
              pyproject = ./pyproject.toml;
              poetrylock = ./poetry.lock;
            })
          ];
        };
      });
}

test.py:

# See http://pycurl.io/docs/latest/quickstart.html#retrieving-a-network-resource
import pycurl
import re
try:
    from io import BytesIO
except ImportError:
    from StringIO import StringIO as BytesIO

headers = {}
def header_function(header_line):
    # HTTP standard specifies that headers are encoded in iso-8859-1.
    # On Python 2, decoding step can be skipped.
    # On Python 3, decoding step is required.
    header_line = header_line.decode('iso-8859-1')

    # Header lines include the first status line (HTTP/1.x ...).
    # We are going to ignore all lines that don't have a colon in them.
    # This will botch headers that are split on multiple lines...
    if ':' not in header_line:
        return

    # Break the header line into header name and value.
    name, value = header_line.split(':', 1)

    # Remove whitespace that may be present.
    # Header lines include the trailing newline, and there may be whitespace
    # around the colon.
    name = name.strip()
    value = value.strip()

    # Header names are case insensitive.
    # Lowercase name here.
    name = name.lower()

    # Now we can actually record the header name and value.
    # Note: this only works when headers are not duplicated, see below.
    headers[name] = value

buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.URL, 'http://pycurl.io')
c.setopt(c.WRITEFUNCTION, buffer.write)
# Set our header function.
c.setopt(c.HEADERFUNCTION, header_function)
c.perform()
c.close()

# Figure out what encoding was sent with the response, if any.
# Check against lowercased header name.
encoding = None
if 'content-type' in headers:
    content_type = headers['content-type'].lower()
    match = re.search('charset=(\S+)', content_type)
    if match:
        encoding = match.group(1)
        print('Decoding using %s' % encoding)
if encoding is None:
    # Default encoding for HTML is iso-8859-1.
    # Other content types may have different default encoding,
    # or in case of binary data, may have no encoding at all.
    encoding = 'iso-8859-1'
    print('Assuming encoding is %s' % encoding)

body = buffer.getvalue()
# Decode using the encoding we figured out.
print(body.decode(encoding))

Test the build and functionality:

$ nix develop -i -c python test.py
[ ... expected HTML output ]

[andersk]

This should not need to be propagated—after PycURL is built, the C headers for curl are not needed to use PycURL.

[takeda]

Similarly to the above comment. I believe all you need is just specify buildInputs and add pkgs.curl there.

nativeBuildInputs is meant for tooling that are typically tools that are needed during build time. Its purpose is clearer when you do cross compiling. So you would specify there things like cmake, for python it would be setuptools, poetry-core etc.

BTW: I see netcdf4 using propagatedBuildInputs but that likely is incorrect.

[andersk]

This may need to be native for the curl-config script. (But still needn’t be propagated.)

[adisbladis]

Then the solution is to add curl as both a build input and a native input.

[takeda]

ah, my bad.

[Chaostheorie]

Thank you for the review and explanation regarding the build inputs. I'll try to make the changes tomorrow.

[takeda]

@bzadm you should also rename old.propagatedBuildInputs to old.buildInputs

[l0b0]

Thanks! Would you also be willing to create tests for this package? If so, see

• tests/default.nix for the test entry and
• the tests/mutmut directory for a module test or tests/matplotlib-post-3-7 for a library test.