carbon install via pip doesn't show up in "pip freeze" Edit

[mleinart]

Imported from https://bugs.launchpad.net/bugs/1022576

Reported by	jlintz
Date Created	Jul 09, 2012
Tags	[u'carbon', u'install', u'pip', u'setup']

[mleinart]

This appears to be a side effect of a default /opt/graphite install. This works:
$ PYTHONPATH=/opt/graphite/lib pip freeze

This is perhaps another argument for getting rid of /opt/graphite as a default install location, but that should be covered elsewhere

[robinsmidsrod]

@mleinart That workaround is not really an option when you use a configuration management system like saltstack/salt. The pip.installed command (actually state) checks if pip freeze has "carbon" in its list before it tries to install it. This causes the pip.installed commands to reinstall carbon every single time the configuration state check is executed.

Please reopen and link as dependent on said fix mentioned above. If you install to /opt/graphite, shouldn't the global pip index still be updated, or doesn't pip work like that?

[mleinart]

Yeah, there's no index - pip (actually distribute or setuptools) is looking through sys.path to find installed packages.

I'll reopen to track it but I dont think there's anything we can do in this case right now. Installing in a Virtualenv or in the system site-packages might be workarounds, but will require some extra flags passed to pip to override the default /opt/graphite prefix. See:
http://graphite.readthedocs.org/en/0.9.x/install-pip.html#installing-carbon-in-a-custom-location
It looks like install_options can be passed in here: http://docs.saltstack.org/en/latest/ref/modules/all/salt.modules.pip.html#salt.modules.pip.install

[robinsmidsrod]

This is basically the workaround I've had to set up to avoid carbon and graphite-web being reinstalled every single time state.highstate (in salt) is run:

pip.whisper:
    pip.installed:
        - name: whisper
        - require:
            - pkg: python-pip
            - pkg: python-dev

pip.carbon:
    cmd.run:
        - name: "pip install carbon"
        - unless: "test -e /opt/graphite/lib/carbon-0.9.10-py2.7.egg-info"
        - require:
            - pkg: python-pip
            - pip: pip.whisper
            - file: graphite.dir

pip.graphite-web:
    cmd.run:
        - name: "pip install graphite-web"
        - unless: "test -e /opt/graphite/webapp/graphite_web-0.9.10-py2.7.egg-info"
        - require:
            - pkg: python-pip
            - pip: pip.whisper
            - file: graphite.dir

As you can see, whisper installs cleanly, but for the two others I've had to resort to a manual test instead (which I'm sure could be rewritten to something cleaner if I knew more about pip).

[berndmweber]

Another way of dealing with this is by creating symlinks of the .egg-info files in the directory pip freeze expects them to be. On Ubuntu 12.04 you'd do something like:
sudo ln -s /opt/graphite/lib/carbon-0.9.10-py2.7.egg-info /usr/lib/python2.7/dist-packages/carbon-0.9.10-py2.7.egg-info
Or in Puppet:

  if $graphite::params::pkg_provider == 'pip' {
    file { '/usr/lib/python2.7/dist-packages/carbon-0.9.10-py2.7.egg-info' :
      ensure => link,
      target => '/opt/graphite/lib/carbon-0.9.10-py2.7.egg-info',
      require => Package [ $graphite::params::package_carbon ],
    }
  }

Same for graphite-web...

[robinsmidsrod]

@bernd-copperfroghosting Thanks for that tip. Adding in that link should most likely take care of things. I'll report back when I've had time to try it out.

[robinsmidsrod]

Doing this with in my graphite.sls file seems to work as expected.

carbon.egg-info.symlink:
    file.symlink:
        - name: /usr/lib/python2.7/dist-packages/carbon-0.9.10-py2.7.egg-info
        - target: /opt/graphite/lib/carbon-0.9.10-py2.7.egg-info
        - force: true

[amosshapira]

This issue is a major bummer since Puppet doesn't support passing arguments to pip, causing it never to find and keep re-installing the package.
Could you please address this?

[trbs]

In systems that allow you to control the shell environment in which pip executes you can also set PYTHONPATH to include /opt/graphite/lib and /opt/graphite/webapp.

PYTHONPATH="/opt/graphite/lib/:/opt/graphite/webapp/"

Which has the advantage that you do not have to symlink a specific version and remember to fix the symlink after upgrading.

In Ansible this works like:

---
# file: roles/graphite/tasks/pip.yml

- name: /opt/graphite/requirements.txt
  template: src=requirements.txt dest={{ graphite_root }}/requirements.txt owner=root group=root mode=0440
  tags: pip

- name: ensure graphite is installed (virtualenv /opt/graphite)
  pip: requirements=/opt/graphite/requirements.txt use_mirrors=yes virtualenv=/opt/graphite virtualenv_site_packages=no state=present
  environment:
    PYTHONPATH: "/opt/graphite/lib:/opt/graphite/webapp"
  tags: pip

[dmichel1]

@trbs thanks for the workaround

[lastorset]

mleinart:

This is perhaps another argument for getting rid of /opt/graphite as a default install location, but that should be covered elsewhere

This is issue #74.

[jssjr]

I'm really tempted to not address this in 0.9.13. Everyone already has a flavor of workaround that works and in 0.10.x we can begin to refactor away from the /opt/graphite prefix default.

Thoughts?

[amosshapira]

I'm OK with that.

On 27 September 2014 06:29, Scott Sanders [email protected] wrote:

I'm really tempted to not address this in 0.9.13. Everyone already has a
flavor of workaround that works and in 0.10.x we can begore to refactor
away from the /opt/graphite prefix default.

Thoughts?

—
Reply to this email directly or view it on GitHub
#86 (comment)
.

http://au.linkedin.com/in/gliderflyer

[jssjr]

Moved this to 0.10.0 milestone.

[obfuscurity]

We need to fix this for 0.9.13. After further consideration I just don't think it's reasonable to ask users to set PYTHONPATH like this to fix problems in both Carbon and Graphite-Web (graphite-project/graphite-web#651).

Note that Whisper works fine.

[obfuscurity]

Sorry for the milestone tom-foolery, this is indeed 1.0.0 material due to the prefix changes required.

[obfuscurity]

The installation docs now formally instruct the user to set PYTHONPATH before installing via pip. Fixed in graphite-project/graphite-web@1904348.