memcached

#!/usr/bin/perl

# Commandline for dev testing:
# MUNIN_CAP_MULTIGRAPH=1 host=memcached-vip perl ./memcached

=head1 MEMCACHED

Memcached Multi - A Plugin to monitor Memcached Servers.  This is the
collect all the things all the time version using multigraphing.  See
also memcached_multi_ and memcached_.

The difference between this memcached Munin plugin and others that
exists, is that others don't expose slab information from memcached,
so you can better tune your memcached interaction / stability / etc.
With this plugin we leverage multigraph capabilities in Munin to
"hide" the slab information underneath of their parent graphs.

=head2 Compatability

This plugin provide the same kinds of stats as other older memcached
plugins, but it does not provide compatability with the memcached_
plugin provided in earlier versions of munin because the differences
are too great.  This plugin is compatible with I<some> but not all
versions of the memcache_multi_ plugin. This is because the different
versions have been using incompatbile graph names.

This plugin is compatible with the latest version provided in Matt
West's github repository.

So if you use most other versions of old memcached* plugins you'll
loose your history unless you retain the old plugin as well.  They
will co-exist peacefuly.

=head1 MUNIN NODE CONFIGURATION

The following configuration information can be overridden by placing
environment definitions like shown here, in a file located in
/etc/munin/plugin-conf.d.  The following are the default values:

  [memcached_multi_*]
    env.host 127.0.0.1
    env.port 11211
    env.timescale 3
    env.cmds get set delete incr decr touch
    env.leitime -1

=head2 MUNIN NODE ENVIRONMENT CONFIGURATION EXPLANATION

=over 4

=item host

host we are going to monitor, this can be used to specify a unix
socket.

=item port

port we are connecting to, in order to gather stats

=item timescale

what time frame do we want to format our graphs too.  The meaning of
the numbers are as follows:

=over 8

=item 1

Seconds

=item 2

Minutes

=item 3

Hours (default as shown)

=item 4

Days

=back

=item cmds

cmd types to display on cmd graph, remove cmds you don't want
displayed from list.

=item leitime

Setting this to 1 will re-enable slab eviction time graphs.

The slab information for LEI is disabled by default since I believe
the counters to be inaccurate, or perhaps not being updated as often I
thought they would be. They can be re-enabled by setting this
environment variable.


=back

=head2 BASIC TROUBLESHOOTING

Please make sure you can telnet to your memcache servers and issue the
following commands: stats, stats settings, stats items and stats
slabs.

=head2 PLUGIN INFORMATION

Available Graphs contained in this Plugin:

=over 4

=item bytes

This graphs the current network traffic in and out

=item commands

This graphs the current commands being issued to the memcache machine.
Provides drilldown to per slab stats.

=item conns

This graphs the current, max connections as well as avg conns per sec,
avg conns per sec, and is derived from total_conns / uptime.

=item evictions

This graphs the current evictions on the node. Provides drilldown to
per slab stats.

=item items

This graphs the current items and total items in the
memcached node. Provides drilldown to per slab stats.

=item memory

This graphs the current and max memory allocation. Provides drilldown to
per slab stats.

=item unfetched

This graphs the number of items that were never touched by a
get/incr/append/etc before being evicted or expiring from the
cache. Provides drilldown to per slab stats.

=back

You will find that some of the graphs have the text "LEI" on
them. This was done in order to save room on space, it means B<Last
Evicted Item>.

=head1 ACKNOWLEDGEMENTS

Thanks to dormando for putting up with me ;)

=head1 AUTHOR

Matt West < https://github.com/mhwest13/Memcached-Munin-Plugin >

Nicolai Langfeldt < https://github.com/niclang/ > perturbed and added
to Matts work.

=head1 LICENSE

GPLv2

=head1 MAGIC MARKERS

#%# family=auto
#%# capabilities=autoconf

=cut

use Carp;
use strict;
use warnings;
use IO::Socket;
use Munin::Plugin;
use File::Basename;

if ( basename($0) !~ /(?:([^\/]+)_)?memcached$/ ) {
    print
"This script needs to be named (prefix_)?memcached to run properly.\n";
    exit 1;
}

# tell munin about our multigraph capabilities
need_multigraph();

=head1 Variable Declarations

This section of code is to declare the variables used throughout the
plugin Some of them are imported as environment variables from munin
plugin conf.d file, others are hashes used for storing information
that comes from the stats commands issued to memcached.

=cut

# lets import environment variables for the plugin or use the default
my $host = $ENV{host} || "127.0.0.1";
my $port = $ENV{port} || 11211;
my $connection;

# This gives us the ability to control the timescale our graphs are displaying.
# The default it set to divide by hours, if you want to get seconds set it to 1.
# Options: 1 = seconds, 2 = minutes, 3 = hours, 4 = days
my $timescale = $ENV{timescale} || 3;

# This gives us the ability to turn the Last Evicted Item time slab graph on.
# It was removed because I believe the counter / response to be broken but
# perhaps this was useful to someone.
my $leitime = $ENV{leitime} || -1;

# This gives us the ability to specify which commands we want to display on the
# command graph. Allowing finer control since some environments don't leverage
# every command possible in memcached.
# Options: get set delete incr decr cas touch flush
my $commands = $ENV{cmds} || "get set delete incr decr touch";

# This hash contains the information contained in two memcache commands
# stats and stats settings.
my %stats;

# This gives us eviction rates and other hit stats per slab
# We track this so we can see if something was evicted earlier than necessary
my %items;

# This gives us the memory size and usage per slab We track this so we
# can see what slab is being used the most and has no free chunks so
# we can re-tune memcached to allocate more pages for the specified
# chunk size
my %chnks;

# Variable for setting up a quick access map for plugin configurations
# / version adherence
my $globalmap;

=head2 Graph Declarations

This block of code builds up all of the graph info for all root /
subgraphs.

  %graphs: is a container for all of the graph definition
           information. In here is where you'll find the
           configuration information for munin's graphing procedure.
Format:

    $graph{graph_name} => {
        config => {
            You'll find the main graph config stored here
            { key => value },
            { ... },
        },
        datasrc => [
            Name: name given to data value
            Attr: Attribute and value, attribute must be valid plugin argument
            { name => 'Name', info => 'info about graph', ... },
            { ... },
        ],
    }

=cut

my %graphs;

# main graph for memcached item count
$graphs{items} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => 'Items in Memcached',
        category => 'memcached',
        title    => 'Items',
        info     => 'Number of items in use by memcached',
    },
    datasrc => [
        { name => 'curr_items', label => 'Current Items', min => '0' },
        {
            name  => 'total_items',
            label => 'New Items',
            min   => '0',
            type  => 'DERIVE'
        },
    ],
};

# main graph for memcached memory usage
$graphs{memory} = {
    config => {
        args     => '--base 1024 --lower-limit 0',
        vlabel   => 'Bytes',
        category => 'memcached',
        title    => 'Memcached Memory Usage',
        info     => 'Memory consumption of memcached',
    },
    datasrc => [
        {
            name  => 'limit_maxbytes',
            draw  => 'AREA',
            label => 'Maximum',
            min   => '0'
        },
        {
            name  => 'bytes',
            draw  => 'AREA',
            label => 'Current',
            min   => '0'
        },
    ],
};

# main graph for memcached network usage
$graphs{bytes} = {
    config => {
        args     => '--base 1000',
        vlabel   => 'bits in (-) / out (+) per ${graph_period}',
        title    => 'Memcached Network Traffic',
        category => 'memcached',
        info     => 'Network traffic in / out of memcached',
        order    => 'bytes_read bytes_written',
    },
    datasrc => [
        {
            name  => 'bytes_read',
            type  => 'DERIVE',
            label => 'Network Traffic coming in (-)',
            graph => 'no',
            cdef  => 'bytes_read,8,*',
            min   => '0'
        },
        {
            name     => 'bytes_written',
            type     => 'DERIVE',
            label    => 'Traffic',
            negative => 'bytes_read',
            cdef     => 'bytes_written,8,*',
            min      => '0'
        },
    ],
};


$graphs{uptime} = {
    config => {
	vlabel   => 'Uptime (days)',
	category => 'memcached',
	title    => 'Memcached Uptime',
    },
    datasrc => [
	{ name  => 'uptime',
	  label => 'Uptime',
	  cdef  => 'uptime,86400,/' }
    ]
};


$graphs{errors} = {
    config => {
	vlabel    => 'Times',
	title     => 'Memcached Errors',
	category  => 'memcached',
    },
    datasrc => [
	{ name  => 'listen_disabled_num', 
	  label => 'Times unavailable',
	  min   => '0',
	  info  => 'Number of times new connections has had to wait' },
	{ name  => 'conn_yields',
	  label => 'Times throttled',
	  min   => '0',
	  info  => 'Number of times a client has been throttled' }
	]
};

# graph for memcached connections
$graphs{conns} = {
    config => {
        vlabel   => 'Number',
        category => 'memcached',
        title    => 'Memcached connections',
        info     => 'Connection stats',
        order    => 'curr_conns avg_conns max_conns',
    },
    datasrc => [
        { name => 'curr_conns', label => 'Current', min => '0' },
        { name => 'max_conns',  label => 'Maximum', min => '0',
	  info => 'Configured maximum connections' },
        { name => 'avg_conns',  label => 'Average', min => '0',
	  info => 'Average connections computed by memcached' },
    ],
};

# main graph for memcached commands issued
$graphs{commands} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => 'Commands per ${graph_period}',
        category => 'memcached',
        title    => 'Commands',
        info     => 'Number of commands being handled by memcached',
    },
    datasrc => [
        {
            name  => 'cmd_get',
            type  => 'DERIVE',
            label => 'Gets',
            info  => 'Cumulative number of retrieval reqs',
            min   => '0'
        },
        {
            name  => 'cmd_set',
            type  => 'DERIVE',
            label => 'Sets',
            info  => 'Cumulative number of storage reqs',
            min   => '0'
        },
        {
            name  => 'cmd_flush',
            type  => 'DERIVE',
            label => 'Flushes',
            info  => 'Cumulative number of flush reqs',
            min   => '0'
        },
        {
            name  => 'cmd_touch',
            type  => 'DERIVE',
            label => 'Touches',
            info  => 'Cumulative number of touch reqs',
            min   => '0'
        },
        {
            name  => 'get_hits',
            type  => 'DERIVE',
            label => 'Get Hits',
            info  => 'Number of keys that were requested and found',
            min   => '0'
        },
        {
            name  => 'get_misses',
            type  => 'DERIVE',
            label => 'Get Misses',
            info  => 'Number of keys there were requested and not found',
            min   => '0'
        },
        {
            name  => 'delete_hits',
            type  => 'DERIVE',
            label => 'Delete Hits',
            info =>
              'Number of delete requests that resulted in a deletion of a key',
            min => '0'
        },
        {
            name  => 'delete_misses',
            type  => 'DERIVE',
            label => 'Delete Misses',
            info  => 'Number of delete requests for missing key',
            min   => '0'
        },
        {
            name  => 'incr_hits',
            type  => 'DERIVE',
            label => 'Increment Hits',
            info  => 'Number of successful increment requests',
            min   => '0'
        },
        {
            name  => 'incr_misses',
            type  => 'DERIVE',
            label => 'Increment Misses',
            info  => 'Number of unsuccessful increment requests',
            min   => '0'
        },
        {
            name  => 'decr_hits',
            type  => 'DERIVE',
            label => 'Decrement Hits',
            info  => 'Number of successful decrement requests',
            min   => '0'
        },
        {
            name  => 'decr_misses',
            type  => 'DERIVE',
            label => 'Decrement Misses',
            info  => 'Number of unsuccessful decrement requests',
            min   => '0'
        },
        {
            name  => 'cas_misses',
            type  => 'DERIVE',
            label => 'CAS Misses',
            info  => 'Number of Compare and Swap requests against missing keys',
            min   => '0'
        },
        {
            name  => 'cas_hits',
            type  => 'DERIVE',
            label => 'CAS Hits',
            info  => 'Number of successful Compare and Swap requests',
            min   => '0'
        },
        {
            name  => 'cas_badval',
            type  => 'DERIVE',
            label => 'CAS Badval',
            info  => 'Number of unsuccessful Compare and Swap requests',
            min   => '0'
        },
        {
            name  => 'touch_hits',
            type  => 'DERIVE',
            label => 'Touch Hits',
            info  => 'Number of successfully touched keys',
            min   => '0'
        },
        {
            name  => 'touch_misses',
            type  => 'DERIVE',
            label => 'Touch Misses',
            info  => 'Number of unsuccessful touch keys',
            min   => '0'
        },
    ],
};

# main graph for memcached eviction rates
$graphs{evictions} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => 'Evictions per ${graph_period}',
        category => 'memcached',
        title    => 'Evictions',
        info     => 'Number of evictions per second',
    },
    datasrc => [
        {
            name  => 'evictions',
            type  => 'DERIVE',
            label => 'Evictions',
            info  => 'Cumulative Evictions Across All Slabs',
            min   => '0'
        },
        {
            name  => 'evicted_nonzero',
            type  => 'DERIVE',
            label => 'Evictions prior to Expire',
            info => 'Cumulative Evictions forced to expire prior to expiration',
            min  => '0'
        },
        {
            name  => 'reclaimed',
            type  => 'DERIVE',
            label => 'Reclaimed Items',
            info  => 'Cumulative Reclaimed Item Entries Across All Slabs',
            min   => '0'
        },
    ],
};

# main graph for memcached eviction rates
$graphs{unfetched} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => 'Unfetched Items per ${graph_period}',
        category => 'memcached',
        title    => 'Unfetched Items',
        info =>
'Number of items that were never touched get/incr/append/etc before X occured',
    },
    datasrc => [
        {
            name  => 'expired_unfetched',
            type  => 'DERIVE',
            label => 'Expired',
            min   => '0',
            info =>
'Number of items that expired and never had get/incr/append/etc performed'
        },
        {
            name  => 'evicted_unfetched',
            type  => 'DERIVE',
            label => 'Evicted',
            min   => '0',
            info =>
'Number of items that evicted and never had get/incr/append/etc performed'
        },
    ],
};

# subgraph for breaking memory info down by slab ( subgraph of memory )
$graphs{slabchnks} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => 'Available Chunks for this Slab',
        category => 'memcached',
        title    => 'Chunk Usage for Slab: ',
        info => 'This graph shows you the chunk usage for this memory slab.',
    },
    datasrc => [
        {
            name  => 'total_chunks',
            label => 'Total Chunks Available',
            min   => '0'
        },
        { name => 'used_chunks', label => 'Total Chunks in Use', min => '0' },
        {
            name  => 'free_chunks',
            label => 'Total Chunks Not in Use (Free)',
            min   => '0'
        },
    ],
};

# subgraph for breaking commands down by slab ( subgraph of commands )
$graphs{slabhits} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => 'Hits per Slab per ${graph_period}',
        category => 'memcached',
        title    => 'Hits for Slab: ',
        info =>
          'This graph shows you the successful hit rate for this memory slab.',
    },
    datasrc => [
        {
            name  => 'get_hits',
            label => 'Get Requests',
            type  => 'DERIVE',
            min   => '0'
        },
        {
            name  => 'cmd_set',
            label => 'Set Requests',
            type  => 'DERIVE',
            min   => '0'
        },
        {
            name  => 'delete_hits',
            label => 'Delete Requests',
            type  => 'DERIVE',
            min   => '0'
        },
        {
            name  => 'incr_hits',
            label => 'Increment Requests',
            type  => 'DERIVE',
            min   => '0'
        },
        {
            name  => 'decr_hits',
            label => 'Decrement Requests',
            type  => 'DERIVE',
            min   => '0'
        },
        {
            name  => 'cas_hits',
            label => 'Sucessful CAS Requests',
            type  => 'DERIVE',
            min   => '0'
        },
        {
            name  => 'cas_badval',
            label => 'UnSucessful CAS Requests',
            type  => 'DERIVE',
            min   => '0'
        },
        {
            name  => 'touch_hits',
            label => 'Touch Requests',
            type  => 'DERIVE',
            min   => '0'
        },
    ],
};

# subgraph for breaking evictions down by slab ( subgraph of evictions )
$graphs{slabevics} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => 'Evictions per Slab per ${graph_period}',
        category => 'memcached',
        title    => 'Evictions for Slab: ',
        info => 'This graph shows you the eviction rate for this memory slab.',
    },
    datasrc => [
        {
            name  => 'evicted',
            label => 'Total Evictions',
            type  => 'DERIVE',
            min   => '0',
            info  => 'Items evicted from memory slab'
        },
        {
            name  => 'evicted_nonzero',
            type  => 'DERIVE',
            label => 'Evictions from LRU Prior to Expire',
            info  => 'Items evicted from memory slab before ttl expiration',
            min   => '0'
        },
        {
            name  => 'reclaimed',
            type  => 'DERIVE',
            label => 'Reclaimed Expired Items',
            info =>
              'Number of times an item was stored in expired memory slab space',
            min => '0'
        },
    ],
};

# subgraph for showing the time between an item was last evicted and requested ( subgraph of evictions )
$graphs{slabevictime} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => ' since Request for LEI',
        category => 'memcached',
        title    => 'Eviction Request Time for Slab: ',
        info =>
'This graph shows you the time since we requested the last evicted item',
    },
    datasrc => [
        {
            name  => 'evicted_time',
            label => 'Eviction Time (LEI)',
            info  => 'Time Since Request for Last Evicted Item',
            min   => '0'
        },
    ],
};

# subgraph for breaking items down by slab ( subgraph of items )
$graphs{slabitems} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => 'Items per Slab',
        category => 'memcached',
        title    => 'Items in Slab: ',
        info =>
'This graph shows you the number of items and reclaimed items per slab.',
    },
    datasrc => [
        {
            name  => 'number',
            label => 'Items',
            draw  => 'AREA',
            info  => 'This is the amount of items stored in this slab',
            min   => '0'
        },
    ],
};

# subgraph for showing the age of the eldest item stored in a slab ( subgraph of items )
$graphs{slabitemtime} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => ' since item was stored',
        category => 'memcached',
        title    => 'Age of Eldest Item in Slab: ',
        info => 'This graph shows you the time of the eldest item in this slab',
    },
    datasrc =>
      [ { name => 'age', label => 'Eldest Item\'s Age', min => '0' }, ],
};

# main graph for memcached eviction rates
$graphs{slabunfetched} = {
    config => {
        args     => '--base 1000 --lower-limit 0',
        vlabel   => 'Unfetched Items per ${graph_period}',
        category => 'memcached',
        title    => 'Unfetched Items in Slab: ',
        info =>
'Number of items that were never touched get/incr/append/etc before X occured',
    },
    datasrc => [
        {
            name  => 'expired_unfetched',
            type  => 'DERIVE',
            label => 'Expired Unfetched',
            min   => '0',
            info =>
'Number of items that expired and never had get/incr/append/etc performed'
        },
        {
            name  => 'evicted_unfetched',
            type  => 'DERIVE',
            label => 'Evictioned Unfetched',
            min   => '0',
            info =>
'Number of items that evicted and never had get/incr/append/etc performed'
        },
    ],
};

=head1 Munin Checks

These checks look for config / autoconf / suggest params


=head2 Config Check

This block of code looks at the argument that is possibly supplied,
should it be config, it then checks to make sure the plugin specified
exists, assuming it does, it will run the do_config subroutine for the
plugin specified, otherwise it dies complaining about an unknown
plugin.

=cut

my @rootplugins = ( 'bytes', 'conns', 'commands', 'evictions',
		    'items', 'memory', 'unfetched', 'uptime', 'errors' );


if ( defined $ARGV[0] && $ARGV[0] eq 'config' ) {

# Lets get our plugin from the symlink being called up, we'll also verify its a valid
# plugin that we have graph information for
    $0 =~ /(?:([^\/]+)_)?memcached(.+)$/;
    my $prefix = $1 ? $1 : '';

    # We need to fetch the stats before we do any config, cause its
    # needed for multigraph subgraphs which use slab information for title
    # / info per slab
    fetch_stats();
    $globalmap = buildglobalmap();

    for my $plugin (@rootplugins) {
	# Now lets go ahead and print out our config.
	do_config( $prefix, $plugin );
    }
    exit 0;
}

=head2 Autoconf Check

This block of code looks at the argument that is possibly supplied,
should it be autoconf, we will attempt to connect to the memcached
service specified on the host:port, upon successful connection it
prints yes, otherwise it prints no.

=cut

if ( defined $ARGV[0] && $ARGV[0] eq 'autoconf' ) {

    # Lets attempt to connect to memcached
    my $s = get_conn();

    # Lets verify that we did connect to memcached
    if ( defined($s) ) {
        print "yes\n";
        exit 0;
    }
    else {
        print "no (unable to connect to $connection)\n";
        exit 0;
    }
}

=head1 Output Subroutines

Output Subroutine calls to output data values

=head2 fetch_output

This subroutine is the main call for printing data for the plugin.  No
parameters are taken as this is the default call if no arguments are
supplied from the command line.

=cut

# Well, no arguments were supplied that we know about, so lets print
# some data
$0 =~ /(?:([^\/]+)_)?memcached$/;
my $prefix = $1 ? $1 : '';

fetch_stats();
$globalmap = buildglobalmap();

for my $plugin (@rootplugins) {
    fetch_output( $prefix, $plugin );
}

sub fetch_output {
    my ( $prefix, $plugin ) = (@_);

    # Now lets go ahead and print out our output.
    my @subgraphs;
    if ( $plugin eq 'memory' ) {
        @subgraphs = ('slabchnks');
        foreach my $slabid ( sort { $a <=> $b } keys %chnks ) {
            print_submulti_output( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_rootmulti_output( $prefix, $plugin );
    }
    elsif ( $plugin eq 'commands' ) {
        @subgraphs = ('slabhits');
        foreach my $slabid ( sort { $a <=> $b } keys %chnks ) {
            print_submulti_output( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_rootmulti_output( $prefix, $plugin );
    }
    elsif ( $plugin eq 'evictions' ) {
        @subgraphs = ('slabevics');
        if ( $leitime == 1 ) { push( @subgraphs, 'slabevictime' ); }
        foreach my $slabid ( sort { $a <=> $b } keys %items ) {
            print_submulti_output( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_rootmulti_output( $prefix, $plugin );
    }
    elsif ( $plugin eq 'items' ) {
        @subgraphs = ( 'slabitems', 'slabitemtime' );
        foreach my $slabid ( sort { $a <=> $b } keys %items ) {
            print_submulti_output( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_rootmulti_output( $prefix, $plugin );
    }
    elsif ( $plugin eq 'unfetched' ) {
        @subgraphs = ('slabunfetched');
        foreach my $slabid ( sort { $a <=> $b } keys %items ) {
            print_submulti_output( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_rootmulti_output( $prefix, $plugin );
    }
    else {
        print_rootmulti_output($prefix, $plugin);
    }

    return;
}

=head2 print_multigraph

Helper subroutine to very simply print the multigraph header of each
graph.

=cut

sub print_multigraph {
    my ($prefix, $plugin, $extended) = @_;

    confess "No plugin" if !defined $plugin;

    if ($plugin eq 'evictions') {
	$plugin .= ".global$plugin";
    }
    
    $prefix   = '' if ! defined $prefix;
    $extended = '' if ! defined $extended;

    # The "multi" part is to stay compatible with the _multi_ plugin.

    if ($prefix) {
        print "multigraph $prefix\_memcached_multi_$plugin$extended\n";
    }
    else {
        print "multigraph memcached_multi_$plugin$extended\n";
    }
    
}

=head2 print_root_output

This subroutine prints out the return values for our non-multigraph
root graphs.  It takes one parameter $plugin and returns when
completed.

        $plugin;    graph we are calling up to print data values for 

Example: print_root_output($plugin);

=cut

sub print_root_output {

    # Lets get our plugin, set our graph reference and print out info for Munin
    my ($prefix, $plugin) = (@_);
    my $graph = $graphs{$plugin};

    print_multigraph($prefix, $plugin);

    # The conns plugin has some specific needs, looking for plugin type
    if ( $plugin ne 'conns' ) {
        foreach my $dsrc ( @{ $graph->{datasrc} } ) {
            my %datasrc = %$dsrc;
            while ( my ( $key, $value ) = each(%datasrc) ) {
                next if ( $key ne 'name' );
                my $output = $stats{$value};
                print "$dsrc->{name}.value $output\n";
            }
        }
    }
    else {
        my $output;
        foreach my $dsrc ( @{ $graph->{datasrc} } ) {
            my %datasrc = %$dsrc;
            while ( my ( $key, $value ) = each(%datasrc) ) {
                if ( $value eq 'max_conns' ) {
                    $output = $stats{maxconns};
                }
                elsif ( $value eq 'curr_conns' ) {
                    $output = $stats{curr_connections};
                }
                elsif ( $value eq 'avg_conns' ) {
                    $output = sprintf( "%02d",
                        $stats{total_connections} / $stats{uptime} );
                }
                else {
                    next;
                }
                print "$dsrc->{name}.value $output\n";
            }
        }
    }
    return;
}

=head2 print_rootmulti_output

This subroutine prints out the return values for our multigraph root
graphs.  It takes one parameter $plugin and returns when completed.

        $plugin;    root graph we are calling up to print data values for

Example: print_rootmulti_output($plugin);

=cut

sub print_rootmulti_output {

    # Lets get our plugin, set our graph reference and print out info
    # for Munin
    my ( $prefix, $plugin ) = (@_);
    my $graph = $graphs{$plugin};

    print_multigraph($prefix, $plugin);

    # Lets print our data values with their appropriate name
    foreach my $dsrc ( @{ $graph->{datasrc} } ) {
        my $output  = 0;
        my %datasrc = %$dsrc;
        while ( my ( $key, $value ) = each(%datasrc) ) {
            next if ( $key ne 'name' );
            next
              if ( ( $plugin eq 'evictions' )
                && ( !exists( $globalmap->{globalevics}->{ $dsrc->{name} } ) )
              );
            next
              if ( ( $plugin eq 'commands' )
                && ( !exists( $globalmap->{globalcmds}->{ $dsrc->{name} } ) ) );
            if ( ( $plugin eq 'evictions' ) && ( $value eq 'evicted_nonzero' ) )
            {
                foreach my $slabid ( sort { $a <=> $b } keys %items ) {
                    $output += $items{$slabid}->{evicted_nonzero};
                }
            }
            else {
		$output = 'U';
                $output = $stats{$value} if defined $stats{$value};
            }
            print "$dsrc->{name}.value $output\n";
        }
    }
    return;
}

=head2 print_submulti_output

This subroutine prints out the return values for our multigraph
subgraphs. It takes three parameters $slabid, $plugin, @subgraphs and
then rReturns when completed.

    $slabid;    slab id that we will use to grab info from and print out
    $plugin;    root graph being called, used for multigraph output and slab id
    @subgraphs; graphs we are actually trying to print data values for

Example: print_submulti_output($slabid,$plugin,@subgraphs);

=cut

sub print_submulti_output {

    # Lets get our slabid, plugin, and subgraphs
    my ( $prefix, $slabid, $plugin, @subgraphs ) = (@_);
    my $currslab = undef;

    # Time to loop over our subgraphs array
    foreach my $sgraph (@subgraphs) {

 # Lets set our graph reference for quick calling, and print some info for munin
        my $graph = $graphs{$sgraph};

	print_multigraph($prefix, $plugin, ".$sgraph\_$slabid");
	
        # Lets figure out what slab info we are trying to call up
        if (   ( $plugin eq 'evictions' )
            || ( $plugin eq 'items' )
            || ( $plugin eq 'unfetched' ) )
        {
            $currslab = $items{$slabid};
        }
        elsif ( ( $plugin eq 'memory' ) || ( $plugin eq 'commands' ) ) {
            $currslab = $chnks{$slabid};
        }
        else {
            return;
        }

        # Lets print our data values with their appropriate name
        foreach my $dsrc ( @{ $graph->{datasrc} } ) {
            my %datasrc = %$dsrc;
            while ( my ( $key, $value ) = each(%datasrc) ) {
                next if ( $key ne 'name' );
                next
                  if ( ( $sgraph eq 'slabevics' )
                    && ( !exists( $globalmap->{slabevics}->{ $dsrc->{name} } ) )
                  );
                next
                  if ( ( $plugin eq 'commands' )
                    && ( !exists( $globalmap->{slabcmds}->{ $dsrc->{name} } ) )
                  );
                my $output = $currslab->{$value};
                if (   ( $sgraph eq 'slabevictime' )
                    || ( $sgraph eq 'slabitemtime' ) )
                {
                    $output = time_scale( 'data', $output );
                }
                print "$dsrc->{name}.value $output\n";
            }
        }
    }
    return;
}

=head1 Config Subroutines

These subroutines handle the config portion of munin calls.

=head2 do_config

This is the main call issued assuming we call up config and plugin
specified exists The subroutine takes one parameter $plugin, and
returns when completed.

     $plugin; root graph being called

Example: do_config($prefix, $plugin);

=cut

sub do_config {
    my ( $prefix, $plugin ) = (@_);
    my @subgraphs;
    if ( $plugin eq 'memory' ) {
        @subgraphs = ('slabchnks');
        foreach my $slabid ( sort { $a <=> $b } keys %chnks ) {
            print_submulti_config( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_subrootmulti_config( $prefix, $plugin );
        # print_rootmulti_config( $prefix, $plugin );
    }
    elsif ( $plugin eq 'commands' ) {
        @subgraphs = ('slabhits');
        foreach my $slabid ( sort { $a <=> $b } keys %chnks ) {
            print_submulti_config( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_subrootmulti_config( $prefix, $plugin );
        # print_rootmulti_config( $prefix, $plugin );
    }
    elsif ( $plugin eq 'evictions' ) {
        @subgraphs = ('slabevics');
        if ( $leitime == 1 ) { push( @subgraphs, 'slabevictime' ); }
        foreach my $slabid ( sort { $a <=> $b } keys %items ) {
            print_submulti_config( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_subrootmulti_config( $prefix, $plugin );
        # print_rootmulti_config( $prefix, $plugin );
    }
    elsif ( $plugin eq 'items' ) {
        @subgraphs = ( 'slabitems', 'slabitemtime' );
        foreach my $slabid ( sort { $a <=> $b } keys %items ) {
            print_submulti_config( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_subrootmulti_config( $prefix, $plugin );
        # print_rootmulti_config( $prefix, $plugin );
    }
    elsif ( $plugin eq 'unfetched' ) {
        @subgraphs = ('slabunfetched');
        foreach my $slabid ( sort { $a <=> $b } keys %items ) {
            print_submulti_config( $prefix, $slabid, $plugin, @subgraphs );
        }
        print_subrootmulti_config( $prefix, $plugin );
        # print_rootmulti_config( $prefix, $plugin );
    }
    else {
        print_root_config( $prefix, $plugin );
    }

    return;
}

=head2 print_root_config

This subroutine prints out the config information for all of the
(multigraph) root graphs.  It takes one parameter, $plugin, returns
when completed.

        $prefix;    possible prefix used to allow multiple plugins per machine
        $plugin;    root graph used for multigraph call

Example:  print_root_config($prefix,$plugin);

=cut

sub print_root_config {

    # Lets get our plugin, set our graph reference and our graph config info
    my ( $prefix, $plugin ) = (@_);
    my $graph     = $graphs{$plugin};
    my %graphconf = %{ $graph->{config} };

    print_multigraph($prefix, $plugin);

    # Lets tell munin about the graph we are referencing and print the
    # main config
    while ( my ( $key, $value ) = each(%graphconf) ) {
        if ( $key eq 'title' ) {
            if ($prefix) {
                print "graph_$key " . ucfirst($prefix) . " $value\n";
            }
            else {
                print "graph_$key $value\n";
            }
        }
        else {
            print "graph_$key $value\n";
        }
    }

    # Lets tell munin about our data values and how to treat them
    foreach my $dsrc ( @{ $graph->{datasrc} } ) {
        my %datasrc = %$dsrc;
        while ( my ( $key, $value ) = each(%datasrc) ) {
            next if ( $key eq 'name' );
            print "$dsrc->{name}.$key $value\n";
        }
    }
    return;
}

=head2 print_rootmulti_config

This subroutine prints out the config information for all of the
multigraph root graphs.  It takes one parameter, $plugin, returns when
completed.

        $prefix;    possible prefix used to allow multiple plugins per machine
        $plugin;    root graph used for multigraph call

Example:  print_rootmulti_config($prefix,$plugin);

=cut

sub print_rootmulti_config {

    # Lets get out plugin, set our graph reference and our graph config info
    my ( $prefix, $plugin ) = (@_);
    my $graph     = $graphs{$plugin};
    my %graphconf = %{ $graph->{config} };

    # Lets tell munin about the graph we are referencing and print the
    # main config
    print_multigraph($prefix, $plugin);

    while ( my ( $key, $value ) = each(%graphconf) ) {
        if ( $key eq 'title' ) {
            if ($prefix) {
                print "graph_$key " . ucfirst($prefix) . " $value\n";
            }
            else {
                print "graph_$key $value\n";
            }
        }
        else {
            print "graph_$key $value\n";
        }
    }

    # Lets tell munin about our data values and how to treat them
    foreach my $dsrc ( @{ $graph->{datasrc} } ) {
        my %datasrc = %$dsrc;
        while ( my ( $key, $value ) = each(%datasrc) ) {
            next if ( $key eq 'name' );
            next
              if ( ( $plugin eq 'evictions' )
                && ( !exists( $globalmap->{globalevics}->{ $dsrc->{name} } ) )
              );
            next
              if ( ( $plugin eq 'commands' )
                && ( !exists( $globalmap->{globalcmds}->{ $dsrc->{name} } ) ) );
            print "$dsrc->{name}.$key $value\n";
        }
    }
    return;
}

=head2 print_subrootmulti_config

This subroutine prints out the config information for all of the
multigraph root graph, only this graph of the data will display on the
subpage made by the multigraph capabilities of munin and the
plugin. It takes one parameter, $plugin, returns when completed.

   $prefix;    possible prefix used to allow multiple plugins per machine
   $plugin;    root graph used for multigraph call

Example:  print_rootmulti_config($prefix,$plugin);

=cut

sub print_subrootmulti_config {

    # Lets get out plugin, set our graph reference and our graph config info
    my ( $prefix, $plugin ) = (@_);
    my $graph     = $graphs{$plugin};
    my %graphconf = %{ $graph->{config} };

    print_multigraph($prefix, $plugin);

    while ( my ( $key, $value ) = each(%graphconf) ) {
        if ( $key eq 'title' ) {
            if ($prefix) {
                print "graph_$key " . ucfirst($prefix) . " $value\n";
            }
            else {
                print "graph_$key $value\n";
            }
        }
        else {
            print "graph_$key $value\n";
        }
    }

    # Lets tell munin about our data values and how to treat them
    foreach my $dsrc ( @{ $graph->{datasrc} } ) {
        my %datasrc = %$dsrc;
        while ( my ( $key, $value ) = each(%datasrc) ) {
            next if ( $key eq 'name' );
            next
              if ( ( $plugin eq 'evictions' )
                && ( !exists( $globalmap->{globalevics}->{ $dsrc->{name} } ) )
              );
            next
              if ( ( $plugin eq 'commands' )
                && ( !exists( $globalmap->{globalcmds}->{ $dsrc->{name} } ) ) );
            print "$dsrc->{name}.$key $value\n";
        }
    }
    return;
}

=head2 print_submulti_config

This subroutine prints out the config information for all of the
multigraph subgraphs.  It takes three parameters, $slabid, $plugin and
@subgraphs, returns when completed.

    $prefix;    possible prefix used to allow multiple plugins per machine
    $slabid;    slab id that we will use to grab info from and print out
    $plugin;    root graph being called, used for multigraph output and slab id
    @subgraphs; graphs we are actually trying to print data values for

Example:  print_submulti_config($prefix,$slabid,$plugin,@subgraphs);

=cut

sub print_submulti_config {

    # Lets get our slabid, plugin, and subgraphs
    my ( $prefix, $slabid, $plugin, @subgraphs ) = (@_);
    my ( $slabitems, $slabchnks ) = undef;

    # Time to loop over our subgraphs array
    foreach my $sgraph (@subgraphs) {

        # Lets set our graph reference, and main graph config for easy handling
        my $graph     = $graphs{$sgraph};
        my %graphconf = %{ $graph->{config} };

	# Lets tell munin which graph we are graphing, and what our
	# main graph config info is
	print_multigraph($prefix, $plugin, ".$sgraph\_$slabid");

        while ( my ( $key, $value ) = each(%graphconf) ) {
            if ( $key eq 'title' ) {
                if ($prefix) {
                    print "graph_$key "
                      . ucfirst($prefix)
                      . " $value"
                      . "$slabid"
                      . " ($chnks{$slabid}->{chunk_size} Bytes)\n";
                }
                else {
                    print "graph_$key $value"
                      . "$slabid"
                      . " ($chnks{$slabid}->{chunk_size} Bytes)\n";
                }
            }
            elsif (
                ( $key eq 'vlabel' )
                && (   ( $sgraph eq 'slabevictime' )
                    || ( $sgraph eq 'slabitemtime' ) )
              )
            {
                $value = time_scale( 'config', $value );
                print "graph_$key $value\n";
            }
            else {
                print "graph_$key $value\n";
            }
        }

        # Lets tell munin about our data values and how to treat them
        foreach my $dsrc ( @{ $graph->{datasrc} } ) {
            my %datasrc = %$dsrc;
            while ( my ( $key, $value ) = each(%datasrc) ) {
                next if ( $key eq 'name' );
                next
                  if ( ( $sgraph eq 'slabevics' )
                    && ( !exists( $globalmap->{slabevics}->{ $dsrc->{name} } ) )
                  );
                next
                  if ( ( $plugin eq 'commands' )
                    && ( !exists( $globalmap->{slabcmds}->{ $dsrc->{name} } ) )
                  );
                print "$dsrc->{name}.$key $value\n";
            }
        }
    }
    return;
}

=head1 Misc Subroutines

These subroutines are misc ones, and are referenced inside of the
code. They should never be called up by Munin.

=head2 get_conn

This subroutine returns a socket connection.

=cut

sub get_conn {
    my $s = undef;

    # check if we want to use sockets instead of tcp
    my ($sock) = ( $host =~ /unix:\/\/(.+)*$/ );

    if ($sock) {
        $connection = "unix:\/\/$sock";
        $s = IO::Socket::UNIX->new( Peer => $sock );
    }
    else {
        $connection = "$host:$port";
        $s          = IO::Socket::INET->new(
            Proto    => "tcp",
            PeerAddr => $host,
            PeerPort => $port,
            Timeout  => 10,
        );
    }
    return $s;
}

=head2 fetch_stats

This subroutine fetches the information from memcached and stores it
into our hashes for later referencing throughout the graph. Returns
when completed.

=cut

sub fetch_stats {

    # Lets try and connect to memcached
    my $s = get_conn();

    # Die if we can't establish a connection to memcached
    die "Error: Unable to Connect to $connection\n" unless $s;

    # Lets print the stats command and store the info from the output
    print $s "stats\r\n";
    while ( my $line = <$s> ) {
        if ( $line =~ /STAT\s(.+?)\s((\w|\d|\S)+)/ ) {
            my ( $skey, $svalue ) = ( $1, $2 );
            $stats{$skey} = $svalue;
        }
        last if $line =~ /^END/;
    }

    # Lets print the stats settings command and store the info from the output
    print $s "stats settings\r\n";
    while ( my $line = <$s> ) {
        if ( $line =~ /STAT\s(.+?)\s((\w|\d|\S)+)/ ) {
            my ( $skey, $svalue ) = ( $1, $2 );
            if ( $skey eq 'evictions' ) {
                $skey = 'evictions_active';
            }
            $stats{$skey} = $svalue;
        }
        last if $line =~ /^END/;
    }

    # Lets print the stats slabs command and store the info from the output
    print $s "stats slabs\r\n";
    while ( my $line = <$s> ) {
        if ( $line =~ /STAT\s(\d+):(.+)\s(\d+)/ ) {
            my ( $slabid, $slabkey, $slabvalue ) = ( $1, $2, $3 );
            $chnks{$slabid}->{$slabkey} = $slabvalue;
        }
        last if $line =~ /^END/;
    }

    # Lets print the stats items command and store the info from the output
    print $s "stats items\r\n";
    while ( my $line = <$s> ) {
        if ( $line =~ /STAT\sitems:(\d+):(.+?)\s(\d+)/ ) {
            my ( $itemid, $itemkey, $itemvalue ) = ( $1, $2, $3 );
            $items{$itemid}->{$itemkey} = $itemvalue;
        }
        last if $line =~ /^END/;
    }
}

=head2 time_scale

This subroutine is here for me to adjust the timescale of the time
graphs for last evicted item and age of eldest item in cache.

Please note, after long usage I have noticed these counters may not be
accurate, I believe the developers are aware and have submitted a
patch upstream.

=cut

sub time_scale {

    # Lets get our config option and value to adjust
    my ( $configopt, $origvalue ) = (@_);
    my $value;

    # If config is defined, it returns the config info for time scale
    # If data is defined, it returns the original value after its been adjusted
    if ( $configopt eq 'config' ) {
        if ( $timescale == 1 ) {
            $value = "Seconds" . $origvalue;
        }
        elsif ( $timescale == 2 ) {
            $value = "Minutes" . $origvalue;
        }
        elsif (( $timescale == 3 )
            || ( $timescale > 4 )
            || ( !defined($timescale) ) )
        {
            $value = "Hours" . $origvalue;
        }
        elsif ( $timescale == 4 ) {
            $value = "Days" . $origvalue;
        }
    }
    elsif ( $configopt eq 'data' ) {
        if ( $timescale == 1 ) {
            $value = sprintf( "%02.2f", $origvalue / 1 );
        }
        elsif ( $timescale == 2 ) {
            $value = sprintf( "%02.2f", $origvalue / 60 );
        }
        elsif (( $timescale == 3 )
            || ( $timescale > 4 )
            || ( !defined($timescale) ) )
        {
            $value = sprintf( "%02.2f", $origvalue / 3600 );
        }
        elsif ( $timescale == 4 ) {
            $value = sprintf( "%02.2f", $origvalue / 86400 );
        }
    }
    else {
        die "Unknown time_scale option given: either [config/data]\n";
    }
    return $value;
}

=head2 buildglobalmap

This subroutine looks at the specified commands inputted, and
generates a hashref containing two arrays, one for global command keys
and one for slab command keys.

=cut

sub buildglobalmap {
    my $results;
    my @cmds = split( ' ', $commands );
    foreach my $cmd (@cmds) {
        if ( $cmd eq "get" ) {
            $results->{globalcmds}->{cmd_get}    = 1;
            $results->{globalcmds}->{get_hits}   = 1;
            $results->{globalcmds}->{get_misses} = 1;
            $results->{slabcmds}->{get_hits}     = 1;
        }
        elsif ( $cmd eq "set" ) {
            $results->{globalcmds}->{cmd_set} = 1;
            $results->{slabcmds}->{cmd_set}   = 1;
        }
        elsif ( $cmd eq "delete" ) {
            $results->{globalcmds}->{delete_hits}   = 1;
            $results->{globalcmds}->{delete_misses} = 1;
            $results->{slabcmds}->{delete_hits}     = 1;
        }
        elsif ( $cmd eq "incr" ) {
            $results->{globalcmds}->{incr_hits}   = 1;
            $results->{globalcmds}->{incr_misses} = 1;
            $results->{slabcmds}->{incr_hits}     = 1;
        }
        elsif ( $cmd eq "decr" ) {
            $results->{globalcmds}->{decr_hits}   = 1;
            $results->{globalcmds}->{decr_misses} = 1;
            $results->{slabcmds}->{decr_hits}     = 1;
        }
        elsif ( $cmd eq "cas" ) {
            $results->{globalcmds}->{cas_hits}   = 1;
            $results->{globalcmds}->{cas_misses} = 1;
            $results->{globalcmds}->{cas_badval} = 1;
            $results->{slabcmds}->{cas_hits}     = 1;
            $results->{slabcmds}->{cas_badval}   = 1;
        }
        elsif ( $cmd eq "touch" ) {
            if ( $stats{version} !~ /^1\.4\.[0-7]$/ ) {
                $results->{globalcmds}->{cmd_touch}    = 1;
                $results->{globalcmds}->{touch_hits}   = 1;
                $results->{globalcmds}->{touch_misses} = 1;
                $results->{slabcmds}->{touch_hits}     = 1;
            }
        }
        elsif ( $cmd eq "flush" ) {
            if ( $stats{version} !~ /^1\.4\.[0-7]$/ ) {
                $results->{globalcmds}->{cmd_flush} = 1;
            }
        }
        else {

            # Do absolutely nothing...
        }
    }
    $results->{globalevics}->{evictions}       = 1;
    $results->{globalevics}->{evicted_nonzero} = 1;
    $results->{slabevics}->{evicted}           = 1;
    $results->{slabevics}->{evicted_nonzero}   = 1;
    if ( $stats{version} !~ /^1\.4\.[0-2]$/ ) {
        $results->{globalevics}->{reclaimed} = 1;
        $results->{slabevics}->{reclaimed}   = 1;
    }
    return $results;
}