Skip to content

ESX agent prototype

Jump to bottom
Doug MacEachern edited this page Feb 23, 2016 · 2 revisions

Overview

The agent prototype will include a vSocket listener on the ESX side and a vSocket client on the VM guest side. The prototype will focus on vSockets as a transport mechanism, with some form of "hello world" invoked from the client side that returns ESX data from the server side.

See also: issue #38

Language and toolchain

On ESX we are limited to the glibc ABI at runtime, language options include:

  • C

  • C++

  • Python (2.7 on ESX 6.0, Python 3 on ESX 6.5)

  • Rust

Non-options:

  • Go

  • Rust with musl-libc

Bundled OSS toolchain

The agent source code can be compiled against open source toolchain components without any VMware dependencies, such as:

  • glibc (2.12 for ESX 5.5 and higher)

  • OpenSSL

Note that code other than Python would need to be built against the glibc version included on ESX. While this process can be cumbersome, it can be done fully in the open. A few examples compiled on RHEL6 and run on ESX 6.0, using gcc:

#include <stdlib.h>

int main() {
    system("uname -a");
    return 0;
}
strace -e open ./hw-c 2>&1 | grep -v ENOENT
open("/lib64/libc.so.6", O_RDONLY)      = 3
VMkernel w2-xlr8-autoroot-esx004 6.0.0 #1 SMP Release build-3029758 Aug 31 2015 00:54:00 x86_64 x86_64 x86_64 ESXi

Using rustc:

use std::process::Command;

fn main() {
    let output = Command::new("uname").arg("-a").output().unwrap_or_else(|e| {
        panic!("failed to execute process: {}", e)
    });

    println!("{}", String::from_utf8_lossy(&output.stdout));
}
strace -e open ./hw-rs 2>&1 | grep -v ENOENT
open("/lib64/libdl.so.2", O_RDONLY)     = 3
open("/lib64/libpthread.so.0", O_RDONLY) = 3
open("/lib64/libgcc_s.so.1", O_RDONLY)  = 3
open("/lib64/libc.so.6", O_RDONLY)      = 3
open("/lib64/libm.so.6", O_RDONLY)      = 3
open("/lib64/librt.so.1", O_RDONLY)     = 3
VMkernel w2-xlr8-autoroot-esx004 6.0.0 #1 SMP Release build-3029758 Aug 31 2015 00:54:00 x86_64 x86_64 x86_64 ESXi

VMware interfaces

There are several interfaces in library and cli form that are only available on ESX. Those of interest to us may not be available in SDK form, making it troublesome to build against from source in the open.

VMCI vSockets

The vSocket interface is available to the public via a single header file (vmci_sockets.h).

In the case of docker-vmdk-plugin, the use of vSockets requires C code as the Python stdlib requires modification to use the vSocket address family (VMCISock_GetAFValueFd). Likewise for Go on the client side. It appears a Rust version would not require modification of the stdlib, but TBD.

VSI (VMkernel Sysinfo Interface)

We need VSI to map the VM cartel ID (from SO_VMCI_PEER_HOST_VM_ID) to VM uuid. The docker-vmdk-plugin uses the (32-bit only) python vsi binding. We can also use vsish for now to avoid the library dependency. For example, where cartel id is '160708':

/sbin/vsish -e get /userworld/cartel/160708/vmmLeader
60710
/sbin/vsish -e get /vm/160710/vmmGroupInfo

group info {
   process id:-1
   uuid:56 4d b1 63 53 a5 dd e6-60 d5 7a 08 dc 4c c1 9a
   VC uuid:52 3b 01 43 42 4f c8 54-06 c3 18 c8 47 e1 79 28
   display name:dougm-ubuntu1504-docker
   config file path:/vmfs/volumes/544e7a4c-d59e8816-94d7-40a8f03603ae/dougm-ubuntu1504-docker/dougm-ubuntu1504-docker.vmx
}

disklib

For any disk operations not available via vmomi, we should be able to avoid linking against disklib by using the vmkfstools cli.

For the sidecar mechanism to store metadata, we can use the diskTool cli for now:

--sidecar <op,opParams>      : Perform Sidecar operations. Following are
                               the supported operations
                                1. create,key,size -> creates a sidecar
                                   with key and size
                                2. forcecreate,key,size -> opens if exist,
                                   creates a new sidecar with key and size
                                   otherwise.
                                3. write,key,offset,<pattern:length> ->
                                   writes pattern to sidecar with key at
                                   offset/length
                                4. writeString,key,offset,string ->
                                   writes string to sidecar with key at
                                   offset, fills with zeros after string end
                                5. read,key,offset,<pattern:length> ->
                                   read from sidecar with key at
                                   offset/length and compare with pattern
                                6. readString,key,offset,string ->
                                   read string from sidecar with key at
                                   offset and compare with string
                                7. delete,key -> deletes the sidecar with key
                                8. deleteall -> deletes all sidecar
                                9. enum -> Enumerate all sidecars
                               10. updateDDB -> Updates the sidecar names
                                   in DDB based on the disk name.
                               11. resize,key,size -> Truncates sidecar
                                   to a given size.

vmomi

We plan to proxy vmomi requests from the guest side and avoid any direct marshalling of vmomi SOAP messages in the agent. If we did want to do so, the only open option that runs on ESX is pyvmomi. The C++ vmomi library used by hostd is not open, nor is the pure C (glib based) gvmomi library.

Examples

Open (or soon to be) agents running on ESX.

Clone this wiki locally