vmrc-instructions.txt

################################################################ LICENSE
#
# Copyright (c) 2012-2015 Michael Dexter <editor@callfortesting.org>
#
# Permission to use, copy, modify, and distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
#
############################################################ INFORMATION
#
# Title: virtual machine rc script instructions
# Version: v.0.8.8
#
# vmrc usage instructions and developer notes
#
########################################################################

Thank you for trying vmrc, the Virtual Machine rc script.

Latest development snapshot: http://vmrc.bsd.lv/vmrc-snapshot.tar
GitHub repository: https://github.com/michaeldexter/vmrc

NOTABLE CHANGES FROM .0.8.7 to .0.8.8

* multiple bridge support in vm.conf or leave plank for a pre-configured bridge
* bridge MAC addres support
* No default VM NIC: choose tap or vmnet with or without MAC address
* Updated templates
* Misc fixes, as always

NOTABLE CHANGES	from .0.8.6 to .0.8.7

* MeetBSD/DevSummit Input. Lots
* Redirectable info and error output to stdio, logger or nowhere
* Download payload tasting for compression
* See the GitHub visual diffs

.0.8.6: Added ACPI shutdown
.0.8.5: Enabled OpenBSD reboot. Updated flashrd version
.0.8.4: Bug fixes

NOTABLE CHANGES from .0.8.2 to .0.8.3

* Implemented ACPI reset and poweroff
* Fixed echoed "cu" command
* Returned "nmdm" to the templates

NOTABLE CHANGES from .0.8.1 to .0.8.2

* Restored nmdm consoles

NOTABLE CHANGES from .0.8 to .0.8.1

* Fixed "iso" mode

NOTABLE CHANGES from .0.7.1 to .0.8

"Less dev-y, more ops-y"

* Reworked "start" to call rebootable VMs with tmux or screen consoles along
  the lines of vmrun.sh for "always on" use
* (SPECIFY A TMUX OR SCREEN CONSOLE FOR "START" TO WORK)
* Removed "detached" tmux and screen console modes. Both are now detached
* "start" directive will ignore non-backgrounded consoles (smooth host boot!)
* Added runvm.sh-style loop to tmux and screen console modes for hosting use
* All tmux and screen console modes are now "detached" for hosting use
* Disabled nmdm console mode for now
* Removed multiple argument VM start/stop (now all or one VM)
* Various clean-up along the way

NOTABLE CHANGES from .0.7 to .0.7.1

* Fixed several multiple VM start bugs
* Fixed "attach" for tmux and screen
* Multiple nmdm consoles still need work
* Added smarter "nextid" function
* Added a check-for-ID-conflicts function to mkvm.sh

SIGNIFICANT CHANGES from .0.6.1 to .0.7

* Sweeping restructuring and style changes (400+ line reduction in vm length)
* Increased tests and VM cleanup upon exit
* Increased reliance on return codes for continuation logic
* Some template format changes
* Improved GRUB support
* Improved QEMU support
* Added generated ifconfig_cmd.sh, bhyveload_cmd.sh, bhyve_cmd.sh,
* grub_cmd.sh and stop_cmd.sh scripts in the VM directory
* Added zvol support to the 'mount' and 'umount' directives
* Added raw image (VM-IMAGE/flashrd/FreeNAS/pfSense etc.) install option
* Added src/obj install option
* Added fine-grained ZFS feature flag support (11 host would only allow 11 VMs)
* Added GNU screen support

CHANGES from .0.5 to .0.6

* Introduces multiple VM NIC support. Simply add 1-7 in "vm_addtl_nics"
in the VM configuration file. This introduced a limit of 100 VM's but
that can be increased with a documented, one-character change.
* Adds updated pfSense
* Minor cosmetic cleanup

SIGNIFICANT CHANGES from .0.4 to .0.5

* The host_vmroot layout has changed.
* /usr/local/vm/ is now /usr/local/vmrc/vm/
* The "provision" option has been replaced by mkvm.sh which is based on a
library of functions and templates.
* Experimental preflight script and multiple block device support added.
* Experimental ZFS on root support is included but requires the included
zfs-boot components or a recent snapshot.


### OVERVIEW ###

This tool has come a long way from a 2005 jail(8) provisioning and management
script to a 2012 FreeBSD 9.0 bhyve build script to a series of sequential
FreeBSD 10.* VM provisioning and management scripts.

Because bhyve is only the latest multiplicity solution on FreeBSD, this
rc script includes basic jail and QEMU support and could include VirtualBox,
chroot, simh and GXemul support. Of these, QEMU is intended to help prepare
OSs that require VirtIO and console adjustments prior to booting under bhyve.


### REQUIREMENTS ###

The Intel or AMD "POPCNT" (EPT) processor feature is required for bhyve.

vmrc requires the bhyve hypervisor, available in FreeBSD and newer.


### CHEAT SHEET ###

Run "sh install-vmrc.sh" or read it and perform the steps manually, adjusting
the paths to suit your system.

Verify your host_nic and optional host_zpool in /usr/local/etc/vm.conf

The mkvm.sh script provisions Virtual Machines based on the templates in
/usr/local/vmrc/templates

mkvm.sh can be used in a captive manner with:

sh mkvm.sh

or by passing in a template name and a VM name. i.e.:

sh mkvm.sh fbsd10amd64_img_gpt_ufs_dist freebsd

which will create a VM named "freebsd#" where "#" is a unique, auto-generated
VM ID number such as "0", resulting in a VM named "freebsd0" in:

/usr/local/vmrc/vm/freebsd0

The template naming is based on its configuration variables. In this example:

fbsd10: FreeBSD 10 Operating System
amd64:  AMD64 Architecture
img:    Disk Image-based Storage
gpt:    GPT Partitioned Filesystem
ufs:    UFS Formatted Filesystem
dist:   Binary Distribution Set Installation Method

All vmrc VMs are distinguished by unique trailing ID numbers:

NON CONFLICTING: vm0, freebsd1, openbsd2

CONFLICTING: vm0, freebsd0 and openbsd0 (trailing ID number must be unique)

As of version 0.7.1, mkvm.sh will check for and report ID conflicts


Once a VM is provisioned, it is managed with the "vm" rc script, hence "vmrc".

All templates are set to a captive, stdio console which requires them to be
manually loaded and booted until a detached console is set in the .conf file.

To load and boot a new VM:

/usr/local/etc/rc.d/vm oneload freebsd0
/usr/local/etc/rc.d/vm oneboot freebsd0

Once a detached console and vm is enabled in /usr/local/etc/rc.conf :

/usr/local/etc/rc.d/vm start freebsd0

NB! You must set a "tmux" or "screen" console type for the "start" directive
to work and install ether "tmux" or "screen".


VMs can always be loaded, booted and stopped in a linear manner for debugging:

/usr/local/etc/rc.d/vm load freebsd0
/usr/local/etc/rc.d/vm boot freebsd0
/usr/local/etc/rc.d/vm stop freebsd0


### ISO MODE ###

Templates with "iso" in their names are installed by booting to ISO images
and formatting their storage devices with the OS's native installer. These are  
provisioned by booting in "ISO" mode:

service vm iso openbsd4

Once the OS is installed from ISO, these can be booted with the standard
'start' directive:

service vm start openbsd4


### GRUB MODE ###

The 'grub' directive exists to launch the VM in the GRUB boot loader. This is
particularly useful for determining the on-disk layout of the operating system.

Useful GRUB prompt commands are:

grub> ls		# List devices
grub> ls (cd0)		# Get information for cd0
grub> ls (cd0)/		# List contents of cd0
grub> ls (hd0)		# Get information for hd0
grub> ls (hd0)/		# List contents of hd0
grub> set		# Show set arguments

linux (cd0)/vmlinuz
initrc (cd0)/initrd.gz
boot

kopenbsd -h com0 (cd0)/5.6/amd64/bsd.rd
boot

knetbsd -h -r cd0a (cd0)/netbsd
boot

In a perfect world, GNU/Linux distros would use a standard location for
vmlinuz and initrd.gz, but this is not that world. BSD gets this right.

Debian/Ubuntu GNU/Linux are some of the simplest distributions to configure:

iso_grub_r="-r cd0"		# Default to cd0 for ISO boot
img_grub_r="-r hd0,msdos1"	# Default to hd0 for img boot

grub-bhyve then correctly finds /boot/grub/grub.cfg on these devices as they
are the default GRUB locations.

If your VM OS (such as OpenBSD, NetBSD and many GNU/Linux distributions)
requires an external grub.cfg, a \n separated one can be added to the template.
For example:

iso_grub_cfg="linux (cd0)/vmlinuz\ninitrc (cd0)/initrd.gz\nboot" 

This will auto-generate a ~/grub.cfg file with:

linux (cd0)/vmlinuz
initrc (cd0)/initrd.gz
boot

vmrc will also auto-generate a device.map file:

/usr/local/vmrc/vm/centos7amd64iso0/device.map

Reads:

(hd0) /usr/local/vmrc/vm//centos7amd64iso0/centos7amd64iso0.img
(cd0) /usr/local/vmrc/vm//centos7amd64iso0/centos7amd64iso0.iso

Centos7 GRUB with the defice.map file:

grub> ls
(hd0) (hd0,msdos2) (hd0,msdos1) (cd0) (cd0,msdos2) (host) (lvm/centos-root) (lv
m/centos-swap)

Without the device.map:

grub> ls
(host)

If a grub.cfg is not supported by a given GNU/Linux distribution for ISO boot,
the arguments can be entered in at the grub prompt or be read from grub.cfg
on the host filesystem.

For example:

/usr/local/vmrc/vm/centos7amd64iso0/grub.cfg

linux (cd0)/vmlinuz
initrc (cd0)/initrd.gz
boot

Once installed, the CentOS 7 grub.cfg exists in:

(hd0,msdos1)/grub2/grub.cfg

The grub.cfg garguments to use this would be:

/usr/local/sbin/grub-bhyve -r hd0,msdos1 -d /grub2/ \
-m /usr/local/vmrc/vm/centos7amd64iso0/device.map -M 1024 centos7amd64iso0

Other examples:

(cd0)/isolinux/grub.conf

(cd0)/EFI/BOOT/grub.cfg

If an external grub.cfg is used, the -r command can be used to specify the
boot drive, i.e. -r hd0,msdos1 and the commands can be issued without the
(hd0,msdos1) prefix.

Note that GNU/Linux paths may vary depending of the type of filesystem you use.

grub-bhyve kfreebsd support is experimental.


### DIRECTIVE USAGE ###

The 'start', 'stop' and 'status' will act on all VMs in /usr/local/vm/ or
a specified VM i.e.:

service vm onestop                (stop all VMs)
service vm onestatus freebsd0     (status for one VM)


### DIRECTIVES ###

Top level commands without a VM name (i.e. /usr/local/etc/rc.d/vm start)

start		Load and boot all VMs in /usr/local/vm/
stop		Ungracefully terminate all VMs in /usr/local/vm/
status		List running VMs

"reboot" and "shutdown" within VM should work as expected.
An ACPI shutdown is available after the 10.0-RELEASE and is included in vmrc.

VM-specific commands (i.e. /etc/rc.d/vm attach vm0)

start		Load and boot VMs
stop		kill and 'destroy' VMs
load		Load a VM with bhyveload(8) or grub-bhyve()
boot		Boot a VM with bhyve(8)
destroy		'destroy' a VM
restart		Run the stop and start functions on a VM
attach		Attach to a VM's nmdm serial port, tmux or screen session
debug		List bhyvectl(8) --get-all output
iso		Format storage device, fetch an ISO image, boot to it
grub		Launch grub with the VM's disk and ISO images
mount		fsck(8) and mount(8) a FreeBSD VM disk image (UFS only)
umount		umount(8) a FreeBSD VM disk image
jail		Launch a FreeBSD VM's disk image with jail(8)
delete		Delete a VM and its images and directory tree


### VM LAYOUT ###

/usr/local/vmrc/distributions/  OS distribution sets and ISOs
/usr/local/vmrc/templates/      VM templates
/usr/local/vmrc/vm/vm#/vm#.conf    VM config file
/usr/local/vmrc/vm/vm#/vm#.img     VM raw disk image (format or copy in)
/usr/local/vmrc/vm/vm#/vm#.iso     Link to bootable ISO image (fetch or copy in)
/usr/local/vmrc/vm/vm#/mnt/        VM mount point for raw disk image
/usr/local/vmrc/vm/vm#/vm#.log     VM command log

The ifconfig_cmd.sh, bhyveload_cmd.sh and bhyve_cmd.sh shell scripts will also
be created in /usr/local/vmrc/vm/vm#/ at boot time and can be run independently
of vmrc for testing purposes. Note that these will not create the network
bridge or perform any VM cleanup.


### CONSOLES ###

Because bhyve console output is inherently captive, vmrc supports several
"console" options:

stdio	Output the VM console to the console from which a VM is started
tmux	Output the VM console to a detached tmux session
screen	Output the VM console to a detached screen session
nmdm	Use an "nmdm" serial console and cu(1)

The included templates are set to "stdio" and must be set to "tmux" or "screen"
to be used with the "start" directive.

To attach to a tmux or screen console, use the "attach" directive:

/usr/local/etc/rc.d/vm oneattach freebsd3

To detach from each console, type:

tmux: "CTRL-b d"
screen: "C-a DD"
nmdm: "~ CTRL-d"

Notes:

* There is a bug in FreeBSD 10.0R nmdm that would cause the VM to boot only
  upon first attachment.


### NETWORKING ###

By default, vmrc uses on network bridge on one interface on the host as defined
in /usr/local/etc/vm.conf :

host_nic="em0"	                 # Host's active NIC for bridged networking
host_bridge="0"	                 # bridge0 (Fine for most configurations)

Should you want to use multiple bridges that are associated with multiple host
interfaces, you should be able to simply add these arguments to the VM template
as they are sourced sequentially. PLEASE TEST THIS.


### DEPENDENCIES ###

vmrc has been tested on PC-BSD 10-STABLE and FreeBSD 11-CURRENT from
July 1st, 2014 onward.

tmux functionality requires ports/sysutils/tmux
screen functionality requires ports/sysutils/screen
ISO and non-FreeBSD VM support requires ports/sysutils/grub2-bhyve 
QEMU functionality requires ports/emulators/qemu(-devel)


### TEMPLATES ###

The template naming attemps to convey as much information about a VM in as
little space possible. VM names over 31 characters will cause bhyve load to
give the error:

vm_create: Invalid argument

The simple solution is to not use the full template name as a VM name and note
that the automatic numbering done by mkvm.sh will add characters to the end of
a given VM name.

The master_template includes all VM arguments but the examples
will make more sense. 'diff' similar FreeBSD ones to see how they differ.

Many templates have OS-specific notes at the bottom in the Comments section.

Some templates have prefixes: dyn_ err_ reqtty_ and reqvt_:

dyn_     These templates reference frequently-changing download links. These
         should be updated manually before use. Few OSs include /latest.iso etc.
         and the included payloads may be invalid by the time you read this.

err_     These templates are largely complete but have known errors of one form
         or another. Your help debugging these is appreciated.

reqtty_  These FreeBSD-based OSs boot but require the tty fix

reqvt_   These FreeBSD-based OSs boot but require VirtIO drivers.  

The tty console fix and VirtIO drivers are easily added but ideally they will
be included with the upstream OSs.

Note the "required" fields in the templates. They are required.


### MISC ###

Please send questions, bug fixes, love mail and hate mail to:
Michael Dexter <editor@callfortesting.org>

FreeBSD VMs have their disk images fsck'd before boot given that fsck at boot
time is not always comprehensive and may result in VM boot failure. Jailed VMs
with dirty file systems may cause kernel panics.

FreeBSD guests can be provisioned with "iso" mode but it will not set the needed
console. (A "smart" console arrived mid-June 2014 and is available in FreeBSD
CURRENT and some STABLE branches.

cat >> /etc/ttys << EOF
ttyu0 "/usr/libexec/getty 3wire.9600" vt100 on secure
EOF

GNU/Linux users will find the "grub" command very useful for locating disks and
grub configuration files.

A few GNU/Linux-related features did not make it into the FreeBSD 10.0 release:

>3000M RAM support
The -w flag for better CentOS and RHEL performance


### bhyve OVERVIEW ###

The bhyve hypervisor consists of the following components:

/boot/kernel/vmm.ko  Kernel module
/dev/vmm/*           Index of loaded virtual machines
/usr/sbin/bhyveload  Userspace FreeBSD loader
/usr/sbin/bhyve      Virtual Machine boot utility
/usr/sbin/bhyvectl   Virtual Machine shutdown and diagnostics utility
/usr/local/sbin/grub-bhyve  Optional userspace GRUB bootloader

Both bhyveload(8) and bhyve(8) are provided the VM memory allocation and a
bootable block device in the form of a /dev/ device or a disk image plus a
virtual machine name. These must be consistent between both utilities.

bhyve(8) is provided additional arguments describing PCI devices (notably for
VirtIO storage and network) plus miscelanous operational parameters. 

A bhyve(8) virtual machine is shutdown with a reboot(8) command and the
/dev/vmm/<vmname> entry is cleaned up with the bhyvect(8) command.

VM network is primarily achived by providing an if_tap(4) network device that is
associated with an if_bridge(4) bridge. Alternatively a hardware network device
could be passed to the virtual machine but this is not supported by vmrc.

The first bhyve-ready FreeBSD snapshot "raw" VM-IMAGEs arrived on June 15th,
2014 and do not require any modification for use with bhyve. vmrc has templates
for these raw images and they can also be used with the
/usr/share/examples/bhyve/vmrun.sh script that is included with FreeBSD. These
images can be found at ftp://ftp.freebsd.org/snapshots/

Alternatively, virtual machines can be built in a manner similar to jail(8)
systems or hardware-based systems. The FreeBSD ISO install images can be used
for this purpose and this the approach taken by vmrun.sh. Note that all but the
most recent FreeBSD 11.0-CURRENT and 10.0-STABLE releases require a /etc/ttys
modification to work properly. vmrc will peform this modification as needed for
most FreeBSD versions and FreeBSD-derived operating systems.

The vmrc template syntax should be self-explanatory and attempts to accomodate
most desired system configuration options short of complex partitioning and the
creation of users or installation of ports and packages. Once built and booted,
a bhyve virtual machine behaves like a normal system and can be operated with
your tools of choice.


### DEVELOPER NOTES ###

Latest releases are available at: https://github.com/michaeldexter/vmrc

vmrc Design Goals

* "Unix" design philosophy, simple architecture and transparent operation
* Rapid, flexible development using in-base tools
* Zero dependencies for basic operation with significant-value add-ons
* Fundamentally-portable design
* Open architecture to accept additional multiplicity strategies

While patches are welcome but please take the time to understand how this script
works before proposing solutions to problems that might be more simple or
complex than they appear. Ask questions!

vmrc aims to have zero dependencies to operate. It optionally uses external
utilities like grub2-bhyve, tmux, qemu, screen and the like to add
functionality but will hopefully never depend on them.

The vm rc script and mkvm.sh perform two key functions:

1. Programmatically construct virtual machine boot devices (mkvm.sh/"iso" mode)
2. Manage the lifecycle operation of the virtual machines it creates

A "Virtual Machine" is comprised of a directory containing a bootable disk
image, a configuration file derived from a template, a mnt/ directory for the
attachment and mounting of the disk image (FreeBSD only) and an optional
preflight script.

mkvm.sh is largely a simplified FreeBSD installer that is comprised of
preflight, primary and debug functions.

The vm rc script primarily creates bhyveload(8) and bhyve(8) command syntax for
loading and booting virtual machines based on the criteria in the configuration
files such as the VM RAM allocation, block device to boot, network interfaces
etc. The vm rc script will output the bhyveload and bhyve command strings that
it generates prior to executing them. If you copy these out, they should work
on their own along with the preflight script if included.

Each VM directory in /usr/local/vmrc/vm/ is fully self-contained and is
only referenced by the vm script looking for VMs in the vm/ directory and then
using the unique numeric VM ID number at the end of the virtual machine.

For example: vm0 has the arbitrary name of "vm" and the VM ID of "0". VMs can
share the same name (vm0, vm1, vm2) but must have a unique ID (vm0, freenas1,
openbsd3). VMs vm0 and linux0 would have conflicting VM IDs.

This naming is extended to the contents of the vm/ directory (vm/vm0/vm0.img)

The VM ID is used to generate unique, non-conflicting tap# network interfaces.

With the introduction of multiple VM virtual network devices, a new convention
was needed to keep VM tap devices unique while preserving their obvious
association with a given VM. The solution is to add a suffix to the tap device,
indicating the NIC ID (vm0's tap0 -> tap01 or vm3's tap3 ->tap31).

This revealed a problem: ifconfig will now allow for tap ID's to be
prefixed with "0" and thus vm0's tap03 would be invalid. To solve this, vmrc 
emplys a double digit VMID that is prefixed with the arbitrary "8", resulting
in vm0's fourth network interface being:

tap8033 [ 8 | VM ID of 3 | Network Device ID of 4 (counting from 0) ]

Network device "3" is the fourth device because the first one is "0".

Additional network devices could be creatd on additional PCI virtual interfaces
but this functionality is not supported by vmrc.


### SPECIAL THANKS ###

Thank you Neel, Peter, John B. and Tycho for your hard work on bhyve.

Thank you Devin for greatly improving this script and Allan at ScaleEngine for
testing it and providing resources for it and bhyve's improvement.