Skip to content
This repository has been archived by the owner on Apr 14, 2024. It is now read-only.

OSM Sidecar Driver

Jump to bottom
Cybwan edited this page Jul 1, 2022 · 8 revisions

Sidecar代理控制平面是OSM关键组件之一，为sidecar提供持续的配置更新的服务，实现相应的sidecar规范的接口。OSM的Sidecar代理控制平面基于Envoy’s go-control-plane xDS v3 API实现，同envoy强绑定。引入Sidecar驱动的概念，实现Sidecar代理控制平面同Envoy解耦，提供更广泛的Sidecar接入能力。

架构图

Sidecar代理控制平面仅负责Sidecar驱动的生命周期的管理及接口适配，不再同Sidecar直接交互；Sidecar厂商基于标准的Sidecar驱动接口，参照自有的规范，实现Sidecar驱动，为Sidecar提供持续的配置更新的能力。

Sidecar驱动生命周期

Sidecar驱动接口

一个Sidecar驱动需要实现Driver接口，以便同OSM的Injector和Controller集成。

// Driver is an interface that must be implemented by a sidecar driver.
// Patch method is invoked by osm-injector and Start method is invoked by osm-controller
type Driver interface {
	Patch(ctx context.Context) error
	Start(ctx context.Context) (health.Probes, error)
}

Start方法返回Sidecar驱动健康检查的探针。

// HealthProbes is to serve as an indication how to probe the sidecar driver's health status
type HealthProbes struct {
	liveness, readiness, startup *HealthProbe
}

// HealthProbe is an API endpoint to indicate the current status of the sidecar driver.
type HealthProbe struct {
	path      string
	port      int32
	http      bool
	timeout   time.Duration
	tcpSocket bool
}

Patch

当一个POD被部署时，OSM Injector启动时注册的webhook将为这个POD做注入配置，例如为Sidecar创建证书，设置Labels、Annotations、Metrics等；然后webhook调用Patch方法, 这个方法需实现：

  • 1.注入初始化容器，为Sidecar配置网络相关规则，例如网络流量的转发

  • 2.注入Sidecar容器，设置sidecar配置文件的挂载卷，健康检查等

这个方法所需要的信息被封装到InjectorContext上下文中:

type InjectorContext struct {
	context.Context

	Pod                          *corev1.Pod
	MeshName                     string
	OsmNamespace                 string
	PodNamespace                 string
	PodOS                        string
	ProxyCommonName              certificate.CommonName
	ProxyUUID                    uuid.UUID
	Configurator                 configurator.Configurator
	KubeClient                   kubernetes.Interface
	BootstrapCertificate         *certificate.Certificate
	ContainerPullPolicy          corev1.PullPolicy
	InboundPortExclusionList     []int
	OutboundPortExclusionList    []int
	OutboundIPRangeInclusionList []string
	OutboundIPRangeExclusionList []string
	OriginalHealthProbes         HealthProbes

	DryRun bool
}

Start

OSM Controller启动时调用这个Start方法，这个方法需实现：

  • 1.一个Sidecar proxy控制平面服务，将相关SMI的流量策略封装成Sidecar需要的格式

  • 2.方法返回健康检查对象

这个方法所需要的信息被封装到ControllerContext上下文中:

type ControllerContext struct {
	context.Context

	ProxyServerPort  uint32
	ProxyServiceCert *certificate.Certificate
	OsmNamespace     string
	KubeConfig       *rest.Config
	Configurator     configurator.Configurator
	MeshCatalog      catalog.MeshCataloger
	CertManager      certificate.Manager
	MsgBroker        *messaging.Broker
	DebugHandlers    map[string]http.Handler
	CancelFunc       func()
	Stop             chan struct {
	}
}

Context

调用Patch和Start方法时，context.WithCancel将InjectorContext和ControllerContext封装为context.Context:

background := driver.InjectorContext{
...
}
ctx, cancel := context.WithCancel(&background)
defer cancel()

background := driver.ControllerContext{
...
}
ctx, cancel := context.WithCancel(&background)
defer cancel()
background.CancelFunc = cancel

在sidecar驱动的实现中，通过InjectorCtxKey和ControllerCtxKey获取：

parentCtx := ctx.Value(&driver.InjectorCtxKey)
if parentCtx == nil {
	return nil, errors.New("missing Injector Context")
}
injCtx := parentCtx.(*driver.InjectorContext)

parentCtx := ctx.Value(&driver.ControllerCtxKey)
if parentCtx == nil {
	return nil, errors.New("missing Controller Context")
}
ctrlCtx := parentCtx.(*driver.ControllerContext)

参考实现

pkg/sidecar/driver/types.go

Sidecar驱动适配**

变量说明

map对象drivers作为sidecar驱动的对象容器；读写锁driversMutex确保操作drivers时线程安全；engineDriver保持当前正在被使用的Sidecar驱动:

var (
	driversMutex sync.RWMutex
	drivers      = make(map[string]driver.Driver)
	engineDriver driver.Driver
)

sidecar驱动注册

Register方法，完成驱动的注册，驱动多次注册会触发panic，驱动实现时要避免重复注册

// Register makes a sidecar driver available by the provided name.
// If Register is called twice with the same name or if driver is nil,
// it panics.
func Register(name string, driver driver.Driver) {
	driversMutex.Lock()
	defer driversMutex.Unlock()
	if driver == nil {
		panic("sidecar: Register driver is nil")
	}
	if _, dup := drivers[name]; dup {
		panic("sidecar: Register called twice for driver " + name)
	}
	drivers[name] = driver
}

sidecar驱动适配

Patch

// Patch is an adapter method for InjectorDriver.Patch
func Patch(ctx context.Context) error {
	driversMutex.RLock()
	defer driversMutex.RUnlock()
	if engineDriver == nil {
		return errors.New("sidecar: unknown driver (forgot to init?)")
	}
	return engineDriver.Patch(ctx)
}

Start

// Start is an adapter method for ControllerDriver.Start
func Start(ctx context.Context) (health.Probes, error) {
	driversMutex.RLock()
	defer driversMutex.RUnlock()
	if engineDriver == nil {
		return nil, errors.New("sidecar: unknown driver (forgot to init?)")
	}
	return engineDriver.Start(ctx)
}

sidecar驱动启用

// InitDriver is to serve as an indication of the using sidecar driver
func InitDriver(driverName string) error {
	driversMutex.Lock()
	defer driversMutex.Unlock()
	registeredDriver, ok := drivers[driverName]
	if !ok {
		return fmt.Errorf("sidecar: unknown driver %q (forgotten import?)", driverName)
	}
	engineDriver = registeredDriver
	return nil
}

参考实现

pkg/sidecar/adapter.go

Sidecar驱动实现

Envoy 驱动实现

Pipy 驱动实现

驱动同DebugServer集成

在驱动的Start方法中，可以通过ControllerContext.DebugHandlers丰富DebugServer的调试功能

// EnvoySidecarDriver is the envoy sidecar driver
type EnvoySidecarDriver struct {
	ctx *driver.ControllerContext
}

// Start is the implement for ControllerDriver.Start
func (sd EnvoySidecarDriver) Start(ctx context.Context) (health.Probes, error) {
	parentCtx := ctx.Value(&driver.ControllerCtxKey)
	if parentCtx == nil {
		return nil, errors.New("missing Controller Context")
	}
	ctrlCtx := parentCtx.(*driver.ControllerContext)
	...

	ctrlCtx.DebugHandlers["/debug/proxy"] = sd.getProxies(proxyRegistry)
	ctrlCtx.DebugHandlers["/debug/xds"] = sd.getXDSHandler(xdsServer)

	...
}

Sidecar驱动使用

Register

import (
	...
	_ "github.com/openservicemesh/osm/pkg/sidecar/providers/envoy/driver"
	_ "github.com/openservicemesh/osm/pkg/sidecar/providers/pipy/driver"
	...
)

驱动实现时，可以通过在init方法中注册驱动，实现驱动的引用即注册功能

const (
	driverName = `pipy`
)

func init() {
	sidecar.Register(driverName, new(PipySidecarDriver))
}

Install

根据MeshConfig中的sidecarClass，安装相应的sidecar驱动

cfg := configurator.NewConfigurator(...)
err = sidecar.InstallDriver(cfg.GetSidecarClass())
if err != nil {
    events.GenericEventRecorder().FatalEvent(err, events.InitializationError, "Error creating sidecar driver")
}

Patch

func (wh *mutatingWebhook) createPatch(pod *corev1.Pod, req *admissionv1.AdmissionRequest, proxyUUID uuid.UUID) ([]byte, error) {
	...

	background := driver.InjectorContext{
		Pod:                          pod,
		MeshName:                     wh.meshName,
		OsmNamespace:                 wh.osmNamespace,
		PodNamespace:                 namespace,
		PodOS:                        podOS,
		ProxyCommonName:              cn,
		ProxyUUID:                    proxyUUID,
		Configurator:                 wh.configurator,
		KubeClient:                   wh.kubeClient,
		BootstrapCertificate:         bootstrapCertificate,
		ContainerPullPolicy:          wh.osmContainerPullPolicy,
		InboundPortExclusionList:     inboundPortExclusionList,
		OutboundPortExclusionList:    outboundPortExclusionList,
		OutboundIPRangeInclusionList: outboundIPRangeInclusionList,
		OutboundIPRangeExclusionList: outboundIPRangeExclusionList,
		OriginalHealthProbes:         originalHealthProbes,
		DryRun:                       req.DryRun != nil && *req.DryRun,
	}
	ctx, cancel := context.WithCancel(&background)
	defer cancel()

	if err = sidecar.Patch(ctx); err != nil {
		return nil, err
	}

	return json.Marshal(makePatches(req, pod))
}

Start

func main() {
	...

	background := driver.ControllerContext{
		ProxyServerPort:  cfg.GetProxyServerPort(),
		ProxyServiceCert: proxyServiceCert,
		OsmNamespace:     osmNamespace,
		KubeConfig:       kubeConfig,
		Configurator:     cfg,
		MeshCatalog:      meshCatalog,
		CertManager:      certManager,
		MsgBroker:        msgBroker,
		DebugHandlers:    make(map[string]http.Handler),
		Stop:             stop,
	}
	ctx, cancel := context.WithCancel(&background)
	defer cancel()
	background.CancelFunc = cancel

	// Create and start the sidecar proxy service
	healthProbes, err := sidecar.Start(ctx)
	if err != nil {
		events.GenericEventRecorder().FatalEvent(err, events.InitializationError, "Error initializing proxy control server")
	}

	...
	// Health/Liveness probes
	funcProbes := []health.Probes{healthProbes, smi.HealthChecker{DiscoveryClient: clientset.Discovery()}}
	...

	// Create DebugServer and start its config event listener.
	// Listener takes care to start and stop the debug server as appropriate
	debugConfig := debugger.NewDebugConfig(certDebugger, meshCatalog, kubeConfig, kubeClient, cfg, k8sClient, msgBroker)
	go debugConfig.StartDebugServerConfigListener(background.DebugHandlers, stop)

	...
}

参考实现

pkg/injector/patch.go

cmd/osm-injector/osm-injector.go

cmd/osm-controller/osm-controller.go

OSM API调整

SidecarSpec增加SidecarClass和SidecarDriverSpec；SidecarClass指定默认使用的sidecar驱动

// SidecarDriverSpec is the type to represent OSM's sidecar driver define.
type SidecarDriverSpec struct {
	...

	// SidecarClass defines the class used for the proxy sidecar.
	SidecarClass string `json:"sidecarClass,omitempty"`

	// SidecarImage defines the container image used for the proxy sidecar.
	SidecarImage string `json:"sidecarImage,omitempty"`

	// SidecarWindowsImage defines the windows container image used for the proxy sidecar.
	SidecarWindowsImage string `json:"SidecarImageWindowsImage,omitempty"`

	// InitContainerImage defines the container image used for the init container injected to meshed pods.
	InitContainerImage string `json:"initContainerImage,omitempty"`

	// SidecarDrivers defines the sidecar supported.
	SidecarDrivers []SidecarDriverSpec `json:"sidecarDrivers,omitempty"`
	
	...
}

type SidecarDriverSpec struct {
	// SidecarName defines the name of the sidecar driver.
	SidecarName string `json:"sidecarName,omitempty"`

	// SidecarImage defines the container image used for the proxy sidecar.
	SidecarImage string `json:"sidecarImage,omitempty"`

	// SidecarWindowsImage defines the windows container image used for the proxy sidecar.
	SidecarWindowsImage string `json:"SidecarImageWindowsImage,omitempty"`

	// InitContainerImage defines the container image used for the init container injected to meshed pods.
	InitContainerImage string `json:"initContainerImage,omitempty"`

	// ProxyServerPort is the port on which the Discovery Service listens for new connections from Sidecars
	ProxyServerPort uint32 `json:"proxyServerPort"`

	// SidecarDisabledMTLS defines whether mTLS is disabled.
	SidecarDisabledMTLS bool `json:"sidecarDisabledMTLS"`
}

参考实现

pkg/apis/config/v1alpha1/mesh_config.go

pkg/apis/config/v1alpha2/mesh_config.go

Helm Charts调整

values.yaml

osm:
  ...
  # -- The class of the OSM Sidecar Driver
  sidecarClass: pipy
  # -- Sidecar image for Linux workloads
  sidecarImage: flomesh/pipy-nightly:latest
  # -- Sidecar drivers supported by osm
  sidecarDrivers:
    - sidecarName: pipy
      # -- Sidecar image for Linux workloads
      sidecarImage: flomesh/pipy-nightly:latest
      # -- Remote destination port on which the Discovery Service listens for new connections from Sidecars.
      proxyServerPort: 6060
    - sidecarName: envoy
      # -- Sidecar image for Linux workloads
      sidecarImage: envoyproxy/envoy:v1.19.3
      # -- Sidecar image for Windows workloads
      sidecarWindowsImage: envoyproxy/envoy-windows:latest
      # -- Remote destination port on which the Discovery Service listens for new connections from Sidecars.
      proxyServerPort: 15128
   ...

osm下的sidecarImage、sidecarWindowsImage使用的优先级高于osm.sidecarDrivers下的sidecarImage、sidecarWindowsImage。如果osm下的sidecarImage、sidecarWindowsImage未设置，则通过sidecarClass获取相应驱动下的sidecarImage、sidecarWindowsImage设置的镜像。该逻辑的代码实现如下:

// GetSidecarImage returns the sidecar image
func (c *client) GetSidecarImage() string {
	image := c.getMeshConfig().Spec.Sidecar.SidecarImage
	if len(image) == 0 {
		sidecarClass := c.getMeshConfig().Spec.Sidecar.SidecarClass
		sidecarDrivers := c.getMeshConfig().Spec.Sidecar.SidecarDrivers
		for _, sidecarDriver := range sidecarDrivers {
			if strings.EqualFold(strings.ToLower(sidecarClass), strings.ToLower(sidecarDriver.SidecarName)) {
				image = sidecarDriver.SidecarImage
				break
			}
		}
	}
	if len(image) == 0 {
		image = os.Getenv("OSM_DEFAULT_SIDECAR_IMAGE")
	}
	return image
}

preset-mesh-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: preset-mesh-config
  namespace: {{ include "osm.namespace" . }}
data:
  preset-mesh-config.json: |
    {
      "sidecar": {
        "enablePrivilegedInitContainer": {{.Values.osm.enablePrivilegedInitContainer | mustToJson}},
        "logLevel": {{.Values.osm.sidecarLogLevel | mustToJson}},
        "maxDataPlaneConnections": {{.Values.osm.maxDataPlaneConnections | mustToJson}},
        "configResyncInterval": {{.Values.osm.configResyncInterval | mustToJson}},
        "sidecarClass": {{.Values.osm.sidecarClass | mustToJson }},
        "sidecarDrivers": {{.Values.osm.sidecarDrivers | mustToJson }}
      },
      ...
    }
Clone this wiki locally