kubernets CSI example
All CSI project are here
CSI-API
Docs (github)
Here is the spec:
Terminology
-
- CO Container Orchestration system, communicates with Plugins using CSI service RPCs.
- Plugin Supervisor Process that governs the lifecycle of a Plugin, MAY be the CO.
- SP Storage Provider, the vendor of a CSI plugin implementation
Architecture
本规范的主要重点是CO和插件之间的协议。 针对各种各样的部署架构， 应该可能的提供跨CO兼容的插件。 CO应该无差别的来处理集中式插件或无头插件，以及拆散装（split-component）插件或统一插件。
几种可能的方案:
CO "Master" Host
+-------------------------------------------+
| |
| +------------+ +------------+ |
| | CO | gRPC | Controller | |
| | +-----------> Plugin | |
| +------------+ +------------+ |
| |
+-------------------------------------------+
CO "Node" Host(s)
+-------------------------------------------+
| |
| +------------+ +------------+ |
| | CO | gRPC | Node | |
| | +-----------> Plugin | |
| +------------+ +------------+ |
| |
+-------------------------------------------+
图1：插件运行在集群中的所有节点上: 控制器插件运行在CO主节点主机上，节点插件可运行在所有CO节点上（包括master）
CO "Node" Host(s)
+-------------------------------------------+
| |
| +------------+ +------------+ |
| | CO | gRPC | Controller | |
| | +--+--------> Plugin | |
| +------------+ | +------------+ |
| | |
| | |
| | +------------+ |
| | | Node | |
| +--------> Plugin | |
| +------------+ |
| |
+-------------------------------------------+图2：无头插件部署，仅在CO Node主机上运行。 单独提供控制器服务和节点服务的分离插件
CO "Node" Host(s)
+-------------------------------------------+
| |
| +------------+ +------------+ |
| | CO | gRPC | Controller | |
| | +-----------> Node | |
| +------------+ | Plugin | |
| +------------+ |
| |
+-------------------------------------------+
图3：无头插件部署，仅运行在Node主机上， 提供同时为控制器服务和节点服务的统一插件组件
CO "Node" Host(s)
+-------------------------------------------+
| |
| +------------+ +------------+ |
| | CO | gRPC | Node | |
| | +-----------> Plugin | |
| +------------+ +------------+ |
| |
+-------------------------------------------+
图4：无头插件部署，仅运行 Node节点，只有节点插件组件仅提供节点服务。
其GetPluginCapabilities RPC不报告CONTROLLER_SERVICE能力。
Volume Lifecycle
CreateVolume +------------+ DeleteVolume
+------------->| CREATED +--------------+
| +---+----+---+ |
| Controller | | Controller v
+++ Publish | | Unpublish +++
|X| Volume | | Volume | |
+-+ +---v----+---+ +-+
| NODE_READY |
+---+----^---+
Node | | Node
Publish | | Unpublish
Volume | | Volume
+---v----+---+
| PUBLISHED |
+------------+
图5：动态预配置卷的生命周期，从创造到破坏。
CreateVolume +------------+ DeleteVolume
+------------->| CREATED +--------------+
| +---+----+---+ |
| Controller | | Controller v
+++ Publish | | Unpublish +++
|X| Volume | | Volume | |
+-+ +---v----+---+ +-+
| NODE_READY |
+---+----^---+
Node | | Node
Stage | | Unstage
Volume | | Volume
+---v----+---+
| VOL_READY |
+------------+
Node | | Node
Publish | | Unpublish
Volume | | Volume
+---v----+---+
| PUBLISHED |
+------------+图6：动态预配置卷的生命周期，从创建到销毁。当节点插件发布STAGE_UNSTAGE_VOLUME功能的情况下。
Controller Controller
Publish Unpublish
Volume +------------+ Volume
+------------->+ NODE_READY +--------------+
| +---+----^---+ |
| Node | | Node v
+++ Publish | | Unpublish +++
|X| <-+ Volume | | Volume | |
+++ | +---v----+---+ +-+
| | | PUBLISHED |
| | +------------+
+----+
Validate
Volume
Capabilities图7：预配置卷的生命周期。控制器发布publish到节点（ControllerPublishVolume）先于在节点发布publishing（`NodePublishVolume`）。
+-+ +-+
|X| | |
+++ +^+
| |
Node | | Node
Publish | | Unpublish
Volume | | Volume
+---v----+---+
| PUBLISHED |
+------------+
图8：插件可能通过capabilities API会禁忌某些功能来放弃其他生命周期步骤。 插件对volum的操作减少到NodePublishVolume和NodeUnpublishVolume。
RPC Interface
CO与plugin通过RPC通信。
- Node Plugin: 提供CSI RPCs的gRPC endpoint运行在node上，用来publish provisioned volume.
- Controller Plugin: 提供CSI RPCs的gRPC endpoint 可以运行在任何地方.
- 可能在有的情况下，一个单一的gRPC endpoint MAY提供所有的CSI RPCs (see Figure 3 in Architecture).
一共有三组RPCs集合:
- Identity Service: Node和Controller插件必须实现这组RPCs.
- Controller Service: Controller插件必须实现这组RPCs.
- Node Service: Node插件必须实现这组RPCs.
-
service Identity { rpc GetPluginInfo(GetPluginInfoRequest) returns (GetPluginInfoResponse) {} rpc GetPluginCapabilities(GetPluginCapabilitiesRequest) returns (GetPluginCapabilitiesResponse) {} rpc Probe (ProbeRequest) returns (ProbeResponse) {} } service Controller { rpc CreateVolume (CreateVolumeRequest) returns (CreateVolumeResponse) {} rpc DeleteVolume (DeleteVolumeRequest) returns (DeleteVolumeResponse) {} rpc ControllerPublishVolume (ControllerPublishVolumeRequest) returns (ControllerPublishVolumeResponse) {} rpc ControllerUnpublishVolume (ControllerUnpublishVolumeRequest) returns (ControllerUnpublishVolumeResponse) {} rpc ValidateVolumeCapabilities (ValidateVolumeCapabilitiesRequest) returns (ValidateVolumeCapabilitiesResponse) {} rpc ListVolumes (ListVolumesRequest) returns (ListVolumesResponse) {} rpc GetCapacity (GetCapacityRequest) returns (GetCapacityResponse) {} rpc ControllerGetCapabilities (ControllerGetCapabilitiesRequest) returns (ControllerGetCapabilitiesResponse) {} rpc CreateSnapshot (CreateSnapshotRequest) returns (CreateSnapshotResponse) {} rpc DeleteSnapshot (DeleteSnapshotRequest) returns (DeleteSnapshotResponse) {} rpc ListSnapshots (ListSnapshotsRequest) returns (ListSnapshotsResponse) {} rpc ControllerExpandVolume (ControllerExpandVolumeRequest) returns (ControllerExpandVolumeResponse) {} } service Node { rpc NodeStageVolume (NodeStageVolumeRequest) returns (NodeStageVolumeResponse) {} rpc NodeUnstageVolume (NodeUnstageVolumeRequest) returns (NodeUnstageVolumeResponse) {} rpc NodePublishVolume (NodePublishVolumeRequest) returns (NodePublishVolumeResponse) {} rpc NodeUnpublishVolume (NodeUnpublishVolumeRequest) returns (NodeUnpublishVolumeResponse) {} rpc NodeGetVolumeStats (NodeGetVolumeStatsRequest) returns (NodeGetVolumeStatsResponse) {} rpc NodeExpandVolume(NodeExpandVolumeRequest) returns (NodeExpandVolumeResponse) {} rpc NodeGetCapabilities (NodeGetCapabilitiesRequest) returns (NodeGetCapabilitiesResponse) {} rpc NodeGetInfo (NodeGetInfoRequest) returns (NodeGetInfoResponse) {} }
Debugging
General Configuration
- Plugin Supervisor必须给plugin提供
CSI_ENDPOINT环境变量. - Operator 必须配置CO， 通过
CSI_ENDPOINT家里的监听地址连接Plugin. - 敏感数据, Plugin configuration 尽可能通过环境变量指定, 而不会命令行的flag 或者 bind-mounted/injected 文件.
Plugin Bootstrap Example
- Supervisor -> Plugin:
CSI_ENDPOINT=unix:///path/to/unix/domain/socket.sock. - Operator -> CO: use plugin at endpoint
unix:///path/to/unix/domain/socket.sock. - CO: monitor
/path/to/unix/domain/socket.sock. - Plugin: 读取
CSI_ENDPOINT变量, 在指定的路径下面创建 UNIX socket ， 绑定和监听. - CO: observe that socket now exists, establish connection.
- CO: invoke
GetPluginCapabilities.
CSI Hostpath Driver sample
2. csi-driver-host-path/pkg/hostpath/identityserver.go 定义了identityServer
3. csi-driver-host-path/pkg/hostpath/controllerserver.go 定义了 NewControllerServer
4. csi-driver-host-path/pkg/hostpath/nodeserver.go 定义了 NewNodeServer
5. csi-driver-host-path/pkg/hostpath/hostpath.go 的GRPC server（NewNonBlockingGRPCServer）启动了NewIdentityServer，NewNodeServer， NewControllerServer。
Node Driver Registrar
The node-driver-registrar is a sidecar container that registers the CSI driver with Kubelet using the kubelet plugin registration mechanism.
This is necessary because Kubelet is responsible for issuing CSI NodeGetInfo, NodeStageVolume, NodePublishVolumecalls. The node-driver-registrar registers your CSI driver with Kubelet so that it knows which Unix domain socket to issue the CSI calls on.
Another CSI Drivers Examples
extra learn:
mdBook is a utility to create modern online books from Markdown files.
