228 lines
9.8 KiB
ReStructuredText
228 lines
9.8 KiB
ReStructuredText
|
.. SPDX-License-Identifier: GPL-2.0+
|
|||
|
|
|
|||
|
|
======================================================
|
|||
|
|
IBM Virtual Management Channel Kernel Driver (IBMVMC)
|
|||
|
|
======================================================
|
|||
|
|
|
|||
|
|
:Authors:
|
|||
|
|
Dave Engebretsen <engebret@us.ibm.com>,
|
|||
|
|
Adam Reznechek <adreznec@linux.vnet.ibm.com>,
|
|||
|
|
Steven Royer <seroyer@linux.vnet.ibm.com>,
|
|||
|
|
Bryant G. Ly <bryantly@linux.vnet.ibm.com>,
|
|||
|
|
|
|||
|
|
Introduction
|
|||
|
|
============
|
|||
|
|
|
|||
|
|
Note: Knowledge of virtualization technology is required to understand
|
|||
|
|
this document.
|
|||
|
|
|
|||
|
|
A good reference document would be:
|
|||
|
|
|
|||
|
|
https://openpowerfoundation.org/wp-content/uploads/2016/05/LoPAPR_DRAFT_v11_24March2016_cmt1.pdf
|
|||
|
|
|
|||
|
|
The Virtual Management Channel (VMC) is a logical device which provides an
|
|||
|
|
interface between the hypervisor and a management partition. This interface
|
|||
|
|
is like a message passing interface. This management partition is intended
|
|||
|
|
to provide an alternative to systems that use a Hardware Management
|
|||
|
|
Console (HMC) - based system management.
|
|||
|
|
|
|||
|
|
The primary hardware management solution that is developed by IBM relies
|
|||
|
|
on an appliance server named the Hardware Management Console (HMC),
|
|||
|
|
packaged as an external tower or rack-mounted personal computer. In a
|
|||
|
|
Power Systems environment, a single HMC can manage multiple POWER
|
|||
|
|
processor-based systems.
|
|||
|
|
|
|||
|
|
Management Application
|
|||
|
|
----------------------
|
|||
|
|
|
|||
|
|
In the management partition, a management application exists which enables
|
|||
|
|
a system administrator to configure the system’s partitioning
|
|||
|
|
characteristics via a command line interface (CLI) or Representational
|
|||
|
|
State Transfer Application (REST API's).
|
|||
|
|
|
|||
|
|
The management application runs on a Linux logical partition on a
|
|||
|
|
POWER8 or newer processor-based server that is virtualized by PowerVM.
|
|||
|
|
System configuration, maintenance, and control functions which
|
|||
|
|
traditionally require an HMC can be implemented in the management
|
|||
|
|
application using a combination of HMC to hypervisor interfaces and
|
|||
|
|
existing operating system methods. This tool provides a subset of the
|
|||
|
|
functions implemented by the HMC and enables basic partition configuration.
|
|||
|
|
The set of HMC to hypervisor messages supported by the management
|
|||
|
|
application component are passed to the hypervisor over a VMC interface,
|
|||
|
|
which is defined below.
|
|||
|
|
|
|||
|
|
The VMC enables the management partition to provide basic partitioning
|
|||
|
|
functions:
|
|||
|
|
|
|||
|
|
- Logical Partitioning Configuration
|
|||
|
|
- Start, and stop actions for individual partitions
|
|||
|
|
- Display of partition status
|
|||
|
|
- Management of virtual Ethernet
|
|||
|
|
- Management of virtual Storage
|
|||
|
|
- Basic system management
|
|||
|
|
|
|||
|
|
Virtual Management Channel (VMC)
|
|||
|
|
--------------------------------
|
|||
|
|
|
|||
|
|
A logical device, called the Virtual Management Channel (VMC), is defined
|
|||
|
|
for communicating between the management application and the hypervisor. It
|
|||
|
|
basically creates the pipes that enable virtualization management
|
|||
|
|
software. This device is presented to a designated management partition as
|
|||
|
|
a virtual device.
|
|||
|
|
|
|||
|
|
This communication device uses Command/Response Queue (CRQ) and the
|
|||
|
|
Remote Direct Memory Access (RDMA) interfaces. A three-way handshake is
|
|||
|
|
defined that must take place to establish that both the hypervisor and
|
|||
|
|
management partition sides of the channel are running prior to
|
|||
|
|
sending/receiving any of the protocol messages.
|
|||
|
|
|
|||
|
|
This driver also utilizes Transport Event CRQs. CRQ messages are sent
|
|||
|
|
when the hypervisor detects one of the peer partitions has abnormally
|
|||
|
|
terminated, or one side has called H_FREE_CRQ to close their CRQ.
|
|||
|
|
Two new classes of CRQ messages are introduced for the VMC device. VMC
|
|||
|
|
Administrative messages are used for each partition using the VMC to
|
|||
|
|
communicate capabilities to their partner. HMC Interface messages are used
|
|||
|
|
for the actual flow of HMC messages between the management partition and
|
|||
|
|
the hypervisor. As most HMC messages far exceed the size of a CRQ buffer,
|
|||
|
|
a virtual DMA (RMDA) of the HMC message data is done prior to each HMC
|
|||
|
|
Interface CRQ message. Only the management partition drives RDMA
|
|||
|
|
operations; hypervisors never directly cause the movement of message data.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Terminology
|
|||
|
|
-----------
|
|||
|
|
RDMA
|
|||
|
|
Remote Direct Memory Access is DMA transfer from the server to its
|
|||
|
|
client or from the server to its partner partition. DMA refers
|
|||
|
|
to both physical I/O to and from memory operations and to memory
|
|||
|
|
to memory move operations.
|
|||
|
|
CRQ
|
|||
|
|
Command/Response Queue a facility which is used to communicate
|
|||
|
|
between partner partitions. Transport events which are signaled
|
|||
|
|
from the hypervisor to partition are also reported in this queue.
|
|||
|
|
|
|||
|
|
Example Management Partition VMC Driver Interface
|
|||
|
|
=================================================
|
|||
|
|
|
|||
|
|
This section provides an example for the management application
|
|||
|
|
implementation where a device driver is used to interface to the VMC
|
|||
|
|
device. This driver consists of a new device, for example /dev/ibmvmc,
|
|||
|
|
which provides interfaces to open, close, read, write, and perform
|
|||
|
|
ioctl’s against the VMC device.
|
|||
|
|
|
|||
|
|
VMC Interface Initialization
|
|||
|
|
----------------------------
|
|||
|
|
|
|||
|
|
The device driver is responsible for initializing the VMC when the driver
|
|||
|
|
is loaded. It first creates and initializes the CRQ. Next, an exchange of
|
|||
|
|
VMC capabilities is performed to indicate the code version and number of
|
|||
|
|
resources available in both the management partition and the hypervisor.
|
|||
|
|
Finally, the hypervisor requests that the management partition create an
|
|||
|
|
initial pool of VMC buffers, one buffer for each possible HMC connection,
|
|||
|
|
which will be used for management application session initialization.
|
|||
|
|
Prior to completion of this initialization sequence, the device returns
|
|||
|
|
EBUSY to open() calls. EIO is returned for all open() failures.
|
|||
|
|
|
|||
|
|
::
|
|||
|
|
|
|||
|
|
Management Partition Hypervisor
|
|||
|
|
CRQ INIT
|
|||
|
|
---------------------------------------->
|
|||
|
|
CRQ INIT COMPLETE
|
|||
|
|
<----------------------------------------
|
|||
|
|
CAPABILITIES
|
|||
|
|
---------------------------------------->
|
|||
|
|
CAPABILITIES RESPONSE
|
|||
|
|
<----------------------------------------
|
|||
|
|
ADD BUFFER (HMC IDX=0,1,..) _
|
|||
|
|
<---------------------------------------- |
|
|||
|
|
ADD BUFFER RESPONSE | - Perform # HMCs Iterations
|
|||
|
|
----------------------------------------> -
|
|||
|
|
|
|||
|
|
VMC Interface Open
|
|||
|
|
------------------
|
|||
|
|
|
|||
|
|
After the basic VMC channel has been initialized, an HMC session level
|
|||
|
|
connection can be established. The application layer performs an open() to
|
|||
|
|
the VMC device and executes an ioctl() against it, indicating the HMC ID
|
|||
|
|
(32 bytes of data) for this session. If the VMC device is in an invalid
|
|||
|
|
state, EIO will be returned for the ioctl(). The device driver creates a
|
|||
|
|
new HMC session value (ranging from 1 to 255) and HMC index value (starting
|
|||
|
|
at index 0 and ranging to 254) for this HMC ID. The driver then does an
|
|||
|
|
RDMA of the HMC ID to the hypervisor, and then sends an Interface Open
|
|||
|
|
message to the hypervisor to establish the session over the VMC. After the
|
|||
|
|
hypervisor receives this information, it sends Add Buffer messages to the
|
|||
|
|
management partition to seed an initial pool of buffers for the new HMC
|
|||
|
|
connection. Finally, the hypervisor sends an Interface Open Response
|
|||
|
|
message, to indicate that it is ready for normal runtime messaging. The
|
|||
|
|
following illustrates this VMC flow:
|
|||
|
|
|
|||
|
|
::
|
|||
|
|
|
|||
|
|
Management Partition Hypervisor
|
|||
|
|
RDMA HMC ID
|
|||
|
|
---------------------------------------->
|
|||
|
|
Interface Open
|
|||
|
|
---------------------------------------->
|
|||
|
|
Add Buffer _
|
|||
|
|
<---------------------------------------- |
|
|||
|
|
Add Buffer Response | - Perform N Iterations
|
|||
|
|
----------------------------------------> -
|
|||
|
|
Interface Open Response
|
|||
|
|
<----------------------------------------
|
|||
|
|
|
|||
|
|
VMC Interface Runtime
|
|||
|
|
---------------------
|
|||
|
|
|
|||
|
|
During normal runtime, the management application and the hypervisor
|
|||
|
|
exchange HMC messages via the Signal VMC message and RDMA operations. When
|
|||
|
|
sending data to the hypervisor, the management application performs a
|
|||
|
|
write() to the VMC device, and the driver RDMA’s the data to the hypervisor
|
|||
|
|
and then sends a Signal Message. If a write() is attempted before VMC
|
|||
|
|
device buffers have been made available by the hypervisor, or no buffers
|
|||
|
|
are currently available, EBUSY is returned in response to the write(). A
|
|||
|
|
write() will return EIO for all other errors, such as an invalid device
|
|||
|
|
state. When the hypervisor sends a message to the management, the data is
|
|||
|
|
put into a VMC buffer and an Signal Message is sent to the VMC driver in
|
|||
|
|
the management partition. The driver RDMA’s the buffer into the partition
|
|||
|
|
and passes the data up to the appropriate management application via a
|
|||
|
|
read() to the VMC device. The read() request blocks if there is no buffer
|
|||
|
|
available to read. The management application may use select() to wait for
|
|||
|
|
the VMC device to become ready with data to read.
|
|||
|
|
|
|||
|
|
::
|
|||
|
|
|
|||
|
|
Management Partition Hypervisor
|
|||
|
|
MSG RDMA
|
|||
|
|
---------------------------------------->
|
|||
|
|
SIGNAL MSG
|
|||
|
|
---------------------------------------->
|
|||
|
|
SIGNAL MSG
|
|||
|
|
<----------------------------------------
|
|||
|
|
MSG RDMA
|
|||
|
|
<----------------------------------------
|
|||
|
|
|
|||
|
|
VMC Interface Close
|
|||
|
|
-------------------
|
|||
|
|
|
|||
|
|
HMC session level connections are closed by the management partition when
|
|||
|
|
the application layer performs a close() against the device. This action
|
|||
|
|
results in an Interface Close message flowing to the hypervisor, which
|
|||
|
|
causes the session to be terminated. The device driver must free any
|
|||
|
|
storage allocated for buffers for this HMC connection.
|
|||
|
|
|
|||
|
|
::
|
|||
|
|
|
|||
|
|
Management Partition Hypervisor
|
|||
|
|
INTERFACE CLOSE
|
|||
|
|
---------------------------------------->
|
|||
|
|
INTERFACE CLOSE RESPONSE
|
|||
|
|
<----------------------------------------
|
|||
|
|
|
|||
|
|
Additional Information
|
|||
|
|
======================
|
|||
|
|
|
|||
|
|
For more information on the documentation for CRQ Messages, VMC Messages,
|
|||
|
|
HMC interface Buffers, and signal messages please refer to the Linux on
|
|||
|
|
Power Architecture Platform Reference. Section F.
|