198 lines
8.3 KiB
ReStructuredText
198 lines
8.3 KiB
ReStructuredText
|
.. SPDX-License-Identifier: GPL-2.0
|
||
|
|
|
||
|
|
.. include:: <isonum.txt>
|
||
|
|
|
||
|
|
=========================================
|
||
|
|
Rockchip Image Signal Processor (rkisp1)
|
||
|
|
=========================================
|
||
|
|
|
||
|
|
Introduction
|
||
|
|
============
|
||
|
|
|
||
|
|
This file documents the driver for the Rockchip ISP1 that is part of RK3288
|
||
|
|
and RK3399 SoCs. The driver is located under drivers/staging/media/rkisp1
|
||
|
|
and uses the Media-Controller API.
|
||
|
|
|
||
|
|
Revisions
|
||
|
|
=========
|
||
|
|
|
||
|
|
There exist multiple smaller revisions to this ISP that got introduced in
|
||
|
|
later SoCs. Revisions can be found in the enum :c:type:`rkisp1_cif_isp_version`
|
||
|
|
in the UAPI and the revision of the ISP inside the running SoC can be read
|
||
|
|
in the field hw_revision of struct media_device_info as returned by
|
||
|
|
ioctl MEDIA_IOC_DEVICE_INFO.
|
||
|
|
|
||
|
|
Versions in use are:
|
||
|
|
|
||
|
|
- RKISP1_V10: used at least in rk3288 and rk3399
|
||
|
|
- RKISP1_V11: declared in the original vendor code, but not used
|
||
|
|
- RKISP1_V12: used at least in rk3326 and px30
|
||
|
|
- RKISP1_V13: used at least in rk1808
|
||
|
|
|
||
|
|
Topology
|
||
|
|
========
|
||
|
|
.. _rkisp1_topology_graph:
|
||
|
|
|
||
|
|
.. kernel-figure:: rkisp1.dot
|
||
|
|
:alt: Diagram of the default media pipeline topology
|
||
|
|
:align: center
|
||
|
|
|
||
|
|
|
||
|
|
The driver has 4 video devices:
|
||
|
|
|
||
|
|
- rkisp1_mainpath: capture device for retrieving images, usually in higher
|
||
|
|
resolution.
|
||
|
|
- rkisp1_selfpath: capture device for retrieving images.
|
||
|
|
- rkisp1_stats: a metadata capture device that sends statistics.
|
||
|
|
- rkisp1_params: a metadata output device that receives parameters
|
||
|
|
configurations from userspace.
|
||
|
|
|
||
|
|
The driver has 3 subdevices:
|
||
|
|
|
||
|
|
- rkisp1_resizer_mainpath: used to resize and downsample frames for the
|
||
|
|
mainpath capture device.
|
||
|
|
- rkisp1_resizer_selfpath: used to resize and downsample frames for the
|
||
|
|
selfpath capture device.
|
||
|
|
- rkisp1_isp: is connected to the sensor and is responsible for all the isp
|
||
|
|
operations.
|
||
|
|
|
||
|
|
|
||
|
|
rkisp1_mainpath, rkisp1_selfpath - Frames Capture Video Nodes
|
||
|
|
-------------------------------------------------------------
|
||
|
|
Those are the `mainpath` and `selfpath` capture devices to capture frames.
|
||
|
|
Those entities are the DMA engines that write the frames to memory.
|
||
|
|
The selfpath video device can capture YUV/RGB formats. Its input is YUV encoded
|
||
|
|
stream and it is able to convert it to RGB. The selfpath is not able to
|
||
|
|
capture bayer formats.
|
||
|
|
The mainpath can capture both bayer and YUV formats but it is not able to
|
||
|
|
capture RGB formats.
|
||
|
|
Both capture videos support
|
||
|
|
the ``V4L2_CAP_IO_MC`` :ref:`capability <device-capabilities>`.
|
||
|
|
|
||
|
|
|
||
|
|
rkisp1_resizer_mainpath, rkisp1_resizer_selfpath - Resizers Subdevices Nodes
|
||
|
|
----------------------------------------------------------------------------
|
||
|
|
Those are resizer entities for the mainpath and the selfpath. Those entities
|
||
|
|
can scale the frames up and down and also change the YUV sampling (for example
|
||
|
|
YUV4:2:2 -> YUV4:2:0). They also have cropping capability on the sink pad.
|
||
|
|
The resizers entities can only operate on YUV:4:2:2 format
|
||
|
|
(MEDIA_BUS_FMT_YUYV8_2X8).
|
||
|
|
The mainpath capture device supports capturing video in bayer formats. In that
|
||
|
|
case the resizer of the mainpath is set to 'bypass' mode - it just forward the
|
||
|
|
frame without operating on it.
|
||
|
|
|
||
|
|
rkisp1_isp - Image Signal Processing Subdevice Node
|
||
|
|
---------------------------------------------------
|
||
|
|
This is the isp entity. It is connected to the sensor on sink pad 0 and
|
||
|
|
receives the frames using the CSI-2 protocol. It is responsible of configuring
|
||
|
|
the CSI-2 protocol. It has a cropping capability on sink pad 0 that is
|
||
|
|
connected to the sensor and on source pad 2 connected to the resizer entities.
|
||
|
|
Cropping on sink pad 0 defines the image region from the sensor.
|
||
|
|
Cropping on source pad 2 defines the region for the Image Stabilizer (IS).
|
||
|
|
|
||
|
|
.. _rkisp1_stats:
|
||
|
|
|
||
|
|
rkisp1_stats - Statistics Video Node
|
||
|
|
------------------------------------
|
||
|
|
The statistics video node outputs the 3A (auto focus, auto exposure and auto
|
||
|
|
white balance) statistics, and also histogram statistics for the frames that
|
||
|
|
are being processed by the rkisp1 to userspace applications.
|
||
|
|
Using these data, applications can implement algorithms and re-parameterize
|
||
|
|
the driver through the rkisp_params node to improve image quality during a
|
||
|
|
video stream.
|
||
|
|
The buffer format is defined by struct :c:type:`rkisp1_stat_buffer`, and
|
||
|
|
userspace should set
|
||
|
|
:ref:`V4L2_META_FMT_RK_ISP1_STAT_3A <v4l2-meta-fmt-rk-isp1-stat-3a>` as the
|
||
|
|
dataformat.
|
||
|
|
|
||
|
|
.. _rkisp1_params:
|
||
|
|
|
||
|
|
rkisp1_params - Parameters Video Node
|
||
|
|
-------------------------------------
|
||
|
|
The rkisp1_params video node receives a set of parameters from userspace
|
||
|
|
to be applied to the hardware during a video stream, allowing userspace
|
||
|
|
to dynamically modify values such as black level, cross talk corrections
|
||
|
|
and others.
|
||
|
|
|
||
|
|
The buffer format is defined by struct :c:type:`rkisp1_params_cfg`, and
|
||
|
|
userspace should set
|
||
|
|
:ref:`V4L2_META_FMT_RK_ISP1_PARAMS <v4l2-meta-fmt-rk-isp1-params>` as the
|
||
|
|
dataformat.
|
||
|
|
|
||
|
|
|
||
|
|
Capturing Video Frames Example
|
||
|
|
==============================
|
||
|
|
|
||
|
|
In the following example, the sensor connected to pad 0 of 'rkisp1_isp' is
|
||
|
|
imx219.
|
||
|
|
|
||
|
|
The following commands can be used to capture video from the selfpath video
|
||
|
|
node with dimension 900x800 planar format YUV 4:2:2. It uses all cropping
|
||
|
|
capabilities possible, (see explanation right below)
|
||
|
|
|
||
|
|
.. code-block:: bash
|
||
|
|
|
||
|
|
# set the links
|
||
|
|
"media-ctl" "-d" "platform:rkisp1" "-r"
|
||
|
|
"media-ctl" "-d" "platform:rkisp1" "-l" "'imx219 4-0010':0 -> 'rkisp1_isp':0 [1]"
|
||
|
|
"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_selfpath':0 [1]"
|
||
|
|
"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_mainpath':0 [0]"
|
||
|
|
|
||
|
|
# set format for imx219 4-0010:0
|
||
|
|
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1640x1232]'
|
||
|
|
|
||
|
|
# set format for rkisp1_isp pads:
|
||
|
|
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":0 [fmt:SRGGB10_1X10/1640x1232 crop: (0,0)/1600x1200]'
|
||
|
|
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":2 [fmt:YUYV8_2X8/1600x1200 crop: (0,0)/1500x1100]'
|
||
|
|
|
||
|
|
# set format for rkisp1_resizer_selfpath pads:
|
||
|
|
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":0 [fmt:YUYV8_2X8/1500x1100 crop: (300,400)/1400x1000]'
|
||
|
|
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":1 [fmt:YUYV8_2X8/900x800]'
|
||
|
|
|
||
|
|
# set format for rkisp1_selfpath:
|
||
|
|
"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "width=900,height=800,"
|
||
|
|
"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "pixelformat=422P"
|
||
|
|
|
||
|
|
# start streaming:
|
||
|
|
v4l2-ctl "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "--stream-mmap" "--stream-count" "10"
|
||
|
|
|
||
|
|
|
||
|
|
In the above example the sensor is configured to bayer format:
|
||
|
|
`SRGGB10_1X10/1640x1232`. The rkisp1_isp:0 pad should be configured to the
|
||
|
|
same mbus format and dimensions as the sensor, otherwise streaming will fail
|
||
|
|
with 'EPIPE' error. So it is also configured to `SRGGB10_1X10/1640x1232`.
|
||
|
|
In addition, the rkisp1_isp:0 pad is configured to cropping `(0,0)/1600x1200`.
|
||
|
|
|
||
|
|
The cropping dimensions are automatically propagated to be the format of the
|
||
|
|
isp source pad `rkisp1_isp:2`. Another cropping operation is configured on
|
||
|
|
the isp source pad: `(0,0)/1500x1100`.
|
||
|
|
|
||
|
|
The resizer's sink pad `rkisp1_resizer_selfpath` should be configured to format
|
||
|
|
`YUYV8_2X8/1500x1100` in order to match the format on the other side of the
|
||
|
|
link. In addition a cropping `(300,400)/1400x1000` is configured on it.
|
||
|
|
|
||
|
|
The source pad of the resizer, `rkisp1_resizer_selfpath:1` is configured to
|
||
|
|
format `YUYV8_2X8/900x800`. That means that the resizer first crop a window
|
||
|
|
of `(300,400)/1400x100` from the received frame and then scales this window
|
||
|
|
to dimension `900x800`.
|
||
|
|
|
||
|
|
Note that the above example does not uses the stats-params control loop.
|
||
|
|
Therefore the capture frames will not go through the 3A algorithms and
|
||
|
|
probably won't have a good quality, and can even look dark and greenish.
|
||
|
|
|
||
|
|
Configuring Quantization
|
||
|
|
========================
|
||
|
|
|
||
|
|
The driver supports limited and full range quantization on YUV formats,
|
||
|
|
where limited is the default.
|
||
|
|
To switch between one or the other, userspace should use the Colorspace
|
||
|
|
Conversion API (CSC) for subdevices on source pad 2 of the
|
||
|
|
isp (`rkisp1_isp:2`). The quantization configured on this pad is the
|
||
|
|
quantization of the captured video frames on the mainpath and selfpath
|
||
|
|
video nodes.
|
||
|
|
Note that the resizer and capture entities will always report
|
||
|
|
``V4L2_QUANTIZATION_DEFAULT`` even if the quantization is configured to full
|
||
|
|
range on `rkisp1_isp:2`. So in order to get the configured quantization,
|
||
|
|
application should get it from pad `rkisp1_isp:2`.
|
||
|
|
|