NanoPi + OV5640 Camera

Intro

Needed a board that was capable of delivering at least a 720p videostream and went for the NanoPi Duo2. Which should (theoretically) handle 1080p at 30fps. Spoiler alert. It can’t

Cedrus is a part of the Allwinner SOC which accelerates H264 encoding but is a bit of a trick to set up. Especially on mainline kernels. This tutorial is about how to setup your NanoPi (or other Allwinner H3 boards) to recognize the I2C camera and Cedrus device

OV5640 sensor

The NanoPi Duo2 does not recognize the OV5640 image sensor and Cedrus H264 hardware encoder by default. For this to work we need to enable them by editing the Device Tree. Download the kernel source and extract.

Kernel source

sudo apt update
sudo apt upgrade

sudo apt install build-essential
uname -r //check your current kernel version
sudo apt install linux-source-<your current kernel version>

In my case:
sudo apt install linux-source-5.8.6-current-sunxi

cd /usr/src
sudo tar xf linux-source-5.8.6-sunxi.tar.xz

Device tree

Now we have an extracted kernel source sitting in /usr/src. The Device Tree files are located in the arch/arm/boot/dts directory. In that directory we need to edit three files. The first one is to enable the OV5640 camera sensor.

sudo nano arch/arm/boot/dts/sun8i-h3-nanopi-duo2.dts

In that file insert this inside the “model” braces

    cam_xclk: cam-xclk {
                #clock-cells = <0>;
                compatible = "fixed-clock";
                clock-frequency = <24000000>;
                clock-output-names = "cam-xclk";
        };

        reg_cam_avdd: cam-avdd {
                compatible = "regulator-fixed";
                regulator-name = "cam-avdd";
                regulator-min-microvolt = <2800000>;
                regulator-max-microvolt = <2800000>;
                vin-supply = <&reg_vcc3v3>;
        };

        reg_cam_dovdd: cam-dovdd {
                compatible = "regulator-fixed";
                regulator-name = "cam-dovdd";
                regulator-min-microvolt = <1800000>;
                regulator-max-microvolt = <1800000>;
                vin-supply = <&reg_vcc3v3>;
        };

        reg_cam_dvdd: cam-dvdd {
                compatible = "regulator-fixed";
                regulator-name = "cam-dvdd";
                regulator-min-microvolt = <1500000>;
                regulator-max-microvolt = <1500000>;
                vin-supply = <&reg_vcc3v3>;
        };

And this part at the top level

&csi {
    status = "okay";

    port {
        #address-cells = <1>;
        #size-cells = <0>;

        /* Parallel bus endpoint */
        csi_from_ov5640: endpoint {
            remote-endpoint = <&ov5640_to_csi>;
            bus-width = <8>;
            data-shift = <2>;
            hsync-active = <1>; /* Active high */
            vsync-active = <0>; /* Active low */
            data-active = <1>;  /* Active high */
            pclk-sample = <1>;  /* Rising */
        };
    };
};

&i2c2 {
   status = "okay";

   ov5640: camera@3c {
        compatible = "ovti,ov5640";
        reg = <0x3c>;
        clocks = <&cam_xclk>;
        clock-names = "xclk";

        reset-gpios = <&pio 4 14 GPIO_ACTIVE_LOW>;
        powerdown-gpios = <&pio 4 15 GPIO_ACTIVE_HIGH>;
        /* power_en-gpios = <&pio 3 14 GPIO_ACTIVE_LOW>; */
        AVDD-supply = <&reg_cam_avdd>;
        DOVDD-supply = <&reg_cam_dovdd>;
        DVDD-supply = <&reg_cam_dvdd>;

        port {
            ov5640_to_csi: endpoint {
                remote-endpoint = <&csi_from_ov5640>;
                bus-width = <8>;
                data-shift = <2>;
                hsync-active = <1>; /* Active high */
                vsync-active = <0>; /* Active low */
                data-active = <1>;  /* Active high */
                pclk-sample = <1>;  /* Rising */
            };
        };
    };
};

&i2c2_pins {
   bias-pull-up;
};

Compile device tree

Now compile the dts file, copy it and reboot

sudo make dtbs
sudo cp arch/arm/boot/dts/sun8i-h3-nanopi-duo2.dtb /boot/dtb/
sudo reboot

If every went according to plan you should have a working OV5640 sensor. To check out if it works correctly type:

dmesg|grep ov5

You should have something similar to this.

[    9.356102] sun6i-csi 1cb0000.camera: creating ov5640 0-003c:0 -> sun6i-csi:0 link

Cedrus encoder

Next step would be to enable the Cedrus H264 hardware encoder. This is essential to encode a video stream at a decent frame rate while offloading the CPU. Again we need to edit a few Device Tree files and compile the driver.

Device tree

cd /usr/src
sudo nano arch/arm/boot/dts/sun8i-h3.dtsi

At the section above “soc” insert this

        reserved-memory {
                #address-cells = <1>;
                #size-cells = <1>;
                ranges;

                cma_pool: cma@43c00000 {
                        compatible = "shared-dma-pool";
                        reusable;
                        reg = <0x57c00000 0x6000000>;
                        linux,cma-default;
                };
        };

Change the “syscon” section to this

                syscon: system-control@1c00000 {
                        compatible = "allwinner,sun8i-h3-system-control","syscon";
                        reg = <0x01c00000 0x1000>;
                        #address-cells = <1>;
                        #size-cells = <1>;
                        ranges;

                        sram_a2: sram@40000 {
                                compatible = "mmio-sram";
                                reg = <0x00040000 0xc000>;
                                #address-cells = <1>;
                                #size-cells = <1>;
                                ranges = <0 0x00040000 0xc000>;

                                scpi_sram: scp-shmem@bc00 {
                                        compatible = "arm,scp-shmem";
                                        reg = <0xbc00 0x200>;
                                };
                        };

                        sram_c: sram@1d00000 {
                                compatible = "mmio-sram";
                                reg = <0x01d00000 0x80000>;
                                #address-cells = <1>;
                                #size-cells = <1>;
                                ranges = <0 0x01d00000 0x80000>;

                                ve_sram: sram-section@0 {
                                        compatible = "allwinner,sun8i-h3-sram-c1",
                                                     "allwinner,sun4i-a10-sram-c1";
                                        reg = <0x000000 0x80000>;
                                };
                        };
                };

Remove or comment out the “video-codec” section

		/*video-codec@1c0e000 {
			compatible = "allwinner,sun8i-h3-video-engine";
			reg = <0x01c0e000 0x1000>;
			clocks = <&ccu CLK_BUS_VE>, <&ccu CLK_VE>,
				 <&ccu CLK_DRAM_VE>;
			clock-names = "ahb", "mod", "ram";
			resets = <&ccu RST_BUS_VE>;
			interrupts = <GIC_SPI 58 IRQ_TYPE_LEVEL_HIGH>;
			allwinner,sram = <&ve_sram 1>;
		};*/

Above the “crypto” section insert this

                ve: video-engine@01c0e000 {
                        compatible = "allwinner,sunxi-cedar-ve";
                        reg = <0x01c0e000 0x1000>,
                                <0x01c00000 0x10>,
                                <0x01c20000 0x800>;
                        memory-region = <&cma_pool>;
                        syscon = <&syscon>;
                        clocks = <&ccu CLK_BUS_VE>, <&ccu CLK_VE>,
                                <&ccu CLK_DRAM_VE>;
                        clock-names = "ahb", "mod", "ram";
                        resets = <&ccu RST_BUS_VE>;
                        interrupts = <GIC_SPI 58 IRQ_TYPE_LEVEL_HIGH>;
                        allwinner,sram = <&ve_sram 1>;
                };

Next edit the a final Device Tree file

cd /usr/src
sudo nano arch/arm/boot/dts/sun8i-h3-nanopi.dtsi

And insert this at the very end of the file

&ve {
        status = "okay";
};

Compile device tree

Compile the Device Tree once again and copy

sudo make dtbs
sudo cp arch/arm/boot/dts/sun8i-h3-nanopi-duo2.dtb /boot/dtb/

Cedrus driver

Now we need to compile the Cedrus driver. You can do this on the NanoPi Duo2 itself (slow) or cross compile on another machine (fast). I went for the second option so you need to have gnueabihf and the correct kernel source installed.

sudo apt install gcc-7-arm-linux-gnueabihf
wget https://mirrors.dotsrc.org/armbian-apt/pool/main/l/linux-5.8.6-sunxi/linux-image-current-sunxi_20.08.2_armhf.deb
sudo dpkg -i linux-image-current-sunxi_20.08.2_armhf.deb
cd /usr/src
sudo mkdir linux-source-5.8.6-sunxi
sudo tar xf linux-source-5.8.6-sunxi.tar.xz -C linux-source-5.8.6-sunxi

Build the kernel and modules

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- sunxi_defconfig
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs

Get the Cedrus module source

cd /usr/src
git clone https://github.com/uboborov/sunxi-cedar-mainline
cd sunxi-cedar-mainline

Compile driver

And compile with a few extra build flags

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- LOCALVERSION=-sunxi KERNEL_SOURCE=../linux-source-5.8.6-sunxi/ 

Install driver

Now copy the cedar_ve.ko module over to the /lib/modules/5.8.6-sunxi directory of the NanoPi Duo2 and run

sudo depmod

Enable the module at startup

echo cedar_ve | sudo tee -a /etc/modules
sudo reboot

When rebooted the cedar device should be enabled and the module loaded. To check you should see something similar to this

[    7.661866] cedar_ve: loading out-of-tree module taints kernel.
[    7.664733] cedar_ve: module verification failed: signature and/or required key missing - tainting kernel
[    7.666365] sunxi cedar version 0.1 
[    7.666771] [cedar]: install start!!!
[    7.667053] cedar_ve: cedar-ve the get irq is 46
[    7.667087] sunxi-cedar 1c0e000.video-engine: assigned reserved memory node cma@43c00000
[    7.667508] [cedar]: Success claim SRAM
[    7.756248] [cedar]: memory allocated at address PA: 57D00000, VA: D7D00000
[    7.756263] [cedar]: install end!!!

FFmpeg & GStreamer

FFmpeg

So far we’ve enabled both the OV5640 sensor and Cedrus H264 modules. Next we need to build a version of FFmpeg which uses the Cedrus encoder to speed up the frame rate. On the NanoPi Duo2:

libcedrus
sudo apt install libcedrus1-dev

Or building from source

git clone https://github.com/linux-sunxi/libcedrus
cd libcedrus
make && sudo make install
libvdpau-sunxi

Optional – Used for decoding

sudo apt install libvdpau-dev

Or building from source

git clone https://github.com/linux-sunxi/libvdpau-sunxi
cd libvdpau-sunxi
make && sudo make install
sudo ldconfig

Install dependencies

sudo apt install v4l-utils libmp3lame-dev libpulse-dev libv4l-dev
libx264
wget https://download.videolan.org/pub/videolan/x264/snapshots/x264-snapshot-20180121-2245-stable.tar.bz2
tar xf x264-snapshot-20180121-2245-stable.tar.bz2 && cd x264-snapshot-20180121-2245-stable/
./configure --enable-static --enable-shared
make -j4
sudo make install
sudo ldconfig
Clone the FFmpeg repo
git clone --depth=1 https://github.com/stulluk/FFmpeg-Cedrus.git
git clone https://github.com/uboborov/ffmpeg_h264_H3.git
cp ffmpeg_h264_H3/cedrus264.c FFmpeg-Cedrus/libavcodec/cedrus264.c
cp -R ffmpeg_h264_H3/sunxi FFmpeg-Cedrus/libavcodec/arm
Configure and build FFmpeg
cd FFmpeg-Cedrus
./configure --prefix=/usr --enable-nonfree --enable-gpl --enable-version3 --enable-vdpau --enable-libx264 --enable-libmp3lame --enable-libpulse --enable-libv4l2
make -j4
sudo make install
Testing

To test if FFmpeg is working correctly and is using the Cedrus encoder download this file and run

sudo ffmpeg -s 1280x720 -pix_fmt nv12 -f rawvideo -i video.nv12 -pix_fmt nv12 -c:v cedrus264 stream.h264

If that went ok then we can move on to the next step, which is using the OV5640 sensor as an input source. But first we need to set the resolution, frame rate and pixel format, for the OV5640

media-ctl --device /dev/media0 --set-v4l2 '"ov5640 0-003c":0[fmt:UYVY8_2X8/1280x720@1/30]'

To save a stream from the OV5650 sensor run

sudo ffmpeg -f v4l2 -video_size 1280x720 -i /dev/video0 -c:v cedrus264 -pix_fmt nv12 test1.mp4

To push a stream from the OV5640 sensor to Youtube you can use

sudo ffmpeg -f v4l2 -video_size 1280x720 -i /dev/video0 -f lavfi -i anullsrc -c:v cedrus264 -pix_fmt nv12 -b:v 2500k -c:a libmp3lame -f flv rtmp://a.rtmp.youtube.com/live2/<your-youtube-key>

GStreamer

Alternatively you could also use Cedrus with Gstreamer. I’ve contacted the developer of gst-plugin-cedar to request an update to work with Gstreamer 1.14. Which can be downloaded here.

sudo apt install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio  libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev

git clone https://github.com/gtalusan/gst-plugin-cedar.git
cd gst-plugin-cedar
./autogen.sh
make
sudo make install

To push the OV5640 stream to Youtube

sudo gst-launch-1.0 -ve v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,width=1280,height=720,framerate=15/1,bitrate=2500k ! videoconvert ! cedar_h264enc ! h264parse ! mux. audiotestsrc wave=silence freq=200 ! voaacenc ! aacparse ! queue ! mux. flvmux streamable=true name=mux ! rtmpsink location="rtmp://a.rtmp.youtube.com/live2/<your-youtube-key>"

As you (may have) noticed the frame rate for FFmpeg as well as GStreamer is aweful. 720p never exceeds 7fps, even with Cedrus H264 acceleration.

nl_NLDutch