OpenCv and Gstreamer on the Raspberry Pi 5
Fourth Working Example-Gstreamer
Introduction
In the process of upgrading my Raspberry Pi’s and their cameras I commissioned an 8GB Raspberry Pi 5 with the Arducam 16MP IMX519 Camera Module with 122°(D) M12 Lens, Wide Angle Color Rolling shutter. Then, as I have done in the past, I installed Gstreamer and built the latest version of OpenCv (4.10.0) on it.
A lot has changed with respect to camera drivers on the RPi 5 relative to legacy RPi’s. This created some challenges that needed to be solved. In that effort I found that searching the web was sometimes not the help it has been in the past when I developed vision applications using principally the RPi 3 Model B+.
Therefore, the purpose of this note is to share what I did to achieve success and provide some working code as examples using gstreamer, opencv in Python and Cplusplus.
Some Preliminaries
When I first set up the RPi 5 with its Bookworm operating system a couple of items needed attention.
1. U. S. keyboard. The default keyboard is UK. The only way to change this to U. S. is to make the indicated change at the end of the file ~/.config/wayfire.ini and then reboot :
[input]
xkb_options=
xkb_model=pc105
xkb_layout=us (formerly uk)
xkb_variant=
2. The particular camera used must be specified at boot or the camera software will not find it. This is done in the file /boot/firmware/config.txt. At the end of this file add camera as shown and then reboot.
[all]
dtoverlay=imx519
3. Update: It was found that the OpenCv function moveWindow will not work since the default backend is Wayland, not x11. One changes it to x11 using sudo raspi-config and finding it under Advance Options.
Test Camera and Software
Before proceeding one should test the camera software and the camera.
1. This command will verify that the camera is present.
libcamera-hello --list-cameras
Available cameras
-----------------
0 : imx519 [4656x3496 10-bit RGGB] (/base/axi/pcie@120000/rp1/i2c@80000/imx519@1a)
Modes: 'SRGGB10_CSI2P' : 1280x720 [80.01 fps - (65535, 65535)/65535x65535 crop]
1920x1080 [60.05 fps - (65535, 65535)/65535x65535 crop]
2328x1748 [30.00 fps - (65535, 65535)/65535x65535 crop]
3840x2160 [18.00 fps - (65535, 65535)/65535x65535 crop]
4656x3496 [9.00 fps - (65535, 65535)/65535x65535 crop]
2. Further testing and familiarization can be done using the various commands:
rpicam-vid
rpicam-still
and equivalents,
libcamera-vid
libcamera-still
3. For each, one can use the switch -- help
Download And Build OpenCv
The following reference was followed. I chose to build opencv and that way I ended up with the latest version 4.10.0.
https://qengineering.eu/install%20opencv%20on%20raspberry%20pi%205.html
python3
Python 3.11.2 (main, May 2 2024, 11:59:08) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print(cv2.__version__)
4.10.0-dev
>>>
Here are the salient items regarding my build following the above reference.
1. My Rpi 5 has 8 GB RAM, therefore I did not need to increased the SWAPFILE.
2. I installed the third party software using the script.
$ wget https://github.com/Qengineering/Install-OpenCV-Raspberry-Pi-64-bits/raw/main/OpenCV-4-10-0.sh
$ sudo chmod 755 ./OpenCV-4-10-0.sh
$ ./OpenCV-4-10-0.sh
3. I then downloaded the opencv files.
$ cd ~
$ git clone --depth=1 https://github.com/opencv/opencv.git
$ git clone --depth=1 https://github.com/opencv/opencv_contrib.git
4. Build directory created.
$ cd ~/opencv
$ mkdir build
$ cd build
5. Here is the cmake command that I used.
$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-D ENABLE_NEON=ON \
-D WITH_OPENMP=ON \
-D WITH_OPENCL=OFF \
-D BUILD_TIFF=ON \
-D WITH_FFMPEG=ON \
-D WITH_TBB=ON \
-D BUILD_TBB=ON \
-D WITH_GSTREAMER=ON \
-D BUILD_TESTS=OFF \
-D WITH_EIGEN=OFF \
-D WITH_V4L=ON \
-D WITH_LIBV4L=ON \
-D WITH_VTK=OFF \
-D WITH_QT=OFF \
-D WITH_PROTOBUF=ON \
-D OPENCV_ENABLE_NONFREE=ON \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D PYTHON3_PACKAGES_PATH=/usr/lib/python3/dist-packages \
-D OPENCV_GENERATE_PKGCONFIG=ON \
-D BUILD_EXAMPLES=ON ..
6.$ make -j4
7. Final step
$ sudo make install
$ sudo ldconfig
# cleaning (frees 300 KB)
$ make clean
$ sudo apt-get update
Finally, one checks the build information as follows.
python3
Python 3.11.2 (main, May 2 2024, 11:59:08) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print (cv2.getBuildInformation())
General configuration for OpenCV 4.10.0-dev =====================================
Version control: b2e118e
Extra modules:
Location (extra): /home/fdoljack/opencv_contrib/modules
Version control (extra): 80f1ca2
Platform:
Timestamp: 2024-09-21T14:00:19Z
Host: Linux 6.6.31+rpt-rpi-2712 aarch64
CMake: 3.25.1
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/gmake
Configuration: RELEASE
Algorithm Hint: ALGO_HINT_ACCURATE
CPU/HW features:
Baseline: NEON FP16
requested: DETECT
required: NEON
Dispatched code generation: NEON_DOTPROD NEON_FP16 NEON_BF16
requested: NEON_FP16 NEON_BF16 NEON_DOTPROD
NEON_DOTPROD (1 files): + NEON_DOTPROD
NEON_FP16 (2 files): + NEON_FP16
NEON_BF16 (0 files): + NEON_BF16
C/C++:
Built as dynamic libs?: YES
C++ standard: 11
C++ Compiler: /usr/bin/c++ (ver 12.2.0)
C++ flags (Release): -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/bin/cc
C flags (Release): -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fopenmp -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fopenmp -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release): -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined
Linker flags (Debug): -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined
ccache: NO
Precompiled headers: NO
Extra dependencies: dl m pthread rt
3rdparty dependencies:
OpenCV modules:
To be built: aruco bgsegm bioinspired calib3d ccalib core datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency shape signal stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
Disabled: world
Disabled by dependency: -
Unavailable: alphamat cannops cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv java julia matlab ovis python2 sfm viz
Applications: perf_tests examples apps
Documentation: NO
Non-free algorithms: YES
GUI: GTK3
GTK+: YES (ver 3.24.38)
Media I/O:
ZLib: /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.13)
JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 62)
WEBP: /usr/lib/aarch64-linux-gnu/libwebp.so (ver encoder: 0x020f)
PNG: /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.39)
TIFF: build (ver 42 - 4.6.0)
JPEG 2000: build (ver 2.5.0)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES
Video I/O:
FFMPEG: YES
avcodec: YES (59.37.100)
avformat: YES (59.27.100)
avutil: YES (57.28.100)
swscale: YES (6.7.100)
avresample: NO
GStreamer: YES (1.22.0)
v4l/v4l2: YES (linux/videodev2.h)
Parallel framework: TBB (ver 2021.11 interface 12110)
Trace: YES (with Intel ITT)
Other third-party libraries:
Lapack: NO
Custom HAL: YES (carotene (ver 0.0.1))
Protobuf: build (3.19.1)
Flatbuffers: builtin/3rdparty (23.5.9)
Python 3:
Interpreter: /usr/bin/python3 (ver 3.11.2)
Libraries: /usr/lib/aarch64-linux-gnu/libpython3.11.so (ver 3.11.2)
Limited API: NO
numpy: /usr/lib/python3/dist-packages/numpy/core/include (ver 1.24.2)
install path: /usr/lib/python3/dist-packages/cv2/python-3.11
Python (for build): /usr/bin/python3
Java:
ant: /bin/ant (ver 1.10.13)
Java: NO
JNI: NO
Java wrappers: NO
Java tests: NO
Install to: /usr/local
-----------------------------------------------------------------
Building the Samples
The first thing I did to test the build and use the camera was to build the cpp samples found in the directory ~/opencv/samples/cpp.
In the samples directory:
mkdir build
cmake –S samples –B samples/build
cd build
make –j4
When finished do not run make clean. One will want to make changes to a particular source file and then run make again. This will avoid rebuilding all of the already built samples.
Executables are found in samples/build/cpp.
Source files remain in samples/cpp.
Running A Test
The sample with source file videocapture_basic.cpp was executed and an immediate problem was revealed. In the sample is the code(now commented out):
// open the default camera using default API
// cap.open(0);
// OR advance usage: select any API backend
// int deviceID = 19; // 0 = open default camera
// int apiID = cv::CAP_ANY; // 0 = autodetect default API
// open selected camera using selected API
The result was that camera could not be found and is not default 0. I ran:
sudo v4lt-ctl - - list-devices
pispbe (platform:1000880000.pisp_be):
/dev/video20
/dev/video21
/dev/video22
/dev/video23
/dev/video24
/dev/video25
/dev/video26
/dev/video27
/dev/video28
/dev/video29
/dev/video30
/dev/video31
/dev/video32
/dev/video33
/dev/video34
/dev/video35
/dev/video36
/dev/video37
/dev/media2
/dev/media3
rp1-cfe (platform:1f00128000.csi):
/dev/video0
/dev/video1
/dev/video2
/dev/video3
/dev/video4
/dev/video5
/dev/video6
/dev/video7
/dev/media0
rpivid (platform:rpivid):
/dev/video19
/dev/media1
And it is not deviceiD=19 nor does VideoCapture cap(“/dev/video19”); work. The solution was found to be the use of gstreamer to specify the camera and construct a pipeline. One may consult the following reference to learn how to do this. https://forums.raspberrypi.com/viewtopic.php?t=361778
While opencv was built with gstreamer support, gstreamer alone is not installed in Bookworm on the RPi 5. It must separately be installed. And to proceed to make the working code to function, one needs to add two missing gstreamer packages.
Install gstreamer:
apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
Install missing packages:
apt-get install libx264-dev libjpeg-dev
apt-get install gstreamer1.0-libcamera
First Working Example-C++
Here is the code:
/**
@file videocapture_basic.cpp
@brief A very basic sample for using VideoCapture and VideoWriter
@author PkLab.net
@date Aug 24, 2016
*/
#include <opencv2/core.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
#include <iostream>
#include <stdio.h>
using namespace cv;
using namespace std;
int main(int, char**)
{
Mat frame;
const char* gst = "libcamerasrc camera-name=/base/axi/pcie@120000/rp1/i2c@80000/imx519@1a ! video/x-raw,width=1280,height=720,framerate=10/1,format=RGBx ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=BGR ! appsink";
//--- INITIALIZE VIDEOCAPTURE
VideoCapture cap(gst);
// check if we succeeded
if (!cap.isOpened()) {
cerr << "ERROR! Unable to open camera\n";
return -1;
}
//--- GRAB AND WRITE LOOP
cout << "Start grabbing" << endl
<< "Press any key to terminate" << endl;
for (;;)
{
// wait for a new frame from camera and store it into 'frame'
cap.read(frame);
// check if we succeeded
if (frame.empty()) {
cerr << "ERROR! blank frame grabbed\n";
break;
}
// show live and wait for a key with timeout long enough to show images
imshow("Live", frame);
if (waitKey(5) >= 0)
break;
}
// the camera will be deinitialized automatically in VideoCapture destructor
return 0;
}
Where does camera-name come from? Run the following:
rpicam-vid --list-cameras
Available cameras
-----------------
0 : imx519 [4656x3496 10-bit RGGB] (/base/axi/pcie@120000/rp1/i2c@80000/imx519@1a)
Modes: 'SRGGB10_CSI2P' : 1280x720 [80.01 fps - (65535, 65535)/65535x65535 crop]
1920x1080 [60.05 fps - (65535, 65535)/65535x65535 crop]
2328x1748 [30.00 fps - (65535, 65535)/65535x65535 crop]
3840x2160 [18.00 fps - (65535, 65535)/65535x65535 crop]
4656x3496 [9.00 fps - (65535, 65535)/65535x65535 crop]
One can also run an equivalent command.
libcamera-hello --list-cameras
The camera name is /base/axi/pcie@120000/rp1/i2c@80000/imx519@1a
Second Working Example-C++
The above first example was a sample provided by the building of opencv and was specifically compiled and linked within the cmake system and directories provided. However, more generally one wants to build source code in other directories anywhere outside of ~/opencv.
In the directory ~/Videos I put this C++ source file named testcv.cpp:
#include <iostream>
//#include "opencv2/opencv.hpp"
#include <opencv2/core.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
using namespace std;
using namespace cv;
//char gst[] = "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink";
//const char* gst = "libcamerasrc camera-name=/base/axi/pcie@120000/rp1/i2c@80000/imx519@1a ! video/x-raw,width=1280,height=720,framerate=10/1,format=NV12 ! \
//queue ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=NV12 ! appsink";
int main(int, char**)
{
int x=500;
int y=500;
const char* gst = "libcamerasrc camera-name=/base/axi/pcie@120000/rp1/i2c@80000/imx519@1a ! video/x-raw,width=1280,height=720,framerate=10/1,format=NV12 ! \
queue ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=BGR ! appsink";
cout << "OpenCV version : " << CV_VERSION << endl;
cout << "Major version : " << CV_MAJOR_VERSION << endl;
cout << "Minor version : " << CV_MINOR_VERSION << endl;
cout << "Subminor version : " << CV_SUBMINOR_VERSION << endl;
VideoCapture cap(gst);
cap.open(gst, CAP_ANY);
if(!cap.isOpened()){
cout<< "fail to open camera"<<endl;
return -1;
}
cout << "camera is open" <<endl;
namedWindow("Display window", 1);
moveWindow("Display window", x, y);
Mat frame;
while(1)
{
cap.read(frame);
// cout << "read frame" << endl;
// Display frame
imshow("Display window", frame);
cv::waitKey(10); //needed to show frame
}
cap.release();
return 0;
}
The pkg-config command was used to help form the compile and link command since a .pc file was generated for opencv4 when it was built.
pkg-config --cflags --libs opencv4
-I/usr/local/include/opencv4 -L/usr/local/lib -lopencv_gapi -lopencv_stitching -lopencv_aruco -lopencv_bgsegm -lopencv_bioinspired -lopencv_ccalib -lopencv_dnn_objdetect -lopencv_dnn_superres -lopencv_dpm -lopencv_face -lopencv_freetype -lopencv_fuzzy -lopencv_hdf -lopencv_hfs -lopencv_img_hash -lopencv_intensity_transform -lopencv_line_descriptor -lopencv_mcc -lopencv_quality -lopencv_rapid -lopencv_reg -lopencv_rgbd -lopencv_saliency -lopencv_signal -lopencv_stereo -lopencv_structured_light -lopencv_phase_unwrapping -lopencv_superres -lopencv_optflow -lopencv_surface_matching -lopencv_tracking -lopencv_highgui -lopencv_datasets -lopencv_text -lopencv_plot -lopencv_videostab -lopencv_videoio -lopencv_wechat_qrcode -lopencv_xfeatures2d -lopencv_shape -lopencv_ml -lopencv_ximgproc -lopencv_video -lopencv_xobjdetect -lopencv_objdetect -lopencv_calib3d -lopencv_imgcodecs -lopencv_features2d -lopencv_dnn -lopencv_flann -lopencv_xphoto -lopencv_photo -lopencv_imgproc -lopencv_core
This output was then copied/pasted to form this command:
g++ -ggdb -o test testcv.cpp <pkg-config output>
g++ -ggdb -o test testcv.cpp -I/usr/local/include/opencv4 -L/usr/local/lib -lopencv_gapi -lopencv_stitching -lopencv_aruco -lopencv_bgsegm -lopencv_bioinspired -lopencv_ccalib -lopencv_dnn_objdetect -lopencv_dnn_superres -lopencv_dpm -lopencv_face -lopencv_freetype -lopencv_fuzzy -lopencv_hdf -lopencv_hfs -lopencv_img_hash -lopencv_intensity_transform -lopencv_line_descriptor -lopencv_mcc -lopencv_quality -lopencv_rapid -lopencv_reg -lopencv_rgbd -lopencv_saliency -lopencv_signal -lopencv_stereo -lopencv_structured_light -lopencv_phase_unwrapping -lopencv_superres -lopencv_optflow -lopencv_surface_matching -lopencv_tracking -lopencv_highgui -lopencv_datasets -lopencv_text -lopencv_plot -lopencv_videostab -lopencv_videoio -lopencv_wechat_qrcode -lopencv_xfeatures2d -lopencv_shape -lopencv_ml -lopencv_ximgproc -lopencv_video -lopencv_xobjdetect -lopencv_objdetect -lopencv_calib3d -lopencv_imgcodecs -lopencv_features2d -lopencv_dnn -lopencv_flann -lopencv_xphoto -lopencv_photo -lopencv_imgproc -lopencv_core
Note that this source code contains some minor revisions that also work. One can play around a bit with the pipeline to see what works and what does not.
Third Working Example-Python3
The use of a gstreamer pipeline is easily extended to python3. The one important change that was found was the need to insert one or more queues in the pipeline. Otherwise python3 complained that it needed buffers.
import cv2
camera = "/base/axi/pcie@120000/rp1/i2c@80000/imx519@1a"
pipeline = "libcamerasrc camera-name=%s ! video/x-raw,width=3840,height=2160,framerate=9/1,format=NV12 ! queue ! videoconvert ! videoscale ! queue ! video/x-raw,width=1280,height=720,format=RGBx ! appsink" % (camera)
cap = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)
#cap = cv2.VideoCapture(0)
if not cap.isOpened():
print("open camera failed")
exit()
while True:
ret, frame = cap.read()
if ret:
cv2.imshow('frame', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
else:
print("read failed")
break
cap.release()
cv2.destroyAllWindows()
Fourth Working Example-Gstreamer
Gstreamer alone can be used to manipulate video from the camera. The following two pipelines were found to work.
gst-launch-1.0 libcamerasrc ! video/x-raw,width=1920,height=1080,framerate=10/1,format=NV12 ! queue ! videoconvert ! videoscale ! queue ! video/x-raw,width=1920,height=1080,format=NV12 ! autovideosink
gst-launch-1.0 libcamerasrc ! video/x-raw,width=1920,height=1080,framerate=10/1,format=RGBx ! queue ! videoconvert ! videoscale ! queue ! video/x-raw,width=1920,height=1080,format=NV12 ! autovideosink
The queues importantly provide buffering and it seemed that sending to autovideosink only the NV12 format worked. Take note that when inside code the sink must be appsink while to display video outside code the sink must be autovideosink.
Final Note
My effort is documented here to hopefully help others saving time in deploying vision applications using the RPi 5 and cameras such as Arducam cameras.
