I was wondering how to manipulate my video picture for online conferences, e.g.
- show a static picture or
- a prerecorded video
- share the screen or
- a window
- or even run filters on top of the real camera.
Introduction
In the current Corona situation, where I’m working most of the time from home and meet people more virtualy than in person, I saw various artificial backgrounds mostly in Zoom. This should also be possible with available opensource software and recently there was an article on heise.de (PayWall) “Schöner Hintergrund mit OBS Studio". Nevertheless, I do not want to simply replace the background, where I would need a green or blue carpet for. I want to find out, how to replace and manipulate the video. The virtual camera v4l2loopback and ffmpeg were the right tools for it.
Prerequisites
Since I work with Debian, I use apt for package installation. If you use other distributions use
their package management tools. First the required software must be installed as ‘root’ (for Ubuntu users,
that’s what you do with sudo; tip: use sudo -i to permanantly become the root user):
apt install -y v4l2loopback-utils ffmpeg wmctrl
After the installation, the kernel module needs to be loaded.
modprobe v4l2loopback video_nr="42" card_label="Virtual Camera: v4l2loopback"
You could also omit the options. However, then you must find out which video device is the correct one (which
can be done with v4l2-ctl --list-devices). If you want the module to be loaded on each boot, insert it to
/etc/modules.
If an error “Required key not available” occurs, you have to switch off secure boot in the BIOS and reboot (see https://github.com/jlam55555/veikk-linux-driver/issues/3, which is related) or you need to sign the module.
Make use on the virtual camera
Open two terminal windows, one to run the ffmpeg commands shown below and the other one to open and
display the virtual video with ffplay /dev/video42.
Simple redirect
One can simply redirect the real webcam to the virtual camera with:
ffmpeg -i /dev/video0 -f v4l2 /dev/video42
However, that is boring.
Dispaly a picture
Instead one could show a static picture with
ffmpeg -loop 1 -re -i picture.jpg -vcodec rawvideo -pix_fmt yuv420p -f v4l2 /dev/video42
Single video file
… or a prerecorded video, like “Charly Chaplin”.
ffmpeg -re -i video.mp4 -pix_fmt yuv420p -map 0:v -f v4l2 /dev/video42
endless video loop
If you pick out one short scene, this can also be repeated endless. The simple -loop -1 option does not work
here, since with the first repetition the video is not displayed in realtime, but as fast as possible. Therefore
the -filter_complex is used here.
ffmpeg -re -i video.mp4 -filter_complex loop=loop=-1:0:0,realtime -pix_fmt yuv420p -map 0:v -f v4l2 /dev/video42
Screencast
Sharing the whole screen is also possible with
ffmpeg -f x11grab -r 15 -s 1280x720 -i ${DISPLAY}+0,0 -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video42
However, that works only on Xorg, not with wayland, which is the default for e.g. Gnome.
Share a window
To share a single window instead of the full screen, I found only a solution by using gstreamer:
gst-launch-1.0 ximagesrc xname="<unique name>" ! video/x-raw,framerate=5/1 ! videoconvert ! videoscale ! "video/x-raw,format=YUY2,width=1280,height=800" ! v4l2sink device=/dev/video42
If you have more windows with the same name (“title”), you can use the X-window ID instead of the name, by using
the output of wmctrl -l.
gst-launch-1.0 ximagesrc xid=0x01a00003 ! video/x-raw,framerate=5/1 ! videoconvert ! videoscale ! "video/x-raw,format=YUY2,width=1280,height=720" ! v4l2sink device=/dev/video42
Filter real camera
Now let’s come to the more interesting part, manipulating the real webcam picture.
Edge detection
Adjust the values for ‘low’ and ‘high’ to your needs:
ffmpeg -r 15 -s 1280x720 -i /dev/video0 -vf edgedetect=low=0.02:high=0.1 -f v4l2 /dev/video42
replace a color by transparency
You need to adjust the hex-value (here a red tone) of the color to replace:
ffmpeg -i /dev/video0 -vf colorkey=0xBD3B1E:0.3:0.0 -f v4l2 /dev/video42
overlay a picture with your video
This is like in the zoom conferences with the artificial backgrounds where people want to hide their real
background. Choose a picture of your choice, if you have a blue or green background, adjust the hex-value of the
color to replace and make sure the picture used has the same size like your camera (in my case 1280x720 pixel).
If unsure use v4l2-ctl --list-formats-ext to see the available resolutions.
Here the complex filter is required again. The filter starts with an input and ends with an output name in square brackets, sepparated by semicolon. then the filter comes with it’s parameters sepparated by colon.
ffmpeg -i picture.jpg -i /dev/video0 -filter_complex "[1:v]colorkey=0xBD3B1E:0.3:0.0[ckout];[0:v][ckout]overlay[out]" -pix_fmt yuv420p -map "[out]" -f v4l2 /dev/video42
overlay a picture with edgedetect
Since the background of the detected edged is black, I use 0x000000
ffmpeg -i picture.jpg -i /dev/video0 -filter_complex "[1:v]edgedetect=low=0.1:high=0.2[edout];[edout]colorkey=0xBB3B1E:0.3:0.0[ckout];[0:v][ckout]overlay[out]" -pix_fmt yuv420p -map "[out]" -f v4l2 /dev/video42
… and become a ghost in the forest in your next conference (I do not have a foto of a castle available):
There are plenty of video filters on the FFMPEG page. The above shown were in my optinion the nicest ones.