Pimp up your video in conferences

How to manipulate your video picture

 Posted on Dec 19, 2020  |  4 minutes  |  828 words  |  Jörg Steinkamp

I was wondering how to manipulate my video picture for online conferences, e.g.

  • show a static picture or
  • a prerecorded video
  • share the screen or
  • a window
  • or even run filters on top of the real camera.

Introduction

In the current Corona situation, where I’m working most of the time from home and meet people more virtualy than in person, I saw various artificial backgrounds mostly in Zoom. This should also be possible with available opensource software and recently there was an article on heise.de (PayWall) “Schöner Hintergrund mit OBS Studio“. Nevertheless, I do not want to simply replace the background, where I would need a green or blue carpet for. I want to find out, how to replace and manipulate the video. The virtual camera v4l2loopback and ffmpeg were the right tools for it.

Prerequisites

Since I work with Debian, I use apt for package installation. If you use other distributions use their package management tools. First the required software must be installed as ‘root’ (for Ubuntu users, that’s what you do with sudo; tip: use sudo -i to permanantly become the root user):

apt install -y v4l2loopback-utils ffmpeg wmctrl

After the installation, the kernel module needs to be loaded.

modprobe v4l2loopback video_nr="42" card_label="Virtual Camera: v4l2loopback"

You could also omit the options. However, then you must find out which video device is the correct one (which can be done with v4l2-ctl --list-devices). If you want the module to be loaded on each boot, insert it to /etc/modules.

If an error “Required key not available” occurs, you have to switch off secure boot in the BIOS and reboot (see https://github.com/jlam55555/veikk-linux-driver/issues/3, which is related) or you need to sign the module.

Make use on the virtual camera

Open two terminal windows, one to run the ffmpeg commands shown below and the other one to open and display the virtual video with ffplay /dev/video42.

Simple redirect

One can simply redirect the real webcam to the virtual camera with:

ffmpeg -i /dev/video0 -f v4l2 /dev/video42

However, that is boring.

Dispaly a picture

Instead one could show a static picture with

ffmpeg -loop 1 -re -i picture.jpg -vcodec rawvideo -pix_fmt yuv420p -f v4l2 /dev/video42

Single video file

… or a prerecorded video, like “Charly Chaplin”.

ffmpeg -re -i video.mp4 -pix_fmt yuv420p -map 0:v -f v4l2 /dev/video42

endless video loop

If you pick out one short scene, this can also be repeated endless. The simple -loop -1 option does not work here, since with the first repetition the video is not displayed in realtime, but as fast as possible. Therefore the -filter_complex is used here.

ffmpeg -re -i video.mp4 -filter_complex loop=loop=-1:0:0,realtime -pix_fmt yuv420p -map 0:v -f v4l2 /dev/video42

Screencast

Sharing the whole screen is also possible with

ffmpeg -f x11grab -r 15 -s 1280x720 -i ${DISPLAY}+0,0 -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video42

However, that works only on Xorg, not with wayland, which is the default for e.g. Gnome.

Share a window

To share a single window instead of the full screen, I found only a solution by using gstreamer:

gst-launch-1.0 ximagesrc xname="<unique name>" ! video/x-raw,framerate=5/1 ! videoconvert ! videoscale ! "video/x-raw,format=YUY2,width=1280,height=800" ! v4l2sink device=/dev/video42

If you have more windows with the same name (“title”), you can use the X-window ID instead of the name, by using the output of wmctrl -l.

gst-launch-1.0 ximagesrc xid=0x01a00003 ! video/x-raw,framerate=5/1 ! videoconvert ! videoscale ! "video/x-raw,format=YUY2,width=1280,height=720" ! v4l2sink device=/dev/video42

Filter real camera

Now let’s come to the more interesting part, manipulating the real webcam picture.

Edge detection

Adjust the values for ‘low’ and ‘high’ to your needs:

ffmpeg -r 15 -s 1280x720 -i /dev/video0 -vf edgedetect=low=0.02:high=0.1 -f v4l2 /dev/video42

replace a color by transparency

You need to adjust the hex-value (here a red tone) of the color to replace:

ffmpeg -i /dev/video0 -vf colorkey=0xBD3B1E:0.3:0.0 -f v4l2 /dev/video42

overlay a picture with your video

This is like in the zoom conferences with the artificial backgrounds where people want to hide their real background. Choose a picture of your choice, if you have a blue or green background, adjust the hex-value of the color to replace and make sure the picture used has the same size like your camera (in my case 1280x720 pixel). If unsure use v4l2-ctl --list-formats-ext to see the available resolutions.

Here the complex filter is required again. The filter starts with an input and ends with an output name in square brackets, sepparated by semicolon. then the filter comes with it’s parameters sepparated by colon.

ffmpeg -i picture.jpg -i /dev/video0 -filter_complex "[1:v]colorkey=0xBD3B1E:0.3:0.0[ckout];[0:v][ckout]overlay[out]" -pix_fmt yuv420p -map "[out]" -f v4l2 /dev/video42

overlay a picture with edgedetect

Since the background of the detected edged is black, I use 0x000000

ffmpeg -i picture.jpg -i /dev/video0 -filter_complex "[1:v]edgedetect=low=0.1:high=0.2[edout];[edout]colorkey=0xBB3B1E:0.3:0.0[ckout];[0:v][ckout]overlay[out]" -pix_fmt yuv420p -map "[out]" -f v4l2 /dev/video42

… and become a ghost in the forest in your next conference (I do not have a foto of a castle available):

There are plenty of video filters on the FFMPEG page. The above shown were in my optinion the nicest ones.

Linux  Webcam 

See also