[FFmpeg-devel] [Proposal] drawvg filter

[FFmpeg-devel] [Proposal] drawvg filter

[Ayose via ffmpeg-devel]

This is a proposal to add a new filter to FFmpeg: drawvg, to draw
vector graphics on top of a video, using libcairo.

This is my first contribution to FFmpeg, so I don't know if this is
the right channel to make the proposal.

I was thinking on how to write it, and I thought that a FAQ-like
summary may be the easiest way to describe it. Apologies if this is
more confusing than helpful.


== What is drawvg ==

drawvg is a filter to render vector graphics on top of video frames.

The render is done by executing a script written in its own language,
called VGS (Vector Graphics Script). The script consists of a series
of commands to describe 2D graphics, which are rasterized using the
libcairo library.

VGS is not intended to be used as a general-purpose language. Since
its scope is limited, it prioritizes being concise and easy to use.
The syntax is heavily inspired by languages like [1]Magick Vector
Graphics, or [2]SVG's <path>.

    [1] https://imagemagick.org/script/magick-vector-graphics.php
    [2] https://developer.mozilla.org/en-US/docs/Web/SVG/Reference/Element/path

Some features of the syntax (like using whitespaces to separate
arguments) are also present in languages like TCL or shell scripts.
Many command names are taken from PostScript.

Scripts can use FFmpeg expressions to describe graphics dynamically,
so they can compute coordinates based on frame dimensions, frame
metadata, generate random values, read pixel colors, etc.

For example, to draw a blue circle in the middle of a frame:

    circle (w / 2) (h / 2) 100
    setcolor blue
    fill

Then:

    $ ffplay -i example.webm -vf 'drawvg=file=blue-circle.vgs'

There is a [3]website with some examples on how to integrate drawvg
with other FFmpeg filters, and also a [4]playground to experiment with
the capabilities of the language.

    [3] https://ayosec.github.io/ffmpeg-drawvg/
    [4] https://ayosec.github.io/ffmpeg-drawvg/playground/


== Why is it useful for FFmpeg ==

FFmpeg has some filters to modify video frames, like drawbox, xfade,
geq, drawtext, drawgrid, etc.

I guess that most users will only need filters like scale or crop, but
for some other users, the ability to draw vector graphics makes
possible to perform more complex operations, like adding annotations,
or creating masks for custom transitions.

The implementation is much smaller than I was expecting. The current
version of [5]vf_drawvg.c, which contains the whole implementation of
drawvg (the parser and the interpreter for VGS, and the class
definition for the filter) has 2590 lines of code. For comparison,
vf_drawtext.c has 1917 LOC.

    [5] https://github.com/ayosec/ffmpeg-drawvg/blob/drawvg-filter/libavfilter/vf_drawvg.c


== Why not SVG ==

FFmpeg can load SVG images with librsvg. While SVG is much more
powerful than drawvg, the images are static, and can't rely on any
information from FFmpeg itself.

On the other hand, a drawvg script can use FFmpeg expressions, to
compute colors and coordinates by using data available in the frame
(like dimensions, timestamp, colors from pixels of the video, frame
metadata, etc).


== Why libcairo ==

There are two reasons to choose libcairo:

First, it is a well-tested, stable, and portable library.

Second, librsvg depends on libcairo, so any installation of FFmpeg
that supports SVG is already using libcairo. I expect that, since
there are no new dependencies, distro maintainers should not need to
make changes on how their FFmpeg package is built in order to support
the new filter.


== Why a custom language ==

Before starting to work on the implementation I was considering to
reuse an existing interpreter, like Lua or a lightweight JavaScript
implementation (QuickJS, Duktape, MuJS, etc.). But this option was
discarded for three reasons:

First, it adds more external dependencies to FFmpeg.

Second, a filter like this does not need the complexity of a general
purpose language. Even when both Lua and JavaScript are quite simple,
we can have a more minimalistic language to describe what must be
drawn by the filter. And, if someone really needs to create a more
complex script, they may prefer something different to Lua/JS anyways.

Third, by using a custom language, we can have full integration with
the existing language for FFmpeg expressions. Thus, users with some
experience writing complex filtergraphs can use a familiar syntax.

The current version of the language is [6]fully documented.

    [6] https://ayosec.github.io/ffmpeg-drawvg/playground/docs/langref.html


== What is done ==

The filter is [7]already implemented (except for yet-to-be-found bugs).

    [7] https://github.com/ayosec/ffmpeg-drawvg/compare/8426622bb..drawvg-filter

I'm compiling FFmpeg with "--toolchain=clang-asan", so I hope that the
implementation is (mostly) correct, although I'm not a C expert.

The patch already has two FATE tests: one to verify the calls to
libcairo, and another one to compare the output (CMD = video_filter).

There is a [8]repository in GitHub with the changes. The commit
messages are mostly noise (I assume that all commits must be squashed,
so I didn't spent time writing proper messages), but I left the
original commit history, in case it is helpful for reviewing the code.

    [8] https://github.com/ayosec/ffmpeg-drawvg


== What is missing ==

The most important task that is still missing is to port the
documentation to Texinfo. I used Markdown because it is what I'm
most used to. If the filter is accepted, I guess than pandoc can
do most of the required changes, so it should be easy.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".