Re: [FFmpeg-devel] [PATCH] avfilter: Proof of Concept: enable out-of-tree filters

[Leandro Santiago <leandrosansilva@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6921 bytes --]

I've applied a few changes from your feedback, the following is how the feature looks like.

***Disclaimer***

this is still a proof of concept and for now I am looking for your feedback and help. There are lots of rough corners and stuff that does not work. I by no means intend to request the changes as they are to be merged. Call it an RFC if you will.

Also, I am not sending the patch as an attachment, and leave the e-mail body for discussion only. I am not sure whether this is a good approach or goes against the mailing list policy. This is the time I wish we could really contribute using the Pull Request more on a nice web interface.

If you folks want to experiment, I pushed my changes in this PR: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/13

***Finish disclaimer***

I changed the name convention from "extra filters" to "external filters", as it sounds more natural. (Another possibility would be "extension filters", so please let me know what you think).

The build system will automatically pick up any filters in the subdirectory `ext/libavfilter`, and ignore it if it does not exist.

The rationale behind this directory structure is to reserve the possibility of extensions of other libraries as well.

An external filter is composed by at least one Makefile, and this Makefile has some "metadata" encoded in its comments, used by the `./configure` script to generate some code. I'll show how they look like in an example.

Although this way to encode metadata looks very ugly and I am pretty sure can be improved, this is the best I could for now, and I hope you folks can point me directions on how to improve it, as you have more knowledge of the build system than I do.

Here are the metadata types:

- check: this is a command used to check if 3rd party dependencies are available and enabling them within the build system. The command here is executed by `./configure`. This entry is optional and for now there can be only one of them, due to technical limitations of my implementation.

- symbol: this entry can appear multiple times, and indicates the name of the FFFilter objects exported by the filter. There must be one entry per exported symbol. These entries are used by `./configure` to generate the list of enabled filters.

- ldflags: those are extra LDFLAGS to be used to link libavfilter. I am not happy this this one at all, as such information ideally should be encoded as Makefile variables, but I confess I could not manage to do it. I'd be happy to get directions on where in the build system to look for the right "hook", but I'll continue looking for it as well.

As an example, here is how one can create an empty filter. Just to exercise the build system, the example filter depends on an external library, json-c, which is found using pkg-config and will be dynamically linked to libavfilter. In case json-c is not found, `./configure` will fail, as expected.

```

mkdir -p ext/libavfilter/example

```

Then create the a file ext/libavfilter/example/Makefile with the content:

```

# check: require_pkg_config json json-c json-c/json.h json_c_version_num
# symbol: ff_vf_example
# ldflags: -ljson-c

OBJS += vf_example.o
CFLAGS += -DFOO=1234

libavfilter/vf_example.o:
        $(CC) $(EXTERNAL_FILTER_FLAGS) $(CPPFLAGS) $(CFLAGS) -c -o $@ $(EXTERNAL_FILTER_EXAMPLE_LOCATION)/vf_example.c

```

Then, create the file ext/libavfilter/example/Makefile/vf_example.c with the content:


```

#include "libavutil/internal.h"
#include "avfilter.h"
#include "filters.h"
#include "video.h"

#include <json.h>

static int process_command(AVFilterContext *ctx, const char *cmd, const char *args, char *res, int res_len, int flags) {
  // NOTE: I know this makes no sense, I just wanted to use some symbol from json-c
  // in a way the compiler cannot optimize it out
  return json_c_version_num();
}


const FFFilter ff_vf_example = {
    .p.name        = "example",
    .p.description = NULL_IF_CONFIG_SMALL("An external example filter"),
    .p.flags       = AVFILTER_FLAG_METADATA_ONLY,
    FILTER_INPUTS(ff_video_default_filterpad),
    FILTER_OUTPUTS(ff_video_default_filterpad),
    .process_command = process_command,
};

```

(You can skip this manually creating those files by checking out this example filter from gitlab: https://gitlab.com/leandrosansilva/ffmpeg-extra-filter-example)

Now you can compile ffmpeg as usual, with `./configure && make && make install` and use the filter.

In my tests, building ffmpeg libraries either in static and shared mode work, with the extra dependencies properly linked.


Explanation:

- The .o files for the external filters will be generated in the same place as the built-in filters, in libavfilter/. That means that two external filters cannot generate the same .o files.

- A new variable with the format EXTERNAL_FILTER_$(FILTER_SYMBOL_IN_UPPERCASE)_LOCATION will be generated for each FFFilter exported. More explanation about that below.

- The build system offers now a variable `EXTERNAL_FILTER_FLAGS` which adds the paths of the libavfilter private headers to the compiler when building the external filters. The idea here is for code for an external filter to look exactly as if it was a built-in filter: the include paths should be the same. This will make it easier for external filters to be moved as built-in and the other way around, not requiring much code changes.


TODO:

- Find a way to remove the `ldflags` metadata entry to turn it into a Makefile variable.

- Allow multiple `check` metadata entries, to make it easier to check for multiple 3rd party dependencies.

- Better error handling on malformed/invalid filter metadata.

- Find a way to encode conditional builds in the metadata. I am here thinking on filters like the `dnn` family, which can have multiple backends, which are now chosen using `--enable-openvino`, `--enable-libtorch`, `--enable-libtensorflow` or the drawtext filter that has optional dependencies. Maybe the external filters could hook new `--enable-*` options to `./configure`, but this smells like trouble when multiple filters specify the same optional dependencies.

- You can see in the example Makefile that a new "magical" variable named `EXTERNAL_FILTER_EXAMPLE_LOCATION` was created by the build system. It looks ugly to me, but I could not find a way to obtain the path of the current Makefile via make, and I needed a way to specify the path of the filter source files. If you know a cleaner way to do it, please let me know.

- Make it possible to link filters written in other languages. My "final" goal for is Rust, but it should not prevent the use of C++ or Zig, for instance. I believe this can be done in a further iteration. For now my focus is only supporting C.

- Once the implementation and "spec" is agreed, document it.

- Refactor the changes in the build system to improve legibility.


Regards,

Leandro

[-- Attachment #2: 0001-avfilter-Proof-of-Concept-enable-out-of-tree-filters.patch --]
[-- Type: text/x-patch, Size: 7179 bytes --]

From 7486f878b982867667892611e181dc4eab3298c8 Mon Sep 17 00:00:00 2001
From: Leandro Santiago <leandrosansilva@gmail.com>
Date: Wed, 12 Mar 2025 13:42:42 +0100
Subject: [PATCH] avfilter: Proof of Concept: enable out-of-tree filters

This is a POC/prototype that aims to enable out of tree filters on
FFmpeg.

Here is how to test it, with an example filter:

```
mkdir -p ext/libavfilter
pushd ext/libavfilter
git clone https://gitlab.com/leandrosansilva/ffmpeg-extra-filter-example example
popd
```

Then compile ffmpeg as usual:

```
./configure && make && make install
```

Now you can use the filters:

```
ffplay /path/to/file.webm -vf 'foo,bar'
```

What works:

- Building C based filters with no extra, or simple dependencies.
- Multiple filters in the same object file.

What is ugly:

- The filter metadata is hidden in the Makefile comments, and this is
  needed because at `./configure` time, executed before make, some code
  needs to be generated and, although such generation could in theory be
  done via make, this feels like too much change at the moment.

- Among the metadata, there are linker and compiler flags, which would
  look much better if they were simple make variables.

- At the moment it's possible for only one `check` metadata entry to be
  included in the Makefile, but we should support multiple of them, for
  the case when there are multiple extra dependencies.

What was not implemented:

- I believe it would be useful to check if the license of the filter is
  compatible with the license used to build FFmpeg.

- Only extra filters written in C (maybe C++?) are supported for now.
  One of my goals is to enable Rust as well.

- There should be a way to have optional dependencies, for filters that
  can have multiple "backends", for instance, to be chosen at build time.

Signed-off-by: Leandro Santiago <leandrosansilva@gmail.com>
---
 .gitignore               |  4 +++
 configure                | 75 ++++++++++++++++++++++++++++++++++++++++
 libavfilter/Makefile     |  4 +++
 libavfilter/allfilters.c |  1 +
 4 files changed, 84 insertions(+)

diff --git a/.gitignore b/.gitignore
index 430abaf91b..4eae911379 100644
--- a/.gitignore
+++ b/.gitignore
@@ -44,3 +44,7 @@
 /tools/python/__pycache__/
 /libavcodec/vulkan/*.c
 /libavfilter/vulkan/*.c
+/ffbuild/external-filters.txt
+/ffbuild/external-filters.mak
+/libavfilter/external_filters_extern.h
+/ext
diff --git a/configure b/configure
index d84e32196d..f7bb2cc38a 100755
--- a/configure
+++ b/configure
@@ -1798,6 +1798,7 @@ AVDEVICE_COMPONENTS="
 
 AVFILTER_COMPONENTS="
     filters
+    external_filters
 "
 
 AVFORMAT_COMPONENTS="
@@ -4489,6 +4490,50 @@ for opt do
     esac
 done
 
+symbols_from_external_filter_makefile() {
+  grep '^#.*symbol:' < "$1" | sed -e 's|#\s*symbol: \(.*\)|\1|g'
+}
+
+list_external_filter_makefiles() {
+  [ ! -d "ext/libavfilter" ] && return
+
+  for f in ext/libavfilter/*; do
+    [ -f "$f/Makefile" ] && echo $f/Makefile
+  done
+}
+
+list_external_filter_makefiles > ffbuild/external-filters.txt
+
+find_external_filters_extern() {
+  # TODO: handle invalid filter
+  while read Makefile; do
+    symbols_from_external_filter_makefile "$Makefile" | sed 's/^ff_[avfsinkrc]\{2,5\}_\([[:alnum:]_]\{1,\}\)/\1_filter/'
+  done < ffbuild/external-filters.txt
+}
+
+EXTERNAL_FILTER_LIST=$(find_external_filters_extern)
+
+for n in external_filters; do
+    v=$(toupper ${n%s})_LIST
+    eval enable \$$v
+    eval ${n}_if_any="\$$v"
+done
+
+FILTER_LIST="
+    $FILTER_LIST
+    $EXTERNAL_FILTER_LIST
+"
+
+AVFILTER_COMPONENTS_LIST="
+    $AVFILTER_COMPONENTS_LIST
+    $EXTERNAL_FILTER_LIST
+"
+
+ALL_COMPONENTS="
+    $ALL_COMPONENTS
+    $EXTERNAL_FILTER_LIST
+"
+
 for e in $env; do
     eval "export $e"
 done
@@ -7173,6 +7218,11 @@ enabled rkmpp             && { require_pkg_config rkmpp rockchip_mpp  rockchip/r
                              }
 enabled vapoursynth       && require_headers "vapoursynth/VSScript4.h vapoursynth/VapourSynth4.h"
 
+# Check for the dependencies of the external filters
+while read Makefile; do
+  # NOTE: this eval is dangerous, as it allows arbitrary code execution!
+  eval $(grep '^#.*check:' "$Makefile" | sed 's|#\s*check: \(.*\)|\1|g')
+done < ffbuild/external-filters.txt
 
 if enabled gcrypt; then
     GCRYPT_CONFIG="${cross_prefix}libgcrypt-config"
@@ -8250,12 +8300,23 @@ IGNORE_TESTS=$ignore_tests
 VERSION_TRACKING=$version_tracking
 EOF
 
+while read Makefile; do
+  # NOTE: this eval is dangerous, as it allows arbitrary code execution!
+  eval "avfilter_extralibs=\"\$avfilter_extralibs $(grep '^#.*ldflags:' "$Makefile" | sed 's|#\s*ldflags: \(.*\)|\1|g')\""
+done < ffbuild/external-filters.txt
+
 map 'eval echo "${v}_FFLIBS=\$${v}_deps" >> ffbuild/config.mak' $LIBRARY_LIST
 
 for entry in $LIBRARY_LIST $PROGRAM_LIST $EXTRALIBS_LIST; do
     eval echo "EXTRALIBS-${entry}=\$${entry}_extralibs" >> ffbuild/config.mak
 done
 
+echo "" > ffbuild/external-filters.mak
+
+while read Makefile; do
+    echo "include $Makefile" >> ffbuild/external-filters.mak
+done < ffbuild/external-filters.txt
+
 cat > $TMPH <<EOF
 /* Automatically generated by configure - do not modify! */
 #ifndef FFMPEG_CONFIG_H
@@ -8343,6 +8404,20 @@ cp_if_changed $TMPH libavutil/avconfig.h
 # ...
 eval "$(sed -n "s/^extern const FFFilter ff_\([avfsinkrc]\{2,5\}\)_\(.*\);/full_filter_name_\2=\1_\2/p" $source_path/libavfilter/allfilters.c)"
 
+rm -f libavfilter/external_filters_extern.h
+
+# register the symbols of the external filters
+while read Makefile; do
+    eval "$(symbols_from_external_filter_makefile "$Makefile" \
+      | sed 's/^ff_\([avfsinkrc]\{2,5\}\)_\([[:alnum:]]\{1,\}\)$/full_filter_name_\2=\1_\2/')"
+
+    symbols_from_external_filter_makefile "$Makefile" | while read symbol; do
+        echo "extern const FFFilter $symbol;" >> libavfilter/external_filters_extern.h
+        echo "EXTERNAL_FILTER_$(echo $symbol | sed 's/^ff_[avfsinkrc]\{2,5\}_\([[:alnum:]]\{1,\}\)$/\1/' | tr a-z A-Z)_LOCATION = $f" \
+            >> ffbuild/external-filters.mak
+    done
+done < ffbuild/external-filters.txt
+
 # generate the lists of enabled components
 print_enabled_components(){
     file=$1
diff --git a/libavfilter/Makefile b/libavfilter/Makefile
index 7c0d879ec9..877b24c30f 100644
--- a/libavfilter/Makefile
+++ b/libavfilter/Makefile
@@ -27,6 +27,10 @@ OBJS = allfilters.o                                                     \
 include $(SRC_PATH)/libavfilter/dnn/Makefile
 include $(SRC_PATH)/libavfilter/vulkan/Makefile
 
+# external filters handling
+include $(SRC_PATH)/ffbuild/external-filters.mak
+EXTERNAL_FILTER_FLAGS = -I$(PWD) -I$(PWD)/libavfilter
+
 OBJS-$(HAVE_LIBC_MSVCRT)                     += file_open.o
 OBJS-$(HAVE_THREADS)                         += pthread.o
 
diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
index 740d9ab265..c2d576e4be 100644
--- a/libavfilter/allfilters.c
+++ b/libavfilter/allfilters.c
@@ -621,6 +621,7 @@ extern  const FFFilter ff_vsrc_buffer;
 extern  const FFFilter ff_asink_abuffer;
 extern  const FFFilter ff_vsink_buffer;
 
+#include "libavfilter/external_filters_extern.h"
 #include "libavfilter/filter_list.c"
 
 
-- 
2.48.1


[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".