[FFmpeg-devel] Brainstorming out of tree filters

[Leandro Santiago <leandrosansilva@gmail.com>]

I've noticed that there is some kind of discontentment with current state of the libavfilter directory and a wish to reduce the number of mainstreamed filters there, relying on some kind of interface for keeping non essential plugins out of tree.

Al noticed was the desire to make it easier for 3rd parties to integrate to ffmpeg without requiring mainstream changes.

I've been working on a Rust [1] based filter which I do not expect to be mainlined in ffmpeg any time soon, so having such ouf of tree support would benefit me a lot as well.

The approaches I've seen so far are (there are probably other approaches, please suggest them here):

- runtime loadable filters: this would require the current private filter API to be refactored as a public API, and would need to guarantee some ABI stability to make it usable in real life by real filters. This seems to be a lot of work, with questionable benefits and a huge burden to keep API and ABI stability.

- filters implemented in a DSL, at runtime: think on offering for instance a Lua API for the filters. This would allow better control of the API, but require writing, possibly manually, such bindings from C to Lua (or whichever chosen language). Due to the nature of the filters, this approach would likely have big performance overhead on the filters implementations, on the C<->Lua translation layer. There is extra burden on keeping the bindings too.

- sandboxed filters, using webassembly. This shares similar issues with the DSL approach above, but at least would allow filters to be written in any language that compiles to WASM. It would also share the disadvantages of the runtime loadable filters, as it'd require some stable ABI to be useful. It seems (I am probably wrong; have no experience using WASM) that in WASM sharing memory between host and virtual machine is not an easy task yet, requiring either frequent memory copying between host and guest, which would kill performance. WASM specs seem to be changing to address similar issues (the GC proposal), but there seems not to be anything ready, stable and that directly addresses the aforementioned issues.

- out of tree filters, integrated at build time: some projects such as LLVM use it. This approach does not require offering any API or ABI stability, being up to the 3rd party developer to guarantee it compiles with a certain version of FFmpeg. It seems that this approach would also provide the shortest path for extracting currently filters that are not are used by most users or require C++ (dnn, decklnk) to external repos and developed independently from the main FFmpeg repo.

This is not an RFC, just some brainstorming. I believe that the out of tree approach is the best fit for the current codebase, but it comes with challenges/issues:

- build system integration

From someone compiling FFmpeg, I'd like something as simple  as `./configure --extra-filter=/path/to/foo --extra-filter=/path/to/bar`. This would then look at such directories and report an error in case dependencies are not met, etc. The extra filters would then be built directly into libavfilter.so/libavfilter.a.

On the filter perspective, we'd need support for describing the filter properties, dependencies, etc. I am thinking right now on something we can easily integrate into the current build system, like a `filter-manifest.json` file, in this example declaring two filters:

```

[
  {
    "name": "example",
    "description": "An example filter",
    "license": "LGPL3",
    "symbol": "ff_vf_example",
    "ldflags": "-lsome_external_lib -lblah",
    "cflags": "-DFOO=123 -I/usr/local/include/blah",
    "check_command": "is_foo_installed.sh"
    "pre-make": "command-to-run-before-make"
  },
  {
    "name": "blah",
    "description": "baz",
    "license": "LGPL3",
    "symbol": "ff_af_blah",
    "pkg-config": [
      "foo",
      "bar"
    ]
  }
]

```

It would require `jq` to be installed, for querying JSON directly from `sh`.

Additionally, there there should be a Makefile in the directory, for the build process itself, but I am not well versed on make and the FFmpeg build system to suggest how it'd look like right now.

I know it sucks to create a new format, but it seems for me a good balance between a bunch of files spread in a directory + awk/sed/make vs requiring FFmpeg to use a different build system such as meson on cmake.

I haven't put much thought into the filter description though, so I might be very mistaken and there are easier ways within the current FFmpeg codebase.

- Licensing

Some people will be interested in building proprietary filters, so I don't know how much building them directly into libavfilter library would be compatible with FFmpeg licenses. Many companies already link FFmpeg into proprietary products, by building only LGPL code and linking to FFMpeg dynamically, so maybe not much would change by putting their code directly into libavfilter.so

- Stability and support

Building 3rd party code directly into libavfilter.(so|a) could cause users of such FFmpeg builds to report issues caused by the 3rd party code to FFmpeg developers, increasing the burden on them. I think this is mostly a social issue and don't have a possible solution for it. But I wonder how much different this is from the current status of Linux distributions building, shipping and supporting FFmpeg.

- Documentation

It would be great to be able to also generate documentation from such extra filters in the same fashion as the mainstream filters.

I would like to know if any of this sounds interesting to you folks and whether anyone is already working in solving the original problem. Advice from more experienced devs is also appreciated.

In case of no objections, I'd like to start experimenting with it, working on a prototype/POC.

I am also interested in knowing if other anyone else interested in working on it too.

Regards,

Leandro

[1] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/11

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".