[FFmpeg-devel] Proposal: extract DNN filters code to a different repository and rewrite it in C++

[FFmpeg-devel] Proposal: extract DNN filters code to a different repository and rewrite it in C++

[Leandro Santiago]

Dear FFmpeg developers, I've been doing some experiments with out-of-tree filters [1] and I start feeling confident that the approach works well enough and, besides a "demo" filter written in C [2], I've implemented a filter in Rust [3].

That said, I propose:

- extracting the DNN filters (dnn_detect, dnn_classify, dnn_processing) to another repository, within the ffmpeg umbrella, to be integrated to ffmpeg at build time by the users who require this feature.

- rewriting those filters in C++.

My rationale is:

- the DNN code is quite niche, not used by most people.

- There are some files on it written in C++, namely the libtorch backend, so moving it out would reduce the C++ code in the ffmpeg repository.

- Besides the libtorch backend, the other backends (openvino and tensorflow) use C APIs which are essentially wrappers around C++ APIs (as both openvino and tensorflow base APIs are C++). I don't see much benefits on using C wrappers when the C++ API could be used directly, making the code cleaner and more reliable, as the C++ APIs offer more type safety than the C ones and, well, RAII.

Outcomes:

- Less C++ code on ffmpeg codebase.

- Less ~6k LOC from the ffmpeg codebase, reducing a little maintenance burden.

- The DNN filters can evolve separately, and even have a different release cycle, ownership and collaboration model.

- After the C++ rewrite, the DNN code will be type safer, likely shorter, and with less dependencies, as it'll use the original C++ APIs instead of the C wrappers.


Proposed steps:

- polish and merge my patch that enables out-of-tree filters [4].

- extract the libavfilter/*dnn* code to an external repository, say git.ffmpeg.org/dnn-filters.git, keeping the git history.

- adapt the dnn-filters repository to be built "out-of-tree" without breaking any public API.

- rewrite the openvino backend and tensorflow backends to C++, removing the dependency on the C API wrappers.

- rewrite the DNN filters in C++.


Open issues:

- Currently selecting the DNN backend be done via ./configure --enable-(libopenvino|libtensorflow|libtorch), which is hardcoded in the build system. I don't know how yet it can be done via the out-of-tree approach. An easy way is to simply pass the backend via environment, like in `DNN_BACKEND=openvino ./configure`, but I'd like to know how you folks would do it.

- several macros used in the C code won't work on the C++ rewrite, requiring new C++ specific macros to be created. This affects only the last step of the aforementioned steps, though.

- There is no way yet to integrate the documentation of out-of-tree filters into ffmpeg docs.

- There will still be some "remains" of the DNN code in libavutil, namely the detection boxes related API.

- During the extraction process, the DNN code should be "frozen", and any existing non merged patch should either be merged before the extraction, or be adapted to the the new repo.

- Some extra coordination will be needed not to break any production code during the migration. I don't think this is the case, as no public API needs change, but I might be wrong.

- I've performed splitting git repositories before, but never with signed commits, if there are any in the DNN code. It's possible that the signatures will break in the process, but I am not very sure right now.


In any case, I would like to know whether the current proposal aligns with the needs of the community or what needs change.

If it does, I'd like to volunteer myself to head the work on it, depending on the project roadmap.


[1] https://ffmpeg.org/pipermail/ffmpeg-devel/2025-March/341105.html

[2] https://gitlab.com/leandrosansilva/ffmpeg-extra-filter-example

[3] https://gitlab.com/leandrosansilva/ffmpeg-track-sort-filter

[4] https://patchwork.ffmpeg.org/project/ffmpeg/patch/2477728b-6217-446a-9a15-830f028baea0@gmail.com/

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Proposal: extract DNN filters code to a different repository and rewrite it in C++

[softworkz .]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Leandro Santiago
> Sent: Donnerstag, 27. März 2025 15:07
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: [FFmpeg-devel] Proposal: extract DNN filters code to a
> different repository and rewrite it in C++
> 
> Dear FFmpeg developers, I've been doing some experiments with out-of-
> tree filters [1] and I start feeling confident that the approach works
> well enough and, besides a "demo" filter written in C [2], I've
> implemented a filter in Rust [3].
> 
> That said, I propose:
> 
> - extracting the DNN filters (dnn_detect, dnn_classify, dnn_processing)
> to another repository, within the ffmpeg umbrella, to be integrated to
> ffmpeg at build time by the users who require this feature.
> 
> - rewriting those filters in C++.
> 
> My rationale is:
> 
> - the DNN code is quite niche, not used by most people.
> 
> - There are some files on it written in C++, namely the libtorch
> backend, so moving it out would reduce the C++ code in the ffmpeg
> repository.
> 
> - Besides the libtorch backend, the other backends (openvino and
> tensorflow) use C APIs which are essentially wrappers around C++ APIs
> (as both openvino and tensorflow base APIs are C++). I don't see much
> benefits on using C wrappers when the C++ API could be used directly,
> making the code cleaner and more reliable, as the C++ APIs offer more
> type safety than the C ones and, well, RAII.
> 
> Outcomes:
> 
> - Less C++ code on ffmpeg codebase.
> 
> - Less ~6k LOC from the ffmpeg codebase, reducing a little maintenance
> burden.
> 
> - The DNN filters can evolve separately, and even have a different
> release cycle, ownership and collaboration model.
> 
> - After the C++ rewrite, the DNN code will be type safer, likely
> shorter, and with less dependencies, as it'll use the original C++ APIs
> instead of the C wrappers.


Hi Leandro,

it was about three years ago but it's one of the things that have burnt into memory: there was an interesting submission for something related to AI and filtering. It appeared to be done well, but the only response the submitter got was like: "We don't need something using once another AI framework, we have enough already".
All the effort of the submitter - destroyed by a single sentence.

Now, of course many frameworks exist and even more to come, so it's also clear that these cannot all go into the ffmpeg core code. But I think we should have a clear path for such contributions and that's why I think your proposal is a very good direction.

Moving out existing AI code also make sense to me, because not all of the currently supported ones are among the top choices one would make these days. AI is progressing fast and faster than FFmpeg can be.
Moving out existing AI code with an easy method to include them would not only slim down the code base but it would also serve as a blueprint for others who refrained from contributing, assuming these contributions wouldn't be accepted anyway.

sw












_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".