Re: [FFmpeg-devel] Proposal: extract DNN filters code to a different repository and rewrite it in C++

["softworkz ." <softworkz-at-hotmail.com@ffmpeg.org>]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Leandro Santiago
> Sent: Donnerstag, 27. März 2025 15:07
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: [FFmpeg-devel] Proposal: extract DNN filters code to a
> different repository and rewrite it in C++
> 
> Dear FFmpeg developers, I've been doing some experiments with out-of-
> tree filters [1] and I start feeling confident that the approach works
> well enough and, besides a "demo" filter written in C [2], I've
> implemented a filter in Rust [3].
> 
> That said, I propose:
> 
> - extracting the DNN filters (dnn_detect, dnn_classify, dnn_processing)
> to another repository, within the ffmpeg umbrella, to be integrated to
> ffmpeg at build time by the users who require this feature.
> 
> - rewriting those filters in C++.
> 
> My rationale is:
> 
> - the DNN code is quite niche, not used by most people.
> 
> - There are some files on it written in C++, namely the libtorch
> backend, so moving it out would reduce the C++ code in the ffmpeg
> repository.
> 
> - Besides the libtorch backend, the other backends (openvino and
> tensorflow) use C APIs which are essentially wrappers around C++ APIs
> (as both openvino and tensorflow base APIs are C++). I don't see much
> benefits on using C wrappers when the C++ API could be used directly,
> making the code cleaner and more reliable, as the C++ APIs offer more
> type safety than the C ones and, well, RAII.
> 
> Outcomes:
> 
> - Less C++ code on ffmpeg codebase.
> 
> - Less ~6k LOC from the ffmpeg codebase, reducing a little maintenance
> burden.
> 
> - The DNN filters can evolve separately, and even have a different
> release cycle, ownership and collaboration model.
> 
> - After the C++ rewrite, the DNN code will be type safer, likely
> shorter, and with less dependencies, as it'll use the original C++ APIs
> instead of the C wrappers.


Hi Leandro,

it was about three years ago but it's one of the things that have burnt into memory: there was an interesting submission for something related to AI and filtering. It appeared to be done well, but the only response the submitter got was like: "We don't need something using once another AI framework, we have enough already".
All the effort of the submitter - destroyed by a single sentence.

Now, of course many frameworks exist and even more to come, so it's also clear that these cannot all go into the ffmpeg core code. But I think we should have a clear path for such contributions and that's why I think your proposal is a very good direction.

Moving out existing AI code also make sense to me, because not all of the currently supported ones are among the top choices one would make these days. AI is progressing fast and faster than FFmpeg can be.
Moving out existing AI code with an easy method to include them would not only slim down the code base but it would also serve as a blueprint for others who refrained from contributing, assuming these contributions wouldn't be accepted anyway.

sw












_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".