[FFmpeg-devel] I've written a filter in Rust

[FFmpeg-devel] I've written a filter in Rust

[Leandro Santiago]

[insert meme here]

(this will be a long e-mail)

Dear FFmpeg devs,

in the past days I've been experimenting hacking FFmpeg using Rust.

As I am becoming more familiar with the libavfilter, and it is not a dependency for any other of the libav* libs, I decided this is a good candidate.

It's also convenient as I use FFmpeg libs heavily in a commercial product, and one of the features I've been working on involves a basic multi object tracking.

In my case, it does not need to be a "perfect" tracking algorithm, as I need to compromise quality of the result in exchange of performance executing in the CPU only, so most of the algorithms out there that need a GPU are out of my range.

I decided then use as first experiment a filter called `track_sort` that implements the 2016 paper SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC, as known as SORT [1].

The filter already works well based on the `master` branch, but the code itself is in very early stages and far from being "production ready", so please do not read the code assuming it's in its final form. It's ugly and needs lots of refactoring.

I've created a PR on forgejo [4] to make it easier for others to track progress, although I use gitlab.com as my main forge.

Here is a description of the filter:

- It perform only object tracking, needing the object detection to be performed elsewhere. It feeds from the detection boxes generated by `dnn_detect`. That means that the quality of the the tracking is closely related to the quality of the detection.

- SORT is a simple algorithm that uses spatial data only, and it not able to handle cases such as object occlusion. It's good enough for my use case, as I mentioned earlier.

- The filter works with the default options, so you can pass it without any arguments. In this mode, it will try to track any objects from the boxes available. You can change this behaviour by specifying the list of labels to track, for example: `track_sort=labels=person|dog|cat`. Such labels come from the ML model you used in the detection filter. It also has the options `threshold`, `min_hits` and `max_age`, which control how the tracking algorithm works, but the default values should work well on most cases.

- The filter will add the tracking information as label on a new frame side data entry of type `AV_FRAME_DATA_DETECTION_BBOXES`. It **WILL NOT** override the side data from `dnn_detect`,, meaning that the frame will have side data two entries of this type. I've created a PR that make it possible to fetch such entry [2].

- The labels in the detection boxes have the format "track:<track_num>:<track_age>", and this is not the final format. I did this way as a quick hack to have some visual information when drawing the boxes and labels with the `drawtext` and `drawbox` filters. I believe this can be improved by putting the tracking information as metadata of the `AVDetectionBBox`es, but this would on API and ABI breaking, so this is still an open question.

What has not been done so far:

I had quite a few goals in this task:

- 1: get a working and efficient implementation of the SORT algorithm.
- 2: start learning Rust again (it's been ~5 years since I used it)
- 3: learn more about the libavfilter codebase
- 4: evaluate whether Rust could work as a second language for hacking FFMpeg.

Results:

- 1: I managed to reuse lots of high quality code, available on crates (the repository of Rust packages), preventing me of needing to write hairy math heavy code. I personally suck in maths, especially linear algebra. Using the paper and the reference implementation [3] was enough, although I do not understand all the math magic. For instance, I reused an existing crate for Kalman filters that I probably would need to implement by hand, as the alternative in C would probably be using the implementation that OpenCV offers. And I am aware that it's not practical to make OpenCV a dependency of FFmpeg.

- 2: yay! Back to Rust!

- 3: I've learned more not only about avfilter, but a bit about other components as well.

- 4: I have more notes on that later, but it feels for me that Rust is natural candidate for new code in large C codebases, as it integrates quite tell, with some warts. I also have no idea whether the FFmpeg community has discussed about Rust in the codebase in the past and, if, not, why not now?

Some notes on using Rust:

In general I enjoyed using Rust in the project, and if you have a look at the code, you'll notice that I am not reusing any of the nice C macros that make a lot of stuff easier on writing new filters. That means that the Rust code looks like the expanded macro versions from C. And that's a lot of boilerplate and ugly code.

There were some reasons for that: One is that I am still learning Rust macros, and wanted to focus on getting stuff done for now. Second is that Rust has a much more powerful macro system than C does, and avoiding macros now allow me to feel all the pain of writing the manual code. Such pain, I believe, can help a set of Rust macros to "emerge" from the codebase, rather than one designing a set of macros that will probably look like the C ones, which might not be "rusty" enough. And I don't find a good practise to design APIs before having some implementation (looking at you, C++ committee).

I've been developing on Manjaro Linux and for now building FFmpeg statically with `--disable-stripping --enable-debug=3 --disable-optimizations` and the Rust code in `Debug` mode. That means slow code and static builds, which are easy to debug a profile.

Debugging is easy, as I can simply use GDB and it simply works with the Rust and C code mixed. I stil don't have pretty-printer for the Rust part, but this is probably an issue on my setup.

Profiling also works well. Even though the Rust code is in Debug mode, profiling with Hotspot/Perf shows that the tracking code is very efficient (you almost cannot see it in the flamegraph!).

Memory management is a breeze, as the standard library has generic versions of many useful containers, such as Vectors and BTrees. The algorithms there also make transforming and filtering very convenient and type safe.

You get support for unit tests for free. No hassle, no complex setup. Simply write unit tests anywhere and run them with `cargo test`.

It feels very good to get the code to work and not being afraid of things going badly (in the code which is not unsafe, of course!).

WARTS

I did not implement any wrapper on top of the avfilter private API (yay `bindgen`!), so it's used directly on the Rust code. It forces you to write the code as `unsafe` on any interaction with libav* API. Nevertheless, even on unsafe code, working on non owned data is very convenient, as you can turn almost anything into slices, which provide you with lots of convenient algorithms (map, filter, zip, etc.).

Working with C pointers is a very painful and ugly. Especially `**` and `***`. Rust is very verbose on using them in the rust side (they become things like `&*mut *mut *mut`, not really easy to reason about). Rust also does not have the `->` operator, forcing you do do stuff like ``(*foo).bar`, which is simply ugly.

Interacting with the C API is also not trivial, as in Rust one must be explicit about ownership and lifetimes, something which is done implicitely (and often wrongly) in C.

Struct members in Rust must always be explicitely initialized, even for global static variables, which C initializes with zero implicitely.

C unions. Luckily Rust supports them, but they are always unsafe.

`bindgen` does not generate wrappers for `static av_always_inline blah()` functions, as those are... inlined, so when in the need of using those, I had to simply reimplement them in Rust.

In general my impression is that Rust code is more verbose than C in "dangerous" code, but way less verbose in safe code, due to the compiler checks.

WHY? WHY? WHY?????

Ok, why do I, who never really took part on the FFmpeg community come apparently now throwing Rust on your faces? Am I saying you folks should rewrite ffmpeg in rust? I know that especially the Rust community have been involved recently in a lot of conflicts involving large C codebases, and it's not my intention to tell you what or not to do. I recognize having no authority in this group for that and I am essentially just a FFmpeg user.

My intention, first of all, was to get some stuff I needed done. I'm working on a commercial product, and developing in Rust was the quickest way I could get it done (considering my requirements). I've enjoyed a lot working in this project, and I believe my learnings can be useful for the FFmpeg community as a whole.

Demo time

Requirements: Cargo/Rust installed. I am using `1.84.0`, the latest stable, via `rustup`.

You'll need openvino, harfbuzz and freetype installed.

First of all, check out the code from the PR at [4] and compile FFmpeg with:

```sh
./configure ./configure --disable-stripping --enable-debug=3 --disable-optimizations --enable-libopenvino --enable-libharfbuzz --enable-libfreetype --enable-openssl
cargo build && make
```

I added a `--enable-rust` flag to the PR, but at the moment it does nothing :-)

Next you should download a pre-trained YOLO4 model and associated files, for perform the object detections:

```sh
pip install openvino-dev tensorflow
omz_downloader --name yolo-v4-tiny-tf
omz_converter --name yolo-v4-tiny-tf
wget https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/refs/heads/master/data/dataset_classes/coco_80cl.txt
```

Here we'll use a video from MOT Challenge 2016, [5] which is the one shown in the original SORT paper. You can use it with the command:

```sh
./ffplay https://motchallenge.net/sequenceVideos/MOT16-06-raw.webm -vf 'dnn_detect=dnn_backend=openvino:model=public/yolo-v4-tiny-tf/FP16/yolo-v4-tiny-tf.xml:input=image_input:confidence=0.1:model_type=yolov4:anchors=81&82&135&169&344&319:labels=coco_80cl.txt:async=0:nb_classes=80,track_sort=labels=person,drawbox=box_source=side_data_detection_bboxes:color=red:skip=1,drawtext=text_source=side_data_detection_bboxes:fontcolor=yellow:bordercolor=yellow:fontsize=20:fontfile=DroidSans-Bold.ttf:skip=1'
```

The `dnn_detect` options were obtained from the YOLO4 model at [6].

Please also noticed I passed the extra option `skip=1` to both the `drawtext` and the `drawbox` filters. This is to make them render the boxes information from  `track_sort` , instead of the ones from `dnn_detect`. More at [2].

I also recorded a video showing the filter in action [7].

Cheers,

Leandro


[1] https://arxiv.org/pdf/1703.07402
[2] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/10
[3] https://github.com/abewley/sort
[4] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/11
[5] https://motchallenge.net/vis/MOT16-06
[6] https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/yolo-v4-tiny-tf/README.md
[7] https://youtu.be/U_y4-NnaINg

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Leandro Santiago]

Just a correction, I messed up the referenced paper, and I actually should refer to this one:

https://arxiv.org/pdf/1602.00763

As the one I mentioned previously seems to be an extension of the SORT, but with appearance information, which is **NOT** what what the filter I implemented.

An embarrassing mistake, and I will fix it in the code and PR (but it will keep forever in the mailing list :-().

In any case, I probably need to spend more time reading papers.

Also, I hope it's clear that this work is a POC, and I am not suggesting being mainlined, at least not in the short term.

On 2/20/25 14:06, Leandro Santiago wrote:
> [insert meme here]
>
> (this will be a long e-mail)
>
> Dear FFmpeg devs,
>
> in the past days I've been experimenting hacking FFmpeg using Rust.
>
> As I am becoming more familiar with the libavfilter, and it is not a dependency for any other of the libav* libs, I decided this is a good candidate.
>
> It's also convenient as I use FFmpeg libs heavily in a commercial product, and one of the features I've been working on involves a basic multi object tracking.
>
> In my case, it does not need to be a "perfect" tracking algorithm, as I need to compromise quality of the result in exchange of performance executing in the CPU only, so most of the algorithms out there that need a GPU are out of my range.
>
> I decided then use as first experiment a filter called `track_sort` that implements the 2016 paper SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC, as known as SORT [1].
>
> The filter already works well based on the `master` branch, but the code itself is in very early stages and far from being "production ready", so please do not read the code assuming it's in its final form. It's ugly and needs lots of refactoring.
>
> I've created a PR on forgejo [4] to make it easier for others to track progress, although I use gitlab.com as my main forge.
>
> Here is a description of the filter:
>
> - It perform only object tracking, needing the object detection to be performed elsewhere. It feeds from the detection boxes generated by `dnn_detect`. That means that the quality of the the tracking is closely related to the quality of the detection.
>
> - SORT is a simple algorithm that uses spatial data only, and it not able to handle cases such as object occlusion. It's good enough for my use case, as I mentioned earlier.
>
> - The filter works with the default options, so you can pass it without any arguments. In this mode, it will try to track any objects from the boxes available. You can change this behaviour by specifying the list of labels to track, for example: `track_sort=labels=person|dog|cat`. Such labels come from the ML model you used in the detection filter. It also has the options `threshold`, `min_hits` and `max_age`, which control how the tracking algorithm works, but the default values should work well on most cases.
>
> - The filter will add the tracking information as label on a new frame side data entry of type `AV_FRAME_DATA_DETECTION_BBOXES`. It **WILL NOT** override the side data from `dnn_detect`,, meaning that the frame will have side data two entries of this type. I've created a PR that make it possible to fetch such entry [2].
>
> - The labels in the detection boxes have the format "track:<track_num>:<track_age>", and this is not the final format. I did this way as a quick hack to have some visual information when drawing the boxes and labels with the `drawtext` and `drawbox` filters. I believe this can be improved by putting the tracking information as metadata of the `AVDetectionBBox`es, but this would on API and ABI breaking, so this is still an open question.
>
> What has not been done so far:
>
> I had quite a few goals in this task:
>
> - 1: get a working and efficient implementation of the SORT algorithm.
> - 2: start learning Rust again (it's been ~5 years since I used it)
> - 3: learn more about the libavfilter codebase
> - 4: evaluate whether Rust could work as a second language for hacking FFMpeg.
>
> Results:
>
> - 1: I managed to reuse lots of high quality code, available on crates (the repository of Rust packages), preventing me of needing to write hairy math heavy code. I personally suck in maths, especially linear algebra. Using the paper and the reference implementation [3] was enough, although I do not understand all the math magic. For instance, I reused an existing crate for Kalman filters that I probably would need to implement by hand, as the alternative in C would probably be using the implementation that OpenCV offers. And I am aware that it's not practical to make OpenCV a dependency of FFmpeg.
>
> - 2: yay! Back to Rust!
>
> - 3: I've learned more not only about avfilter, but a bit about other components as well.
>
> - 4: I have more notes on that later, but it feels for me that Rust is natural candidate for new code in large C codebases, as it integrates quite tell, with some warts. I also have no idea whether the FFmpeg community has discussed about Rust in the codebase in the past and, if, not, why not now?
>
> Some notes on using Rust:
>
> In general I enjoyed using Rust in the project, and if you have a look at the code, you'll notice that I am not reusing any of the nice C macros that make a lot of stuff easier on writing new filters. That means that the Rust code looks like the expanded macro versions from C. And that's a lot of boilerplate and ugly code.
>
> There were some reasons for that: One is that I am still learning Rust macros, and wanted to focus on getting stuff done for now. Second is that Rust has a much more powerful macro system than C does, and avoiding macros now allow me to feel all the pain of writing the manual code. Such pain, I believe, can help a set of Rust macros to "emerge" from the codebase, rather than one designing a set of macros that will probably look like the C ones, which might not be "rusty" enough. And I don't find a good practise to design APIs before having some implementation (looking at you, C++ committee).
>
> I've been developing on Manjaro Linux and for now building FFmpeg statically with `--disable-stripping --enable-debug=3 --disable-optimizations` and the Rust code in `Debug` mode. That means slow code and static builds, which are easy to debug a profile.
>
> Debugging is easy, as I can simply use GDB and it simply works with the Rust and C code mixed. I stil don't have pretty-printer for the Rust part, but this is probably an issue on my setup.
>
> Profiling also works well. Even though the Rust code is in Debug mode, profiling with Hotspot/Perf shows that the tracking code is very efficient (you almost cannot see it in the flamegraph!).
>
> Memory management is a breeze, as the standard library has generic versions of many useful containers, such as Vectors and BTrees. The algorithms there also make transforming and filtering very convenient and type safe.
>
> You get support for unit tests for free. No hassle, no complex setup. Simply write unit tests anywhere and run them with `cargo test`.
>
> It feels very good to get the code to work and not being afraid of things going badly (in the code which is not unsafe, of course!).
>
> WARTS
>
> I did not implement any wrapper on top of the avfilter private API (yay `bindgen`!), so it's used directly on the Rust code. It forces you to write the code as `unsafe` on any interaction with libav* API. Nevertheless, even on unsafe code, working on non owned data is very convenient, as you can turn almost anything into slices, which provide you with lots of convenient algorithms (map, filter, zip, etc.).
>
> Working with C pointers is a very painful and ugly. Especially `**` and `***`. Rust is very verbose on using them in the rust side (they become things like `&*mut *mut *mut`, not really easy to reason about). Rust also does not have the `->` operator, forcing you do do stuff like ``(*foo).bar`, which is simply ugly.
>
> Interacting with the C API is also not trivial, as in Rust one must be explicit about ownership and lifetimes, something which is done implicitely (and often wrongly) in C.
>
> Struct members in Rust must always be explicitely initialized, even for global static variables, which C initializes with zero implicitely.
>
> C unions. Luckily Rust supports them, but they are always unsafe.
>
> `bindgen` does not generate wrappers for `static av_always_inline blah()` functions, as those are... inlined, so when in the need of using those, I had to simply reimplement them in Rust.
>
> In general my impression is that Rust code is more verbose than C in "dangerous" code, but way less verbose in safe code, due to the compiler checks.
>
> WHY? WHY? WHY?????
>
> Ok, why do I, who never really took part on the FFmpeg community come apparently now throwing Rust on your faces? Am I saying you folks should rewrite ffmpeg in rust? I know that especially the Rust community have been involved recently in a lot of conflicts involving large C codebases, and it's not my intention to tell you what or not to do. I recognize having no authority in this group for that and I am essentially just a FFmpeg user.
>
> My intention, first of all, was to get some stuff I needed done. I'm working on a commercial product, and developing in Rust was the quickest way I could get it done (considering my requirements). I've enjoyed a lot working in this project, and I believe my learnings can be useful for the FFmpeg community as a whole.
>
> Demo time
>
> Requirements: Cargo/Rust installed. I am using `1.84.0`, the latest stable, via `rustup`.
>
> You'll need openvino, harfbuzz and freetype installed.
>
> First of all, check out the code from the PR at [4] and compile FFmpeg with:
>
> ```sh
> ./configure ./configure --disable-stripping --enable-debug=3 --disable-optimizations --enable-libopenvino --enable-libharfbuzz --enable-libfreetype --enable-openssl
> cargo build && make
> ```
>
> I added a `--enable-rust` flag to the PR, but at the moment it does nothing :-)
>
> Next you should download a pre-trained YOLO4 model and associated files, for perform the object detections:
>
> ```sh
> pip install openvino-dev tensorflow
> omz_downloader --name yolo-v4-tiny-tf
> omz_converter --name yolo-v4-tiny-tf
> wget https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/refs/heads/master/data/dataset_classes/coco_80cl.txt
> ```
>
> Here we'll use a video from MOT Challenge 2016, [5] which is the one shown in the original SORT paper. You can use it with the command:
>
> ```sh
> ./ffplay https://motchallenge.net/sequenceVideos/MOT16-06-raw.webm -vf 'dnn_detect=dnn_backend=openvino:model=public/yolo-v4-tiny-tf/FP16/yolo-v4-tiny-tf.xml:input=image_input:confidence=0.1:model_type=yolov4:anchors=81&82&135&169&344&319:labels=coco_80cl.txt:async=0:nb_classes=80,track_sort=labels=person,drawbox=box_source=side_data_detection_bboxes:color=red:skip=1,drawtext=text_source=side_data_detection_bboxes:fontcolor=yellow:bordercolor=yellow:fontsize=20:fontfile=DroidSans-Bold.ttf:skip=1'
> ```
>
> The `dnn_detect` options were obtained from the YOLO4 model at [6].
>
> Please also noticed I passed the extra option `skip=1` to both the `drawtext` and the `drawbox` filters. This is to make them render the boxes information from  `track_sort` , instead of the ones from `dnn_detect`. More at [2].
>
> I also recorded a video showing the filter in action [7].
>
> Cheers,
>
> Leandro
>
>
> [1] https://arxiv.org/pdf/1703.07402
> [2] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/10
> [3] https://github.com/abewley/sort
> [4] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/11
> [5] https://motchallenge.net/vis/MOT16-06
> [6] https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/yolo-v4-tiny-tf/README.md
> [7] https://youtu.be/U_y4-NnaINg
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 791 bytes --]

Hi

On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> [insert meme here]
[...]
> I also recorded a video showing the filter in action [7].
[...
> [7] https://youtu.be/U_y4-NnaINg

cool, it doesnt detect everyone though

also i think this shows how useful a plugin framework would be for ffmpeg

with plugins everyone could use,test and contribute to this today.
without plugins, this needs to be merged in ffmpeg git master. (which
will take some time i suspect)

Rust support in FFmpeg is an interresting idea though, iam in favor of
supporting more "safe" languages

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Nations do behave wisely once they have exhausted all other alternatives. 
-- Abba Eban

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Leandro Santiago]

On 2/20/25 23:49, Michael Niedermayer wrote:
> Hi
>
> On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
>> [insert meme here]
> [...]
>> I also recorded a video showing the filter in action [7].
> [...
>> [7] https://youtu.be/U_y4-NnaINg
> cool, it doesnt detect everyone though

Yes, this has two reasons: one is the quality of the detection depends on the model used. I use in the demo a yolov4-tiny that is very fast but not really accurate, leading to many people not being tracked. There can be bugs in the track_sort code as well.

The other reason is that the `track_sort` at the moment ignores all the detected boxes which have a position, but no area (they seem to be detected as simple points by `dnn_detect`), which are useless for now, as the SORT implementation at the moment depends only the boxes dimensions for computing IOU (intersection over union).

I could think though on extending the code to "fake an area" of a box when it's unknown, which might help to improve the tracking.

>
> also i think this shows how useful a plugin framework would be for ffmpeg
>
> with plugins everyone could use,test and contribute to this today.
> without plugins, this needs to be merged in ffmpeg git master. (which
> will take some time i suspect)

Yes, it could even open space for webassembly based plugins, which would be less of a hassle regarding ABI and potentially be cross platform, although I foresee some challenges on getting wasm runtime to share data with the host without copying, especially if the data in the GPU.

>
> Rust support in FFmpeg is an interresting idea though, iam in favor of
> supporting more "safe" languages
Great to hear that!
>
> thx
>
> [...]
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Tomas Härdin]

tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> Hi
> 
> On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > [insert meme here]
> [...]
> > I also recorded a video showing the filter in action [7].
> [...
> > [7] https://youtu.be/U_y4-NnaINg
> 
> cool, it doesnt detect everyone though
> 
> also i think this shows how useful a plugin framework would be for
> ffmpeg
> 
> with plugins everyone could use,test and contribute to this today.
> without plugins, this needs to be merged in ffmpeg git master. (which
> will take some time i suspect)

Have we not gone over and rejected plugins many times? I recall points
about them encouraging proliferation of proprietary code. I also feel
this project is increasingly forgetting about the power of the UNIX
pipe.

/Tomas

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Tomas Härdin
> Sent: Freitag, 21. Februar 2025 10:02
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> 
> tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > Hi
> >
> > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > [insert meme here]
> > [...]
> > > I also recorded a video showing the filter in action [7].
> > [...
> > > [7] https://youtu.be/U_y4-NnaINg
> >
> > cool, it doesnt detect everyone though
> >
> > also i think this shows how useful a plugin framework would be for
> > ffmpeg
> >
> > with plugins everyone could use,test and contribute to this today.
> > without plugins, this needs to be merged in ffmpeg git master. (which
> > will take some time i suspect)

Hi Thomas,

> Have we not gone over and rejected plugins many times? I recall points
> about them encouraging proliferation of proprietary code. 

I haven't followed earlier discussions, what's bad about that (in a nutshell)?

> I also feel
> this project is increasingly forgetting about the power of the UNIX
> pipe.

That "power" doesn't exist on all platforms, and even on UNIX: does that "power" go that far that it works without memcopy? E.g. - a filter needs to paint just a few pixels on a video frame - doesn't the full uncompressed frame need to be piped forth and back?
Also, filters often do and require more than just a range of data to do their work.

sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Lynne]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 5220 bytes --]

On 20/02/2025 14:06, Leandro Santiago wrote:
> [insert meme here]
> 
> (this will be a long e-mail)
> 
> Dear FFmpeg devs,
> 
> in the past days I've been experimenting hacking FFmpeg using Rust.
> 
> As I am becoming more familiar with the libavfilter, and it is not a dependency for any other of the libav* libs, I decided this is a good candidate.
> 
> It's also convenient as I use FFmpeg libs heavily in a commercial product, and one of the features I've been working on involves a basic multi object tracking.
> 
> In my case, it does not need to be a "perfect" tracking algorithm, as I need to compromise quality of the result in exchange of performance executing in the CPU only, so most of the algorithms out there that need a GPU are out of my range.
> 
> I decided then use as first experiment a filter called `track_sort` that implements the 2016 paper SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC, as known as SORT [1].
> 
> The filter already works well based on the `master` branch, but the code itself is in very early stages and far from being "production ready", so please do not read the code assuming it's in its final form. It's ugly and needs lots of refactoring.
> 
> I've created a PR on forgejo [4] to make it easier for others to track progress, although I use gitlab.com as my main forge.
> 
> Here is a description of the filter:
> 
> - It perform only object tracking, needing the object detection to be performed elsewhere. It feeds from the detection boxes generated by `dnn_detect`. That means that the quality of the the tracking is closely related to the quality of the detection.

We've been looking to deprecate our native DNN stuff for years. Its 
bitrotten NIH code written long ago. Not a good idea to base it on.

> - SORT is a simple algorithm that uses spatial data only, and it not able to handle cases such as object occlusion. It's good enough for my use case, as I mentioned earlier.
> 
> - The filter works with the default options, so you can pass it without any arguments. In this mode, it will try to track any objects from the boxes available. You can change this behaviour by specifying the list of labels to track, for example: `track_sort=labels=person|dog|cat`. Such labels come from the ML model you used in the detection filter. It also has the options `threshold`, `min_hits` and `max_age`, which control how the tracking algorithm works, but the default values should work well on most cases.
> 
> - The filter will add the tracking information as label on a new frame side data entry of type `AV_FRAME_DATA_DETECTION_BBOXES`. It **WILL NOT** override the side data from `dnn_detect`,, meaning that the frame will have side data two entries of this type. I've created a PR that make it possible to fetch such entry [2].
> 
> - The labels in the detection boxes have the format "track:<track_num>:<track_age>", and this is not the final format. I did this way as a quick hack to have some visual information when drawing the boxes and labels with the `drawtext` and `drawbox` filters. I believe this can be improved by putting the tracking information as metadata of the `AVDetectionBBox`es, but this would on API and ABI breaking, so this is still an open question.

That's not a very useful filter. What this needs is a more complex and 
flexible interface than bounding boxes, which can enable processing of 
individual objects and turning the tracking info into something useful, 
for example, splitting up a video into objects.

> What has not been done so far:
> 
> I had quite a few goals in this task:
> 
> - 1: get a working and efficient implementation of the SORT algorithm.
> - 2: start learning Rust again (it's been ~5 years since I used it)
> - 3: learn more about the libavfilter codebase
> - 4: evaluate whether Rust could work as a second language for hacking FFMpeg.

None of us know Rust confidently enough to maintain and work on such code.

> Results:
> 
> - 1: I managed to reuse lots of high quality code, available on crates (the repository of Rust packages), preventing me of needing to write hairy math heavy code. I personally suck in maths, especially linear algebra. Using the paper and the reference implementation [3] was enough, although I do not understand all the math magic. For instance, I reused an existing crate for Kalman filters that I probably would need to implement by hand, as the alternative in C would probably be using the implementation that OpenCV offers. And I am aware that it's not practical to make OpenCV a dependency of FFmpeg.

Regardless of the language, I disagree with using crates in the context 
of FFmpeg, and any use of cargo.
We used to link to OpenCV. The only reason why we dropped it was because 
we used the C interface, which was removed, and no one wanted to or was 
interested in rewriting it.

I also don't agree with a plugin interface. We've been over this a few 
times. A plugin interface would likely turn the project into gstreamer 
or dshow, with dozens of proprietary plugins, whose users would be 
asking us for support.
We bump our ABI once a year, which is far too often for this as well.

[-- Attachment #1.1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 637 bytes --]

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2091 bytes --]

Hi

On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > Hi
> > 
> > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > [insert meme here]
> > [...]
> > > I also recorded a video showing the filter in action [7].
> > [...
> > > [7] https://youtu.be/U_y4-NnaINg
> > 
> > cool, it doesnt detect everyone though
> > 
> > also i think this shows how useful a plugin framework would be for
> > ffmpeg
> > 
> > with plugins everyone could use,test and contribute to this today.
> > without plugins, this needs to be merged in ffmpeg git master. (which
> > will take some time i suspect)
> 
> Have we not gone over and rejected plugins many times? I recall points

no
there was no formal and no public informal vote that i remember.

ive raised the issue with plugins many times. Because it would
allow people and myself to contribute more complex features and
end the stagnation of FFmpeg.
Think of AI filters (like text to image, automated language translation of subtiltes,
image to metadata, audio to subtitle, upscaling, and infinite more ideas)
Neural network based image, video and audio compression
thats both local neural networks and also using some server based APIs

In general innovation needs the innovators to be in charge.
Plugins arent the only way to achieve that of course.


> about them encouraging proliferation of proprietary code. I also feel

Thats nonsense, you can make the plugin system GPL or AGPL if you like.


> this project is increasingly forgetting about the power of the UNIX
> pipe.

The unix pipe gave us gzip and tar. And the world gave it the middle
finger and used pkzip because pkzip did what everyone needs, and the
pipe did not

PS: if someone sends me a pull request that cleanly adds plugin support to
FFmpeg, i will merge it.

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

No great genius has ever existed without some touch of madness. -- Aristotle

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Kieran Kunhya via ffmpeg-devel]

On Fri, 21 Feb 2025, 13:18 Lynne, <dev@lynne.ee> wrote:

> On 20/02/2025 14:06, Leandro Santiago wrote:
> Regardless of the language, I disagree with using crates in the context
> of FFmpeg, and any use of cargo.
>

I have no opinion on rust in FFmpeg but I broadly agree crates and cargo
are not suited for FFmpeg.

Kieran

>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Michael Niedermayer
> Sent: Freitag, 21. Februar 2025 14:22
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> 
> Hi
> 
> On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > Hi
> > >
> > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > [insert meme here]
> > > [...]
> > > > I also recorded a video showing the filter in action [7].
> > > [...
> > > > [7] https://youtu.be/U_y4-NnaINg
> > >
> > > cool, it doesnt detect everyone though
> > >
> > > also i think this shows how useful a plugin framework would be for
> > > ffmpeg
> > >
> > > with plugins everyone could use,test and contribute to this today.
> > > without plugins, this needs to be merged in ffmpeg git master. (which
> > > will take some time i suspect)
> >
> > Have we not gone over and rejected plugins many times? I recall points
> 
> no
> there was no formal and no public informal vote that i remember.
> 
> ive raised the issue with plugins many times. Because it would
> allow people and myself to contribute more complex features and
> end the stagnation of FFmpeg.

Yup, that's exactly one of the reasons why other projects like GStreamer
have gained that much popularity. It might not have happened when
ffmpeg would have been more open and extensible in the first place.

> Think of AI filters (like text to image, automated language translation of
> subtiltes,
> image to metadata, audio to subtitle, upscaling, and infinite more ideas)
> Neural network based image, video and audio compression
> thats both local neural networks and also using some server based APIs

I had similar things in mind already. The graphicsubs2text filter kind of uses
AI already, but it was only possible because it does it through a standard 
lib (tesseract). But as soon as it comes to anything which requires using 
an API that is vendor-specific, that's something which nobody will want 
to have in ffmpeg, and it might also change rapidly in a way that it's already
outdated when its containing ffmpeg version gets packaged by a distro.


> In general innovation needs the innovators to be in charge.
> Plugins arent the only way to achieve that of course.

Totally agree. Saying that we have no solution for these things is a path 
to becoming Irrelevant in the future.


What do you people think about the feasibility of a compile-time plugin
mechanism?
Given that on most platforms (except Windows), you can't easily mix 
binaries from different origins (i.e. built differently), it would be much
easier when plugin binaries could be compiled as part of building ffmpeg
rather than having to set up and maintain separate builds for each plugin.

sw


_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Kieran Kunhya via ffmpeg-devel]

On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-hotmail.com@ffmpeg.org>
wrote:

>
>
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Michael Niedermayer
> > Sent: Freitag, 21. Februar 2025 14:22
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel@ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> >
> > Hi
> >
> > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > Hi
> > > >
> > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > > [insert meme here]
> > > > [...]
> > > > > I also recorded a video showing the filter in action [7].
> > > > [...
> > > > > [7] https://youtu.be/U_y4-NnaINg
> > > >
> > > > cool, it doesnt detect everyone though
> > > >
> > > > also i think this shows how useful a plugin framework would be for
> > > > ffmpeg
> > > >
> > > > with plugins everyone could use,test and contribute to this today.
> > > > without plugins, this needs to be merged in ffmpeg git master. (which
> > > > will take some time i suspect)
> > >
> > > Have we not gone over and rejected plugins many times? I recall points
> >
> > no
> > there was no formal and no public informal vote that i remember.
> >
> > ive raised the issue with plugins many times. Because it would
> > allow people and myself to contribute more complex features and
> > end the stagnation of FFmpeg.
>
> Yup, that's exactly one of the reasons why other projects like GStreamer
> have gained that much popularity. It might not have happened when
> ffmpeg would have been more open and extensible in the first place.
>

It's quite the opposite. Gstreamer is successful in its vertical because it
allows easy inclusion of vendor binary blobs. This is not "open and
extensible".

Don't confuse the history of avisynth in the 2000s with today's reality.

Kieran

>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Kieran Kunhya via ffmpeg-devel
> Sent: Freitag, 21. Februar 2025 15:53
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Cc: Kieran Kunhya <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> 
> On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
> hotmail.com@ffmpeg.org>
> wrote:
> 
> >
> >
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > Michael Niedermayer
> > > Sent: Freitag, 21. Februar 2025 14:22
> > > To: FFmpeg development discussions and patches <ffmpeg-
> > > devel@ffmpeg.org>
> > > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> > >
> > > Hi
> > >
> > > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > > Hi
> > > > >
> > > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > > > [insert meme here]
> > > > > [...]
> > > > > > I also recorded a video showing the filter in action [7].
> > > > > [...
> > > > > > [7] https://youtu.be/U_y4-NnaINg
> > > > >
> > > > > cool, it doesnt detect everyone though
> > > > >
> > > > > also i think this shows how useful a plugin framework would be for
> > > > > ffmpeg
> > > > >
> > > > > with plugins everyone could use,test and contribute to this today.
> > > > > without plugins, this needs to be merged in ffmpeg git master. (which
> > > > > will take some time i suspect)
> > > >
> > > > Have we not gone over and rejected plugins many times? I recall points
> > >
> > > no
> > > there was no formal and no public informal vote that i remember.
> > >
> > > ive raised the issue with plugins many times. Because it would
> > > allow people and myself to contribute more complex features and
> > > end the stagnation of FFmpeg.
> >
> > Yup, that's exactly one of the reasons why other projects like GStreamer
> > have gained that much popularity. It might not have happened when
> > ffmpeg would have been more open and extensible in the first place.
> >
> 
> It's quite the opposite. Gstreamer is successful in its vertical because it
> allows easy inclusion of vendor binary blobs. This is not "open and
> extensible".

Open means it's extensible for everybody, including vendors. I fail to see
what's bad about it. Do we have a fight against everything commercial?

What are you afraid of to happen?

Just wanna understand the position, thanks.
sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Stephen Hutchinson]

On 2/21/25 9:53 AM, Kieran Kunhya via ffmpeg-devel wrote:
> On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-hotmail.com@ffmpeg.org>
> wrote:
> 
>>
>>
>>> -----Original Message-----
>>> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
>>> Michael Niedermayer
>>> Sent: Freitag, 21. Februar 2025 14:22
>>> To: FFmpeg development discussions and patches <ffmpeg-
>>> devel@ffmpeg.org>
>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>>
>>> Hi
>>>
>>> On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
>>>> tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
>>>>> Hi
>>>>>
>>>>> On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
>>>>>> [insert meme here]
>>>>> [...]
>>>>>> I also recorded a video showing the filter in action [7].
>>>>> [...
>>>>>> [7] https://youtu.be/U_y4-NnaINg
>>>>>
>>>>> cool, it doesnt detect everyone though
>>>>>
>>>>> also i think this shows how useful a plugin framework would be for
>>>>> ffmpeg
>>>>>
>>>>> with plugins everyone could use,test and contribute to this today.
>>>>> without plugins, this needs to be merged in ffmpeg git master. (which
>>>>> will take some time i suspect)
>>>>
>>>> Have we not gone over and rejected plugins many times? I recall points
>>>
>>> no
>>> there was no formal and no public informal vote that i remember.
>>>
>>> ive raised the issue with plugins many times. Because it would
>>> allow people and myself to contribute more complex features and
>>> end the stagnation of FFmpeg.
>>
>> Yup, that's exactly one of the reasons why other projects like GStreamer
>> have gained that much popularity. It might not have happened when
>> ffmpeg would have been more open and extensible in the first place.
>>
> 
> It's quite the opposite. Gstreamer is successful in its vertical because it
> allows easy inclusion of vendor binary blobs. This is not "open and
> extensible".
> 
> Don't confuse the history of avisynth in the 2000s with today's reality.
> 

And for that matter, AviSynth got bit **hard** when supporting more than
just 32-bit x86 became necessary, so any early success in attracting
plugin authors by having a Classpath-style exemption was quickly offset
by getting stuck with those decisions to maintain compatibility, and
suffering a lack of established plugins on new architectures/platforms
due to how many authors in the early days kept their plugins closed 
(thankfully most(?) of the regularly used and useful plugins now are
under FOSS licenses, but I don't even know how many plugins exist that
simply can't be ported because they're essentially closed-source
abandonware tied to an ancient version of AviSynth).

So it's just as much of a cautionary tale about how not to set up plugin
licensing and how the upstream handles compatibility in regard to it.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Tomas Härdin]

fre 2025-02-21 klockan 14:18 +0100 skrev Lynne:
> On 20/02/2025 14:06, Leandro Santiago wrote:
> 
> > - 1: I managed to reuse lots of high quality code, available on
> > crates (the repository of Rust packages), preventing me of needing
> > to write hairy math heavy code. I personally suck in maths,
> > especially linear algebra. Using the paper and the reference
> > implementation [3] was enough, although I do not understand all the
> > math magic. For instance, I reused an existing crate for Kalman
> > filters that I probably would need to implement by hand, as the
> > alternative in C would probably be using the implementation that
> > OpenCV offers. And I am aware that it's not practical to make
> > OpenCV a dependency of FFmpeg.
> 
> Regardless of the language, I disagree with using crates in the context 
> of FFmpeg, and any use of cargo.

In addition, this almost certainly will create headaches for package
maintainers. The proper way to do this is to use one's OS' package
manager, not go down the godawful path Python and every other modern
language has of rolling their own package managers.

The above said, I'm not against Rust. It has some nice properties. But
it does not seem very "stable" so far. Perhaps this has changed in
recent years..

If we're in the habit of allowing other languages I'd be in favor of
allowing C++, so that we can make use of the STL containers rather than
rolling our own.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Kieran Kunhya via ffmpeg-devel]

On Fri, 21 Feb 2025, 15:02 Soft Works, <softworkz@hotmail.com> wrote:

>
>
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Kieran Kunhya via ffmpeg-devel
> > Sent: Freitag, 21. Februar 2025 15:53
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel@ffmpeg.org>
> > Cc: Kieran Kunhya <kieran618@googlemail.com>
> > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> >
> > On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
> > hotmail.com@ffmpeg.org>
> > wrote:
> >
> > >
> > >
> > > > -----Original Message-----
> > > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > > Michael Niedermayer
> > > > Sent: Freitag, 21. Februar 2025 14:22
> > > > To: FFmpeg development discussions and patches <ffmpeg-
> > > > devel@ffmpeg.org>
> > > > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> > > >
> > > > Hi
> > > >
> > > > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > > > Hi
> > > > > >
> > > > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > > > > [insert meme here]
> > > > > > [...]
> > > > > > > I also recorded a video showing the filter in action [7].
> > > > > > [...
> > > > > > > [7] https://youtu.be/U_y4-NnaINg
> > > > > >
> > > > > > cool, it doesnt detect everyone though
> > > > > >
> > > > > > also i think this shows how useful a plugin framework would be
> for
> > > > > > ffmpeg
> > > > > >
> > > > > > with plugins everyone could use,test and contribute to this
> today.
> > > > > > without plugins, this needs to be merged in ffmpeg git master.
> (which
> > > > > > will take some time i suspect)
> > > > >
> > > > > Have we not gone over and rejected plugins many times? I recall
> points
> > > >
> > > > no
> > > > there was no formal and no public informal vote that i remember.
> > > >
> > > > ive raised the issue with plugins many times. Because it would
> > > > allow people and myself to contribute more complex features and
> > > > end the stagnation of FFmpeg.
> > >
> > > Yup, that's exactly one of the reasons why other projects like
> GStreamer
> > > have gained that much popularity. It might not have happened when
> > > ffmpeg would have been more open and extensible in the first place.
> > >
> >
> > It's quite the opposite. Gstreamer is successful in its vertical because
> it
> > allows easy inclusion of vendor binary blobs. This is not "open and
> > extensible".
>
> Open means it's extensible for everybody, including vendors. I fail to see
> what's bad about it. Do we have a fight against everything commercial?
>
> What are you afraid of to happen?
>
> Just wanna understand the position, thanks.
> sw
>

What will happen is all the binary blob vendors who have to currently ship
a custom ffmpeg and tell their users to compile from source can just ship
their blob as a plugin.

FFmpeg will get all the bug reports from crashes (this is why the kernel
had to add the "tainted") flag.

I have a deep dislike for binary blobs.

Thinking the avisynth filter development of the 2000s will reappear when
ffmpeg gets plugins is wishful to say the least.

Kieran

>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Soft Works]

From: Kieran Kunhya <kieran618@googlemail.com>
Sent: Freitag, 21. Februar 2025 20:27
To: Soft Works <softworkz@hotmail.com>
Cc: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Subject: Re: [FFmpeg-devel] I've written a filter in Rust


On Fri, 21 Feb 2025, 15:02 Soft Works, <softworkz@hotmail.com<mailto:softworkz@hotmail.com>> wrote:


> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ffmpeg.org>> On Behalf Of
> Kieran Kunhya via ffmpeg-devel
> Sent: Freitag, 21. Februar 2025 15:53
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
> Cc: Kieran Kunhya <kieran618@googlemail.com<mailto:kieran618@googlemail.com>>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>
> On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
> hotmail.com@ffmpeg.org<mailto:hotmail.com@ffmpeg.org>>
> wrote:
>
> >
> >
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ffmpeg.org>> On Behalf Of
> > > Michael Niedermayer
> > > Sent: Freitag, 21. Februar 2025 14:22
> > > To: FFmpeg development discussions and patches <ffmpeg-
> > > devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
> > > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> > >
> > > Hi
> > >
> > > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > > Hi
> > > > >
> > > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > > > [insert meme here]
> > > > > [...]
> > > > > > I also recorded a video showing the filter in action [7].
> > > > > [...
> > > > > > [7] https://youtu.be/U_y4-NnaINg
> > > > >
> > > > > cool, it doesnt detect everyone though
> > > > >
> > > > > also i think this shows how useful a plugin framework would be for
> > > > > ffmpeg
> > > > >
> > > > > with plugins everyone could use,test and contribute to this today.
> > > > > without plugins, this needs to be merged in ffmpeg git master. (which
> > > > > will take some time i suspect)
> > > >
> > > > Have we not gone over and rejected plugins many times? I recall points
> > >
> > > no
> > > there was no formal and no public informal vote that i remember.
> > >
> > > ive raised the issue with plugins many times. Because it would
> > > allow people and myself to contribute more complex features and
> > > end the stagnation of FFmpeg.
> >
> > Yup, that's exactly one of the reasons why other projects like GStreamer
> > have gained that much popularity. It might not have happened when
> > ffmpeg would have been more open and extensible in the first place.
> >
>
> It's quite the opposite. Gstreamer is successful in its vertical because it
> allows easy inclusion of vendor binary blobs. This is not "open and
> extensible".

Open means it's extensible for everybody, including vendors. I fail to see
what's bad about it. Do we have a fight against everything commercial?

What are you afraid of to happen?

Just wanna understand the position, thanks.
sw


Hi Kieran,

What will happen is all the binary blob vendors who have to currently ship a custom ffmpeg and tell their users to compile from source can just ship their blob as a plugin.

We currently have 200 reasons (commits) for needing to ship a custom ffmpeg and a plugin model would not change that.
It’s hard for everybody to get even good changes in ffmpeg and there are other changes for which there’s clearly no place in the ffmpeg codebase. There should be an equalization for this situation.
Also, I think it’s a bit unfair and oversimplifying to blame all vendors’ main interest being to ship binaries as extension just for the sake of keeping them closed source – even though those cases might occur.


FFmpeg will get all the bug reports from crashes (this is why the kernel had to add the "tainted") flag.

Is this really a big problem? It seems to me that such cases can be easily identified (the presence of loaded plugins could also be indicated in the version header).

I have a deep dislike for binary blobs.

That I can understand. But on one side rejecting a lot of source code and also not allowing compiled code for extending is a pretty fencing policy es well.

What do you think about a compile-time plugin model?

PS:I don’t know the avisynth back story...

Thanks
sw


_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".