[FFmpeg-devel] I've written a filter in Rust

[FFmpeg-devel] I've written a filter in Rust

[Leandro Santiago]

[insert meme here]

(this will be a long e-mail)

Dear FFmpeg devs,

in the past days I've been experimenting hacking FFmpeg using Rust.

As I am becoming more familiar with the libavfilter, and it is not a dependency for any other of the libav* libs, I decided this is a good candidate.

It's also convenient as I use FFmpeg libs heavily in a commercial product, and one of the features I've been working on involves a basic multi object tracking.

In my case, it does not need to be a "perfect" tracking algorithm, as I need to compromise quality of the result in exchange of performance executing in the CPU only, so most of the algorithms out there that need a GPU are out of my range.

I decided then use as first experiment a filter called `track_sort` that implements the 2016 paper SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC, as known as SORT [1].

The filter already works well based on the `master` branch, but the code itself is in very early stages and far from being "production ready", so please do not read the code assuming it's in its final form. It's ugly and needs lots of refactoring.

I've created a PR on forgejo [4] to make it easier for others to track progress, although I use gitlab.com as my main forge.

Here is a description of the filter:

- It perform only object tracking, needing the object detection to be performed elsewhere. It feeds from the detection boxes generated by `dnn_detect`. That means that the quality of the the tracking is closely related to the quality of the detection.

- SORT is a simple algorithm that uses spatial data only, and it not able to handle cases such as object occlusion. It's good enough for my use case, as I mentioned earlier.

- The filter works with the default options, so you can pass it without any arguments. In this mode, it will try to track any objects from the boxes available. You can change this behaviour by specifying the list of labels to track, for example: `track_sort=labels=person|dog|cat`. Such labels come from the ML model you used in the detection filter. It also has the options `threshold`, `min_hits` and `max_age`, which control how the tracking algorithm works, but the default values should work well on most cases.

- The filter will add the tracking information as label on a new frame side data entry of type `AV_FRAME_DATA_DETECTION_BBOXES`. It **WILL NOT** override the side data from `dnn_detect`,, meaning that the frame will have side data two entries of this type. I've created a PR that make it possible to fetch such entry [2].

- The labels in the detection boxes have the format "track:<track_num>:<track_age>", and this is not the final format. I did this way as a quick hack to have some visual information when drawing the boxes and labels with the `drawtext` and `drawbox` filters. I believe this can be improved by putting the tracking information as metadata of the `AVDetectionBBox`es, but this would on API and ABI breaking, so this is still an open question.

What has not been done so far:

I had quite a few goals in this task:

- 1: get a working and efficient implementation of the SORT algorithm.
- 2: start learning Rust again (it's been ~5 years since I used it)
- 3: learn more about the libavfilter codebase
- 4: evaluate whether Rust could work as a second language for hacking FFMpeg.

Results:

- 1: I managed to reuse lots of high quality code, available on crates (the repository of Rust packages), preventing me of needing to write hairy math heavy code. I personally suck in maths, especially linear algebra. Using the paper and the reference implementation [3] was enough, although I do not understand all the math magic. For instance, I reused an existing crate for Kalman filters that I probably would need to implement by hand, as the alternative in C would probably be using the implementation that OpenCV offers. And I am aware that it's not practical to make OpenCV a dependency of FFmpeg.

- 2: yay! Back to Rust!

- 3: I've learned more not only about avfilter, but a bit about other components as well.

- 4: I have more notes on that later, but it feels for me that Rust is natural candidate for new code in large C codebases, as it integrates quite tell, with some warts. I also have no idea whether the FFmpeg community has discussed about Rust in the codebase in the past and, if, not, why not now?

Some notes on using Rust:

In general I enjoyed using Rust in the project, and if you have a look at the code, you'll notice that I am not reusing any of the nice C macros that make a lot of stuff easier on writing new filters. That means that the Rust code looks like the expanded macro versions from C. And that's a lot of boilerplate and ugly code.

There were some reasons for that: One is that I am still learning Rust macros, and wanted to focus on getting stuff done for now. Second is that Rust has a much more powerful macro system than C does, and avoiding macros now allow me to feel all the pain of writing the manual code. Such pain, I believe, can help a set of Rust macros to "emerge" from the codebase, rather than one designing a set of macros that will probably look like the C ones, which might not be "rusty" enough. And I don't find a good practise to design APIs before having some implementation (looking at you, C++ committee).

I've been developing on Manjaro Linux and for now building FFmpeg statically with `--disable-stripping --enable-debug=3 --disable-optimizations` and the Rust code in `Debug` mode. That means slow code and static builds, which are easy to debug a profile.

Debugging is easy, as I can simply use GDB and it simply works with the Rust and C code mixed. I stil don't have pretty-printer for the Rust part, but this is probably an issue on my setup.

Profiling also works well. Even though the Rust code is in Debug mode, profiling with Hotspot/Perf shows that the tracking code is very efficient (you almost cannot see it in the flamegraph!).

Memory management is a breeze, as the standard library has generic versions of many useful containers, such as Vectors and BTrees. The algorithms there also make transforming and filtering very convenient and type safe.

You get support for unit tests for free. No hassle, no complex setup. Simply write unit tests anywhere and run them with `cargo test`.

It feels very good to get the code to work and not being afraid of things going badly (in the code which is not unsafe, of course!).

WARTS

I did not implement any wrapper on top of the avfilter private API (yay `bindgen`!), so it's used directly on the Rust code. It forces you to write the code as `unsafe` on any interaction with libav* API. Nevertheless, even on unsafe code, working on non owned data is very convenient, as you can turn almost anything into slices, which provide you with lots of convenient algorithms (map, filter, zip, etc.).

Working with C pointers is a very painful and ugly. Especially `**` and `***`. Rust is very verbose on using them in the rust side (they become things like `&*mut *mut *mut`, not really easy to reason about). Rust also does not have the `->` operator, forcing you do do stuff like ``(*foo).bar`, which is simply ugly.

Interacting with the C API is also not trivial, as in Rust one must be explicit about ownership and lifetimes, something which is done implicitely (and often wrongly) in C.

Struct members in Rust must always be explicitely initialized, even for global static variables, which C initializes with zero implicitely.

C unions. Luckily Rust supports them, but they are always unsafe.

`bindgen` does not generate wrappers for `static av_always_inline blah()` functions, as those are... inlined, so when in the need of using those, I had to simply reimplement them in Rust.

In general my impression is that Rust code is more verbose than C in "dangerous" code, but way less verbose in safe code, due to the compiler checks.

WHY? WHY? WHY?????

Ok, why do I, who never really took part on the FFmpeg community come apparently now throwing Rust on your faces? Am I saying you folks should rewrite ffmpeg in rust? I know that especially the Rust community have been involved recently in a lot of conflicts involving large C codebases, and it's not my intention to tell you what or not to do. I recognize having no authority in this group for that and I am essentially just a FFmpeg user.

My intention, first of all, was to get some stuff I needed done. I'm working on a commercial product, and developing in Rust was the quickest way I could get it done (considering my requirements). I've enjoyed a lot working in this project, and I believe my learnings can be useful for the FFmpeg community as a whole.

Demo time

Requirements: Cargo/Rust installed. I am using `1.84.0`, the latest stable, via `rustup`.

You'll need openvino, harfbuzz and freetype installed.

First of all, check out the code from the PR at [4] and compile FFmpeg with:

```sh
./configure ./configure --disable-stripping --enable-debug=3 --disable-optimizations --enable-libopenvino --enable-libharfbuzz --enable-libfreetype --enable-openssl
cargo build && make
```

I added a `--enable-rust` flag to the PR, but at the moment it does nothing :-)

Next you should download a pre-trained YOLO4 model and associated files, for perform the object detections:

```sh
pip install openvino-dev tensorflow
omz_downloader --name yolo-v4-tiny-tf
omz_converter --name yolo-v4-tiny-tf
wget https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/refs/heads/master/data/dataset_classes/coco_80cl.txt
```

Here we'll use a video from MOT Challenge 2016, [5] which is the one shown in the original SORT paper. You can use it with the command:

```sh
./ffplay https://motchallenge.net/sequenceVideos/MOT16-06-raw.webm -vf 'dnn_detect=dnn_backend=openvino:model=public/yolo-v4-tiny-tf/FP16/yolo-v4-tiny-tf.xml:input=image_input:confidence=0.1:model_type=yolov4:anchors=81&82&135&169&344&319:labels=coco_80cl.txt:async=0:nb_classes=80,track_sort=labels=person,drawbox=box_source=side_data_detection_bboxes:color=red:skip=1,drawtext=text_source=side_data_detection_bboxes:fontcolor=yellow:bordercolor=yellow:fontsize=20:fontfile=DroidSans-Bold.ttf:skip=1'
```

The `dnn_detect` options were obtained from the YOLO4 model at [6].

Please also noticed I passed the extra option `skip=1` to both the `drawtext` and the `drawbox` filters. This is to make them render the boxes information from  `track_sort` , instead of the ones from `dnn_detect`. More at [2].

I also recorded a video showing the filter in action [7].

Cheers,

Leandro


[1] https://arxiv.org/pdf/1703.07402
[2] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/10
[3] https://github.com/abewley/sort
[4] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/11
[5] https://motchallenge.net/vis/MOT16-06
[6] https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/yolo-v4-tiny-tf/README.md
[7] https://youtu.be/U_y4-NnaINg

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Leandro Santiago]

Just a correction, I messed up the referenced paper, and I actually should refer to this one:

https://arxiv.org/pdf/1602.00763

As the one I mentioned previously seems to be an extension of the SORT, but with appearance information, which is **NOT** what what the filter I implemented.

An embarrassing mistake, and I will fix it in the code and PR (but it will keep forever in the mailing list :-().

In any case, I probably need to spend more time reading papers.

Also, I hope it's clear that this work is a POC, and I am not suggesting being mainlined, at least not in the short term.

On 2/20/25 14:06, Leandro Santiago wrote:
> [insert meme here]
>
> (this will be a long e-mail)
>
> Dear FFmpeg devs,
>
> in the past days I've been experimenting hacking FFmpeg using Rust.
>
> As I am becoming more familiar with the libavfilter, and it is not a dependency for any other of the libav* libs, I decided this is a good candidate.
>
> It's also convenient as I use FFmpeg libs heavily in a commercial product, and one of the features I've been working on involves a basic multi object tracking.
>
> In my case, it does not need to be a "perfect" tracking algorithm, as I need to compromise quality of the result in exchange of performance executing in the CPU only, so most of the algorithms out there that need a GPU are out of my range.
>
> I decided then use as first experiment a filter called `track_sort` that implements the 2016 paper SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC, as known as SORT [1].
>
> The filter already works well based on the `master` branch, but the code itself is in very early stages and far from being "production ready", so please do not read the code assuming it's in its final form. It's ugly and needs lots of refactoring.
>
> I've created a PR on forgejo [4] to make it easier for others to track progress, although I use gitlab.com as my main forge.
>
> Here is a description of the filter:
>
> - It perform only object tracking, needing the object detection to be performed elsewhere. It feeds from the detection boxes generated by `dnn_detect`. That means that the quality of the the tracking is closely related to the quality of the detection.
>
> - SORT is a simple algorithm that uses spatial data only, and it not able to handle cases such as object occlusion. It's good enough for my use case, as I mentioned earlier.
>
> - The filter works with the default options, so you can pass it without any arguments. In this mode, it will try to track any objects from the boxes available. You can change this behaviour by specifying the list of labels to track, for example: `track_sort=labels=person|dog|cat`. Such labels come from the ML model you used in the detection filter. It also has the options `threshold`, `min_hits` and `max_age`, which control how the tracking algorithm works, but the default values should work well on most cases.
>
> - The filter will add the tracking information as label on a new frame side data entry of type `AV_FRAME_DATA_DETECTION_BBOXES`. It **WILL NOT** override the side data from `dnn_detect`,, meaning that the frame will have side data two entries of this type. I've created a PR that make it possible to fetch such entry [2].
>
> - The labels in the detection boxes have the format "track:<track_num>:<track_age>", and this is not the final format. I did this way as a quick hack to have some visual information when drawing the boxes and labels with the `drawtext` and `drawbox` filters. I believe this can be improved by putting the tracking information as metadata of the `AVDetectionBBox`es, but this would on API and ABI breaking, so this is still an open question.
>
> What has not been done so far:
>
> I had quite a few goals in this task:
>
> - 1: get a working and efficient implementation of the SORT algorithm.
> - 2: start learning Rust again (it's been ~5 years since I used it)
> - 3: learn more about the libavfilter codebase
> - 4: evaluate whether Rust could work as a second language for hacking FFMpeg.
>
> Results:
>
> - 1: I managed to reuse lots of high quality code, available on crates (the repository of Rust packages), preventing me of needing to write hairy math heavy code. I personally suck in maths, especially linear algebra. Using the paper and the reference implementation [3] was enough, although I do not understand all the math magic. For instance, I reused an existing crate for Kalman filters that I probably would need to implement by hand, as the alternative in C would probably be using the implementation that OpenCV offers. And I am aware that it's not practical to make OpenCV a dependency of FFmpeg.
>
> - 2: yay! Back to Rust!
>
> - 3: I've learned more not only about avfilter, but a bit about other components as well.
>
> - 4: I have more notes on that later, but it feels for me that Rust is natural candidate for new code in large C codebases, as it integrates quite tell, with some warts. I also have no idea whether the FFmpeg community has discussed about Rust in the codebase in the past and, if, not, why not now?
>
> Some notes on using Rust:
>
> In general I enjoyed using Rust in the project, and if you have a look at the code, you'll notice that I am not reusing any of the nice C macros that make a lot of stuff easier on writing new filters. That means that the Rust code looks like the expanded macro versions from C. And that's a lot of boilerplate and ugly code.
>
> There were some reasons for that: One is that I am still learning Rust macros, and wanted to focus on getting stuff done for now. Second is that Rust has a much more powerful macro system than C does, and avoiding macros now allow me to feel all the pain of writing the manual code. Such pain, I believe, can help a set of Rust macros to "emerge" from the codebase, rather than one designing a set of macros that will probably look like the C ones, which might not be "rusty" enough. And I don't find a good practise to design APIs before having some implementation (looking at you, C++ committee).
>
> I've been developing on Manjaro Linux and for now building FFmpeg statically with `--disable-stripping --enable-debug=3 --disable-optimizations` and the Rust code in `Debug` mode. That means slow code and static builds, which are easy to debug a profile.
>
> Debugging is easy, as I can simply use GDB and it simply works with the Rust and C code mixed. I stil don't have pretty-printer for the Rust part, but this is probably an issue on my setup.
>
> Profiling also works well. Even though the Rust code is in Debug mode, profiling with Hotspot/Perf shows that the tracking code is very efficient (you almost cannot see it in the flamegraph!).
>
> Memory management is a breeze, as the standard library has generic versions of many useful containers, such as Vectors and BTrees. The algorithms there also make transforming and filtering very convenient and type safe.
>
> You get support for unit tests for free. No hassle, no complex setup. Simply write unit tests anywhere and run them with `cargo test`.
>
> It feels very good to get the code to work and not being afraid of things going badly (in the code which is not unsafe, of course!).
>
> WARTS
>
> I did not implement any wrapper on top of the avfilter private API (yay `bindgen`!), so it's used directly on the Rust code. It forces you to write the code as `unsafe` on any interaction with libav* API. Nevertheless, even on unsafe code, working on non owned data is very convenient, as you can turn almost anything into slices, which provide you with lots of convenient algorithms (map, filter, zip, etc.).
>
> Working with C pointers is a very painful and ugly. Especially `**` and `***`. Rust is very verbose on using them in the rust side (they become things like `&*mut *mut *mut`, not really easy to reason about). Rust also does not have the `->` operator, forcing you do do stuff like ``(*foo).bar`, which is simply ugly.
>
> Interacting with the C API is also not trivial, as in Rust one must be explicit about ownership and lifetimes, something which is done implicitely (and often wrongly) in C.
>
> Struct members in Rust must always be explicitely initialized, even for global static variables, which C initializes with zero implicitely.
>
> C unions. Luckily Rust supports them, but they are always unsafe.
>
> `bindgen` does not generate wrappers for `static av_always_inline blah()` functions, as those are... inlined, so when in the need of using those, I had to simply reimplement them in Rust.
>
> In general my impression is that Rust code is more verbose than C in "dangerous" code, but way less verbose in safe code, due to the compiler checks.
>
> WHY? WHY? WHY?????
>
> Ok, why do I, who never really took part on the FFmpeg community come apparently now throwing Rust on your faces? Am I saying you folks should rewrite ffmpeg in rust? I know that especially the Rust community have been involved recently in a lot of conflicts involving large C codebases, and it's not my intention to tell you what or not to do. I recognize having no authority in this group for that and I am essentially just a FFmpeg user.
>
> My intention, first of all, was to get some stuff I needed done. I'm working on a commercial product, and developing in Rust was the quickest way I could get it done (considering my requirements). I've enjoyed a lot working in this project, and I believe my learnings can be useful for the FFmpeg community as a whole.
>
> Demo time
>
> Requirements: Cargo/Rust installed. I am using `1.84.0`, the latest stable, via `rustup`.
>
> You'll need openvino, harfbuzz and freetype installed.
>
> First of all, check out the code from the PR at [4] and compile FFmpeg with:
>
> ```sh
> ./configure ./configure --disable-stripping --enable-debug=3 --disable-optimizations --enable-libopenvino --enable-libharfbuzz --enable-libfreetype --enable-openssl
> cargo build && make
> ```
>
> I added a `--enable-rust` flag to the PR, but at the moment it does nothing :-)
>
> Next you should download a pre-trained YOLO4 model and associated files, for perform the object detections:
>
> ```sh
> pip install openvino-dev tensorflow
> omz_downloader --name yolo-v4-tiny-tf
> omz_converter --name yolo-v4-tiny-tf
> wget https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/refs/heads/master/data/dataset_classes/coco_80cl.txt
> ```
>
> Here we'll use a video from MOT Challenge 2016, [5] which is the one shown in the original SORT paper. You can use it with the command:
>
> ```sh
> ./ffplay https://motchallenge.net/sequenceVideos/MOT16-06-raw.webm -vf 'dnn_detect=dnn_backend=openvino:model=public/yolo-v4-tiny-tf/FP16/yolo-v4-tiny-tf.xml:input=image_input:confidence=0.1:model_type=yolov4:anchors=81&82&135&169&344&319:labels=coco_80cl.txt:async=0:nb_classes=80,track_sort=labels=person,drawbox=box_source=side_data_detection_bboxes:color=red:skip=1,drawtext=text_source=side_data_detection_bboxes:fontcolor=yellow:bordercolor=yellow:fontsize=20:fontfile=DroidSans-Bold.ttf:skip=1'
> ```
>
> The `dnn_detect` options were obtained from the YOLO4 model at [6].
>
> Please also noticed I passed the extra option `skip=1` to both the `drawtext` and the `drawbox` filters. This is to make them render the boxes information from  `track_sort` , instead of the ones from `dnn_detect`. More at [2].
>
> I also recorded a video showing the filter in action [7].
>
> Cheers,
>
> Leandro
>
>
> [1] https://arxiv.org/pdf/1703.07402
> [2] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/10
> [3] https://github.com/abewley/sort
> [4] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/11
> [5] https://motchallenge.net/vis/MOT16-06
> [6] https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/yolo-v4-tiny-tf/README.md
> [7] https://youtu.be/U_y4-NnaINg
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 791 bytes --]

Hi

On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> [insert meme here]
[...]
> I also recorded a video showing the filter in action [7].
[...
> [7] https://youtu.be/U_y4-NnaINg

cool, it doesnt detect everyone though

also i think this shows how useful a plugin framework would be for ffmpeg

with plugins everyone could use,test and contribute to this today.
without plugins, this needs to be merged in ffmpeg git master. (which
will take some time i suspect)

Rust support in FFmpeg is an interresting idea though, iam in favor of
supporting more "safe" languages

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Nations do behave wisely once they have exhausted all other alternatives. 
-- Abba Eban

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Leandro Santiago]

On 2/20/25 23:49, Michael Niedermayer wrote:
> Hi
>
> On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
>> [insert meme here]
> [...]
>> I also recorded a video showing the filter in action [7].
> [...
>> [7] https://youtu.be/U_y4-NnaINg
> cool, it doesnt detect everyone though

Yes, this has two reasons: one is the quality of the detection depends on the model used. I use in the demo a yolov4-tiny that is very fast but not really accurate, leading to many people not being tracked. There can be bugs in the track_sort code as well.

The other reason is that the `track_sort` at the moment ignores all the detected boxes which have a position, but no area (they seem to be detected as simple points by `dnn_detect`), which are useless for now, as the SORT implementation at the moment depends only the boxes dimensions for computing IOU (intersection over union).

I could think though on extending the code to "fake an area" of a box when it's unknown, which might help to improve the tracking.

>
> also i think this shows how useful a plugin framework would be for ffmpeg
>
> with plugins everyone could use,test and contribute to this today.
> without plugins, this needs to be merged in ffmpeg git master. (which
> will take some time i suspect)

Yes, it could even open space for webassembly based plugins, which would be less of a hassle regarding ABI and potentially be cross platform, although I foresee some challenges on getting wasm runtime to share data with the host without copying, especially if the data in the GPU.

>
> Rust support in FFmpeg is an interresting idea though, iam in favor of
> supporting more "safe" languages
Great to hear that!
>
> thx
>
> [...]
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Tomas Härdin]

tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> Hi
> 
> On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > [insert meme here]
> [...]
> > I also recorded a video showing the filter in action [7].
> [...
> > [7] https://youtu.be/U_y4-NnaINg
> 
> cool, it doesnt detect everyone though
> 
> also i think this shows how useful a plugin framework would be for
> ffmpeg
> 
> with plugins everyone could use,test and contribute to this today.
> without plugins, this needs to be merged in ffmpeg git master. (which
> will take some time i suspect)

Have we not gone over and rejected plugins many times? I recall points
about them encouraging proliferation of proprietary code. I also feel
this project is increasingly forgetting about the power of the UNIX
pipe.

/Tomas

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Tomas Härdin
> Sent: Freitag, 21. Februar 2025 10:02
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> 
> tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > Hi
> >
> > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > [insert meme here]
> > [...]
> > > I also recorded a video showing the filter in action [7].
> > [...
> > > [7] https://youtu.be/U_y4-NnaINg
> >
> > cool, it doesnt detect everyone though
> >
> > also i think this shows how useful a plugin framework would be for
> > ffmpeg
> >
> > with plugins everyone could use,test and contribute to this today.
> > without plugins, this needs to be merged in ffmpeg git master. (which
> > will take some time i suspect)

Hi Thomas,

> Have we not gone over and rejected plugins many times? I recall points
> about them encouraging proliferation of proprietary code. 

I haven't followed earlier discussions, what's bad about that (in a nutshell)?

> I also feel
> this project is increasingly forgetting about the power of the UNIX
> pipe.

That "power" doesn't exist on all platforms, and even on UNIX: does that "power" go that far that it works without memcopy? E.g. - a filter needs to paint just a few pixels on a video frame - doesn't the full uncompressed frame need to be piped forth and back?
Also, filters often do and require more than just a range of data to do their work.

sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2091 bytes --]

Hi

On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > Hi
> > 
> > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > [insert meme here]
> > [...]
> > > I also recorded a video showing the filter in action [7].
> > [...
> > > [7] https://youtu.be/U_y4-NnaINg
> > 
> > cool, it doesnt detect everyone though
> > 
> > also i think this shows how useful a plugin framework would be for
> > ffmpeg
> > 
> > with plugins everyone could use,test and contribute to this today.
> > without plugins, this needs to be merged in ffmpeg git master. (which
> > will take some time i suspect)
> 
> Have we not gone over and rejected plugins many times? I recall points

no
there was no formal and no public informal vote that i remember.

ive raised the issue with plugins many times. Because it would
allow people and myself to contribute more complex features and
end the stagnation of FFmpeg.
Think of AI filters (like text to image, automated language translation of subtiltes,
image to metadata, audio to subtitle, upscaling, and infinite more ideas)
Neural network based image, video and audio compression
thats both local neural networks and also using some server based APIs

In general innovation needs the innovators to be in charge.
Plugins arent the only way to achieve that of course.


> about them encouraging proliferation of proprietary code. I also feel

Thats nonsense, you can make the plugin system GPL or AGPL if you like.


> this project is increasingly forgetting about the power of the UNIX
> pipe.

The unix pipe gave us gzip and tar. And the world gave it the middle
finger and used pkzip because pkzip did what everyone needs, and the
pipe did not

PS: if someone sends me a pull request that cleanly adds plugin support to
FFmpeg, i will merge it.

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

No great genius has ever existed without some touch of madness. -- Aristotle

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Michael Niedermayer
> Sent: Freitag, 21. Februar 2025 14:22
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> 
> Hi
> 
> On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > Hi
> > >
> > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > [insert meme here]
> > > [...]
> > > > I also recorded a video showing the filter in action [7].
> > > [...
> > > > [7] https://youtu.be/U_y4-NnaINg
> > >
> > > cool, it doesnt detect everyone though
> > >
> > > also i think this shows how useful a plugin framework would be for
> > > ffmpeg
> > >
> > > with plugins everyone could use,test and contribute to this today.
> > > without plugins, this needs to be merged in ffmpeg git master. (which
> > > will take some time i suspect)
> >
> > Have we not gone over and rejected plugins many times? I recall points
> 
> no
> there was no formal and no public informal vote that i remember.
> 
> ive raised the issue with plugins many times. Because it would
> allow people and myself to contribute more complex features and
> end the stagnation of FFmpeg.

Yup, that's exactly one of the reasons why other projects like GStreamer
have gained that much popularity. It might not have happened when
ffmpeg would have been more open and extensible in the first place.

> Think of AI filters (like text to image, automated language translation of
> subtiltes,
> image to metadata, audio to subtitle, upscaling, and infinite more ideas)
> Neural network based image, video and audio compression
> thats both local neural networks and also using some server based APIs

I had similar things in mind already. The graphicsubs2text filter kind of uses
AI already, but it was only possible because it does it through a standard 
lib (tesseract). But as soon as it comes to anything which requires using 
an API that is vendor-specific, that's something which nobody will want 
to have in ffmpeg, and it might also change rapidly in a way that it's already
outdated when its containing ffmpeg version gets packaged by a distro.


> In general innovation needs the innovators to be in charge.
> Plugins arent the only way to achieve that of course.

Totally agree. Saying that we have no solution for these things is a path 
to becoming Irrelevant in the future.


What do you people think about the feasibility of a compile-time plugin
mechanism?
Given that on most platforms (except Windows), you can't easily mix 
binaries from different origins (i.e. built differently), it would be much
easier when plugin binaries could be compiled as part of building ffmpeg
rather than having to set up and maintain separate builds for each plugin.

sw


_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Kieran Kunhya via ffmpeg-devel]

On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-hotmail.com@ffmpeg.org>
wrote:

>
>
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Michael Niedermayer
> > Sent: Freitag, 21. Februar 2025 14:22
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel@ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> >
> > Hi
> >
> > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > Hi
> > > >
> > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > > [insert meme here]
> > > > [...]
> > > > > I also recorded a video showing the filter in action [7].
> > > > [...
> > > > > [7] https://youtu.be/U_y4-NnaINg
> > > >
> > > > cool, it doesnt detect everyone though
> > > >
> > > > also i think this shows how useful a plugin framework would be for
> > > > ffmpeg
> > > >
> > > > with plugins everyone could use,test and contribute to this today.
> > > > without plugins, this needs to be merged in ffmpeg git master. (which
> > > > will take some time i suspect)
> > >
> > > Have we not gone over and rejected plugins many times? I recall points
> >
> > no
> > there was no formal and no public informal vote that i remember.
> >
> > ive raised the issue with plugins many times. Because it would
> > allow people and myself to contribute more complex features and
> > end the stagnation of FFmpeg.
>
> Yup, that's exactly one of the reasons why other projects like GStreamer
> have gained that much popularity. It might not have happened when
> ffmpeg would have been more open and extensible in the first place.
>

It's quite the opposite. Gstreamer is successful in its vertical because it
allows easy inclusion of vendor binary blobs. This is not "open and
extensible".

Don't confuse the history of avisynth in the 2000s with today's reality.

Kieran

>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Kieran Kunhya via ffmpeg-devel
> Sent: Freitag, 21. Februar 2025 15:53
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Cc: Kieran Kunhya <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> 
> On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
> hotmail.com@ffmpeg.org>
> wrote:
> 
> >
> >
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > Michael Niedermayer
> > > Sent: Freitag, 21. Februar 2025 14:22
> > > To: FFmpeg development discussions and patches <ffmpeg-
> > > devel@ffmpeg.org>
> > > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> > >
> > > Hi
> > >
> > > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > > Hi
> > > > >
> > > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > > > [insert meme here]
> > > > > [...]
> > > > > > I also recorded a video showing the filter in action [7].
> > > > > [...
> > > > > > [7] https://youtu.be/U_y4-NnaINg
> > > > >
> > > > > cool, it doesnt detect everyone though
> > > > >
> > > > > also i think this shows how useful a plugin framework would be for
> > > > > ffmpeg
> > > > >
> > > > > with plugins everyone could use,test and contribute to this today.
> > > > > without plugins, this needs to be merged in ffmpeg git master. (which
> > > > > will take some time i suspect)
> > > >
> > > > Have we not gone over and rejected plugins many times? I recall points
> > >
> > > no
> > > there was no formal and no public informal vote that i remember.
> > >
> > > ive raised the issue with plugins many times. Because it would
> > > allow people and myself to contribute more complex features and
> > > end the stagnation of FFmpeg.
> >
> > Yup, that's exactly one of the reasons why other projects like GStreamer
> > have gained that much popularity. It might not have happened when
> > ffmpeg would have been more open and extensible in the first place.
> >
> 
> It's quite the opposite. Gstreamer is successful in its vertical because it
> allows easy inclusion of vendor binary blobs. This is not "open and
> extensible".

Open means it's extensible for everybody, including vendors. I fail to see
what's bad about it. Do we have a fight against everything commercial?

What are you afraid of to happen?

Just wanna understand the position, thanks.
sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Kieran Kunhya via ffmpeg-devel]

On Fri, 21 Feb 2025, 15:02 Soft Works, <softworkz@hotmail.com> wrote:

>
>
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Kieran Kunhya via ffmpeg-devel
> > Sent: Freitag, 21. Februar 2025 15:53
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel@ffmpeg.org>
> > Cc: Kieran Kunhya <kieran618@googlemail.com>
> > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> >
> > On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
> > hotmail.com@ffmpeg.org>
> > wrote:
> >
> > >
> > >
> > > > -----Original Message-----
> > > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > > Michael Niedermayer
> > > > Sent: Freitag, 21. Februar 2025 14:22
> > > > To: FFmpeg development discussions and patches <ffmpeg-
> > > > devel@ffmpeg.org>
> > > > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> > > >
> > > > Hi
> > > >
> > > > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > > > Hi
> > > > > >
> > > > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > > > > [insert meme here]
> > > > > > [...]
> > > > > > > I also recorded a video showing the filter in action [7].
> > > > > > [...
> > > > > > > [7] https://youtu.be/U_y4-NnaINg
> > > > > >
> > > > > > cool, it doesnt detect everyone though
> > > > > >
> > > > > > also i think this shows how useful a plugin framework would be
> for
> > > > > > ffmpeg
> > > > > >
> > > > > > with plugins everyone could use,test and contribute to this
> today.
> > > > > > without plugins, this needs to be merged in ffmpeg git master.
> (which
> > > > > > will take some time i suspect)
> > > > >
> > > > > Have we not gone over and rejected plugins many times? I recall
> points
> > > >
> > > > no
> > > > there was no formal and no public informal vote that i remember.
> > > >
> > > > ive raised the issue with plugins many times. Because it would
> > > > allow people and myself to contribute more complex features and
> > > > end the stagnation of FFmpeg.
> > >
> > > Yup, that's exactly one of the reasons why other projects like
> GStreamer
> > > have gained that much popularity. It might not have happened when
> > > ffmpeg would have been more open and extensible in the first place.
> > >
> >
> > It's quite the opposite. Gstreamer is successful in its vertical because
> it
> > allows easy inclusion of vendor binary blobs. This is not "open and
> > extensible".
>
> Open means it's extensible for everybody, including vendors. I fail to see
> what's bad about it. Do we have a fight against everything commercial?
>
> What are you afraid of to happen?
>
> Just wanna understand the position, thanks.
> sw
>

What will happen is all the binary blob vendors who have to currently ship
a custom ffmpeg and tell their users to compile from source can just ship
their blob as a plugin.

FFmpeg will get all the bug reports from crashes (this is why the kernel
had to add the "tainted") flag.

I have a deep dislike for binary blobs.

Thinking the avisynth filter development of the 2000s will reappear when
ffmpeg gets plugins is wishful to say the least.

Kieran

>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Soft Works]

From: Kieran Kunhya <kieran618@googlemail.com>
Sent: Freitag, 21. Februar 2025 20:27
To: Soft Works <softworkz@hotmail.com>
Cc: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Subject: Re: [FFmpeg-devel] I've written a filter in Rust


On Fri, 21 Feb 2025, 15:02 Soft Works, <softworkz@hotmail.com<mailto:softworkz@hotmail.com>> wrote:


> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ffmpeg.org>> On Behalf Of
> Kieran Kunhya via ffmpeg-devel
> Sent: Freitag, 21. Februar 2025 15:53
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
> Cc: Kieran Kunhya <kieran618@googlemail.com<mailto:kieran618@googlemail.com>>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>
> On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
> hotmail.com@ffmpeg.org<mailto:hotmail.com@ffmpeg.org>>
> wrote:
>
> >
> >
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ffmpeg.org>> On Behalf Of
> > > Michael Niedermayer
> > > Sent: Freitag, 21. Februar 2025 14:22
> > > To: FFmpeg development discussions and patches <ffmpeg-
> > > devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
> > > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> > >
> > > Hi
> > >
> > > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > > Hi
> > > > >
> > > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > > > [insert meme here]
> > > > > [...]
> > > > > > I also recorded a video showing the filter in action [7].
> > > > > [...
> > > > > > [7] https://youtu.be/U_y4-NnaINg
> > > > >
> > > > > cool, it doesnt detect everyone though
> > > > >
> > > > > also i think this shows how useful a plugin framework would be for
> > > > > ffmpeg
> > > > >
> > > > > with plugins everyone could use,test and contribute to this today.
> > > > > without plugins, this needs to be merged in ffmpeg git master. (which
> > > > > will take some time i suspect)
> > > >
> > > > Have we not gone over and rejected plugins many times? I recall points
> > >
> > > no
> > > there was no formal and no public informal vote that i remember.
> > >
> > > ive raised the issue with plugins many times. Because it would
> > > allow people and myself to contribute more complex features and
> > > end the stagnation of FFmpeg.
> >
> > Yup, that's exactly one of the reasons why other projects like GStreamer
> > have gained that much popularity. It might not have happened when
> > ffmpeg would have been more open and extensible in the first place.
> >
>
> It's quite the opposite. Gstreamer is successful in its vertical because it
> allows easy inclusion of vendor binary blobs. This is not "open and
> extensible".

Open means it's extensible for everybody, including vendors. I fail to see
what's bad about it. Do we have a fight against everything commercial?

What are you afraid of to happen?

Just wanna understand the position, thanks.
sw


Hi Kieran,

What will happen is all the binary blob vendors who have to currently ship a custom ffmpeg and tell their users to compile from source can just ship their blob as a plugin.

We currently have 200 reasons (commits) for needing to ship a custom ffmpeg and a plugin model would not change that.
It’s hard for everybody to get even good changes in ffmpeg and there are other changes for which there’s clearly no place in the ffmpeg codebase. There should be an equalization for this situation.
Also, I think it’s a bit unfair and oversimplifying to blame all vendors’ main interest being to ship binaries as extension just for the sake of keeping them closed source – even though those cases might occur.


FFmpeg will get all the bug reports from crashes (this is why the kernel had to add the "tainted") flag.

Is this really a big problem? It seems to me that such cases can be easily identified (the presence of loaded plugins could also be indicated in the version header).

I have a deep dislike for binary blobs.

That I can understand. But on one side rejecting a lot of source code and also not allowing compiled code for extending is a pretty fencing policy es well.

What do you think about a compile-time plugin model?

PS:I don’t know the avisynth back story...

Thanks
sw


_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Tomas Härdin]

fre 2025-02-21 klockan 20:10 +0000 skrev Soft Works:
> 
> 
> From: Kieran Kunhya <kieran618@googlemail.com>
> Sent: Freitag, 21. Februar 2025 20:27
> To: Soft Works <softworkz@hotmail.com>
> Cc: FFmpeg development discussions and patches
> <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> 
> 
> On Fri, 21 Feb 2025, 15:02 Soft Works,
> <softworkz@hotmail.com<mailto:softworkz@hotmail.com>> wrote:
> 
> 
> > -----Original Message-----
> > From: ffmpeg-devel
> > <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ffmpeg
> > .org>> On Behalf Of
> > Kieran Kunhya via ffmpeg-devel
> > Sent: Freitag, 21. Februar 2025 15:53
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
> > Cc: Kieran Kunhya
> > <kieran618@googlemail.com<mailto:kieran618@googlemail.com>>
> > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> > 
> > On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
> > hotmail.com@ffmpeg.org<mailto:hotmail.com@ffmpeg.org>>
> > wrote:
> > 
> > > 
> > > 
> > > > -----Original Message-----
> > > > From: ffmpeg-devel
> > > > <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ff
> > > > mpeg.org>> On Behalf Of
> > > > Michael Niedermayer
> > > > Sent: Freitag, 21. Februar 2025 14:22
> > > > To: FFmpeg development discussions and patches <ffmpeg-
> > > > devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
> > > > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> > > > 
> > > > Hi
> > > > 
> > > > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > > > Hi
> > > > > > 
> > > > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago
> > > > > > wrote:
> > > > > > > [insert meme here]
> > > > > > [...]
> > > > > > > I also recorded a video showing the filter in action [7].
> > > > > > [...
> > > > > > > [7] https://youtu.be/U_y4-NnaINg
> > > > > > 
> > > > > > cool, it doesnt detect everyone though
> > > > > > 
> > > > > > also i think this shows how useful a plugin framework would
> > > > > > be for
> > > > > > ffmpeg
> > > > > > 
> > > > > > with plugins everyone could use,test and contribute to this
> > > > > > today.
> > > > > > without plugins, this needs to be merged in ffmpeg git
> > > > > > master. (which
> > > > > > will take some time i suspect)
> > > > > 
> > > > > Have we not gone over and rejected plugins many times? I
> > > > > recall points
> > > > 
> > > > no
> > > > there was no formal and no public informal vote that i
> > > > remember.
> > > > 
> > > > ive raised the issue with plugins many times. Because it would
> > > > allow people and myself to contribute more complex features and
> > > > end the stagnation of FFmpeg.
> > > 
> > > Yup, that's exactly one of the reasons why other projects like
> > > GStreamer
> > > have gained that much popularity. It might not have happened when
> > > ffmpeg would have been more open and extensible in the first
> > > place.
> > > 
> > 
> > It's quite the opposite. Gstreamer is successful in its vertical
> > because it
> > allows easy inclusion of vendor binary blobs. This is not "open and
> > extensible".
> 
> Open means it's extensible for everybody, including vendors. I fail
> to see
> what's bad about it. Do we have a fight against everything
> commercial?
> 
> What are you afraid of to happen?

I think Kieran already expressed quite well the issue:

1) it encourages proprietary blobs
2) it creates pressure on us to maintain compatibility with said blobs

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Zhao Zhili]

> On Feb 26, 2025, at 21:50, Tomas Härdin <git@haerdin.se> wrote:
> 
> fre 2025-02-21 klockan 20:10 +0000 skrev Soft Works:
>> 
>> 
>> From: Kieran Kunhya <kieran618@googlemail.com>
>> Sent: Freitag, 21. Februar 2025 20:27
>> To: Soft Works <softworkz@hotmail.com>
>> Cc: FFmpeg development discussions and patches
>> <ffmpeg-devel@ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>> 
>> 
>> On Fri, 21 Feb 2025, 15:02 Soft Works,
>> <softworkz@hotmail.com<mailto:softworkz@hotmail.com>> wrote:
>> 
>> 
>>> -----Original Message-----
>>> From: ffmpeg-devel
>>> <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ffmpeg
>>> .org>> On Behalf Of
>>> Kieran Kunhya via ffmpeg-devel
>>> Sent: Freitag, 21. Februar 2025 15:53
>>> To: FFmpeg development discussions and patches <ffmpeg-
>>> devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
>>> Cc: Kieran Kunhya
>>> <kieran618@googlemail.com<mailto:kieran618@googlemail.com>>
>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>> 
>>> On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
>>> hotmail.com@ffmpeg.org<mailto:hotmail.com@ffmpeg.org>>
>>> wrote:
>>> 
>>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: ffmpeg-devel
>>>>> <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ff
>>>>> mpeg.org>> On Behalf Of
>>>>> Michael Niedermayer
>>>>> Sent: Freitag, 21. Februar 2025 14:22
>>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>>> devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
>>>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>>>> 
>>>>> Hi
>>>>> 
>>>>> On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
>>>>>> tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
>>>>>>> Hi
>>>>>>> 
>>>>>>> On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago
>>>>>>> wrote:
>>>>>>>> [insert meme here]
>>>>>>> [...]
>>>>>>>> I also recorded a video showing the filter in action [7].
>>>>>>> [...
>>>>>>>> [7] https://youtu.be/U_y4-NnaINg
>>>>>>> 
>>>>>>> cool, it doesnt detect everyone though
>>>>>>> 
>>>>>>> also i think this shows how useful a plugin framework would
>>>>>>> be for
>>>>>>> ffmpeg
>>>>>>> 
>>>>>>> with plugins everyone could use,test and contribute to this
>>>>>>> today.
>>>>>>> without plugins, this needs to be merged in ffmpeg git
>>>>>>> master. (which
>>>>>>> will take some time i suspect)
>>>>>> 
>>>>>> Have we not gone over and rejected plugins many times? I
>>>>>> recall points
>>>>> 
>>>>> no
>>>>> there was no formal and no public informal vote that i
>>>>> remember.
>>>>> 
>>>>> ive raised the issue with plugins many times. Because it would
>>>>> allow people and myself to contribute more complex features and
>>>>> end the stagnation of FFmpeg.
>>>> 
>>>> Yup, that's exactly one of the reasons why other projects like
>>>> GStreamer
>>>> have gained that much popularity. It might not have happened when
>>>> ffmpeg would have been more open and extensible in the first
>>>> place.
>>>> 
>>> 
>>> It's quite the opposite. Gstreamer is successful in its vertical
>>> because it
>>> allows easy inclusion of vendor binary blobs. This is not "open and
>>> extensible".
>> 
>> Open means it's extensible for everybody, including vendors. I fail
>> to see
>> what's bad about it. Do we have a fight against everything
>> commercial?
>> 
>> What are you afraid of to happen?
> 
> I think Kieran already expressed quite well the issue:
> 
> 1) it encourages proprietary blobs
> 2) it creates pressure on us to maintain compatibility with said blobs

The first issue has a solution: use gpl or more strict license for the plugin
interface support, like obs-studio.

For the second issue, we can just not give a stable API/ABI guarantee for
the plugin interface, for example, reject to load a plugin if the version doesn’t
match.

I have some feature in mind which suit for a plugin but not for a builtin implementation.
For example, different AI model have different pre- and post-process. Current
dnn in libavfilter only support some particular model like yolov4. We can’t write some
code for each model, which can be outdated soon.

VLC and obs-studio are both good example of plugin based frameworks.

> 
> /Tomas
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org <mailto:ffmpeg-devel@ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org <mailto:ffmpeg-devel-request@ffmpeg.org> with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Rémi Denis-Courmont]

Hi,

Le 26 février 2025 16:18:20 GMT+02:00, Zhao Zhili <quinkblack-at-foxmail.com@ffmpeg.org> a écrit :
>
>
>> On Feb 26, 2025, at 21:50, Tomas Härdin <git@haerdin.se> wrote:
>> 
>> fre 2025-02-21 klockan 20:10 +0000 skrev Soft Works:
>>> 
>>> 
>>> From: Kieran Kunhya <kieran618@googlemail.com>
>>> Sent: Freitag, 21. Februar 2025 20:27
>>> To: Soft Works <softworkz@hotmail.com>
>>> Cc: FFmpeg development discussions and patches
>>> <ffmpeg-devel@ffmpeg.org>
>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>> 
>>> 
>>> On Fri, 21 Feb 2025, 15:02 Soft Works,
>>> <softworkz@hotmail.com<mailto:softworkz@hotmail.com>> wrote:
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: ffmpeg-devel
>>>> <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ffmpeg
>>>> .org>> On Behalf Of
>>>> Kieran Kunhya via ffmpeg-devel
>>>> Sent: Freitag, 21. Februar 2025 15:53
>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>> devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
>>>> Cc: Kieran Kunhya
>>>> <kieran618@googlemail.com<mailto:kieran618@googlemail.com>>
>>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>>> 
>>>> On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
>>>> hotmail.com@ffmpeg.org<mailto:hotmail.com@ffmpeg.org>>
>>>> wrote:
>>>> 
>>>>> 
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: ffmpeg-devel
>>>>>> <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ff
>>>>>> mpeg.org>> On Behalf Of
>>>>>> Michael Niedermayer
>>>>>> Sent: Freitag, 21. Februar 2025 14:22
>>>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>>>> devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
>>>>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>>>>> 
>>>>>> Hi
>>>>>> 
>>>>>> On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
>>>>>>> tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
>>>>>>>> Hi
>>>>>>>> 
>>>>>>>> On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago
>>>>>>>> wrote:
>>>>>>>>> [insert meme here]
>>>>>>>> [...]
>>>>>>>>> I also recorded a video showing the filter in action [7].
>>>>>>>> [...
>>>>>>>>> [7] https://youtu.be/U_y4-NnaINg
>>>>>>>> 
>>>>>>>> cool, it doesnt detect everyone though
>>>>>>>> 
>>>>>>>> also i think this shows how useful a plugin framework would
>>>>>>>> be for
>>>>>>>> ffmpeg
>>>>>>>> 
>>>>>>>> with plugins everyone could use,test and contribute to this
>>>>>>>> today.
>>>>>>>> without plugins, this needs to be merged in ffmpeg git
>>>>>>>> master. (which
>>>>>>>> will take some time i suspect)
>>>>>>> 
>>>>>>> Have we not gone over and rejected plugins many times? I
>>>>>>> recall points
>>>>>> 
>>>>>> no
>>>>>> there was no formal and no public informal vote that i
>>>>>> remember.
>>>>>> 
>>>>>> ive raised the issue with plugins many times. Because it would
>>>>>> allow people and myself to contribute more complex features and
>>>>>> end the stagnation of FFmpeg.
>>>>> 
>>>>> Yup, that's exactly one of the reasons why other projects like
>>>>> GStreamer
>>>>> have gained that much popularity. It might not have happened when
>>>>> ffmpeg would have been more open and extensible in the first
>>>>> place.
>>>>> 
>>>> 
>>>> It's quite the opposite. Gstreamer is successful in its vertical
>>>> because it
>>>> allows easy inclusion of vendor binary blobs. This is not "open and
>>>> extensible".
>>> 
>>> Open means it's extensible for everybody, including vendors. I fail
>>> to see
>>> what's bad about it. Do we have a fight against everything
>>> commercial?
>>> 
>>> What are you afraid of to happen?
>> 
>> I think Kieran already expressed quite well the issue:
>> 
>> 1) it encourages proprietary blobs
>> 2) it creates pressure on us to maintain compatibility with said blobs
>
>The first issue has a solution: use gpl or more strict license for the plugin
>interface support, like obs-studio.
>
>For the second issue, we can just not give a stable API/ABI guarantee for
>the plugin interface, for example, reject to load a plugin if the version doesn’t
>match.
>
>I have some feature in mind which suit for a plugin but not for a builtin implementation.
>For example, different AI model have different pre- and post-process. Current
>dnn in libavfilter only support some particular model like yolov4. We can’t write some
>code for each model, which can be outdated soon.
>
>VLC and obs-studio are both good example of plugin based frameworks.

Or the Linux kernel.

Though VLC does not have that "unofficial" plugins, which aren't backports of official plugins from a newer version. So I'm not convinced that that's a good example.

Even then proprietary vendors still managed to screw it up by failing to release new versions of their binary blobs for new versions of (Windows) VLC.

In my opinion, VLC plugins are more about reducing binary load times and load sizes by breaking the code up, and helping Linux distro break dependencies down, than about enabling external (proprietary or not) plugins.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Zhao Zhili]

> On Feb 26, 2025, at 23:32, Rémi Denis-Courmont <remi@remlab.net> wrote:
> 
> Hi,
> 
> Le 26 février 2025 16:18:20 GMT+02:00, Zhao Zhili <quinkblack-at-foxmail.com@ffmpeg.org> a écrit :
>> 
>> 
>>> On Feb 26, 2025, at 21:50, Tomas Härdin <git@haerdin.se> wrote:
>>> 
>>> fre 2025-02-21 klockan 20:10 +0000 skrev Soft Works:
>>>> 
>>>> 
>>>> From: Kieran Kunhya <kieran618@googlemail.com>
>>>> Sent: Freitag, 21. Februar 2025 20:27
>>>> To: Soft Works <softworkz@hotmail.com>
>>>> Cc: FFmpeg development discussions and patches
>>>> <ffmpeg-devel@ffmpeg.org>
>>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>>> 
>>>> 
>>>> On Fri, 21 Feb 2025, 15:02 Soft Works,
>>>> <softworkz@hotmail.com<mailto:softworkz@hotmail.com>> wrote:
>>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: ffmpeg-devel
>>>>> <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ffmpeg
>>>>> .org>> On Behalf Of
>>>>> Kieran Kunhya via ffmpeg-devel
>>>>> Sent: Freitag, 21. Februar 2025 15:53
>>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>>> devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
>>>>> Cc: Kieran Kunhya
>>>>> <kieran618@googlemail.com<mailto:kieran618@googlemail.com>>
>>>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>>>> 
>>>>> On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-
>>>>> hotmail.com@ffmpeg.org<mailto:hotmail.com@ffmpeg.org>>
>>>>> wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: ffmpeg-devel
>>>>>>> <ffmpeg-devel-bounces@ffmpeg.org<mailto:ffmpeg-devel-bounces@ff
>>>>>>> mpeg.org>> On Behalf Of
>>>>>>> Michael Niedermayer
>>>>>>> Sent: Freitag, 21. Februar 2025 14:22
>>>>>>> To: FFmpeg development discussions and patches <ffmpeg-
>>>>>>> devel@ffmpeg.org<mailto:devel@ffmpeg.org>>
>>>>>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>>>>>> 
>>>>>>> Hi
>>>>>>> 
>>>>>>> On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
>>>>>>>> tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
>>>>>>>>> Hi
>>>>>>>>> 
>>>>>>>>> On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago
>>>>>>>>> wrote:
>>>>>>>>>> [insert meme here]
>>>>>>>>> [...]
>>>>>>>>>> I also recorded a video showing the filter in action [7].
>>>>>>>>> [...
>>>>>>>>>> [7] https://youtu.be/U_y4-NnaINg
>>>>>>>>> 
>>>>>>>>> cool, it doesnt detect everyone though
>>>>>>>>> 
>>>>>>>>> also i think this shows how useful a plugin framework would
>>>>>>>>> be for
>>>>>>>>> ffmpeg
>>>>>>>>> 
>>>>>>>>> with plugins everyone could use,test and contribute to this
>>>>>>>>> today.
>>>>>>>>> without plugins, this needs to be merged in ffmpeg git
>>>>>>>>> master. (which
>>>>>>>>> will take some time i suspect)
>>>>>>>> 
>>>>>>>> Have we not gone over and rejected plugins many times? I
>>>>>>>> recall points
>>>>>>> 
>>>>>>> no
>>>>>>> there was no formal and no public informal vote that i
>>>>>>> remember.
>>>>>>> 
>>>>>>> ive raised the issue with plugins many times. Because it would
>>>>>>> allow people and myself to contribute more complex features and
>>>>>>> end the stagnation of FFmpeg.
>>>>>> 
>>>>>> Yup, that's exactly one of the reasons why other projects like
>>>>>> GStreamer
>>>>>> have gained that much popularity. It might not have happened when
>>>>>> ffmpeg would have been more open and extensible in the first
>>>>>> place.
>>>>>> 
>>>>> 
>>>>> It's quite the opposite. Gstreamer is successful in its vertical
>>>>> because it
>>>>> allows easy inclusion of vendor binary blobs. This is not "open and
>>>>> extensible".
>>>> 
>>>> Open means it's extensible for everybody, including vendors. I fail
>>>> to see
>>>> what's bad about it. Do we have a fight against everything
>>>> commercial?
>>>> 
>>>> What are you afraid of to happen?
>>> 
>>> I think Kieran already expressed quite well the issue:
>>> 
>>> 1) it encourages proprietary blobs
>>> 2) it creates pressure on us to maintain compatibility with said blobs
>> 
>> The first issue has a solution: use gpl or more strict license for the plugin
>> interface support, like obs-studio.
>> 
>> For the second issue, we can just not give a stable API/ABI guarantee for
>> the plugin interface, for example, reject to load a plugin if the version doesn’t
>> match.
>> 
>> I have some feature in mind which suit for a plugin but not for a builtin implementation.
>> For example, different AI model have different pre- and post-process. Current
>> dnn in libavfilter only support some particular model like yolov4. We can’t write some
>> code for each model, which can be outdated soon.
>> 
>> VLC and obs-studio are both good example of plugin based frameworks.
> 
> Or the Linux kernel.
> 
> Though VLC does not have that "unofficial" plugins, which aren't backports of official plugins from a newer version. So I'm not convinced that that's a good example.

Maybe not much but I have noticed some unofficial plugins,

https://github.com/mfkl/awesome-vlc

And there is a Debian package for vlc-plugin-bittorrent.

Plugin interface can be abused, but at the same time, they can fulfill many legitimate
user needs. Not everyone can write a program based on libavcodec/libavformat/libavfilter,
and get the same function as fftools. They can write a filter plugin, and let ffmpeg cmd to
do the heavy lifting.

> 
> Even then proprietary vendors still managed to screw it up by failing to release new versions of their binary blobs for new versions of (Windows) VLC.
> 
> In my opinion, VLC plugins are more about reducing binary load times and load sizes by breaking the code up, and helping Linux distro break dependencies down, than about enabling external (proprietary or not) plugins.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[martin schitter]

On 2/26/25 17:03, Zhao Zhili wrote:
> Plugin interface can be abused, but at the same time, they can fulfill many legitimate
> user needs. Not everyone can write a program based on libavcodec/libavformat/libavfilter,
> and get the same function as fftools. They can write a filter plugin, and let ffmpeg cmd to
> do the heavy lifting.

Perhaps it could make sense to also consider the benefits of WASM based 
platform independent extensions in this debate. Right now that's often 
handled by help of extism (https://extism.org/) but WASIp3 will be very 
soon another interesting more standardized option for this kind of 
modular components and software extensibility.

Martin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Nicolas George]

Soft Works (HE12025-02-21):
> Open means it's extensible for everybody, including vendors. I fail to see
> what's bad about it. Do we have a fight against everything commercial?

I am with Kieran and Tomas on this. This feature would make it
extensible for everybody, but considering that it is already extensible
for everybody who agrees to release their source code, it mostly helps
the people who do not want to release their source code.

Why should we expend efforts to help people who do not want to release
their source code?

If, out of 20 projects who would have used them, the lack of plugins
leads 5 to abuse the license, 5 to find a loophole, 5 to use gstreamer,
4 to give up and 1 to release their source code, then from the point of
view of the Libre Software community, that 1 at the end makes it a win.

What we could do is make it easier to build from source with extra
components:

- Build scripts that detect available libraries. Possibly build scripts
  that install devel packages for these libraries. Possibly even a build
  script that installs a Debian in a VM/container.

- Tweak the build system to let add a component by just dropping it in a
  subdirectory.

I think you suggested something similar in another mail.

Regards,

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Nicolas George
> Sent: Mittwoch, 26. Februar 2025 15:08
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Cc: Kieran Kunhya <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> 
> Soft Works (HE12025-02-21):
> > Open means it's extensible for everybody, including vendors. I fail to see
> > what's bad about it. Do we have a fight against everything commercial?
> 
> I am with Kieran and Tomas on this. This feature would make it
> extensible for everybody, but considering that it is already extensible
> for everybody who agrees to release their source code, it mostly helps
> the people who do not want to release their source code.

Hi Nicolas,

I see it somewhat more liberal and I would see the primary purpose 
in helping people who have code which cannot get into ffmpeg 
(for whatever reason). But surely it would have that side-effect of
making it easier to deliver binaries without source. 
The tag word seems to be "binary blobs", yet I wonder whether 
that's even a sustainable approach for such vendors. On Windows
it's pretty easy to build a single dll (per architecture) which can have
its own set of dependency and C runtime libs. I'm not sure for Linux
& Co - is it even feasible for a vendor to create a "binary blob" that
would work universally? What I mean is, how many different 
"binary blobs" might a vendor need to build, then?
(not sure if dlmopen() or RTLD_LOCAL would allow something
similar like on Windows)

Anyway, I'm not intending to fight against open-source ideas,
it should be up to the developers to decide in which ways their
works are being used; personally, I wouldn't mind this kind of
of usage, but I understand that others see it differently.

> What we could do is make it easier to build from source with extra
> components:

[..]


> - Build scripts that detect available libraries. Possibly build scripts
>   that install devel packages for these libraries. Possibly even a build
>   script that installs a Debian in a VM/container.
> 
> - Tweak the build system to let add a component by just dropping it in a
>   subdirectory.

Yes, that's exactly what I mean. Currently, when adding custom filters
(or codecs, muxers etc.), you need to make changes at various places
and each time when rebasing or merging things around and managing
versions, these things are causing conflicts.
Or when you want to provide a filter/component to the community 
(as source code), you can't provide a patch that could be applied to
arbitrary ffmpeg versions.

Having a "plugins" directory that is automatically scanned for plugin
code, or a command line parameter like

./configure --plugindir=../myplugins

would be fine IMO.

And obviously much more agreeable for everybody than runtime
plugins, even though I would favor having both 😊

Thanks,
sw




_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Stephen Hutchinson]

On 2/21/25 9:53 AM, Kieran Kunhya via ffmpeg-devel wrote:
> On Fri, 21 Feb 2025, 14:30 Soft Works, <softworkz-at-hotmail.com@ffmpeg.org>
> wrote:
> 
>>
>>
>>> -----Original Message-----
>>> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
>>> Michael Niedermayer
>>> Sent: Freitag, 21. Februar 2025 14:22
>>> To: FFmpeg development discussions and patches <ffmpeg-
>>> devel@ffmpeg.org>
>>> Subject: Re: [FFmpeg-devel] I've written a filter in Rust
>>>
>>> Hi
>>>
>>> On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
>>>> tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
>>>>> Hi
>>>>>
>>>>> On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
>>>>>> [insert meme here]
>>>>> [...]
>>>>>> I also recorded a video showing the filter in action [7].
>>>>> [...
>>>>>> [7] https://youtu.be/U_y4-NnaINg
>>>>>
>>>>> cool, it doesnt detect everyone though
>>>>>
>>>>> also i think this shows how useful a plugin framework would be for
>>>>> ffmpeg
>>>>>
>>>>> with plugins everyone could use,test and contribute to this today.
>>>>> without plugins, this needs to be merged in ffmpeg git master. (which
>>>>> will take some time i suspect)
>>>>
>>>> Have we not gone over and rejected plugins many times? I recall points
>>>
>>> no
>>> there was no formal and no public informal vote that i remember.
>>>
>>> ive raised the issue with plugins many times. Because it would
>>> allow people and myself to contribute more complex features and
>>> end the stagnation of FFmpeg.
>>
>> Yup, that's exactly one of the reasons why other projects like GStreamer
>> have gained that much popularity. It might not have happened when
>> ffmpeg would have been more open and extensible in the first place.
>>
> 
> It's quite the opposite. Gstreamer is successful in its vertical because it
> allows easy inclusion of vendor binary blobs. This is not "open and
> extensible".
> 
> Don't confuse the history of avisynth in the 2000s with today's reality.
> 

And for that matter, AviSynth got bit **hard** when supporting more than
just 32-bit x86 became necessary, so any early success in attracting
plugin authors by having a Classpath-style exemption was quickly offset
by getting stuck with those decisions to maintain compatibility, and
suffering a lack of established plugins on new architectures/platforms
due to how many authors in the early days kept their plugins closed 
(thankfully most(?) of the regularly used and useful plugins now are
under FOSS licenses, but I don't even know how many plugins exist that
simply can't be ported because they're essentially closed-source
abandonware tied to an ancient version of AviSynth).

So it's just as much of a cautionary tale about how not to set up plugin
licensing and how the upstream handles compatibility in regard to it.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Vittorio Giovara]

On Fri, Feb 21, 2025 at 3:30 PM Soft Works <
softworkz-at-hotmail.com@ffmpeg.org> wrote:

>
>
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Michael Niedermayer
> > Sent: Freitag, 21. Februar 2025 14:22
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel@ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] I've written a filter in Rust
> >
> > Hi
> >
> > On Fri, Feb 21, 2025 at 10:01:56AM +0100, Tomas Härdin wrote:
> > > tor 2025-02-20 klockan 23:49 +0100 skrev Michael Niedermayer:
> > > > Hi
> > > >
> > > > On Thu, Feb 20, 2025 at 02:06:47PM +0100, Leandro Santiago wrote:
> > > > > [insert meme here]
> > > > [...]
> > > > > I also recorded a video showing the filter in action [7].
> > > > [...
> > > > > [7] https://youtu.be/U_y4-NnaINg
> > > >
> > > > cool, it doesnt detect everyone though
> > > >
> > > > also i think this shows how useful a plugin framework would be for
> > > > ffmpeg
> > > >
> > > > with plugins everyone could use,test and contribute to this today.
> > > > without plugins, this needs to be merged in ffmpeg git master. (which
> > > > will take some time i suspect)
> > >
> > > Have we not gone over and rejected plugins many times? I recall points
> >
> > no
> > there was no formal and no public informal vote that i remember.
> >
> > ive raised the issue with plugins many times. Because it would
> > allow people and myself to contribute more complex features and
> > end the stagnation of FFmpeg.
>
> Yup, that's exactly one of the reasons why other projects like GStreamer
> have gained that much popularity. It might not have happened when
> ffmpeg would have been more open and extensible in the first place.
>

The popularity was gained because it is easier to bundle in closed source
applications which I think we all agree it's not the direction we should be
going towards. Also their community is welcoming developers instead of
actively pushing them away with conspiracy or governance talks.

What do you people think about the feasibility of a compile-time plugin
> mechanism?


Bad idea, as others have expressed in more detail.
-- 
Vittorio
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Lynne]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 5220 bytes --]

On 20/02/2025 14:06, Leandro Santiago wrote:
> [insert meme here]
> 
> (this will be a long e-mail)
> 
> Dear FFmpeg devs,
> 
> in the past days I've been experimenting hacking FFmpeg using Rust.
> 
> As I am becoming more familiar with the libavfilter, and it is not a dependency for any other of the libav* libs, I decided this is a good candidate.
> 
> It's also convenient as I use FFmpeg libs heavily in a commercial product, and one of the features I've been working on involves a basic multi object tracking.
> 
> In my case, it does not need to be a "perfect" tracking algorithm, as I need to compromise quality of the result in exchange of performance executing in the CPU only, so most of the algorithms out there that need a GPU are out of my range.
> 
> I decided then use as first experiment a filter called `track_sort` that implements the 2016 paper SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC, as known as SORT [1].
> 
> The filter already works well based on the `master` branch, but the code itself is in very early stages and far from being "production ready", so please do not read the code assuming it's in its final form. It's ugly and needs lots of refactoring.
> 
> I've created a PR on forgejo [4] to make it easier for others to track progress, although I use gitlab.com as my main forge.
> 
> Here is a description of the filter:
> 
> - It perform only object tracking, needing the object detection to be performed elsewhere. It feeds from the detection boxes generated by `dnn_detect`. That means that the quality of the the tracking is closely related to the quality of the detection.

We've been looking to deprecate our native DNN stuff for years. Its 
bitrotten NIH code written long ago. Not a good idea to base it on.

> - SORT is a simple algorithm that uses spatial data only, and it not able to handle cases such as object occlusion. It's good enough for my use case, as I mentioned earlier.
> 
> - The filter works with the default options, so you can pass it without any arguments. In this mode, it will try to track any objects from the boxes available. You can change this behaviour by specifying the list of labels to track, for example: `track_sort=labels=person|dog|cat`. Such labels come from the ML model you used in the detection filter. It also has the options `threshold`, `min_hits` and `max_age`, which control how the tracking algorithm works, but the default values should work well on most cases.
> 
> - The filter will add the tracking information as label on a new frame side data entry of type `AV_FRAME_DATA_DETECTION_BBOXES`. It **WILL NOT** override the side data from `dnn_detect`,, meaning that the frame will have side data two entries of this type. I've created a PR that make it possible to fetch such entry [2].
> 
> - The labels in the detection boxes have the format "track:<track_num>:<track_age>", and this is not the final format. I did this way as a quick hack to have some visual information when drawing the boxes and labels with the `drawtext` and `drawbox` filters. I believe this can be improved by putting the tracking information as metadata of the `AVDetectionBBox`es, but this would on API and ABI breaking, so this is still an open question.

That's not a very useful filter. What this needs is a more complex and 
flexible interface than bounding boxes, which can enable processing of 
individual objects and turning the tracking info into something useful, 
for example, splitting up a video into objects.

> What has not been done so far:
> 
> I had quite a few goals in this task:
> 
> - 1: get a working and efficient implementation of the SORT algorithm.
> - 2: start learning Rust again (it's been ~5 years since I used it)
> - 3: learn more about the libavfilter codebase
> - 4: evaluate whether Rust could work as a second language for hacking FFMpeg.

None of us know Rust confidently enough to maintain and work on such code.

> Results:
> 
> - 1: I managed to reuse lots of high quality code, available on crates (the repository of Rust packages), preventing me of needing to write hairy math heavy code. I personally suck in maths, especially linear algebra. Using the paper and the reference implementation [3] was enough, although I do not understand all the math magic. For instance, I reused an existing crate for Kalman filters that I probably would need to implement by hand, as the alternative in C would probably be using the implementation that OpenCV offers. And I am aware that it's not practical to make OpenCV a dependency of FFmpeg.

Regardless of the language, I disagree with using crates in the context 
of FFmpeg, and any use of cargo.
We used to link to OpenCV. The only reason why we dropped it was because 
we used the C interface, which was removed, and no one wanted to or was 
interested in rewriting it.

I also don't agree with a plugin interface. We've been over this a few 
times. A plugin interface would likely turn the project into gstreamer 
or dshow, with dozens of proprietary plugins, whose users would be 
asking us for support.
We bump our ABI once a year, which is far too often for this as well.

[-- Attachment #1.1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 637 bytes --]

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Kieran Kunhya via ffmpeg-devel]

On Fri, 21 Feb 2025, 13:18 Lynne, <dev@lynne.ee> wrote:

> On 20/02/2025 14:06, Leandro Santiago wrote:
> Regardless of the language, I disagree with using crates in the context
> of FFmpeg, and any use of cargo.
>

I have no opinion on rust in FFmpeg but I broadly agree crates and cargo
are not suited for FFmpeg.

Kieran

>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Tomas Härdin]

fre 2025-02-21 klockan 14:18 +0100 skrev Lynne:
> On 20/02/2025 14:06, Leandro Santiago wrote:
> 
> > - 1: I managed to reuse lots of high quality code, available on
> > crates (the repository of Rust packages), preventing me of needing
> > to write hairy math heavy code. I personally suck in maths,
> > especially linear algebra. Using the paper and the reference
> > implementation [3] was enough, although I do not understand all the
> > math magic. For instance, I reused an existing crate for Kalman
> > filters that I probably would need to implement by hand, as the
> > alternative in C would probably be using the implementation that
> > OpenCV offers. And I am aware that it's not practical to make
> > OpenCV a dependency of FFmpeg.
> 
> Regardless of the language, I disagree with using crates in the context 
> of FFmpeg, and any use of cargo.

In addition, this almost certainly will create headaches for package
maintainers. The proper way to do this is to use one's OS' package
manager, not go down the godawful path Python and every other modern
language has of rolling their own package managers.

The above said, I'm not against Rust. It has some nice properties. But
it does not seem very "stable" so far. Perhaps this has changed in
recent years..

If we're in the habit of allowing other languages I'd be in favor of
allowing C++, so that we can make use of the STL containers rather than
rolling our own.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Rémi Denis-Courmont]

Le perjantaina 21. helmikuuta 2025, 20.02.16 UTC+2 Tomas Härdin a écrit :
> The above said, I'm not against Rust. It has some nice properties. But
> it does not seem very "stable" so far. Perhaps this has changed in
> recent years..

IME, it's become very usable for user-space code. Bare metal still pretty much 
requires unstable features, but that's not a problem for FFmpeg.

We may want to depend on the allocator_api unstable feature for fallible 
allocations (or just allow non-fallible allocations in Rust code until that 
gets stabilised). Naked functions would be nice for assembler, but I don't 
think that we have any urgent need to call assembler from Rust (as opposed to 
from C).
 
> If we're in the habit of allowing other languages I'd be in favor of
> allowing C++, so that we can make use of the STL containers rather than
> rolling our own.

Yikes. Rust is actually way saner for type-generic programming than C++.

-- 
Rémi Denis-Courmont
Villeneuve de Tapiola, ex-République finlandaise d´Uusimaa



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Tomas Härdin]

lör 2025-02-22 klockan 14:57 +0200 skrev Rémi Denis-Courmont:
> Le perjantaina 21. helmikuuta 2025, 20.02.16 UTC+2 Tomas Härdin a écrit :
> > The above said, I'm not against Rust. It has some nice properties. But
> > it does not seem very "stable" so far. Perhaps this has changed in
> > recent years..
> 
> IME, it's become very usable for user-space code. Bare metal still pretty much 
> requires unstable features, but that's not a problem for FFmpeg.

I mean more in terms of ABI, and having to have cargo install specific
versions of the Rust compiler and so on.

> > If we're in the habit of allowing other languages I'd be in favor of
> > allowing C++, so that we can make use of the STL containers rather than
> > rolling our own.
> 
> Yikes. Rust is actually way saner for type-generic programming than C++.

No doubt, but STL is still miles better than rolling our own
containers.

Anyway, rather than shoehorning Rust into this codebase it might make
more sense to contribute to NihAV instead. But only if it has a sane
parsing framework

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 1613 bytes --]

Hi

On Sun, Feb 23, 2025 at 10:30:03PM +0100, Tomas Härdin wrote:
> lör 2025-02-22 klockan 14:57 +0200 skrev Rémi Denis-Courmont:
> > Le perjantaina 21. helmikuuta 2025, 20.02.16 UTC+2 Tomas Härdin a écrit :
> > > The above said, I'm not against Rust. It has some nice properties. But
> > > it does not seem very "stable" so far. Perhaps this has changed in
> > > recent years..
> > 
> > IME, it's become very usable for user-space code. Bare metal still pretty much 
> > requires unstable features, but that's not a problem for FFmpeg.
> 
> I mean more in terms of ABI, and having to have cargo install specific
> versions of the Rust compiler and so on.
> 
> > > If we're in the habit of allowing other languages I'd be in favor of
> > > allowing C++, so that we can make use of the STL containers rather than
> > > rolling our own.
> > 
> > Yikes. Rust is actually way saner for type-generic programming than C++.
> 
> No doubt, but STL is still miles better than rolling our own
> containers.
> 

> Anyway, rather than shoehorning Rust into this codebase it might make
> more sense to contribute to NihAV instead. But only if it has a sane
> parsing framework

That misses the point. FFmpeg should support a "safer" language than C
because for some modules its the better choice.

The same way linux benefits from rust. Similarly putting rust code
in a different project than linux doesnt help linux.

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

No great genius has ever existed without some touch of madness. -- Aristotle

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Tomas Härdin]

sön 2025-02-23 klockan 22:51 +0100 skrev Michael Niedermayer:
> Hi
> 
> On Sun, Feb 23, 2025 at 10:30:03PM +0100, Tomas Härdin wrote:
> > lör 2025-02-22 klockan 14:57 +0200 skrev Rémi Denis-Courmont:
> > > Le perjantaina 21. helmikuuta 2025, 20.02.16 UTC+2 Tomas Härdin a écrit :
> > > > The above said, I'm not against Rust. It has some nice properties. But
> > > > it does not seem very "stable" so far. Perhaps this has changed in
> > > > recent years..
> > > 
> > > IME, it's become very usable for user-space code. Bare metal still pretty much 
> > > requires unstable features, but that's not a problem for FFmpeg.
> > 
> > I mean more in terms of ABI, and having to have cargo install specific
> > versions of the Rust compiler and so on.
> > 
> > > > If we're in the habit of allowing other languages I'd be in favor of
> > > > allowing C++, so that we can make use of the STL containers rather than
> > > > rolling our own.
> > > 
> > > Yikes. Rust is actually way saner for type-generic programming than C++.
> > 
> > No doubt, but STL is still miles better than rolling our own
> > containers.
> > 
> 
> > Anyway, rather than shoehorning Rust into this codebase it might make
> > more sense to contribute to NihAV instead. But only if it has a sane
> > parsing framework
> 
> That misses the point. FFmpeg should support a "safer" language than C
> because for some modules its the better choice.

Maybe. We can do a lot by just improving the build system. But if we're
going that route I think we should first try and see how working C++
into more parts of the code works, because we already have support for
C++ for torch and decklink. Doing so would allow us to toss out lots of
code, especially in lavu, which is always nice. Code is a liability.

/Tomas

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Rémi Denis-Courmont]

Hi,

Le 23 février 2025 23:30:03 GMT+02:00, "Tomas Härdin" <git@haerdin.se> a écrit :
>lör 2025-02-22 klockan 14:57 +0200 skrev Rémi Denis-Courmont:
>> Le perjantaina 21. helmikuuta 2025, 20.02.16 UTC+2 Tomas Härdin a écrit :
>> > The above said, I'm not against Rust. It has some nice properties. But
>> > it does not seem very "stable" so far. Perhaps this has changed in
>> > recent years..
>> 
>> IME, it's become very usable for user-space code. Bare metal still pretty much 
>> requires unstable features, but that's not a problem for FFmpeg.
>
>I mean more in terms of ABI, and having to have cargo install specific
>versions of the Rust compiler and so on.

Why? The ABI will never be fully stable. Zero-cost abstractions just don't lend themselves to an ABI.

But what is stable is not going to just break in a future version. We could settle for Rust edition 2024 and not depend on any specific compiler version, AFAICT.

>> > If we're in the habit of allowing other languages I'd be in favor of
>> > allowing C++, so that we can make use of the STL containers rather than
>> > rolling our own.
>> 
>> Yikes. Rust is actually way saner for type-generic programming than C++.
>
>No doubt, but STL is still miles better than rolling our own
>containers.

But STL is not worth switching to C++ for.

If the goal is to enable a language with significantly improved static safety, without compromising on performance (especially no GC) and optimisability, then Rust is pretty much the only option at the moment.

If the goal were to provide a nicer language to work with rather than improving safety, then maybe Zig would be better than Rust. But I don't see a scenario where C++ can be justified/worthwhile.

>Anyway, rather than shoehorning Rust into this codebase it might make
>more sense to contribute to NihAV instead. But only if it has a sane
>parsing framework

That makes sense if the goal is to publish and forget, but otherwise I doubt that NihAV will ever be relevant "in the field".

That being said, I think Rust would make much more sense for decoders and demuxers than for filters and ML stuff.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Tomas Härdin]

mån 2025-02-24 klockan 16:51 +0200 skrev Rémi Denis-Courmont:
> Hi,
> 
> Le 23 février 2025 23:30:03 GMT+02:00, "Tomas Härdin"
> <git@haerdin.se> a écrit :
> > lör 2025-02-22 klockan 14:57 +0200 skrev Rémi Denis-Courmont:
> > > Le perjantaina 21. helmikuuta 2025, 20.02.16 UTC+2 Tomas Härdin a
> > > écrit :
> > > > The above said, I'm not against Rust. It has some nice
> > > > properties. But
> > > > it does not seem very "stable" so far. Perhaps this has changed
> > > > in
> > > > recent years..
> > > 
> > > IME, it's become very usable for user-space code. Bare metal
> > > still pretty much 
> > > requires unstable features, but that's not a problem for FFmpeg.
> > 
> > I mean more in terms of ABI, and having to have cargo install
> > specific
> > versions of the Rust compiler and so on.
> 
> Why? The ABI will never be fully stable. Zero-cost abstractions just
> don't lend themselves to an ABI.

The C ABI is very stable. It's why "C for the API/ABI and C++ for the
implementation" is a such powerful combination. Perhaps the same thing
could be done with Rust, or both Rust and C++..

> > > > If we're in the habit of allowing other languages I'd be in
> > > > favor of
> > > > allowing C++, so that we can make use of the STL containers
> > > > rather than
> > > > rolling our own.
> > > 
> > > Yikes. Rust is actually way saner for type-generic programming
> > > than C++.
> > 
> > No doubt, but STL is still miles better than rolling our own
> > containers.
> 
> But STL is not worth switching to C++ for.

We already use C++ in the codebase. The cost for using it in more
places is very low

> If the goal is to enable a language with significantly improved
> static safety, without compromising on performance (especially no GC)
> and optimisability, then Rust is pretty much the only option at the
> moment.

I have suggested using formal methods (frama-c) to improve the safety
of the C code. So far the reception has been lukewarm at best, even for
less demanding tools such as value analysis, compared to weakest
precondition calculus (WP). Frama-C is also moving towards C++ support.

> > Anyway, rather than shoehorning Rust into this codebase it might
> > make
> > more sense to contribute to NihAV instead. But only if it has a
> > sane
> > parsing framework
> 
> That makes sense if the goal is to publish and forget, but otherwise
> I doubt that NihAV will ever be relevant "in the field".
> 
> That being said, I think Rust would make much more sense for decoders
> and demuxers than for filters and ML stuff.

On this I fully agree. For demuxers it makes perfect sense, and also
for the non-DSP side of codecs. But this is also why, if you're already
willing pay the oxidation cost, contributing to NihAV makes much more
sense, because you're not held back by trying to retain ABI compat.

Far above all this, what is even more important is to do proper
parsing, with say a parser combinator framework, not just implementing
yet another library of shotgun parsers. And of course, using existing
libraries whenever possible. But in this project both of these amount
to tech-heresy.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Leandro Santiago]

(I am replying to the messages in general, not inline, as I got lost with all the subthreads).

My opinion on C++ vs Rust:

I've been using mostly C++ during my career (~15 years) and I confess I kind of lost hope on its future. I no longer see any advantages of writing new code in C++ just because of the language merits.

Unless of course it's "the right/only tool for the job", due to the amount of high quality C++ libraries and large C++ codebases out there, as well as wider compiler/platforms support, to make my position clear.

The main selling point I see C++ has over Rust here is being mostly a superset of C, able to simply include C files.

The good parts of the STL, the containers and algorithms, make it easy to accidentally allocate memory, as well as one never knows really if something will throw an exception, making it harder to work from the C side.

Rust, IMO, prevents most of those issues, due to its explicit nature, especially regarding sharing and allocating/releasing memory.

I also feel that the algorithms and interfaces in the Rust standard library are better designed and provide better ergonomics and performance than the C++ counterparts. C++ has tried hard to improve it with ranges, but the syntax is weird and it seems to have gained little traction.

C++ ranges also do not prevent one of making simple mistakes with object lifetimes.

In my experience on Linux, it's also quite difficult to statically link to lib[std]c++.

Rust, on the other rand, has mature tooling and defaults to static link everything by default. This means no more need to depend on specific versions of GCC, for instance. (in practice, though, ffmpeg will likely link to libstdc++ due to its dependencies).

With Rust (and cargo), 3rd party code vendoring seems to be more mature, so one does not need to depend on an internet connection to build code with cargo.

One of the biggest disadvantages of Rust regarding C compatibility is not being able to simply #include C code. One has to rely on tools such as bindgen for bridging the two ecosystems. In my little experience, it does quite a good job, though, with few bits left to be fixed (things such as complex macros, etc.).

Extra manual work is usually needed, though, to write proper safe and "rusty" wrappers around the unsafe C API.

There is also the question about how to ship and dynamically link Rust libraries. I've played only with static linking and can already foresee challenges linking it dynamically in my application.

While I have no doubts that Rust could benefit core libraries such as avcodec/avformat (which I have very little knowledge of), I imagine that other non core code would benefit as well.

I started playing around with avfilter, but I really think that the fftools would be good candidates for a "RiiR", as they are complex codebases that have no dependents, besides being how most ffmpeg users interact with the project.

In any case, I don't plan to propose mainlining my work on the Rust filters, but I'll continue working on it, and be open to contributions and suggestions. I am not forking ffmpeg as a project, to make it clear. My code is right now very ugly and needs lots of refactoring and cleanup, bugfixing and maturing, being far from being "production ready".

Cheers,

Leandro

On 2/24/25 15:51, Rémi Denis-Courmont wrote:
> Hi,
>
> Le 23 février 2025 23:30:03 GMT+02:00, "Tomas Härdin" <git@haerdin.se> a écrit :
>> lör 2025-02-22 klockan 14:57 +0200 skrev Rémi Denis-Courmont:
>>> Le perjantaina 21. helmikuuta 2025, 20.02.16 UTC+2 Tomas Härdin a écrit :
>>>> The above said, I'm not against Rust. It has some nice properties. But
>>>> it does not seem very "stable" so far. Perhaps this has changed in
>>>> recent years..
>>> IME, it's become very usable for user-space code. Bare metal still pretty much 
>>> requires unstable features, but that's not a problem for FFmpeg.
>> I mean more in terms of ABI, and having to have cargo install specific
>> versions of the Rust compiler and so on.
> Why? The ABI will never be fully stable. Zero-cost abstractions just don't lend themselves to an ABI.
>
> But what is stable is not going to just break in a future version. We could settle for Rust edition 2024 and not depend on any specific compiler version, AFAICT.
>
>>>> If we're in the habit of allowing other languages I'd be in favor of
>>>> allowing C++, so that we can make use of the STL containers rather than
>>>> rolling our own.
>>> Yikes. Rust is actually way saner for type-generic programming than C++.
>> No doubt, but STL is still miles better than rolling our own
>> containers.
> But STL is not worth switching to C++ for.
>
> If the goal is to enable a language with significantly improved static safety, without compromising on performance (especially no GC) and optimisability, then Rust is pretty much the only option at the moment.
>
> If the goal were to provide a nicer language to work with rather than improving safety, then maybe Zig would be better than Rust. But I don't see a scenario where C++ can be justified/worthwhile.
>
>> Anyway, rather than shoehorning Rust into this codebase it might make
>> more sense to contribute to NihAV instead. But only if it has a sane
>> parsing framework
> That makes sense if the goal is to publish and forget, but otherwise I doubt that NihAV will ever be relevant "in the field".
>
> That being said, I think Rust would make much more sense for decoders and demuxers than for filters and ML stuff.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] I've written a filter in Rust

[Rémi Denis-Courmont]

Le perjantaina 21. helmikuuta 2025, 15.18.06 UTC+2 Lynne a écrit :
> > - 1: get a working and efficient implementation of the SORT algorithm.
> > - 2: start learning Rust again (it's been ~5 years since I used it)
> > - 3: learn more about the libavfilter codebase
> > - 4: evaluate whether Rust could work as a second language for hacking
> > FFMpeg.
> 
> None of us know Rust confidently enough to maintain and work on such code.

A project should of course limit the number of distinct programminglanguages 
involved for a number of reasons, but there are people who know Rust quite 
well here as well as in adjacent projects from the OSS multimedia community.

The sustainability problem is really not specific to the programming language. 
The majority of FFmpeg code requires domain-specific knowledge that even most 
FFmpeg maintainers don't have. By that same logic we should not have any 
assembler optimisation for any instruction set other than, maybe, 386 and 
support only the most well-known codecs and containers.

(...)
> Regardless of the language, I disagree with using crates in the context 
> of FFmpeg

I assume that you mean *external* crates because if FFmpeg starts 
incorporating Rust, I would certainly expect it to consist of a workspace with 
multiple crates.

With that said, external crates are just the Rust equivalent of external C 
libraries. I agree that, ideally, FFmpeg should not depend on external 
libraries and carry native implementations of anything other than the 
languages runtimes and hardware abstraction layers...but in actuality, FFmpeg 
has tons of libraries as optional dependencies.

> and any use of cargo.

I fail to see the problem, TBH. You can use rustc directly, but it's really 
inconvenient with little to no benefits. I don't see why we couldn't use Cargo 
to produce static libraries, then have the Makefile import those static 
libraries into the existing linking process.

We can (and most definitely should) restrict adding external crates in the 
Cargo manifests but that does not mean we should ban Cargo altogether.

> We used to link to OpenCV. The only reason why we dropped it was because 
> we used the C interface, which was removed, and no one wanted to or was 
> interested in rewriting it.

I agree that we should not depend on external *frameworks*.

-- 
Rémi Denis-Courmont
Tapio's place new town, former Finnish Republic of Uusimaa



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".