Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
 help / color / mirror / Atom feed
* [FFmpeg-devel] [PATCH FFmpeg 11/15] doc: avgclass Filter Documentation
@ 2025-03-08 15:01 m.kaindl0208
  2025-03-09 19:19 ` Michael Niedermayer
  0 siblings, 1 reply; 3+ messages in thread
From: m.kaindl0208 @ 2025-03-08 15:01 UTC (permalink / raw)
  To: ffmpeg-devel

Try the new filters using my Github Repo https://github.com/MaximilianKaindl/DeepFFMPEGVideoClassification. 

Any Feedback is appreciated!

Signed-off-by: MaximilianKaindl <m.kaindl0208@gmail.com>
---
 doc/filters.texi | 64 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)

diff --git a/doc/filters.texi b/doc/filters.texi
index b6cccbacb6..bd75982d7d 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -30827,6 +30827,70 @@ ffplay -f lavfi 'amovie=input.mp3, asplit [a][out1];
 
 This filter supports the all above options as commands except options @code{size} and @code{rate}.
 
+@section avgclass
+
+Average classification probabilities across multiple frames for both audio and video streams.
+
+This filter analyzes classification data from frame side data (bounding boxes) and calculates average confidence scores for each label. The filter processes classification metadata from the @code{dnn_classify} filter or other sources that generate AVDetectionBBox side data, computing averages over the entire stream.
+
+At the end of the stream (or when manually triggered), the filter outputs the average probability for each detected class, both to console logs and optionally to a CSV file.
+
+@table @option
+@item output_file
+Path to a CSV output file where average classification results will be written. If not specified, results are only printed to log output.
+
+@item v
+Specify the number of video streams (default: 1).
+
+@item a
+Specify the number of audio streams (default: 0).
+@end table
+
+This filter supports the following commands:
+
+@table @option
+@item writeinfo
+Immediately write current average classification results to the log and output file (if specified) without waiting for the stream to end.
+
+@item flush
+Force the filter to write results and flush all its internal state.
+@end table
+
+@subsection Examples
+
+Process a video with object detection and classification, then calculate average classification probabilities:
+@example
+ffmpeg -i input.mp4 -vf "dnn_detect=model=detection.xml:input=data:output=detection_out:confidence=0.5,dnn_classify=model=classification.pt:dnn_backend=torch:tokenizer=tokenizer.json:labels=labels.txt,avgclass=output_file=results.csv" -f null -
+@end example
+
+Process both audio and video classification:
+@example
+ffmpeg -i input.mkv -filter_complex "[0:v]dnn_classify[v0]; [0:a]aformat=sample_fmts=fltp,dnn_classify=dnn_backend=torch:model=clap_model.pt:is_audio=1:tokenizer=tokenizer.json:labels=audio_labels.txt[a0]; [v0][a0]avgclass=v=1:a=1:output_file=av_results.csv" -f null -
+@end example
+
+@subsection Output Format
+
+When the filter completes processing (or when the @code{writeinfo} command is sent), it outputs classification results in this format:
+
+@example
+Classification averages:
+Stream #0:
+  Label: cat: Average probability 0.8765, Appeared 120 times
+  Label: dog: Average probability 0.3421, Appeared 42 times
+Stream #1:
+  Label: music: Average probability 0.9823, Appeared 315 times
+  Label: speech: Average probability 0.1245, Appeared 15 times
+@end example
+
+If an output file is specified, the same data is written in CSV format:
+@example
+stream_id,label,avg_probability,count
+0,cat,0.8765,120
+0,dog,0.3421,42
+1,music,0.9823,315
+1,speech,0.1245,15
+@end example
+
 @section bench, abench
 
 Benchmark part of a filtergraph.
-- 
2.34.1


_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [FFmpeg-devel] [PATCH FFmpeg 11/15] doc: avgclass Filter Documentation
  2025-03-08 15:01 [FFmpeg-devel] [PATCH FFmpeg 11/15] doc: avgclass Filter Documentation m.kaindl0208
@ 2025-03-09 19:19 ` Michael Niedermayer
  2025-03-09 20:24   ` m.kaindl0208
  0 siblings, 1 reply; 3+ messages in thread
From: Michael Niedermayer @ 2025-03-09 19:19 UTC (permalink / raw)
  To: FFmpeg development discussions and patches


[-- Attachment #1.1: Type: text/plain, Size: 1631 bytes --]

Hi Maximilian

On Sat, Mar 08, 2025 at 04:01:40PM +0100, m.kaindl0208@gmail.com wrote:
> Try the new filters using my Github Repo https://github.com/MaximilianKaindl/DeepFFMPEGVideoClassification. 
> 
> Any Feedback is appreciated!
> 
> Signed-off-by: MaximilianKaindl <m.kaindl0208@gmail.com>
> ---
>  doc/filters.texi | 64 ++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 64 insertions(+)
> 
> diff --git a/doc/filters.texi b/doc/filters.texi
> index b6cccbacb6..bd75982d7d 100644
> --- a/doc/filters.texi
> +++ b/doc/filters.texi
> @@ -30827,6 +30827,70 @@ ffplay -f lavfi 'amovie=input.mp3, asplit [a][out1];
[...]
> +@example
> +Classification averages:
> +Stream #0:
> +  Label: cat: Average probability 0.8765, Appeared 120 times
> +  Label: dog: Average probability 0.3421, Appeared 42 times
> +Stream #1:
> +  Label: music: Average probability 0.9823, Appeared 315 times
> +  Label: speech: Average probability 0.1245, Appeared 15 times
> +@end example

Nice!

how exactly does one interpret the average probability ?

I mean if one frame is detecting a cat with 0.99 and one with 0.01
does that give a average of 0.5 ?
iam asking as that seems not the most usefull metric as two frames with
0.5 would be alot weaker indicator than one with 0.99 that there was at
least one cat (if these behave like standard probabilities)

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

No human being will ever know the Truth, for even if they happen to say it
by chance, they would not even known they had done so. -- Xenophanes

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [FFmpeg-devel] [PATCH FFmpeg 11/15] doc: avgclass Filter Documentation
  2025-03-09 19:19 ` Michael Niedermayer
@ 2025-03-09 20:24   ` m.kaindl0208
  0 siblings, 0 replies; 3+ messages in thread
From: m.kaindl0208 @ 2025-03-09 20:24 UTC (permalink / raw)
  To: 'FFmpeg development discussions and patches'

Hi Michael,

You are right. The workflow is that any classification above the confidence value parameter (default 0.5) gets written to the Side data of the Frame, then read by the avgclass filter and averaged. Given the parameter was set to 0.01 or lower, if one frame detects a cat with 0.99 confidence and another with 0.01 confidence, the average would indeed be 0.5 - the same as two frames with 0.5 confidence each, despite these representing very different detection scenarios.

I think the average classification approach makes more sense when the goal is not to classify specific objects in individual frames, but rather to identify general characteristics about the entire video. For my project, I am aiming to classify movies by their Recording System, Genre and Content type. I use CLIP/CLAP to capture the overall "vibe"/facts in the images or audio, which is why I implemented category classification this way. 

Example LLM generated categories file for classifying Recording System, Genre and Content type:
https://github.com/MaximilianKaindl/DeepFFMPEGVideoClassification/blob/main/resources/labels/categories_clip.txt

In my testing, combined with scene classification, this approach works reasonably well for my use case.

For the cat detection example, setting a higher confidence threshold would be more appropriate to ensure it is detecting a cat. I recognize there might be better approaches for specific detection tasks, and I should probably create a new example in the doc that better demonstrates the most useful application cases.

If we could guarantee that only a single animal type appears in the entire video, this averaging approach would be effective. However, this scenario is highly unrealistic outside of controlled settings like Google Lens classifications, where users typically focus the camera on just one specific subject at a time.

Kind regards

-----Original Message-----
From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Michael Niedermayer
Sent: Sunday, 9 March 2025 20:19
To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Subject: Re: [FFmpeg-devel] [PATCH FFmpeg 11/15] doc: avgclass Filter Documentation

Hi Maximilian

On Sat, Mar 08, 2025 at 04:01:40PM +0100, m.kaindl0208@gmail.com wrote:
> Try the new filters using my Github Repo https://github.com/MaximilianKaindl/DeepFFMPEGVideoClassification. 
> 
> Any Feedback is appreciated!
> 
> Signed-off-by: MaximilianKaindl <m.kaindl0208@gmail.com>
> ---
>  doc/filters.texi | 64 
> ++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 64 insertions(+)
> 
> diff --git a/doc/filters.texi b/doc/filters.texi index 
> b6cccbacb6..bd75982d7d 100644
> --- a/doc/filters.texi
> +++ b/doc/filters.texi
> @@ -30827,6 +30827,70 @@ ffplay -f lavfi 'amovie=input.mp3, asplit 
> [a][out1];
[...]
> +@example
> +Classification averages:
> +Stream #0:
> +  Label: cat: Average probability 0.8765, Appeared 120 times
> +  Label: dog: Average probability 0.3421, Appeared 42 times Stream 
> +#1:
> +  Label: music: Average probability 0.9823, Appeared 315 times
> +  Label: speech: Average probability 0.1245, Appeared 15 times @end 
> +example

Nice!

how exactly does one interpret the average probability ?

I mean if one frame is detecting a cat with 0.99 and one with 0.01 does that give a average of 0.5 ?
iam asking as that seems not the most usefull metric as two frames with
0.5 would be alot weaker indicator than one with 0.99 that there was at least one cat (if these behave like standard probabilities)

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

No human being will ever know the Truth, for even if they happen to say it by chance, they would not even known they had done so. -- Xenophanes

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2025-03-09 20:24 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-03-08 15:01 [FFmpeg-devel] [PATCH FFmpeg 11/15] doc: avgclass Filter Documentation m.kaindl0208
2025-03-09 19:19 ` Michael Niedermayer
2025-03-09 20:24   ` m.kaindl0208

Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \
		ffmpegdev@gitmailbox.com
	public-inbox-index ffmpegdev

Example config snippet for mirrors.


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git