From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id 3C2234E084 for ; Sat, 8 Mar 2025 15:01:20 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B0AF568F423; Sat, 8 Mar 2025 17:01:12 +0200 (EET) Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1A73D68F456 for ; Sat, 8 Mar 2025 17:01:06 +0200 (EET) Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-39129017bbbso1220658f8f.1 for ; Sat, 08 Mar 2025 07:01:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741446065; x=1742050865; darn=ffmpeg.org; h=content-language:thread-index:content-transfer-encoding :mime-version:message-id:date:subject:to:from:from:to:cc:subject :date:message-id:reply-to; bh=C+N8LdxikSUURb+RHqbcIs3T/9lT6T9EKX97wsY82R8=; b=dRNYKJ7SYc7N+LNwHcXQ9DBslzzIyOwXN/bS0OpYDMV+2IVmr8WKustgVjlo8g/RiS dozzhlUScWX0Id3gU50BO3fm0TIMd5PX8wmGIiA56YX8I9v/R/0jEGq3J0+1bmvzrMLX mlIYLPHmAxj17bOwOSt4FrlG9iqSBXTm3G2C8DJvmAFWg6N3+66nSeLGcdeZ7zKebCEU J8+gUZhZtTkBTEZpj/5fQq0o20czoO1TqK25CfJapZrSGo7SMxW8PZyd6ynp7OKU1BW0 D8d2eePpukM3c2tTGZz74wU5r275XEaUmXnS65jg2XxekXO5l9BcoY8oprM8msL27qOL fsWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741446065; x=1742050865; h=content-language:thread-index:content-transfer-encoding :mime-version:message-id:date:subject:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=C+N8LdxikSUURb+RHqbcIs3T/9lT6T9EKX97wsY82R8=; b=lhrvcSo1iGBTLzP6qZV2SR/xWKyUtOpi3jW5amJ/t1VkLozvp5Mu2TH0JVYK9IiWai RZrgK6TN5tTikYpSWVeEVxeKzzytit5TJ75Caxui3iLLdLbNlrTrBeqK5lyKKUQIURGm A/Hf/dRhwRA/qMpZUWASN1KsDEO3Z021664N31IkfiKhNKvZif86rPLT677xeekhR26t id7VinH+qRDtbCrzT1q+hOwO+Zsq0jvtR/ty+BMIfBoBJx7Xh5n2/cnZOy+M1t18aD1L i+Rr6tky2FqfAZVLltgZvjBDav6bkKyIxrnH5PjHAEoJGpmwucv57uWvVQ71+e8O4BIU Sa1w== X-Gm-Message-State: AOJu0YzbANATCdlaOcROyaDDdpsBAtkko9VytOm+BFpEwJEprhQFm4Tx Cbp+N7b7nR+onbmmNHRbVTbxc+uUFIRMUkBOkxSYHXCZwFgb3fUWkk0rew== X-Gm-Gg: ASbGncs+CkzkbSZU0jKC7Ylvqo6hsEVzuoGgv4DLiqdKbNuRBviS8Bf2/enOaIKU/Jt gg0ZMRNhqDOyNcWfAIjcd7Kkqg+kw2D4pQwHCbeeaRBVVvjvf0EAnfwtXQmIYH2PGaE3YFpqwNR dPL5vqOE8bBdI6JUUeT0eyFgYa6pNlB9y1vGNb4cI/HaoBNPuVQCK6AER+sv8tGWF4SppBXMgJQ THA07OGuCR4scRe+JjuXqLtKUnDEjSN+P1JcKfdsA/Dd356Mcypch/T/YhQWEvqvC8F1BbW8GZn zl3iWKrvWtITVoXOtzD/Br9iwCO1SvvNQYoGvUmZAIWI+6mUF6078qNjmRcFOfTZKxZHk3VbEpQ VvE34FU2dWrFkOcS4 X-Google-Smtp-Source: AGHT+IFPlorsruLHDPJrJGxgidDCQ1byyiq8TSUgmb9d/QT5MImw2ZcX+0j7XAF4NgdznVAxEjWtpQ== X-Received: by 2002:a05:6000:1846:b0:391:2192:ccd6 with SMTP id ffacd0b85a97d-39132dd6afcmr5651538f8f.39.1741446064503; Sat, 08 Mar 2025 07:01:04 -0800 (PST) Received: from MK2 (80-108-16-220.cable.dynamic.surfer.at. [80.108.16.220]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3912bfdfddcsm9153110f8f.35.2025.03.08.07.01.03 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 08 Mar 2025 07:01:03 -0800 (PST) From: To: Date: Sat, 8 Mar 2025 16:01:07 +0100 Message-ID: <007b01db903a$ebc0ddf0$c34299d0$@gmail.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 16.0 Thread-Index: AduQOIrsQocxUq8qRxCMELWO2OQ+FQ== Content-Language: en-at Subject: [FFmpeg-devel] [PATCH FFmpeg 9/15] doc: Filters.texi updated classify X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: Try the new filters using my Github Repo https://github.com/MaximilianKaindl/DeepFFMPEGVideoClassification. Any Feedback is appreciated! Signed-off-by: MaximilianKaindl --- doc/filters.texi | 106 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 76 insertions(+), 30 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index 0ba7d3035f..b6cccbacb6 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -11971,43 +11971,89 @@ ffmpeg -i INPUT -f lavfi -i nullsrc=hd720,geq='r=128+80*(sin(sqrt((X-W/2)*(X-W/2 @end itemize @section dnn_classify +Analyze media (video frames or audio) using deep neural networks to apply classifications based on the content. +This filter supports three classification modes: -Do classification with deep neural networks based on bounding boxes. +@itemize @bullet +@item Standard image classification (OpenVINO backend) +@item CLIP (Contrastive Language-Image Pre-training) classification (Torch backend) +@item CLAP (Contrastive Language-Audio Pre-training) classification (Torch backend) +@end itemize The filter accepts the following options: - @table @option @item dnn_backend -Specify which DNN backend to use for model loading and execution. This option accepts -only openvino now, tensorflow backends will be added. - -@item model -Set path to model file specifying network architecture and its parameters. -Note that different backends use different file formats. - -@item input -Set the input name of the dnn network. - -@item output -Set the output name of the dnn network. - +Specify which DNN backend to use for model loading and execution. Currently supports: +@table @samp +@item openvino +Use OpenVINO backend (standard image classification only). +@item torch +Use LibTorch backend (supports CLIP for images and CLAP for audio). +@end table @item confidence -Set the confidence threshold (default: 0.5). - +Set the confidence threshold (default: 0.5). Classifications with confidence below this value will be filtered out. @item labels -Set path to label file specifying the mapping between label id and name. -Each label name is written in one line, tailing spaces and empty lines are skipped. -The first line is the name of label id 0, -and the second line is the name of label id 1, etc. -The label id is considered as name if the label file is not provided. - -@item backend_configs -Set the configs to be passed into backend - -For tensorflow backend, you can set its configs with @option{sess_config} options, -please use tools/python/tf_sess_config.py to get the configs for your system. - -@end table +Set path to a label file specifying classification labels. This is required for standard classification and can be used for CLIP/CLAP classification. +Each label is written on a separate line in the file. Trailing spaces and empty lines are skipped. +@item categories +Path to a categories file for hierarchical classification (CLIP/CLAP only). This allows classification to be organized into multiple category units with individual categories containing related labels. +@item tokenizer +Path to the text tokenizer.json file (CLIP/CLAP only). Required for text embedding generation. +@item target +Specify which objects to classify. When omitted, the entire frame is classified. When specified, only bounding boxes with detection labels matching this value are classified. +@item is_audio +Enable audio processing mode for CLAP models (default: 0). Set to 1 to process audio input instead of video frames. +@item logit_scale +Logit scale for similarity calculation in CLIP/CLAP (default: 4.6052 for CLIP, 33.37 for CLAP). Values below 0 use the default. +@item temperature +Softmax temperature for CLIP/CLAP models (default: 1.0). Lower values make the output more peaked, higher values make it smoother. +@item forward_order +Order of forward output for CLIP/CLAP: 0 for media-text order, 1 for text-media order (default depends on model type). +@item normalize +Whether to normalize the input tensor for CLIP/CLAP (default depends on model type). Some scripted models already do this in the forward, so this is not necessary in some cases. +@item input_res +Expected input resolution for video processing models (default: automatically detected). +@item sample_rate +Expected sample rate for audio processing models (default: 44100). +@item sample_duration +Expected sample duration in seconds for audio processing models (default: 7). +@item token_dimension +Dimension of token vector for text embeddings (default: 77). +@item optimize +Enable graph executor optimization (0: disabled, 1: enabled). +@end table +@subsection Category Files Format +For CLIP/CLAP models, a hierarchical categories file can be provided with the following format: +@example +[RecordingSystem] +(Professional) +a photo with high level of detail +a professionally recorded sound +(HomeRecording) +a photo with low level of detail +an amateur recording +[ContentType] +(Nature) +trees +mountains +birds singing +(Urban) +buildings +street noise +traffic sounds +@end example +Each unit enclosed in square brackets [] creates a classification group. Within each group, categories are defined with parentheses () and the labels under each category are used to classify the input. +@subsection Examples +@example +Classify video using OpenVINO +ffmpeg -i input.mp4 -vf "dnn_classify=dnn_backend=openvino:model=model.xml:labels=labels.txt" output.mp4 +Classify video using CLIP +ffmpeg -i input.mp4 -vf "dnn_classify=dnn_backend=torch:model=clip_model.pt:categories=categories.txt:tokenizer=tokenizer.json" output.mp4 +Classify only person objects in a video +ffmpeg -i input.mp4 -vf "dnn_detect=model=detection.xml:input=data:output=detection_out:confidence=0.5,dnn_classify=model=clip_model.pt:dnn_backend=torch:tokenizer=tokenizer.json:labels=labels.txt:target=person" output.mp4 +Classify audio using CLAP +ffmpeg -i input.mp3 -af "dnn_classify=dnn_backend=torch:model=clap_model.pt:categories=audio_categories.txt:tokenizer=tokenizer.json:is_audio=1:sample_rate=44100:sample_duration=7" output.mp3 +@end example @section dnn_detect -- 2.34.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".