From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id C87F14E322 for ; Mon, 10 Mar 2025 19:54:38 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0187668E093; Mon, 10 Mar 2025 21:54:27 +0200 (EET) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C860E68C1FB for ; Mon, 10 Mar 2025 21:54:20 +0200 (EET) Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-38f403edb4eso2538929f8f.3 for ; Mon, 10 Mar 2025 12:54:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741636460; x=1742241260; darn=ffmpeg.org; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:to:from:from:to:cc:subject :date:message-id:reply-to; bh=lNae+hhfM9jytFKssEYhTSStjHcuWLjnQQiauVcI2qw=; b=Ah63F0NuhD7LW+EW/fDhgCBSF6dyeTXupyH6IbZqIoNNYO8Jx9jeaPgYhUO/eQaA4T S8k1dDFmxUU8VnFpvrf7yXsinyD1I8sQC0iGnExaTTyZjFYCfv9pJNA84TZqTtlZCHsI RX2w7CuaNVwqVZI3Mi6Gh0WyJP+RYV7wPmXgIu02NuNXsdJbvCP2CvEreMEoSaNn6Jlr QRefItScDECJZGUB+rUGAGDkTvvc+aOfSJyp7COPXysEjpM3Qda0QteAp+1USVd6cGRr lGV7v/DJHXkVpwQNSJGtVIcd18lKMIvSSyO7VxRyN6earep4M5vtTQH81KHh6u47WvgM e6oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741636460; x=1742241260; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=lNae+hhfM9jytFKssEYhTSStjHcuWLjnQQiauVcI2qw=; b=Masy8r+PxmR32QOQ7xXJhBT3hPt9/MXmSpuZ4CT0iahP/XXZ6gzsx6Moc9Tqd2VKxL wGTG5opbBwcE3/4n5cQ4DLZD4QQLefIgFR5cUYamyA7H5pj7Uivo+3OC055DIpavb8Gs gtU8xCAseE/1Us//mL2Qqbboc7Uv+fc5LcSr2Y6CkbOICf+2Dy7p2EP1tnu3aZUnvuLK zoBC0gSsVvKXSy4z3s/A2L39tHqmFCDnYsVfCGgQXYssKHoDqnsQ2hm5HunxV5GrOL7I e3Yy5d26CH5vRR4gHfvlQmxzc17ac9Wbm5xW4q4pMAybBILE6fOczPYuoQq/ETrFP+lC PoHQ== X-Gm-Message-State: AOJu0YyhAe3DgpMhD1gUv6cFcRkk/P+oeBcyB41eQsECbz9nbJlAk6P3 NaQ/KlId+dBbom3DxALm5WghAIBbfcOfvXOFNPxrXHu4ZcniwZNVVB+QYw== X-Gm-Gg: ASbGncvlX/GRWIo/LFZRUd4bKaCAedFdNTUeAODBOBk5H6FGFbcmmAZiTJz30kgENXB ttzXunDomycdgE5BWTme8wNmj1BIiZpeMqfI9eykGc2QPmUiGqeyz/ARxC+SY691Nqp7sR/1BwI lPuuRL/w1LWI6ke3KbpRVEBl8eZqR+G1YlZtrEpl2Yo5Iij0SY+h6AuArY1VqlmnyZVABIrv4V8 bq/pa4ECjM773cevWS69t7AySo5eto0k0Y6eZJlpHsah4VIoGZuIXwHfhASi8QGkBRqWSn6V1Gz PEXEkY6PfBJgdo1qxfq/ziX0zQ+iAIgKQVGHyq0k8SGfY7YouhMGGiL4JV1c+db+RoNUm3FRqpC W11l9MMjKsAEgoHd8 X-Google-Smtp-Source: AGHT+IH9+e6r/QuZAAWoHvMf4BYmeLyMsxri9QNvpOCiIOrsN/gGX7yX51DKBw3DRchuq4G8gybEiw== X-Received: by 2002:a05:6000:4184:b0:391:4559:8773 with SMTP id ffacd0b85a97d-3914559896fmr4142824f8f.51.1741636459681; Mon, 10 Mar 2025 12:54:19 -0700 (PDT) Received: from MK2 (80-108-16-220.cable.dynamic.surfer.at. [80.108.16.220]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-43d020b7e5asm4765765e9.22.2025.03.10.12.54.19 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Mar 2025 12:54:19 -0700 (PDT) From: To: Date: Mon, 10 Mar 2025 20:54:19 +0100 Message-ID: <003f01db91f6$36603a60$a320af20$@gmail.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 16.0 Content-Language: en-at Thread-Index: AduR9jPuviQ39k2UQB6qdzuC/X92mw== Subject: [FFmpeg-devel] [PATCH v2 FFmpeg 11/20] libavfilter/dnn_filter_common: dnn execute functions for CLIP and CLAP X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: Signed-off-by: MaximilianKaindl --- libavfilter/dnn_filter_common.c | 81 +++++++++++++++++++++++---------- libavfilter/dnn_filter_common.h | 34 +++++++------- 2 files changed, 76 insertions(+), 39 deletions(-) diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c index 899e869cca..f3117748ae 100644 --- a/libavfilter/dnn_filter_common.c +++ b/libavfilter/dnn_filter_common.c @@ -1,20 +1,20 @@ /* - * This file is part of FFmpeg. - * - * FFmpeg is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 2.1 of the License, or (at your option) any later version. - * - * FFmpeg is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public - * License along with FFmpeg; if not, write to the Free Software - * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA - */ +* This file is part of FFmpeg. +* +* FFmpeg is free software; you can redistribute it and/or +* modify it under the terms of the GNU Lesser General Public +* License as published by the Free Software Foundation; either +* version 2.1 of the License, or (at your option) any later version. +* +* FFmpeg is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* Lesser General Public License for more details. +* +* You should have received a copy of the GNU Lesser General Public +* License along with FFmpeg; if not, write to the Free Software +* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +*/ #include "dnn_filter_common.h" #include "libavutil/avstring.h" @@ -87,10 +87,10 @@ static int ff_dnn_init_priv(DnnContext *ctx, DNNFunctionType func_type, AVFilter if (backend == DNN_TH) { if (ctx->model_inputname) av_log(filter_ctx, AV_LOG_WARNING, "LibTorch backend do not require inputname, "\ - "inputname will be ignored.\n"); + "inputname will be ignored.\n"); if (ctx->model_outputnames) av_log(filter_ctx, AV_LOG_WARNING, "LibTorch backend do not require outputname(s), "\ - "all outputname(s) will be ignored.\n"); + "all outputname(s) will be ignored.\n"); #if (CONFIG_LIBTOKENIZERS == 0) if ((func_type == DFT_ANALYTICS_CLIP || func_type == DFT_ANALYTICS_CLAP)) { @@ -126,14 +126,14 @@ static int ff_dnn_init_priv(DnnContext *ctx, DNNFunctionType func_type, AVFilter void *child = NULL; av_log(filter_ctx, AV_LOG_WARNING, - "backend_configs is deprecated, please set backend options directly\n"); + "backend_configs is deprecated, please set backend options directly\n"); while (child = ff_dnn_child_next(ctx, child)) { if (*(const AVClass **)child == &ctx->dnn_module->clazz) { int ret = av_opt_set_from_string(child, ctx->backend_options, - NULL, "=", "&"); + NULL, "=", "&"); if (ret < 0) { av_log(filter_ctx, AV_LOG_ERROR, "failed to parse options \"%s\"\n", - ctx->backend_options); + ctx->backend_options); return ret; } } @@ -199,9 +199,9 @@ int ff_dnn_get_input(DnnContext *ctx, DNNData *input) int ff_dnn_get_output(DnnContext *ctx, int input_width, int input_height, int *output_width, int *output_height) { char * output_name = ctx->model_outputnames && ctx->backend_type != DNN_TH ? - ctx->model_outputnames[0] : NULL; + ctx->model_outputnames[0] : NULL; return ctx->model->get_output(ctx->model, ctx->model_inputname, input_width, input_height, - (const char *)output_name, output_width, output_height); + (const char *)output_name, output_width, output_height); } int ff_dnn_execute_model(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame) @@ -231,6 +231,41 @@ int ff_dnn_execute_model_classification(DnnContext *ctx, AVFrame *in_frame, AVFr return (ctx->dnn_module->execute_model)(ctx->model, &class_params.base); } +int ff_dnn_execute_model_clip(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame, const char **labels, int label_count, const char* tokenizer_path, char *target) +{ + DNNExecZeroShotClassificationParams class_params = { + { + .input_name = ctx->model_inputname, + .output_names = (const char **)ctx->model_outputnames, + .nb_output = ctx->nb_outputs, + .in_frame = in_frame, + .out_frame = out_frame, + }, + .labels = labels, + .label_count = label_count, + .tokenizer_path = tokenizer_path, + .target = target, + }; + return (ctx->dnn_module->execute_model)(ctx->model, &class_params.base); +} + +int ff_dnn_execute_model_clap(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame, const char **labels, int label_count, const char* tokenizer_path) +{ + DNNExecZeroShotClassificationParams class_params = { + { + .input_name = ctx->model_inputname, + .output_names = (const char **)ctx->model_outputnames, + .nb_output = ctx->nb_outputs, + .in_frame = in_frame, + .out_frame = out_frame, + }, + .labels = labels, + .label_count = label_count, + .tokenizer_path = tokenizer_path, + }; + return (ctx->dnn_module->execute_model)(ctx->model, &class_params.base); +} + DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame, AVFrame **out_frame) { return (ctx->dnn_module->get_result)(ctx->model, in_frame, out_frame); diff --git a/libavfilter/dnn_filter_common.h b/libavfilter/dnn_filter_common.h index 301441a37a..a782f64a1f 100644 --- a/libavfilter/dnn_filter_common.h +++ b/libavfilter/dnn_filter_common.h @@ -1,20 +1,20 @@ /* - * This file is part of FFmpeg. - * - * FFmpeg is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 2.1 of the License, or (at your option) any later version. - * - * FFmpeg is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public - * License along with FFmpeg; if not, write to the Free Software - * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA - */ +* This file is part of FFmpeg. +* +* FFmpeg is free software; you can redistribute it and/or +* modify it under the terms of the GNU Lesser General Public +* License as published by the Free Software Foundation; either +* version 2.1 of the License, or (at your option) any later version. +* +* FFmpeg is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* Lesser General Public License for more details. +* +* You should have received a copy of the GNU Lesser General Public +* License along with FFmpeg; if not, write to the Free Software +* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +*/ /** * @file @@ -63,6 +63,8 @@ int ff_dnn_get_input(DnnContext *ctx, DNNData *input); int ff_dnn_get_output(DnnContext *ctx, int input_width, int input_height, int *output_width, int *output_height); int ff_dnn_execute_model(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame); int ff_dnn_execute_model_classification(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame, const char *target); +int ff_dnn_execute_model_clip(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame, const char **labels, int label_count, const char* tokenizer_path, char *target); +int ff_dnn_execute_model_clap(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame, const char **labels, int label_count, const char* tokenizer_path); DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame, AVFrame **out_frame); int ff_dnn_flush(DnnContext *ctx); void ff_dnn_uninit(DnnContext *ctx); -- 2.34.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".