From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.ffmpeg.org (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id 686874755B for ; Thu, 25 Dec 2025 10:38:38 +0000 (UTC) Authentication-Results: ffbox; dkim=fail (body hash mismatch (got b'MGfrIbnmt66kLd2g628brkZSH9WxEjFeKV0Vlq1x6vw=', expected b'XPb7nxL3dTOcCF1Gd4I8/Too1pR++qrryJDHdogq6S4=')) header.d=gmail.com header.a=rsa-sha256 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ffmpeg.org; i=@ffmpeg.org; q=dns/txt; s=mail; t=1766659101; h=to : date : message-id : mime-version : reply-to : subject : list-id : list-archive : list-archive : list-help : list-owner : list-post : list-subscribe : list-unsubscribe : from : cc : content-type : content-transfer-encoding : from; bh=mybakqbXC9qt/filSIaomKMXUKYsciEYzD5iw2pzFnk=; b=lbEDXEyFaREaOIeTc5Fmu9HL+/JPMlO5ThoX7SPQQeOXnZGvSKX3lXhlzOVC+FkfTN6ob 5MPVGYt+Gfr0WNsBXBHyI9mBMTFJel8R4MwrkJKX0/ALqTCUJ6c5nfPNN2kpUjBnmAqVLae x4xDTAu7sDRBNCDv/BrHUoKyJSlb732+ddFd+VfhTBJkFM+224h14kHzKfHTBBTtEGvtECr g5ez7Ly2o+3TdGMUNmEDrz9Vd95bS2sfDIBde8WX06W0lgdM/MrW23KZq2WMwKl4hLXlJCn p9jy9wQipoLCeULRxjsggmrlg5mWMt1C+jZqqDpKggffld3g+TcJLKl1gOAw== Received: from [172.20.0.4] (unknown [172.20.0.4]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTP id 26178690B8F; Thu, 25 Dec 2025 12:38:21 +0200 (EET) ARC-Seal: i=1; cv=none; a=rsa-sha256; d=ffmpeg.org; s=arc; t=1766659086; b=oLt6d/OhsgyGRz3AR3tqF0bvo6n/jlCxQVZ+2j/jPC1g/d8ubDU9CHPiy8qd43eVGuUAu CQzmwJfYUcylDUlO/iz2pNze+tMKl7kDiUfQXRAxXHPlokK0QUg+CQjE4+VOQOBOJ1o9P4y cUDlJ0sWQvvLG57fY58afT9rec/kUq8/jivDommZP8Y4BomTO5+YB1dVO5rOIEyIWzUa/H1 4h59mjDxheG2zEgCi0NqicKmifviCoH6wQl6GkI22dPr3HO0IVVrOLAsNwb5DVeVMsOnJU9 wmLmoKhkOe/k/XElUr0AipPK8a0UQdVOkjXSuV1scMcq+A/ZSjxhmTZOrxpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=ffmpeg.org; s=arc; t=1766659086; h=from : sender : reply-to : subject : date : message-id : to : cc : mime-version : content-type : content-transfer-encoding : content-id : content-description : resent-date : resent-from : resent-sender : resent-to : resent-cc : resent-message-id : in-reply-to : references : list-id : list-help : list-unsubscribe : list-subscribe : list-post : list-owner : list-archive; bh=MGfrIbnmt66kLd2g628brkZSH9WxEjFeKV0Vlq1x6vw=; b=pXkJEM/yo0Gu/Zj7NOBDy1MiRu27KoX/3mLFybyHxwgzabsQ5oRC6tjIxZk2mJTeaIIDr kEXIzOWMNHRS171iPh/E5FrUc4jAYhAVFbWUyONUhJMDck+HDJURPkkT5DfQhUu6eEh1GWk iu24OfMgNjyjCIiufezFx1W7QlbmNajN0TzESRk+5WEMIethLPRo608gMj1xxvbeED0vgLj myuKn3Z1x5AlGmWQAURgOVyYfVEHS9UFhCZDMzTEGD0wPXs5pvpAHBjf/niSr/kwNUYPPzU 0CkGztByfT0sVQC+ZjD5O2qEezoy0nOB2f04XNRKuBc4S/Q43XeuK1XgFWKg== ARC-Authentication-Results: i=1; ffmpeg.org; dkim=pass header.d=gmail.com; arc=none; dmarc=pass header.from=gmail.com policy.dmarc=quarantine Authentication-Results: ffmpeg.org; dkim=pass header.d=gmail.com; arc=none (Message is not ARC signed); dmarc=pass (Used From Domain Record) header.from=gmail.com policy.dmarc=quarantine Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTPS id 0329768FD19 for ; Thu, 25 Dec 2025 12:37:52 +0200 (EET) Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2a0bae9aca3so94697495ad.3 for ; Thu, 25 Dec 2025 02:37:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766659070; x=1767263870; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=XPb7nxL3dTOcCF1Gd4I8/Too1pR++qrryJDHdogq6S4=; b=ROJpek0NuuMPIs2XT6SWvu3erombh5Xe9e34THlBPBwC3dDJTx92VQZYqUx62UKkk3 lqwoQDahRJ/xp8lZRcT8SZUx5V1lqzFil3A1kV3UngjvH+KjBUwGvUXKwWo7zSeS5fTP 9FInhivTF0XJNz3gFBPMzel07O2BPdoBEqsGjtSmqAr9NmCMjkALYTYrHHbi3gWvRUH/ BDzEv2c4myhvUIwrZoFFDpHDWOB/98tRUgmL+0sm101gnSQoVhuFQLgUvqBcdxnMOyWy dKR94rifVWKPpI45wpOU7ovMBxrAk6hkZctOdRh+ZLCSVdlR2hgGYOSoKQ94AjKu5FWu nWSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766659070; x=1767263870; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=XPb7nxL3dTOcCF1Gd4I8/Too1pR++qrryJDHdogq6S4=; b=TxqZV1Rj8WOYosB+iQBslqUAmtr38o5ds9iCge+mFvR1wf58eBXHS7OvqN3+wjZlSE bMDGu0A6/0u0l0KWf94oMsDXzGeV5DP5/N5u9OY+ePkkRVIhnRymSf/fkdDbSKDtH842 ZwfPEKaH2HbItC+wOcYwfNMrwXC3nk1ukQ9jNSsZ4x8czlaGuCnT55RPDFeVatFl6BK8 2Lap//nRpaG49k+H/AZBRBjoP+dPMTbOEQ0JGQ27v2Bn/09idhMChOdtN38ZkNpQwOQq GMCVJ+h/hbhA/KGpczaWVLAXtdKWa6kdIcOexy3NMmHzCr/rPw6jV96J1uXErazjzSEY OeIw== X-Gm-Message-State: AOJu0Yx21nALpWsu0ugFqwTURXIaxLiytuckRxVpSWd5L8qCRufyj2Ff dfc2QH/5BKxjjvjwsSUPFVYlp/KiQf6uOJ9x7pprE9Po2S1ldNLB1/uvqFQ/1Q== X-Gm-Gg: AY/fxX7WkaX3oZFmuMSFulFtnDxxlb/XBbzUpdW2WGw5I8WuGpBvDRthEz216nThwJY 05U4Dmi7g+mchGcFuDJmfaoCO75KxaITbWdg5MZUbTuO9A1sEebP9rA08jvj9od1et6yDnqBAxt 3+YbEiQMmiCZ0tWj6Tk/yCxYXVeqOieTr4JF3e7HE3d2sZJFzkLiJCUtrbSe96Qv/A3F+YJEv+s ORS8Hy47S4iQ6mD/8DDEIhauSxFYWe/xB+ksFtP8vAtZDhyFqeOrh/3mubhmXmBxRVsRo9+IBnn L5VBs2XEaXcMfOm7+15xN+/l/hcLogjGbJ1GJjctdXUou5FkvEqaCmTNsXkzD3myR+3vV8P0i+A JZWpOMy5iVgigCGj7/0OsVtL1GK283UWNAs2bZIaPJBA9bLF8Ls+B5QEPOxKYQAd4+E6JjMd68w WRzh0y+cL+rNHofjSQ+RV++NYhb0tRpM0Cgs+MAPSy X-Google-Smtp-Source: AGHT+IHyfjA/r82FzAPV4Un/ULnDBiDePa0T0ZqgFwZQNaY+MT/YGVDXAtdDO8afiGcvAGxeaP7Bfg== X-Received: by 2002:a17:903:41c8:b0:2a0:ba6d:d101 with SMTP id d9443c01a7336-2a2f24248b5mr213579055ad.21.1766659070385; Thu, 25 Dec 2025 02:37:50 -0800 (PST) Received: from Raja-Rathour-ASUS-TUF-Gaming-A15.. ([103.240.236.29]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3c88d0asm175472595ad.40.2025.12.25.02.37.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Dec 2025 02:37:49 -0800 (PST) To: ffmpeg-devel@ffmpeg.org Date: Thu, 25 Dec 2025 16:03:19 +0530 Message-ID: <20251225103606.58855-1-imraja729@gmail.com> X-Mailer: git-send-email 2.48.1 MIME-Version: 1.0 Message-ID-Hash: KYZERFCI3BXWP6ZDO56FI5GHANYC7GBG X-Message-ID-Hash: KYZERFCI3BXWP6ZDO56FI5GHANYC7GBG X-MailFrom: SRS0=vXVA=67=gmail.com=imraja729@ffmpeg.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; header-match-ffmpeg-devel.ffmpeg.org-0; header-match-ffmpeg-devel.ffmpeg.org-1; header-match-ffmpeg-devel.ffmpeg.org-2; header-match-ffmpeg-devel.ffmpeg.org-3; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list Reply-To: FFmpeg development discussions and patches Subject: [FFmpeg-devel] [PATCH 1/3] avfilter/dnn_backend_torch: implement async execution List-Id: FFmpeg development discussions and patches Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Raja Rathour via ffmpeg-devel Cc: Raja Rathour Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Archived-At: List-Archive: List-Post: Signed-off-by: Raja Rathour This patch implements asynchronous model execution for the LibTorch backend in FFmpeg's DNN module. Key changes: - Integrated a worker thread and a pending queue to handle inference. - Prevents the main filter thread from blocking during model execution. - Aligns LibTorch backend behavior with the existing OpenVINO async implementation. - Improves overall throughput for deep learning filters using LibTorch. The implementation has been tested with various torch models to ensure stability and correct frame synchronization. --- libavfilter/dnn/dnn_backend_torch.cpp | 115 +++++++++++++++++++++++++- 1 file changed, 111 insertions(+), 4 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index 2e4326d9d4..ad81aff8da 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -25,6 +25,10 @@ #include #include +#include +#include +#include +#include extern "C" { #include "dnn_io_proc.h" @@ -39,9 +43,16 @@ typedef struct THModel { DNNModel model; DnnContext *ctx; torch::jit::Module *jit_model; - SafeQueue *request_queue; + SafeQueue *request_queue; // Holds available/idle request slots Queue *task_queue; Queue *lltask_queue; + + // --- Async Support --- + SafeQueue *pending_queue; // Holds requests waiting for inference + std::thread *worker_thread; // The background worker + std::mutex *mutex; // Protects the condition variable + std::condition_variable *cond; // Wakes up worker when new task arrives + std::atomic worker_stop; // Flag to stop the thread } THModel; typedef struct THInferRequest { @@ -119,6 +130,32 @@ static void dnn_free_model_th(DNNModel **model) return; th_model = (THModel *) (*model); + + // --- Stop and Join Worker Thread --- + if (th_model->worker_thread) { + { + std::lock_guard lock(*th_model->mutex); + th_model->worker_stop = true; + } + th_model->cond->notify_all(); + + if (th_model->worker_thread->joinable()) { + th_model->worker_thread->join(); + } + delete th_model->worker_thread; + delete th_model->mutex; + delete th_model->cond; + } + + if (th_model->pending_queue) { + // Clear remaining items (if any) + while (ff_safe_queue_size(th_model->pending_queue) != 0) { + ff_safe_queue_pop_front(th_model->pending_queue); + } + ff_safe_queue_destroy(th_model->pending_queue); + } + // ----------------------------------- + while (ff_safe_queue_size(th_model->request_queue) != 0) { THRequestItem *item = (THRequestItem *)ff_safe_queue_pop_front(th_model->request_queue); destroy_request_item(&item); @@ -318,6 +355,41 @@ err: } } +// --- Worker Thread Function --- +static void th_worker_thread(THModel *th_model) { + while (true) { + THRequestItem *request = NULL; + + { + // Acquire lock to check condition + std::unique_lock lock(*th_model->mutex); + + // Wait until: We are told to stop OR there is work in the queue + th_model->cond->wait(lock, [&]{ + return th_model->worker_stop || ff_safe_queue_size(th_model->pending_queue) > 0; + }); + + // If stopped and no work left, exit + if (th_model->worker_stop && ff_safe_queue_size(th_model->pending_queue) == 0) { + break; + } + + // Get work + request = (THRequestItem *)ff_safe_queue_pop_front(th_model->pending_queue); + } + + // Process work (Lock released so we don't block submission) + if (request) { + int ret = th_start_inference(request); + if (ret != 0) { + av_log(th_model->ctx, AV_LOG_ERROR, "Async inference failed\n"); + } + // Always callback to clean up and notify FFmpeg + infer_completion_callback(request); + } + } +} + static int execute_model_th(THRequestItem *request, Queue *lltask_queue) { THModel *th_model = NULL; @@ -343,9 +415,24 @@ static int execute_model_th(THRequestItem *request, Queue *lltask_queue) if ( ret != 0) { goto err; } + + // --- EXECUTION LOGIC (ASYNC vs SYNC) --- if (task->async) { - avpriv_report_missing_feature(th_model->ctx, "LibTorch async"); + // 1. Acquire lock + std::lock_guard lock(*th_model->mutex); + + // 2. Push to pending queue + if (ff_safe_queue_push_back(th_model->pending_queue, request) < 0) { + return AVERROR(ENOMEM); + } + + // 3. Wake up worker + th_model->cond->notify_one(); + + // 4. Return immediately (Success) + return 0; } else { + // Synchronous fallback ret = th_start_inference((void *)(request)); if (ret != 0) { goto err; @@ -484,6 +571,25 @@ static DNNModel *dnn_load_model_th(DnnContext *ctx, DNNFunctionType func_type, A goto fail; } + // --- INITIALIZE ASYNC QUEUE AND THREAD --- + th_model->pending_queue = ff_safe_queue_create(); + if (!th_model->pending_queue) { + av_log(ctx, AV_LOG_ERROR, "Failed to create pending queue\n"); + goto fail; + } + + try { + th_model->mutex = new std::mutex(); + th_model->cond = new std::condition_variable(); + th_model->worker_stop = false; + + // Start worker thread + th_model->worker_thread = new std::thread(th_worker_thread, th_model); + } catch (...) { + av_log(ctx, AV_LOG_ERROR, "Failed to initialize worker thread or mutexes\n"); + goto fail; + } + model->get_input = &get_input_th; model->get_output = &get_output_th; model->filter_ctx = filter_ctx; @@ -519,7 +625,8 @@ static int dnn_execute_model_th(const DNNModel *model, DNNExecBaseParams *exec_p return AVERROR(ENOMEM); } - ret = ff_dnn_fill_task(task, exec_params, th_model, 0, 1); + // Set 'async' flag based on context (ctx->async) instead of hardcoded 0 + ret = ff_dnn_fill_task(task, exec_params, th_model, ctx->async, 1); if (ret != 0) { av_freep(&task); av_log(ctx, AV_LOG_ERROR, "unable to fill task.\n"); @@ -580,4 +687,4 @@ extern const DNNModule ff_dnn_backend_torch = { .get_result = dnn_get_result_th, .flush = dnn_flush_th, .free_model = dnn_free_model_th, -}; +}; \ No newline at end of file -- 2.48.1 _______________________________________________ ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org