From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <ffmpeg-devel-bounces@ffmpeg.org> Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id 42B974BA12 for <ffmpegdev@gitmailbox.com>; Thu, 27 Mar 2025 14:07:06 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B0CEF687C6A; Thu, 27 Mar 2025 16:07:02 +0200 (EET) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 95D60687B8A for <ffmpeg-devel@ffmpeg.org>; Thu, 27 Mar 2025 16:06:55 +0200 (EET) Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-43cfecdd8b2so8506765e9.2 for <ffmpeg-devel@ffmpeg.org>; Thu, 27 Mar 2025 07:06:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743084415; x=1743689215; darn=ffmpeg.org; h=content-transfer-encoding:subject:from:to:content-language :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=p4TlC5Z8caXTx9kKGGSBdGB2KTDTNmwfcJ92AtTpYC0=; b=fmKGM7acwbZGOqn33CWzqw3dmKInsAY/BUw8onxIqJMeSzVRmZg6vU4pBjQByIdXvj WlGbQLbHRJRcAVNzIgRkzi0l2ojYyvhnU6rhC3i4GMb7dD5BdiiCxQGDYFUYqFMJ3W96 ziZoIGzosODBuoDRrJVaoBI33ZQ4ZjjtKSJLlS4bGtLTFjhfVp/Nn4dI68PpOhToSHb/ kaT9eNNoxcKqDfD1C0KqMa9D6Ph/n7rqjoJLF8kvti+PQL4W9RLngmrcCJXazjy1qjDx Sg8oirxbkx7WKuer6PmuOTWI4ltQvLs/swPEe7bgxoiaPIYl/IkKLi4B+gPlrkwAM68C 2GCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743084415; x=1743689215; h=content-transfer-encoding:subject:from:to:content-language :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=p4TlC5Z8caXTx9kKGGSBdGB2KTDTNmwfcJ92AtTpYC0=; b=gnTB81uZiECcIT6zTz8H6/CTY4gP6PSpy8MfYl6YP1Q2gGLuBKGm8s2/xihasDQmdQ mN/TFfAdn83vYPblM2ZiOhw5ktUIiGXVYMutBpr+T6qhdFX9DOCK83nDm3RWLOSwIZN0 fHaZ1VpaP40//XaLSWaPiS9tw4DOUcPI0QWXnbYDY49bUz4qddzlA1omiNQkPTypD2pG SC0mZz8iWkVJwcEPqusJSHezdydq1uPzzkQeNxh5F3SMb/QCxqbXNCZsHnYRM+iGnVJZ IWEyA4rWg8jFu/LB4iCr8NbA1nKXvzmkQsx5oX8Vh5iBqHUhl/9rhNXublnzOsqeK0HR Kp8Q== X-Gm-Message-State: AOJu0YxqXB3K57ZJTryvBBGLtfLuBKSvu4niSDWYFk0RwYxqKU04s++4 bg8abaquHP//8bA4ihS3hRZjZShhTCEqCZnfpnVj8eIMBzukMpx274vqOQ== X-Gm-Gg: ASbGncukqiQh5FJmyH69e+c2Zi9hbRnPmx+fl/31zmNSpUu/82xEH+OV3+EHwIw3Ym3 rBG5m+RHGCc6W42U9fAboEg78XEkxDMEBCbN27EHEBGpt1f670IAFIDDGFj6p49o7/jCmVpKnP0 QOhah8GC9og1BFqMVjkbRpzczPCwGHafautXLV650qt1vdNAm3Na6RLphg9+HbKhQ6YnWudX1DY 0w76/3ccnYVVYr2t4xdAlI3VBPvcpOP/Nt7viLAmutSdWrxQneCeciMGft1NdnvpWpzw453oYZx rsWIKRcNVOzZJLcQX1qnkr/Mo4NETaCJLwxT8E5n/gfP3EoyDiS4ILo6lRWj52wqr8FitUkcDNp IZx4Fim9voclBo7JExk9X8l0= X-Google-Smtp-Source: AGHT+IEYsARN4jGpbit65KVdwnuDlQe/Xtg+B7GCgZ2xVxrz0bfaJrwThMRqcbhwen0o9KViCrfW7Q== X-Received: by 2002:adf:9c88:0:b0:391:4bf2:6f0d with SMTP id ffacd0b85a97d-39ad179b06emr2589773f8f.52.1743084414496; Thu, 27 Mar 2025 07:06:54 -0700 (PDT) Received: from ?IPV6:2001:9e8:1db3:5e00:37df:8f46:1873:74a8? ([2001:9e8:1db3:5e00:37df:8f46:1873:74a8]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-39ac80e6814sm8555932f8f.56.2025.03.27.07.06.53 for <ffmpeg-devel@ffmpeg.org> (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 27 Mar 2025 07:06:53 -0700 (PDT) Message-ID: <5912ca0f-2a2d-497d-9747-bb630b6ec2d6@gmail.com> Date: Thu, 27 Mar 2025 15:06:53 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org> From: Leandro Santiago <leandrosansilva@gmail.com> Subject: [FFmpeg-devel] Proposal: extract DNN filters code to a different repository and rewrite it in C++ X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches <ffmpeg-devel.ffmpeg.org> List-Unsubscribe: <https://ffmpeg.org/mailman/options/ffmpeg-devel>, <mailto:ffmpeg-devel-request@ffmpeg.org?subject=unsubscribe> List-Archive: <https://ffmpeg.org/pipermail/ffmpeg-devel> List-Post: <mailto:ffmpeg-devel@ffmpeg.org> List-Help: <mailto:ffmpeg-devel-request@ffmpeg.org?subject=help> List-Subscribe: <https://ffmpeg.org/mailman/listinfo/ffmpeg-devel>, <mailto:ffmpeg-devel-request@ffmpeg.org?subject=subscribe> Reply-To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" <ffmpeg-devel-bounces@ffmpeg.org> Archived-At: <https://master.gitmailbox.com/ffmpegdev/5912ca0f-2a2d-497d-9747-bb630b6ec2d6@gmail.com/> List-Archive: <https://master.gitmailbox.com/ffmpegdev/> List-Post: <mailto:ffmpegdev@gitmailbox.com> Dear FFmpeg developers, I've been doing some experiments with out-of-tree filters [1] and I start feeling confident that the approach works well enough and, besides a "demo" filter written in C [2], I've implemented a filter in Rust [3]. That said, I propose: - extracting the DNN filters (dnn_detect, dnn_classify, dnn_processing) to another repository, within the ffmpeg umbrella, to be integrated to ffmpeg at build time by the users who require this feature. - rewriting those filters in C++. My rationale is: - the DNN code is quite niche, not used by most people. - There are some files on it written in C++, namely the libtorch backend, so moving it out would reduce the C++ code in the ffmpeg repository. - Besides the libtorch backend, the other backends (openvino and tensorflow) use C APIs which are essentially wrappers around C++ APIs (as both openvino and tensorflow base APIs are C++). I don't see much benefits on using C wrappers when the C++ API could be used directly, making the code cleaner and more reliable, as the C++ APIs offer more type safety than the C ones and, well, RAII. Outcomes: - Less C++ code on ffmpeg codebase. - Less ~6k LOC from the ffmpeg codebase, reducing a little maintenance burden. - The DNN filters can evolve separately, and even have a different release cycle, ownership and collaboration model. - After the C++ rewrite, the DNN code will be type safer, likely shorter, and with less dependencies, as it'll use the original C++ APIs instead of the C wrappers. Proposed steps: - polish and merge my patch that enables out-of-tree filters [4]. - extract the libavfilter/*dnn* code to an external repository, say git.ffmpeg.org/dnn-filters.git, keeping the git history. - adapt the dnn-filters repository to be built "out-of-tree" without breaking any public API. - rewrite the openvino backend and tensorflow backends to C++, removing the dependency on the C API wrappers. - rewrite the DNN filters in C++. Open issues: - Currently selecting the DNN backend be done via ./configure --enable-(libopenvino|libtensorflow|libtorch), which is hardcoded in the build system. I don't know how yet it can be done via the out-of-tree approach. An easy way is to simply pass the backend via environment, like in `DNN_BACKEND=openvino ./configure`, but I'd like to know how you folks would do it. - several macros used in the C code won't work on the C++ rewrite, requiring new C++ specific macros to be created. This affects only the last step of the aforementioned steps, though. - There is no way yet to integrate the documentation of out-of-tree filters into ffmpeg docs. - There will still be some "remains" of the DNN code in libavutil, namely the detection boxes related API. - During the extraction process, the DNN code should be "frozen", and any existing non merged patch should either be merged before the extraction, or be adapted to the the new repo. - Some extra coordination will be needed not to break any production code during the migration. I don't think this is the case, as no public API needs change, but I might be wrong. - I've performed splitting git repositories before, but never with signed commits, if there are any in the DNN code. It's possible that the signatures will break in the process, but I am not very sure right now. In any case, I would like to know whether the current proposal aligns with the needs of the community or what needs change. If it does, I'd like to volunteer myself to head the work on it, depending on the project roadmap. [1] https://ffmpeg.org/pipermail/ffmpeg-devel/2025-March/341105.html [2] https://gitlab.com/leandrosansilva/ffmpeg-extra-filter-example [3] https://gitlab.com/leandrosansilva/ffmpeg-track-sort-filter [4] https://patchwork.ffmpeg.org/project/ffmpeg/patch/2477728b-6217-446a-9a15-830f028baea0@gmail.com/ _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".