From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 1B3CD49F3F for ; Fri, 15 Mar 2024 02:01:58 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0866968D03D; Fri, 15 Mar 2024 04:01:55 +0200 (EET) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CFA1168CDA0 for ; Fri, 15 Mar 2024 04:01:47 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710468113; x=1742004113; h=from:to:subject:date:message-id:references:in-reply-to: content-transfer-encoding:mime-version; bh=hI3izK2FupFQbngqNk5Fhgg7RgUaYORGtP+Hl6U9BAc=; b=lMnI7Rrcu/31zgpgOggZeAyx5Z5CsETgyxUw1Mab8s++5Fwv8Fzji/vV HxlnFvM7wRBthQzJsGPVDukHU1U0kGRb6JdMW16MKLBRJvDBcJQiLqLXz 87a6/hLIZXsfEFjeburp5Lwoey8Ol+6q6KZVZQmQjms7Q3njwOjZ796O/ 0sS16JA979ee4o6bC732DHmQ9uZ/TH59aMOEFkRs3JgxXtZSFBmeikb9f 1xlMZsT0pO2iFXifo0WcZddYuvDMkqM/5xPeKKDfUFWONMTrEs352DJCC FQpZC6wS4Dov92JZ64Amc7grESJq+2oqvLhlMvE27XGNDU9WZ1fFuYKdz w==; X-IronPort-AV: E=McAfee;i="6600,9927,11013"; a="16473282" X-IronPort-AV: E=Sophos;i="6.07,127,1708416000"; d="scan'208";a="16473282" Received: from fmviesa001.fm.intel.com ([10.60.135.141]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Mar 2024 19:01:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,127,1708416000"; d="scan'208";a="43531374" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by fmviesa001.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 14 Mar 2024 19:01:43 -0700 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Thu, 14 Mar 2024 19:01:43 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Thu, 14 Mar 2024 19:01:43 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Thu, 14 Mar 2024 19:01:43 -0700 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.168) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Thu, 14 Mar 2024 19:01:42 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KUcBwvZZakf/ruLhaCNXsCiYh2qSmo608Aq1rZRdZtEVJkF5thSoDa6PpjwFLsHMx82CKS4PDzDm4OvLp2SF862YM4t2/ilgYYqgnoFldNUgR/h51LS1e0YQCpqesGi20ucSQp9hdbLeJC8xbE5ceq2sMkzW7qPkSeWH/Lkz9QWgAWTGlQwHiT3oU+ZZgS+ZOFHbv+uyGTf1YiHHDNW+1nGNLdodjwleyFgfo98xgokYsjK3GEtiFinahxL4cJetJH8lFdT+Z+lGr1lsWLY1XSwCzxtz/LNHOBOhRdQWLlxp3E1ev6lELsLPZq/AMvdZuXPg4xBc8SsGjcZjIqNBeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SKRl8iJXM2oqLttbQ+xwfCWUPhkm3Gji5r0WesgpITM=; b=MxAg+I4HMyssOYUHwTwVs15QlxyCkWjb8cJP3umMdJGDDH9kGpxN4/8C4jhGI/YHogQKRHzSzbjr1wanh3RGOz+lVevDv+m18fB5LGPup0GH7xpfByNx3XdioKmZ+Jpulk6+7enUwPOFMXaxyMjcdDHb0aB9xnDPJeNT/LfqzU8JUY0e1QZvQfsJN7F3HYlJoDyr9pg8a36upnMEc9D6U27lVPb4QBleg6SIj2RdpLb/GvTfs4z0S9c8ZW2Voi7lAmsmYrP1yqnatn0vSTXlXtM8EzJoaaTBMnAEtIExsmRSDGk/tom8871ZDzLOCDOyWZ2cEpzJFGg/Uya0Sixa2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from IA1PR11MB6396.namprd11.prod.outlook.com (2603:10b6:208:3ab::10) by CY8PR11MB7362.namprd11.prod.outlook.com (2603:10b6:930:85::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.20; Fri, 15 Mar 2024 02:01:40 +0000 Received: from IA1PR11MB6396.namprd11.prod.outlook.com ([fe80::1245:6bac:7bec:d07c]) by IA1PR11MB6396.namprd11.prod.outlook.com ([fe80::1245:6bac:7bec:d07c%5]) with mapi id 15.20.7386.017; Fri, 15 Mar 2024 02:01:40 +0000 From: "Chen, Wenbin" To: FFmpeg development discussions and patches Thread-Topic: [FFmpeg-devel] [PATCH v5] libavfi/dnn: add LibTorch as one of DNN backend Thread-Index: AQHac3FfvmBbTCGwp0yNWkYwpDSgo7E3IV2AgADucdA= Date: Fri, 15 Mar 2024 02:01:40 +0000 Message-ID: References: <20240311050229.1692658-1-wenbin.chen@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: IA1PR11MB6396:EE_|CY8PR11MB7362:EE_ x-ms-office365-filtering-correlation-id: a701efa0-4bc6-4651-dbb0-08dc4493db01 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: QAUoc5+59opF8ay6Q6Nh8KmiUI63GGQ4ISLM0bBLIh7ljJsusgkcgrdnu4/PHP8hBpO2NpSryySZyhtbX0+C9No9F7BMolwJwYoHYR9YpPX6oerbubaXNekiIn7v68kv3HOZCWaEgljNDA4kjcw99s5T3E/7vPmoNqxGLnBph12s2v6CJqfwo+gACRkTYKboheBHenXowLvXl/V8VK3arcDXYL7yJVUTSPTa18zLXsaTBHtbtpI0fVk4GjVvePGpAZDkHAgVfXe14CJkUgzmI1Xi6fRXlIL8JZylfhXMomiK3IBGGQroXgxHlLu4b7RdXkVandV7Pv+CcvMUFWgcqu5BeX10KBxvx24FqKLIF7KlWmBgqaGs7tjYuVP55S/BxOaGVZn4dwFKDTACrsFsQcrCiqR3E0+NU5EQ5999qBpoASznsSn8gkKvm0/FmO+ufKboQpsjXFmUoOwRplWsuTLgmTGoPGQ6Vbsr2R6qjSs23QZNo/EK3+aI2qL2ja1aWvtDa7n6t5nSrQGc8lEUhEiMAaKxycopONA/OfNNSJGpk4CHqIcw0NXMUEQXh/lAtRZIUGvazc7y34MOb1hRKqHyNpA9BG7FNK6HhtJoyr5wT6fSlt6jxAYzjmKatcSanaWAGYHydT3bm2l/BAaaqBPbJsX0SMTMfXbreFh1yRg= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA1PR11MB6396.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366007)(376005)(1800799015)(38070700009); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Fb5oOkoMe5slzMm4tAerS/+/gQkN6VNpSrkSibtR9BTCOxoJH/u0QdUA8b3k?= =?us-ascii?Q?I64l0OJae/j7h7NPIaTD+VrCCDc6Ts06+OwbEXfgZaqCeWpO+hRe2ThjIE7w?= =?us-ascii?Q?hu/+kolt3ixvvKIIYboayZ5b8cXZWCMtDopViqu6csZqnUvEYPMMkDz92Gn1?= =?us-ascii?Q?zwgezk5qHCVvOSyMyofkpM8x0X91Cnh31bRGox3XagWwxsesm1wGQL/fPfhC?= =?us-ascii?Q?k5GSyO1G9sKoC2L7RPhjLLFvjYuv7kBFpdJSPJgV0JjTaJq6daHoj4yky42A?= =?us-ascii?Q?L9qHcwE/AYPJPBOhO0bcz1Xa5OIopqPKZq8TY1pJnU4Awth1kjoizGny6ZbN?= =?us-ascii?Q?77xqOl7xOI0qiJ+61+gVNMczSZy4nsSwZmQzIsDpJBdymHIi/5y2Cdelw3Ss?= =?us-ascii?Q?7dvoZoAmqRNTTsWhi2HbGSr701f/uu9S3vobdjFr6jlLQ2mi6mdi+F1Z+it4?= =?us-ascii?Q?MMdeJ+nz0LTmFK6P9q172XknDL1cK60kUnVcZqWUdPZLEglt3UJcIqFqRirh?= =?us-ascii?Q?KM8JDaZWt9la94E/oFSpq8tpYrUc8Cz1+/G1ykhr/AJVlZL/lDHnr/3yptxJ?= =?us-ascii?Q?/IgHjKTxq13KX61V+uLFww7NBwz1Jre8cWCq5CAdNXMeW0Ys1OY0+ZxJz8q4?= =?us-ascii?Q?qQ83x4QIQNuQO3OQ0Vn/6x2/wCh9Rzjx9StiHCbEPvQMvb82QO2fjL7VQEEf?= =?us-ascii?Q?DjVdIYUCC5ee4jA3mntbTeZMRbFLaE48Ro74DKPuPgMr9YgJnCpQUQBSsZEb?= =?us-ascii?Q?a0xud08IqRPB0KdS1Kz3az7YG81lnnVsn3AUsiOohC4AOdFft9jgMJ0MhKcr?= =?us-ascii?Q?tCzT/yPecHY7Q/fjItRC/2wqgTq9schaLiG4Yev6lqbahuhsUhh5t14K+shN?= =?us-ascii?Q?zsvGXEyNN94AjZiPkiWWyciIsc2JndNX6kT7J7XH0/AQDxJfZZDe5OOgd8ZP?= =?us-ascii?Q?BU0ZDgtvYfIsulnittv7KFBJVCzBYJcNhorQ3KaZwgO8lZqTPPOktcwx5CQz?= =?us-ascii?Q?QtV2CA+AjEPer97q2yL6uo9JRBX0W1YGJJ0nDj4k+p5IWSNDF2ilsCRccXwk?= =?us-ascii?Q?IelIF32EmT/hLA5SQQqKHQyvVF0apBYBhui5ZnpByVeKLsXCkOlE3U5MCeX8?= =?us-ascii?Q?lUKbl+hhp5/uvGdq0jwHaJd82Y44QCu7GZlyXhGHCOfbAOGMEfu3Kv2a5ls3?= =?us-ascii?Q?7zKxWFEfurSVPGp5HCSsQ4VpuzEx0CqxVhiQ1tcVbbCOWnhdlpgoA8NfrLub?= =?us-ascii?Q?5qiXG1f4K2/SPiwEpFVOAy+fj5zQCy9fwMIHTrZGTA8u4FBXK9f4B3pjEosV?= =?us-ascii?Q?svMfShNldtWL9+4PPoSh1PN1zR1lMa4Y5Wmpn7xsNs7ZnZytmzxMR3l8rvKm?= =?us-ascii?Q?47fvwQQlEAIe3tx9TeBXGYLC3/c5GkK5BkgxBGZUqAKPaDYeCCIRd09+EQvt?= =?us-ascii?Q?o6L+8NK3XrkBPJIrXnMbZuQNHpTpx6LUoaiwV7LG+rjN99b97KKTP9VDTKGr?= =?us-ascii?Q?GkHfJS/JH3wNaQ+i3XmAHgmv1DLq/4TXNE4zhn2L6veyr/Fggmz7m5gQGHQU?= =?us-ascii?Q?/6QT6W6tpKDvZKQ5CmPRcPfzbjdAX0RG/Y600B0Q?= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: IA1PR11MB6396.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a701efa0-4bc6-4651-dbb0-08dc4493db01 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Mar 2024 02:01:40.4514 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: GrOZkypeiFVPk5OydREpz+3SYzPAvSC+2VNUUs4P8NfB4gjb+YSRsTKYOO9nxDWrSIv0A5BqZpo+4WZyUNHHIw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7362 X-OriginatorOrg: intel.com Subject: Re: [FFmpeg-devel] [PATCH v5] libavfi/dnn: add LibTorch as one of DNN backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: > > -----Original Message----- > > From: ffmpeg-devel On Behalf Of > > wenbin.chen-at-intel.com@ffmpeg.org > > Sent: Monday, March 11, 2024 1:02 PM > > To: ffmpeg-devel@ffmpeg.org > > Subject: [FFmpeg-devel] [PATCH v5] libavfi/dnn: add LibTorch as one of DNN > > backend > > > > From: Wenbin Chen > > > > PyTorch is an open source machine learning framework that accelerates > > the path from research prototyping to production deployment. Official > > website: https://pytorch.org/. We call the C++ library of PyTorch as > > LibTorch, the same below. > > > > To build FFmpeg with LibTorch, please take following steps as reference: > > 1. download LibTorch C++ library in https://pytorch.org/get-started/locally/, > > please select C++/Java for language, and other options as your need. > > Please download cxx11 ABI version (libtorch-cxx11-abi-shared-with-deps- > > *.zip). > > 2. unzip the file to your own dir, with command > > unzip libtorch-shared-with-deps-latest.zip -d your_dir > > 3. export libtorch_root/libtorch/include and > > libtorch_root/libtorch/include/torch/csrc/api/include to $PATH > > export libtorch_root/libtorch/lib/ to $LD_LIBRARY_PATH > > 4. config FFmpeg with ../configure --enable-libtorch --extra-cflag=- > > I/libtorch_root/libtorch/include --extra-cflag=- > > I/libtorch_root/libtorch/include/torch/csrc/api/include --extra-ldflags=- > > L/libtorch_root/libtorch/lib/ > > 5. make > > > > To run FFmpeg DNN inference with LibTorch backend: > > ./ffmpeg -i input.jpg -vf > > dnn_processing=dnn_backend=torch:model=LibTorch_model.pt -y > output.jpg > > The LibTorch_model.pt can be generated by Python with torch.jit.script() > api. > > Please note, torch.jit.trace() is not recommanded, since it does not support > > ambiguous input size. > > Can you provide more detail (maybe a link from pytorch) about the > libtorch_model.py generation and so we can have a try. > This is a guide from pytorch: https://pytorch.org/tutorials/advanced/cpp_export.html I will add it into commit log. I didn't find a ready-made torchscript model to download. I'm afraid you'll have to export the model yourself to test. > > > > Signed-off-by: Ting Fu > > Signed-off-by: Wenbin Chen > > --- > > configure | 5 +- > > libavfilter/dnn/Makefile | 1 + > > libavfilter/dnn/dnn_backend_torch.cpp | 597 > > ++++++++++++++++++++++++++ > > libavfilter/dnn/dnn_interface.c | 5 + > > libavfilter/dnn_filter_common.c | 15 +- > > libavfilter/dnn_interface.h | 2 +- > > libavfilter/vf_dnn_processing.c | 3 + > > 7 files changed, 624 insertions(+), 4 deletions(-) > > create mode 100644 libavfilter/dnn/dnn_backend_torch.cpp > > > > +static int fill_model_input_th(THModel *th_model, THRequestItem > *request) > > +{ > > + LastLevelTaskItem *lltask = NULL; > > + TaskItem *task = NULL; > > + THInferRequest *infer_request = NULL; > > + DNNData input = { 0 }; > > + THContext *ctx = &th_model->ctx; > > + int ret, width_idx, height_idx, channel_idx; > > + > > + lltask = (LastLevelTaskItem *)ff_queue_pop_front(th_model- > > >lltask_queue); > > + if (!lltask) { > > + ret = AVERROR(EINVAL); > > + goto err; > > + } > > + request->lltask = lltask; > > + task = lltask->task; > > + infer_request = request->infer_request; > > + > > + ret = get_input_th(th_model, &input, NULL); > > + if ( ret != 0) { > > + goto err; > > + } > > + width_idx = dnn_get_width_idx_by_layout(input.layout); > > + height_idx = dnn_get_height_idx_by_layout(input.layout); > > + channel_idx = dnn_get_channel_idx_by_layout(input.layout); > > + input.dims[height_idx] = task->in_frame->height; > > + input.dims[width_idx] = task->in_frame->width; > > + input.data = av_malloc(input.dims[height_idx] * input.dims[width_idx] * > > + input.dims[channel_idx] * sizeof(float)); > > + if (!input.data) > > + return AVERROR(ENOMEM); > > + infer_request->input_tensor = new torch::Tensor(); > > + infer_request->output = new torch::Tensor(); > > + > > + switch (th_model->model->func_type) { > > + case DFT_PROCESS_FRAME: > > + input.scale = 255; > > + if (task->do_ioproc) { > > + if (th_model->model->frame_pre_proc != NULL) { > > + th_model->model->frame_pre_proc(task->in_frame, &input, > > th_model->model->filter_ctx); > > + } else { > > + ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); > > + } > > + } > > + break; > > + default: > > + avpriv_report_missing_feature(NULL, "model function type %d", > > th_model->model->func_type); > > + break; > > + } > > + *infer_request->input_tensor = torch::from_blob(input.data, > > + {1, 1, input.dims[channel_idx], input.dims[height_idx], > > input.dims[width_idx]}, > > An extra dimension is added to support multiple frames for algorithms > such as VideoSuperResolution, besides batch size, channel, height and width. > > Let's first support the regular dimension for NCHW/NHWC, and then > add support for multiple frames. OK, I will update it in patch version 6, and submit another patchset to support multiple frame input. Thanks for the review. Wenbin > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".