From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 3E60649EF9 for ; Thu, 14 Mar 2024 11:38:39 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 45F5068D18A; Thu, 14 Mar 2024 13:38:37 +0200 (EET) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E612A68D12C for ; Thu, 14 Mar 2024 13:38:30 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710416316; x=1741952316; h=from:to:subject:date:message-id:references:in-reply-to: content-transfer-encoding:mime-version; bh=MRk0uK5kkd88hTzuMDU0KQw9MrELuZbvwhfQDfhfP5U=; b=VWA87lanBmdgFmCUIlEtSwXl8vCURnYBxfOM0cADp44zpLkfx9y0Qkn0 5UWlPCCPJPOBr8sGgs27vu0D3CrvG5FohVXMLTTJ3/zL5RCdkW112vhqO fYOSqKUmalgCH1394AmQ1zpqV4spJzCysY3dQ8Uz6eTuWufJd2ccYQsrt bFu2syTgL+lLVQ+X1//rULb6hrDsX65/Rff/zuwWPTM309538dh73rCSh tcvJnEJM7S6xO0kk02bMKJyoNr/fZZQOywtD4vMJsnl915oCS0g2qIT7e Z8HEJGQwBYpkBtUJ/M/ZcJ87T2lZshibN9Ku778g8AWvQnCqgbFlZMw3q Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11012"; a="5085565" X-IronPort-AV: E=Sophos;i="6.07,125,1708416000"; d="scan'208";a="5085565" Received: from orviesa003.jf.intel.com ([10.64.159.143]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Mar 2024 04:38:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,125,1708416000"; d="scan'208";a="16914398" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orviesa003.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 14 Mar 2024 04:38:28 -0700 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Thu, 14 Mar 2024 04:38:27 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Thu, 14 Mar 2024 04:38:27 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.100) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Thu, 14 Mar 2024 04:38:27 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZcqQFNMOC4v5TjoP4dRgpPb37Yd8j1MxMsvmKV/1VSzjiMbuh4/fPb9C8H/0BKQtIEQ99BdGv3jpmgtLUfJTx3YdzccCKOpdqeAsVVJ4rRXJ0/fXMRUuwulcuqWL/fTItcCcmeIboNxcEyHeT5qnEdwrHGKPVFctfCYIMenr01kvS3TLImj551VzWSNxJscmcLJ7CyG1z8VrcrFhqtlQTE1l23R3jlYFM118DX8MjrQzSoRSckQgjvDMW3+ejErwKbSeClibVt37hMtdBJJdbDRPwDb77HFgoku5a6KI3bAdmMz0G4yQvS70S/d58ncGasYo8L4PIH0E+Ni9bwaxVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LETLgNnmEL+lQ+rfUtOVR/qIwQoUmWagDRETgfYSvlw=; b=od1ZX9aWmkWcmZ4O5XFJZet4afVEv8nzO58zdevD7k3GBUpQDUhYZaw2LGnx68GHHAIj0lJs5Lg/gPuTxf0FORP+lS397QwMfWgffu6N4Xk/EErKAbiLGgyeAz1wxV9kSrvaoblC3EOyVAID/MUOpiDVwdpYNjBTtlSKiBmjb8eKmG80VJGuBxJwkJRniR1JvvZ4s91erm9w20VdrRePyxv1nzoYsSlGM+YNN9v6GkTHl+X//aZr85veEmNyBdTnIma/FmhEdVNpigCDCOZ7GJ2oQq/ZfzuvnO3qFcZkdsSd9G6sairwGmo/cWywgrHQUfPjPycd0S+YVorfi3pfKA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from PH7PR11MB5957.namprd11.prod.outlook.com (2603:10b6:510:1e0::14) by PH0PR11MB5176.namprd11.prod.outlook.com (2603:10b6:510:3f::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.20; Thu, 14 Mar 2024 11:38:25 +0000 Received: from PH7PR11MB5957.namprd11.prod.outlook.com ([fe80::f0b2:5d:e8b4:1489]) by PH7PR11MB5957.namprd11.prod.outlook.com ([fe80::f0b2:5d:e8b4:1489%6]) with mapi id 15.20.7386.015; Thu, 14 Mar 2024 11:38:25 +0000 From: "Guo, Yejun" To: FFmpeg development discussions and patches Thread-Topic: [FFmpeg-devel] [PATCH v5] libavfi/dnn: add LibTorch as one of DNN backend Thread-Index: AQHac3FhW4MifaiQeEKbSn7GgLi+ZbE3EqHw Date: Thu, 14 Mar 2024 11:38:25 +0000 Message-ID: References: <20240311050229.1692658-1-wenbin.chen@intel.com> In-Reply-To: <20240311050229.1692658-1-wenbin.chen@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH7PR11MB5957:EE_|PH0PR11MB5176:EE_ x-ms-office365-filtering-correlation-id: 19eb7853-8dcd-4141-dcd3-08dc441b42bd x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Vk4qNJ9yVcyiPl9J/d+KpIJ7hpdpJ2elQN3/5KnkMukH6fJEeDWRrtLBzCZIAKjM9lSHf6CWgaJx9pr0tfcNsQu2kUufTaNyqB5CaiR8TYW6O+/A+q7q9fZsrlg8pyp41vZBfyp30//QyJYry7OHTYGdvJtVLITM36r8VqkzcbG5SIWSSL7KLbMt00m+FnghxjkVFulU9aV1eF7PPBczgu55z6sZZupwkh2u/Fnbft+r+LWvJyoJb5sDCglW3yqdEcYg8dhMxTrxDy72RGAPf4l8lafvHuFBgfpCSdi2UlyPmZFc6aSlnbxp1db8mItYIo1gbAG7m76vfuxsBetohNphTuHQkwkJU3f7xI7KEdStk9MVNrMpRVJf0TrQ9gzvt31vmACHrnaK9j37Rr0izq1mx8z6c/eZzWY32PIqOnAwHOrEL9fp4eWM2WdS8bMCWK01u76akkT+RLiRDR407EzHGaX2G4yQKEPPO2xAjeO5NBmZEVguXRix9AsF2dKiP1DSt9guUvZGPMHWRyHOl2+UFMjdLUvbWtGWn88KVb4GVot2hZoL2Or6kvP9sbNPCgbj0vxEEfgRTyGqHMZEoKJSSh/GF0ILFaMkcltePbLzeoGJnv+Ab/7WQ3j/buzM3Dl7qGMtfkr8QznA5cHOk7cMlQG0t6VlwChqrMlys98= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH7PR11MB5957.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(1800799015)(376005)(38070700009); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?56i0KsMBYL5XDelBIrnnyPknvJSNmyDltwdFqUUCpWtJ4AV0lwse3yytUFma?= =?us-ascii?Q?XG/qhXThEygeNWyGEOuEFlvVkqQeF3a7NYTR33H+tLWn65A+NSNlhPJbkdtm?= =?us-ascii?Q?EEDQriRql05cGkJZG2yniQOAT5gDUF4FovK7eO0RuTeNFg9TDqjqoVw5Z35X?= =?us-ascii?Q?srFlcDtxTCwDTKcDVZCPy9KiiLTKhAmSZ7ee2BBjxVrpoAbQLiDj392kxWQM?= =?us-ascii?Q?Os0GElOgOhrLLfHnozcGoc/ra6pKIC10RG6biXYHq7V5UdLXlxVuYOhFwQ6x?= =?us-ascii?Q?+4zEw6bM248xLPeTEudPPZqJLQVPjnmCL//IA/a7LxM99UWO8Pf6i3JWnLEM?= =?us-ascii?Q?Q1NmIqKD5NxRcJwThiEW0ISflxXz5yWkIbUHFOY9PGNzz1l7iuFw4Xj9tafy?= =?us-ascii?Q?NoFu6lHSmS369XqFEZcvU1K7VxKnLSLfYsCq+/W0bvdIEHYgLLpFUnMSWDYF?= =?us-ascii?Q?xusp6nMfdcDVNVm2taIW/yHFi4g+TVpdqGYs0j4Rh7f446ba9JNJA+IqiiA6?= =?us-ascii?Q?gyXLQN30XIcE85kCmpxoi78I7AmGHHQxRM4BQwO0xLIvIcbuibhlhX52yzGW?= =?us-ascii?Q?INJQV4lpsex8QxvzxmzaJT3LsGwnfRCuOcHxumUKWZOBL4Sp/MB2ELk06Eio?= =?us-ascii?Q?G/CU5b0sgOjqdmRt/aovrRX+EtUYyGwYxodkzcekLi83aBeDwy7qZ0qOR5E4?= =?us-ascii?Q?/m/zENzD3nxvRHkpePS19c0pKysufx7mgZXSfI0xprmRm74mp+43Er6oM41J?= =?us-ascii?Q?giif1ALhF1R9Gz9g5OK434y4nLCXhbz+FpmT0FpegwERRdLQy9ON2LAoruMd?= =?us-ascii?Q?1I+7mOm24mNNr7+bJi0SLlBs2of+VqmD66STpUeV7fI54pMaQ3G60MGqYrvo?= =?us-ascii?Q?hIODDT+P8Hyq/vTEDmI4R2mhUHmyAH3a0S28VJGwm9YpIkP40/oG2BjGtuxI?= =?us-ascii?Q?kB3tnBZtp1YQTNQk5eAEcBdEKlwXhsG8hb/LR5udUoD6chCKPhi7bSRwBMm6?= =?us-ascii?Q?5cIGnqRDh0+pRbFza+lgOxQDjab72KDk0z/DASEL1FRQiRD07UXx4HTTKa5L?= =?us-ascii?Q?am+2tbX+irkm5f6Q5jLDILK4aF7tD3UiAtpAQlmMi82D7KO/JgeqZTTorleA?= =?us-ascii?Q?8j9qBax/ZLnghrdI7uBxcdPJQTxz5QXkK4TBjJ4YXqLzugoSHQWRLzNYrlOe?= =?us-ascii?Q?FcizBCuTYrewIOQgu3FWjSdeO4DU4NjSLsSNicrqP72m6H25rnT6ivUvtCYM?= =?us-ascii?Q?4lPyZXEbe8HFgjyh9VKjXVo6wegxFPzGXxnaF3NA/1DQWE0ecPfO/46geOSA?= =?us-ascii?Q?JeMBSoptD6zLrnhtRJr22+RkLebhiP4cD2Q/VrQxqAVVQQR+k9CNnBMjbZFH?= =?us-ascii?Q?xQiSLDbh7bNFTq4T1mARwKUpehZY22EuMlBa9a9whR5GwP8MCeiRMhEHfRur?= =?us-ascii?Q?maf1SRHJZzBL19yVtEUlwGvo5Mj17XyYfGI44FgcCxFRJgKWgPmaw/8KazJJ?= =?us-ascii?Q?tPFxiJBNpdlvEedI21giBAuFiuDq12gQI5qgdyxoFpLac5bvMztc47LoVocu?= =?us-ascii?Q?Pgr4MJdmMlyRYXO9UUXjAARxyZ5osDG6E6UzDKVz?= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5957.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 19eb7853-8dcd-4141-dcd3-08dc441b42bd X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Mar 2024 11:38:25.3978 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: N9VqsH/8TmIR0MK1g/u+5VZqiGUfW6OHxP+401yGV0mnEdtgyEwy4IVyFErh2MuZ0vWK+v2uLBXg6rw8+dWHsg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5176 X-OriginatorOrg: intel.com Subject: Re: [FFmpeg-devel] [PATCH v5] libavfi/dnn: add LibTorch as one of DNN backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: > -----Original Message----- > From: ffmpeg-devel On Behalf Of > wenbin.chen-at-intel.com@ffmpeg.org > Sent: Monday, March 11, 2024 1:02 PM > To: ffmpeg-devel@ffmpeg.org > Subject: [FFmpeg-devel] [PATCH v5] libavfi/dnn: add LibTorch as one of DNN > backend > > From: Wenbin Chen > > PyTorch is an open source machine learning framework that accelerates > the path from research prototyping to production deployment. Official > website: https://pytorch.org/. We call the C++ library of PyTorch as > LibTorch, the same below. > > To build FFmpeg with LibTorch, please take following steps as reference: > 1. download LibTorch C++ library in https://pytorch.org/get-started/locally/, > please select C++/Java for language, and other options as your need. > Please download cxx11 ABI version (libtorch-cxx11-abi-shared-with-deps- > *.zip). > 2. unzip the file to your own dir, with command > unzip libtorch-shared-with-deps-latest.zip -d your_dir > 3. export libtorch_root/libtorch/include and > libtorch_root/libtorch/include/torch/csrc/api/include to $PATH > export libtorch_root/libtorch/lib/ to $LD_LIBRARY_PATH > 4. config FFmpeg with ../configure --enable-libtorch --extra-cflag=- > I/libtorch_root/libtorch/include --extra-cflag=- > I/libtorch_root/libtorch/include/torch/csrc/api/include --extra-ldflags=- > L/libtorch_root/libtorch/lib/ > 5. make > > To run FFmpeg DNN inference with LibTorch backend: > ./ffmpeg -i input.jpg -vf > dnn_processing=dnn_backend=torch:model=LibTorch_model.pt -y output.jpg > The LibTorch_model.pt can be generated by Python with torch.jit.script() api. > Please note, torch.jit.trace() is not recommanded, since it does not support > ambiguous input size. Can you provide more detail (maybe a link from pytorch) about the libtorch_model.py generation and so we can have a try. > > Signed-off-by: Ting Fu > Signed-off-by: Wenbin Chen > --- > configure | 5 +- > libavfilter/dnn/Makefile | 1 + > libavfilter/dnn/dnn_backend_torch.cpp | 597 > ++++++++++++++++++++++++++ > libavfilter/dnn/dnn_interface.c | 5 + > libavfilter/dnn_filter_common.c | 15 +- > libavfilter/dnn_interface.h | 2 +- > libavfilter/vf_dnn_processing.c | 3 + > 7 files changed, 624 insertions(+), 4 deletions(-) > create mode 100644 libavfilter/dnn/dnn_backend_torch.cpp > > +static int fill_model_input_th(THModel *th_model, THRequestItem *request) > +{ > + LastLevelTaskItem *lltask = NULL; > + TaskItem *task = NULL; > + THInferRequest *infer_request = NULL; > + DNNData input = { 0 }; > + THContext *ctx = &th_model->ctx; > + int ret, width_idx, height_idx, channel_idx; > + > + lltask = (LastLevelTaskItem *)ff_queue_pop_front(th_model- > >lltask_queue); > + if (!lltask) { > + ret = AVERROR(EINVAL); > + goto err; > + } > + request->lltask = lltask; > + task = lltask->task; > + infer_request = request->infer_request; > + > + ret = get_input_th(th_model, &input, NULL); > + if ( ret != 0) { > + goto err; > + } > + width_idx = dnn_get_width_idx_by_layout(input.layout); > + height_idx = dnn_get_height_idx_by_layout(input.layout); > + channel_idx = dnn_get_channel_idx_by_layout(input.layout); > + input.dims[height_idx] = task->in_frame->height; > + input.dims[width_idx] = task->in_frame->width; > + input.data = av_malloc(input.dims[height_idx] * input.dims[width_idx] * > + input.dims[channel_idx] * sizeof(float)); > + if (!input.data) > + return AVERROR(ENOMEM); > + infer_request->input_tensor = new torch::Tensor(); > + infer_request->output = new torch::Tensor(); > + > + switch (th_model->model->func_type) { > + case DFT_PROCESS_FRAME: > + input.scale = 255; > + if (task->do_ioproc) { > + if (th_model->model->frame_pre_proc != NULL) { > + th_model->model->frame_pre_proc(task->in_frame, &input, > th_model->model->filter_ctx); > + } else { > + ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); > + } > + } > + break; > + default: > + avpriv_report_missing_feature(NULL, "model function type %d", > th_model->model->func_type); > + break; > + } > + *infer_request->input_tensor = torch::from_blob(input.data, > + {1, 1, input.dims[channel_idx], input.dims[height_idx], > input.dims[width_idx]}, An extra dimension is added to support multiple frames for algorithms such as VideoSuperResolution, besides batch size, channel, height and width. Let's first support the regular dimension for NCHW/NHWC, and then add support for multiple frames. _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".