From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 3A3B648337 for ; Sun, 28 Jan 2024 04:10:35 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3DC2068D158; Sun, 28 Jan 2024 06:10:32 +0200 (EET) Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.93]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EB11D68CDE3 for ; Sun, 28 Jan 2024 06:10:25 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706415031; x=1737951031; h=from:to:subject:date:message-id:references:in-reply-to: content-transfer-encoding:mime-version; bh=iyE3MTHfXFD9FckXsUpENJ//P0Jwym/nclx7OBa3Txo=; b=HbCf6Rvzlin3WVqwGz+bn3bIue+IVDvqqhM2hOC3/v21VbvG/mB6U9qE iqR77U/aKcT3FVWVGthSjQNvq2CJLoern8G+QUGIw4U4XQaoN7/52RZh4 HCxNsmsstOMqy6CODcglKy901D35/oge9vqMw0JL8tPBo49mFjd7Yb8YO GKxL1zJUZ0auxjrQ095hBbN7A+IF/ReYl/2VW8aTpqLfjPpQ7NCjVqOLr Z98rpekxpM0JSmGivqm56fWbkC5vl5F+QftHuWZ756M/w4AXPFowBqBTu MbtFvqRHTLhicX8VplOH7pL07CkLVJKxa2vPajOVk9cmsO8Op6w49p1ii A==; X-IronPort-AV: E=McAfee;i="6600,9927,10966"; a="399894393" X-IronPort-AV: E=Sophos;i="6.05,220,1701158400"; d="scan'208";a="399894393" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jan 2024 20:10:23 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10966"; a="930756411" X-IronPort-AV: E=Sophos;i="6.05,220,1701158400"; d="scan'208";a="930756411" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by fmsmga001.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 27 Jan 2024 20:10:23 -0800 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Sat, 27 Jan 2024 20:10:23 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Sat, 27 Jan 2024 20:10:22 -0800 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Sat, 27 Jan 2024 20:10:22 -0800 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.101) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Sat, 27 Jan 2024 20:10:10 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Qsza5K0zGR2i4vn5jLc5hHBFzj+9F8jeEWT8/sBkJhLrqOsJkJpHsocmlAWZUMdlKm6VeizItx/SswRa0jBFeRwhScwYTcJJlVl3x1ILdMPog8ifi0sZoC9Du/Y9Pcb+Pt2M8RjRV8/HwJqAIEhpWy1rlraR3Eha6nW2O/oPu7Z2pKW3gLRYiLL6moXhFEj9PJvAR1eorABoSZiLLBLD9S+0lqNMkIF166K8UBCCy1c6GM5I6e9sliLfgreyK7KjKtCWTMllgzehAyWEcBrWwKK4gs/dHi4vI80ngxBC8kX7IdX8q4v+PFqFZpaPE4v/2HLF3hlXYz7R7e3ypRrRdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UNdrafR/A2sPMAxelmlD98AI+b3zd5GdKLUIMr5JEes=; b=VRTGt/z1dA1L4InhokH6hPf+zXb1J3ft3tcodgKmC+lUGKe/52XLFgVLYAeW2gFTHyF5OFosVi/u4svE3HzWoxMvKEfAONIZbnoJsKfNRmQJ1ShUri8djSMbrpXokS8IHazqcMRIPA/DZw4nJ9xqU5lHruaYMo1+jVlD2HBQzq+LocHhKFD7A9JeIdVMAJJEk4DnIiqxiv+ABJzozLMiPH+FZgafm3IfxiVBIQszmvzwkLo343gmqcWDGtIUK+VotRCsbXk2mtmC0C0474VWiiurjh9kfAv5/vmu10ztpEPj4kMq+JVSoRSWI+YaEoDWFwOCWFfc7I+jeKtae8aTIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from PH7PR11MB5957.namprd11.prod.outlook.com (2603:10b6:510:1e0::14) by DM8PR11MB5637.namprd11.prod.outlook.com (2603:10b6:8:33::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.31; Sun, 28 Jan 2024 04:10:08 +0000 Received: from PH7PR11MB5957.namprd11.prod.outlook.com ([fe80::7557:d261:5bd0:cbff]) by PH7PR11MB5957.namprd11.prod.outlook.com ([fe80::7557:d261:5bd0:cbff%4]) with mapi id 15.20.7228.029; Sun, 28 Jan 2024 04:10:08 +0000 From: "Guo, Yejun" To: FFmpeg development discussions and patches Thread-Topic: [FFmpeg-devel] [PATCH] libavfi/dnn: add LibTorch as one of DNN backend Thread-Index: AQHaTPnWSBdAq+qRZ02fF8pVRhCY2LDupBwg Date: Sun, 28 Jan 2024 04:10:08 +0000 Message-ID: References: <20240122061057.2844720-1-wenbin.chen@intel.com> In-Reply-To: <20240122061057.2844720-1-wenbin.chen@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH7PR11MB5957:EE_|DM8PR11MB5637:EE_ x-ms-office365-filtering-correlation-id: ba79ebe0-cbf5-451c-48ff-08dc1fb703ed x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Y0IAFc+3tfevPzmfulCTpffahLTdMrMgd0lVgFAtQT65xgODCfJYxEHkSeK17eJAkf/n0jRjP3YTKBCUWL3ibuwGJ4KaWzhx/LImY+0naOd7m3nEzcBspmYS4TSkWPS52Mb5XjyFFqmVwu3mQuS7Wqf3k0oxGOS2a5vIRHakDFax/H1PgKUEPQsclDwTdTGNs/8pbS0ObXmuPsbd51yHzh2ZJK/yTeJaw36W+IG9zEGCA0UcUhjvT7Ct0nTCKaSppt1rkEisGqWDlRPULsDC2aqlWaSiXx2jrKBI5j9exl1nHLBZEK405Pme8QXUJeyBfQe1eEllWz85FQ9t1zkshrCXuf8bUW4DnYLZEkhxHewM0KnQA6A3VeMH1qnU8zFXu1O59gNrR4i2K4tRp/esNrokJswqEvyx8oxJsteEIWMkYK2j9zwJLmDYcqs3DEO7SuMa6Cx3bV93HDrsb2SBRUycsWQsLa/W790igqfIuqFzxemLe1QbpKMQdXOtcKnYzeg1ZVGJI/a9vwHb7fOE9PwFue+GCT2+VALdKiHIN4UAhnwRaKvADX99KmzIczMoAxHeFVduf+zS7xIHDd+Z8g84Vksc5DKcFWI6y7c/vjA= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH7PR11MB5957.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(39860400002)(376002)(396003)(366004)(346002)(136003)(230922051799003)(451199024)(1800799012)(186009)(64100799003)(26005)(83380400001)(53546011)(6506007)(7696005)(9686003)(38100700002)(122000001)(5660300002)(8676002)(8936002)(52536014)(41300700001)(2906002)(71200400001)(966005)(478600001)(316002)(6916009)(64756008)(66446008)(66476007)(66556008)(66946007)(76116006)(33656002)(86362001)(82960400001)(38070700009)(55016003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?SIFbYXDwxKqdjLe5/p45+Jf7ts9bv3a2EUZFcCxQpw4X4OZgkIP/L3idPRFJ?= =?us-ascii?Q?oObF+1C0ySbWqlKYhJ4aBVQgtCp3542//NAwYXTHJAmJBlwP6cbH6ACfVFpO?= =?us-ascii?Q?HlFudvbbRk1nHZIeq7FSM3iUZzO40fAIUmxEX6GD8ruwooK05eXgT5rftXum?= =?us-ascii?Q?xV2Qxv2ixcJaXct4rlp9sw4GiUdtNLXUjbha49c9WRPDvMb9xaBmX7C7+97l?= =?us-ascii?Q?PXKlYeu11fOgtfCpv8XpMdJ0ifA5kXrZrEaiYjUJggb7DXbrBu6ww78N2Hlr?= =?us-ascii?Q?qq+ZRKyqUuWWdONZItac8AZs4Hms3Vg3GsAL+IQoIAu+lEsB/7s5pVSAD43w?= =?us-ascii?Q?H6nZ8+C2IakYESclMWeXOT2LpstKHlr1L71pRNaRphhAuQzHi9f1YUcdlNeY?= =?us-ascii?Q?L7NDaWzz5Njn10JpSGdEr5htagNHnmBk9/EeWZVdvGL/xGoAhVK28Uuj5Uqs?= =?us-ascii?Q?JtbMPj5j8psxxngadNsTzaWED0qHmTcXnDe8X3faup5cut+CNK2uOr6u6Gf3?= =?us-ascii?Q?8BI9JOvsHTqYAsj1BvIsbQhCvJwNCPPLTv5+UeFER3Dsz+AQ26QMVDo28dF9?= =?us-ascii?Q?LoG4TJ3uztJfo2mult8H9WirfSXrpShwdxx9ea0gd7+2ZdEeyaTJ5JR082t/?= =?us-ascii?Q?QoFIe3dYRCPEDEVutc9H2KtkeNQ/SLGfyhE+CQ/MfXJaOgF+IUvXu+CJ39sV?= =?us-ascii?Q?UNsxkAseegw6Y1eDAb02ZwCVUzJ6OVSTMMLNMIZCeRdx8OiMAYMt8ngJvq6h?= =?us-ascii?Q?0isXTkaqZaCGdpyVZYXnx2pOjEF/vZeRTaZj9BTyA97W5SPRv21OCCscMUKA?= =?us-ascii?Q?gS19qGQizTF4A1wnJRgmCDzFDS+f5/hGLScK65UA7tcNwUfgK2bvvxn2XS6N?= =?us-ascii?Q?NCNjIgAtL2Rfq2pfbtktSh0Z7UafFBrRJBnHHWBubzwR8OrDhheZYDqOJp0v?= =?us-ascii?Q?mjKYHMYFF4XKgoessjdr1tuZYmO/ZaT2D3HYoj0YLs+90Ma5il3KA1DpqWXL?= =?us-ascii?Q?EFsWmj3XUhGV/H+1LMLU2VTJbHiGquuX8YQjl+Ul2XN8UKrhnU3ulNPJqmNw?= =?us-ascii?Q?hr1Et9fBIHV9HTUC6ElnpSz6r8Q3+G1136Q4K7P6GHr5UhkXODmdBjkFUZUJ?= =?us-ascii?Q?HcH4t6G/xP5HSoe7POQGRM5CoVDzWsS+YKytZNNR86vgmg8no3CcFO5Fk6pC?= =?us-ascii?Q?FYly2dBxripSGrYFFoppgQqAaQoUTZ0S2b4EPjLrCDkopOFD7kBIDqddBDGE?= =?us-ascii?Q?EFzDyJIq4pJHC9FsS4MAxHJtPSQnV8hirivXH/mgwlPXYPt7DScs+fHxUVnw?= =?us-ascii?Q?wI4Nh0QRkI3XmA6NohKl43bkqyevibRgcULjYzshIUZ+u1rXGXoIqGbrvSfy?= =?us-ascii?Q?bHWrCNx3MbHa6c9Os8WJLVEKHyQhUzz3aqnlVw6b9ZjaWnthi6/nHU2gQyDJ?= =?us-ascii?Q?RHFrBiU7UcFO+LngttHrMaMaYHyUyiG2QarKM4wfOZI6zqAOovNDUpybW6bS?= =?us-ascii?Q?vZnNMxyOxVpm8O23CCS/cqf7y0rgH5OAi9ou2YUaK7hQh65n1uhlS2iNrL95?= =?us-ascii?Q?+1tilCEoJbxwwgwwLIBMNQL80Eyks8avzKZNm4w5?= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5957.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ba79ebe0-cbf5-451c-48ff-08dc1fb703ed X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jan 2024 04:10:08.4981 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: +sfKwZU5B2a0vE3VrEsAfNcLaPRyhriMhoZxYHC9w+wkqaIVAV1Yu/fBD9IAvaDEDdWIq2rK1FcLCm49UuxMUg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR11MB5637 X-OriginatorOrg: intel.com Subject: Re: [FFmpeg-devel] [PATCH] libavfi/dnn: add LibTorch as one of DNN backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: > -----Original Message----- > From: ffmpeg-devel On Behalf Of > wenbin.chen-at-intel.com@ffmpeg.org > Sent: Monday, January 22, 2024 2:11 PM > To: ffmpeg-devel@ffmpeg.org > Subject: [FFmpeg-devel] [PATCH] libavfi/dnn: add LibTorch as one of DNN > backend > > From: Wenbin Chen > > PyTorch is an open source machine learning framework that accelerates the > path from research prototyping to production deployment. Official > websit: https://pytorch.org/. We call the C++ library of PyTorch as LibTorch, > the same below. > > To build FFmpeg with LibTorch, please take following steps as reference: > 1. download LibTorch C++ library in https://pytorch.org/get-started/locally/, > please select C++/Java for language, and other options as your need. > 2. unzip the file to your own dir, with command unzip libtorch-shared-with- > deps-latest.zip -d your_dir 3. export libtorch_root/libtorch/include and > libtorch_root/libtorch/include/torch/csrc/api/include to $PATH export > libtorch_root/libtorch/lib/ to $LD_LIBRARY_PATH 4. config FFmpeg > with ../configure --enable-libtorch --extra-cflag=- > I/libtorch_root/libtorch/include --extra-cflag=- > I/libtorch_root/libtorch/include/torch/csrc/api/include --extra-ldflags=- > L/libtorch_root/libtorch/lib/ > 5. make > > To run FFmpeg DNN inference with LibTorch backend: > ./ffmpeg -i input.jpg -vf > dnn_processing=dnn_backend=torch:model=LibTorch_model.pt -y output.jpg > The LibTorch_model.pt can be generated by Python with torch.jit.script() api. > Please note, torch.jit.trace() is not recommanded, since it does not support > ambiguous input size. > > Signed-off-by: Ting Fu > Signed-off-by: Wenbin Chen > --- > configure | 5 +- > libavfilter/dnn/Makefile | 1 + > libavfilter/dnn/dnn_backend_torch.cpp | 585 ++++++++++++++++++++++++++ > libavfilter/dnn/dnn_interface.c | 5 + > libavfilter/dnn_filter_common.c | 31 +- > libavfilter/dnn_interface.h | 2 +- > libavfilter/vf_dnn_processing.c | 3 + > 7 files changed, 621 insertions(+), 11 deletions(-) create mode 100644 > libavfilter/dnn/dnn_backend_torch.cpp > I'm glad to see the libtorch as a new dnn backend personally, due to the fact that more and more deep learning models are trained with PyTorch. PyTorch is a necessary in the AI domain, including analysis/processing of image, video, audio and subtitle (text) and even putting them together. _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".