From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 8810E400EC for ; Sat, 3 Dec 2022 09:32:55 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E02B768B1CE; Sat, 3 Dec 2022 11:32:51 +0200 (EET) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 79DD968B109 for ; Sat, 3 Dec 2022 11:32:44 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670059969; x=1701595969; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=rhPP13eUc3AXH3ju4Bd+jJO40TtHo/xKXd9RNdOQUlE=; b=cKiVKjaBROjVEzPKB4kBbunKw+yiCbZV/F1b3mgg+dLV7wA0PD+f85dM uucxcgLqxCBKJQSbFmzW8rOUqFOsJS28OLaSI0/hWC2D318exYwolgytN p7gkN+DpCgO6drgEDslhfZEJ+eTr4RMVJ8F+keUMWInUUyThVUEFGmQ9V zOwWv8k7fzgnfOvk3RY+kOi0E8cLN7KHtqJKunzIqv2m1Kpw/M4uIk0no cSRE3V6UvRi2njtTXF87VXVUf8A+1QL4K/Cy3I2c9X9iX63Pax2N8m/lz jupZmZ6m/UEMuUNn8VOm4VcqmDH98WRyK9y11LNeIn1Z3Ld1QODkZUPYw w==; X-IronPort-AV: E=McAfee;i="6500,9779,10549"; a="313749086" X-IronPort-AV: E=Sophos;i="5.96,214,1665471600"; d="scan'208";a="313749086" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Dec 2022 01:32:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10549"; a="622969362" X-IronPort-AV: E=Sophos;i="5.96,214,1665471600"; d="scan'208";a="622969362" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by orsmga006.jf.intel.com with ESMTP; 03 Dec 2022 01:32:41 -0800 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16; Sat, 3 Dec 2022 01:32:40 -0800 Received: from orsmsx612.amr.corp.intel.com (10.22.229.25) by ORSMSX610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16; Sat, 3 Dec 2022 01:32:40 -0800 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx612.amr.corp.intel.com (10.22.229.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16 via Frontend Transport; Sat, 3 Dec 2022 01:32:40 -0800 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.172) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.16; Sat, 3 Dec 2022 01:32:40 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RtsLM0Flk7MQ+wCNUV+bUzSVVihKeEO0DFymyYqu4TFi0Gmhas4h6V+SpMO/3k0jQYK9fxnn2KPvF6xERFSCiYVlpE/iDV2ljx55DtWKjZPsbCukhUufaAOXALaIb+hf8zLDtXXS1h7b9K9jw9orCFZ4QopBMayNBwpuA4b/CFu8lJOBFMbspeC/t8qg6XAb9GeWY51wIa2ZhxbtW/8MIYX2zDpuVoMj/phyfIgYWE2LIG+UwGY2B23MSEO2ymMxWk14ptn/5V7yzTwJtQtMo8am7GvTVJjn+Z1sMPa/6DdII4KkCE0LXoFyL2l/ulYnlgVXp7F1meWHPOmh+HYjeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lzxbbL6cCnULCASN1yA5ydNwgORJvdy1XecVsvQbhhA=; b=Bnu/SMZNftY5b5TPnw9a3Va5vSbmylWJBcsME1fVmsiftf0gmkgOxs13STSWIklM6yy9qI0Yy8XdfUZrl4j9Dcnsbr06pz3zu6tXQFju57b3wG5VJAaBP+BEKRvCCapsg/Zi4EpTUk634EBRxh2lTxji+qZymb0N3dFmVm0h9FI+/oNsfLH+FPnc3JuC7QOrYsisi4WakfIoW5SvoqnGszahKNmEbk54iz3wXuhpclT1s5WG6oj/AH6LXoP5YpMgIoNJnwD9D/m9TUqZK/lD8pvLSVMuM2yfM/9VPv7BHxCSJZ6UD/+OVMgOoEzpEonEtC8Jj840sHzfxjol1AMdLw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from PH7PR11MB5957.namprd11.prod.outlook.com (2603:10b6:510:1e0::14) by SN7PR11MB7589.namprd11.prod.outlook.com (2603:10b6:806:34a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5880.11; Sat, 3 Dec 2022 09:32:37 +0000 Received: from PH7PR11MB5957.namprd11.prod.outlook.com ([fe80::5cf:771f:9487:4f92]) by PH7PR11MB5957.namprd11.prod.outlook.com ([fe80::5cf:771f:9487:4f92%8]) with mapi id 15.20.5880.011; Sat, 3 Dec 2022 09:32:37 +0000 From: "Guo, Yejun" To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: backends of libavfilter/dnn module Thread-Index: AQHZBvoui4BiZ426vEydwj9P7BD9kw== Date: Sat, 3 Dec 2022 09:32:37 +0000 Message-ID: References: <20221129213636.1504-1-jamrial@gmail.com> <23dfad2f-db16-cf49-b12e-d43fd0edeae5@gmail.com> In-Reply-To: <23dfad2f-db16-cf49-b12e-d43fd0edeae5@gmail.com> Accept-Language: en-US Content-Language: en-US X-Mentions: bygrandao@gmail.com,ting.fu@intel.com X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH7PR11MB5957:EE_|SN7PR11MB7589:EE_ x-ms-office365-filtering-correlation-id: 11182a8a-a5d8-40f9-bef7-08dad51150cd x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: H3a4brwKDryqDQqOwzs2hUyGMk9QxyslcCXE57AKVvlrS/vZJn1x3TZLbtny4JoHtHSrfdtcF9s3omUbfEhmHubJyhymnFJrY9UIKq+mXcP/1CmRuVO0Qa9c2c5WIEx6msj5temn47dlmUy+6Sq/+4ElVdOMeGRtgSEGoVbk7I9yHuaT2lSvA7UhJiJh6TfNT41VfZ611boJPUtpIEMFMfEvPJ0vbhfo2qY8WrUI3ysoVgxKp/4UG1Zxtu78CaVwpcpq18rUr9xgHFvk0+iG5iYO4QEt9ilfkVOfdaF9mtM0xWqUVmhy26q71NyPnZmABVODnE1HAPTeRI7LXbms32Mm+4EFQoHFrQCOseXQMulhCWcUZJM8BjkHyxLBQg8o1K0Ke7Kp41qvCjSdvFT1CvG8DNcL8OWL/+KEuA0vY0k0GpElb7TO9rIxYXpSwYGEQaOf9KZ4VZjpjCvNT4Bs2kM1VAZCeI6Zu0Cd3JOLAEbTob4wBOADEpNVyAv4LzLbZDIK5RMQU1m0HCait1HTKT1OzdwfDDJ57SDDCT1vgObSWFm9vf/Y+F2ZZnp+JMVxP8tKduqn5DZthSevjCFXVChhKH6ZrbeLpM+QfIUi1ecb0PYrVNr03OHL6CTgarBMyEbVysWU5cfYVmje+E2YPybGJp+NMtEW3HsNqpuuMrMeKwixe6+fMJ4QQssZJSeeHy5LefoKl2IA6yP9UMyUStEtPCHUqKFSo8L9pVM4TIA= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH7PR11MB5957.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(396003)(376002)(39860400002)(346002)(136003)(366004)(451199015)(82960400001)(38070700005)(966005)(122000001)(478600001)(86362001)(107886003)(71200400001)(6506007)(7696005)(3480700007)(316002)(2906002)(76116006)(66946007)(66556008)(4326008)(8676002)(66446008)(64756008)(66476007)(55016003)(54906003)(6916009)(41300700001)(38100700002)(33656002)(5660300002)(52536014)(8936002)(26005)(9686003)(186003)(83380400001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?10qgHazia3RBc/qTBUFbWIL58rcwbPAJxTc7soon4TxcXevXrtInRj2CKvB2?= =?us-ascii?Q?gTCevzKJKqAUn7KH006p0kVaLFAM7qMhXWjrMoaQrsV0UAn4YeNLN1+tcCjE?= =?us-ascii?Q?7hbZhisv8D8ldADAxYe42x0FRr8uXcthUGFgwx3nOZgLkNkQRJntjijOL8Uh?= =?us-ascii?Q?Sd/A1IePxZ7GlBXZR4zUj1ts8F8TbTNcGEkHdioZiWqGCtIMKahpJxvJGLc6?= =?us-ascii?Q?tUGsXBFU9LYCoMdK+JvsTefrO5n5CJ5XmbwwOu/xLVmpqU1PQato4EglEg99?= =?us-ascii?Q?dVegkh80IKgCKFn7D0Lhi5ILHMqJpoefO5NEoXGBgoWDhU46P04kBwuGotme?= =?us-ascii?Q?4iuUNEj4aqNo4K3xB2aJYLlrphmCcGEfYOSoipCtM1Ww6F1Pv6oGfp5KhASY?= =?us-ascii?Q?QZFuU6ElY23Qtmopyge0x1iKszscu7oQG26AoEJ8LYXpkU0P6cNYGe08O365?= =?us-ascii?Q?V4blt8pm8ZVwbLwnXl6d6YNMoqkVDiDQW9xGhZZ6tn4DFgRNLVrAB04I3UW/?= =?us-ascii?Q?oBpIofe7nFW9J4USCNWTzZ0IcVwEESqQgJI82FS0RRNbFC9zWZrBPTG86Wf0?= =?us-ascii?Q?33gzvsgKIirBVmY/c+HcbwJoFRMSHDjW6dzSCjqNRHxRanKN8u0Deu9CJ1Bh?= =?us-ascii?Q?DlBylBEJiM2md11BQ9Dk8eFfzhSRvbROV2u3+WMBvvGFwpzKx9C1TOGCXXBO?= =?us-ascii?Q?K7t6LzQqlNZbZWUEwEJj2Z3et5f1I3G+Mome5tTRdw6kq5NZ9cPQhl9ZAXcy?= =?us-ascii?Q?jjxPZBchyEJY+asHd5cHXU98AAKQvhpuWOWXblS9Yjm6AiKleV3Bp9f8UqrB?= =?us-ascii?Q?L9fqDJmL3j8dyC/ZuSZQgRfOQYOtCMAjhGCJROagLy+Wg+Fgm+9h3Ys83Kg8?= =?us-ascii?Q?e8ph3Ul/BljeAPZ0c/Q2ItYKgqmV5OKpbLg8oEHC+pIXuRVDWqLMbR/oC0Hv?= =?us-ascii?Q?xty65B2nXCWdD6M99OB5mxBCMOi/rMiGHVx3iaRUbzKJRuHkJoZVQJMhqYvY?= =?us-ascii?Q?6dE/Hn2KVALpSb5tf8jEHTldcXM71uz6ngO8Sr0zicEz1boMIgHms4HHrxBs?= =?us-ascii?Q?clpLk+O5p/dYQtBf90OqNRkK51kMwFZ4vplIQm82c23E13xth5IBibM52OI0?= =?us-ascii?Q?wv+6eRTgAqEJoxoYvB7yT6rz5y80p9pmNqjKSWqhy+j0Vi5sU5+f2TrrB+7y?= =?us-ascii?Q?4UWRchEWrMMmKzEYrwReBaRQWifKYFok3LxAPoKOgHbScPvemR+MubpaVJ2q?= =?us-ascii?Q?dGekVYqiEnSlhOCwrleb9W7tHQb/Rbs3X0GznVhKvytsY8/E7Uv071M7BTCv?= =?us-ascii?Q?yjJlMl4ZqyCAKw3xAUCnJMo9nZ3lIbd2zw1j0mWB1Xnep0/52rB3Wb4coqdc?= =?us-ascii?Q?BIU1etokOEJFbULaH3jYFQHBSBiHReo6vGAP+P2U1JFB+tnFCtyPLXHBPfWX?= =?us-ascii?Q?ya0P+klBwUaBl1+SbkzjQYcLf4d94OLy6P5O99aiUJBX6pMtlsb1HA97LPWy?= =?us-ascii?Q?N6LqP7/x3V4BD2lcv8Oiwvc5QIGfTDMcrbLiZqf/5ldQRMQFO/einuEmtr22?= =?us-ascii?Q?hMSlBmVvtFjK5+JyU/4oZt63H+et6Wb1NQUgzxOh?= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5957.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 11182a8a-a5d8-40f9-bef7-08dad51150cd X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2022 09:32:37.2543 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: KYjTwUlfzKJbqPFaM5WSrqOP/cWJI0lwPadJoYcfaN2CGJBNnvbhCsFg2jdPrv6PgFgSyI3sDwMfLRg8r3J/yQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7589 X-OriginatorOrg: intel.com Subject: [FFmpeg-devel] backends of libavfilter/dnn module X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: "Fu, Ting" , Pedro Arthur Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: Hi, There are discussions about dnn module at https://etherpad.mit.edu/p/FF_dev_meeting_20221202. 1) Regarding "Delete the native backend? Yes", I agree since it is performance sensitive, and I can take this effort. And want to get advice on how to delete it. Usually, we'd mark it as deprecated, and then delete after a long time. As for this case, my idea is: Step1: delete code of native backend under libavfilter/dnn, and also add error message in filters to let end user get a clear message that it is not supported. Step2: after a long time (maybe next major release), delete the 'error message' (all code relative to native backend) in filters. 2) Regarding "Move to a separate library? Delete? Move to somewhere else?", @Pedro Arthur any comment? And I have two other opens about the backend. 3) Adding libtorch as a new backend after native backend is deleted. Deep learning models are usually developed on different frameworks and so the model files are in different format. The current most popular framework is probably pytorch and we see lots of new model files in pytorch format. My idea is to embrace libtorch as a new backend, and @Fu, Ting has finished the code and is willing to upstream it and also adding a new feature in vf_dnn_processing.c with basic VSR (video super resolution) whose model file is now only available in pytorch format. 4) how about many other deep learning frameworks? There are also many other promising dnn inference frameworks, and how do we support them? One method is that they add a glue layer to implement the interface in libavfilter/dnn_interface.h, and we (ffmpeg) can dlopen the glue layer library. Actually, the implementation can be another framework, or even be a service. Anyway, this can be a long term solution, and can be discussed in more detail when this requirement appears. Thanks Yejun _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".