From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id DA6A642A14 for ; Fri, 10 Jun 2022 23:54:54 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 12FD268B567; Sat, 11 Jun 2022 02:54:52 +0300 (EEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11olkn2089.outbound.protection.outlook.com [40.92.18.89]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8037C68B03D for ; Sat, 11 Jun 2022 02:54:45 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TaMR7Nj4DjvuF9z2K6KbnaDFydYaQLyLyd0oeIWqYIt+8rWBBvc7Q4ZYO8MMftr0imKOcZiNhaGMY2VjnwdE8njwDvPE/KyqbQyedCvah7MOtg+apfFsgH6hF/q1OkziIH4f2AvvMj30wR8G3ekBKpq3dxw+Rz/jDMotUQJIzP9NLNmf9qabO0oMIEFr12iXd/Uiy3jiMsk/+G4YzrBznpFWeK8azt2Vji1+4BaDTHldSAr8dJOP4dA4ZrLuz7pZhydZeDDbA6LmqCDlRt+5Rrw4Vy9tzP6AsMc58BABaxCCiZl9C0ox3OQXi4OVRmfcwS4dQTPTkokA66YkTijlag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sLsEsXJrlREHmXtrj0MMSZffqqlRz583q+rw6vAYRqc=; b=NeHXVvc+igzOabM0NjhTFJSXjtyINd1klc+G0Vu9f7xHWCOx/OSyLgHTHhvHuwprXBWj9HqFg7Moq4qwAB1vCSo3gIVYWlvi0KvLQeZOwKJeSQjN6O2BjVRZkWNa+5EpezgE66dW0BMm/FeK7Ehi8nlQ7kXzVwQ+FfrhNHyO1UXkt+UAddf2fBAUUi/xlVR7c96lL7xlYQeTxdSr2C0Ft+BCFxUj30kpWAl2qCgnqf6J3UcM5bDYRldn0Cqngvunh0kk03rCBIyIU5aTKay2LH/QxOxKXUxhX1OUHg/PRq27oTC2gCDT07qSpA27TVOm162Us45qxeWe5Pw/WWQWFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sLsEsXJrlREHmXtrj0MMSZffqqlRz583q+rw6vAYRqc=; b=oOMTrRAdiqJ9Jnd+U8bD0m1mkOUw6OcHS8R3c8Kcl58+HHnlFxz2QdzNMkuekHfO0xljspvRK01/j6yod8evsWQhY/H0mOE2YYX3C+/HlHIoCjm6YwC7W6i48xRqNNYFm9t1DzhcPf4wr0pcpqbdTVAtvqjO35a4+J8AfzdSa8YVC0poRVBIMz5LLKVucELI05QDhWv3j/YXyBl0AHCGvEA2LOjYMYLgnuI06er4SZqvOLCk1o5UIjNsQ1MnKcS2DPoUdLPGBWVwhUlp8ZbkN/4XCEN5QyYJ+jMIjJM/NQ4I7BGqMZznubXBLG7VVt/xnJ4daSeyR8HIKOhx6/acQQ== Received: from DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM (2603:10b6:8:b::20) by BL3P223MB0193.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:34a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.13; Fri, 10 Jun 2022 23:54:41 +0000 Received: from DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM ([fe80::f144:21f9:cdb1:5c8a]) by DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM ([fe80::f144:21f9:cdb1:5c8a%9]) with mapi id 15.20.5332.015; Fri, 10 Jun 2022 23:54:40 +0000 From: Soft Works To: FFmpeg development discussions and patches Thread-Topic: [FFmpeg-devel] [PATCH] avcodec/qsvenc: make QSV encoder encode VAAPI and D3D11 frames directly Thread-Index: AQHYelBN7yIjDSNIUki2xnbTqv8tca1E7eiQgABEkACAAB144IABVREAgAKfjPA= Date: Fri, 10 Jun 2022 23:54:40 +0000 Message-ID: References: <20220607092216.405-1-tong1.wu@intel.com> <078794290ef4b70ba19dfd436313a58c2ab8bdff.camel@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-tmn: [yF5Pk0pamxSV2Ux8BGoSqylzBcQ+0m5s] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 90285b2f-8b81-4d56-27a4-08da4b3c95bf x-ms-traffictypediagnostic: BL3P223MB0193:EE_ x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 8L+/aYbMAiy2Sj/4838/5HsyTYT6YRz/wj8EwOyh2z+XdOvgJntLlo8iTfPDBvZHW3XT9o6CfGrAwIQEwsr5247ehb8xoZL9RmKNBh54JMu6DsLynRFXdD2n6n8J3P9Sw3xj277zoC+LplWB2kPa9nAYLA5DeOEXlP85lPM9AsV+qt7jkAHYo3x9Ro4X9FeF+8OYONv6KKbvoIxvY1+sYRTQYxSHjfPBUnlp1qPdadmm3L26SdtKMHVNBPwsJjaG4eycUYtEwM7UEofv0XNjWXHCaJsUJG5Hy3ihl1oe+QtYANMs9EB3k/403+Ii0IryX0hPOs1KVG5y+KnH73v64wtx56YseN9NkpwCgpR+L3IzCKdZFcQmUhN3A61aOUPG0xeCnDgojBV6X+FCwwfYmnWxUYRSZ6wS+MoTYDsRIRbXkfMqJLrRWarWXANG8moJNNaB/yxN6396LoRYcEZtPyuERFwyFzhvlB56U1nrqPeAg/jffx15ZEv3dnMQf+M9t3PptGwzu1dBLF+eLyxVShTLMLQxypkXYuH1R9iK9r8mJccIl4Y8myjUX6it70MTs1HrU24Ug8+0z2CEl9pxqr5Dzig3z7iyoB/u/sKK0Dl0MgjLiz/B83SEq/lFfegN x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?UzFrUHRzaUJHUVJSUFEzS0huZlNOb0NoNk9RTlZCQitMWFhvOFExU21BeVFz?= =?utf-8?B?OHRkQXJrbmtsOWt6RktTTE1hblJEQmZOYzgxSlFUSDRleTJVUVRDTGpVeXdP?= =?utf-8?B?SXd2WEwvTm1WalY5UllxQjBVME9mY2xSZE0zM0F6aTczRUcrb080cndPMGVX?= =?utf-8?B?Zzl0VldadVZ2RlBjRmozY1k0WW1zbjB2QVhxemdxUmRDNTZwUUJwcVZsTWVM?= =?utf-8?B?RlBtYkY4L2hKT0JMV3B4UEdQUkFySmEreDZGeEJld2tRem52RGlkNitjQkpF?= =?utf-8?B?ZjczalEySTNCbDRLaThPMWpUUVlCaDhGRGxucG1YYzJMUUM3dllxeHhDbUdZ?= =?utf-8?B?UTlsV0k2WjVaT1FndmpxRGhXZ3MvV3U3YncwdGVLSlZwZ2hrcnJmQkVEWTlB?= =?utf-8?B?aFE1Q0dVdEdub1JVUnh2WG1lTWEwa3IwVDNBV0JGK2NEcmcvLzRvclpodFZX?= =?utf-8?B?bWU5NlVkcHRZVm9IVXFmY0N4UnpSRVBHR0ZuZGd5VjJabDY1ZHdqMDErNG83?= =?utf-8?B?WCtOcUVkZHRVZE9OMjUvcTdaRmNqcC8xY2RWMEFsYlkxVmp3Q1F0dGVMOWEx?= =?utf-8?B?M2NpUm5SbEFzZUxLWUE4S2ZKTUZGUVZ2SEk1OTZGeXFRYXltcDdncTQvK0la?= =?utf-8?B?NWQxRzF3MXdMbU4raVJWaE9WYmIxbmtoTjBjcHRURWh0QmZqdy9xS3B4NUl5?= =?utf-8?B?aktBZzJ4TEpraE94TTBoeVN1VnZLSUNQcklMTFB6dDN0TTdmT05wMS9lYXd5?= =?utf-8?B?R2FIMDR5TzdERXViRGZJQi9lTC9ycWZMRDUyNHFCYlJZTDlzb3hTMmY3eTJv?= =?utf-8?B?WlJwdFRzdVE0L2RFaGxaS2pzZy9zd0tNeDRyNDdtUVNYaUFMVEJ1aURTTTk1?= =?utf-8?B?SmkrZGdCc3lzM1lFbVpYZEFwM2IrQ2MyVmZ1dDNxVHJqTmR5cnBZU2xSNmov?= =?utf-8?B?czdnZmFMUXJWbXN1VUR5YzVSaXNzYk5DK0JFcitVS3dSbzcvVTNHaWhLWHNF?= =?utf-8?B?RDlKMExzbTh6SXhHWVJ6TzkycHd1ZS9oMmsxWjVxaDlIR29PdjVFeHNTS3Jk?= =?utf-8?B?eHdCQzdObDRZV2syKy93Q2UzOGFOLys5b3NrWVFhUEFjdjFUbDcyRHM2MlIy?= =?utf-8?B?bjJQYnRnYTRhT3VrYWc1ZFdhTytsNTZXS0JDZUFWd1RXVWo2cFE4SnRrYjN6?= =?utf-8?B?NUl5OG40bVRQRy9nM0N5ZG04L0NWdXRFSmJzVVNIQ0toRDlkUGRDa2ZRZUht?= =?utf-8?B?MzVIdCtYMm00a0RPaW4xRVhiclgxd3FPSS8xeFJDUFRoUnlKbmo3Q3paN01w?= =?utf-8?B?UVRZM0RsTVZxZTA4b2orRnpGOS9WaXBLMDU4L2IyQW1kS255TVNrNk84ZHk5?= =?utf-8?B?VG8rNlUzc3lLS2c3N0kzNUxEYWxMa25UenYzMlVUem5PS2Z4UkhOb1YrZnRp?= =?utf-8?B?OHBsWFpNZisrRFNEMU4xcHNvSGFzaE50NGFXVXowMkhKQjlkc21XbE1ZYU54?= =?utf-8?B?VUNWVjhnUnh4VlhpbFFZeStmZkJnSGpqbW9FVCs1RGE2b1ZUL29hMDc2ekhC?= =?utf-8?B?Tk41ZlRtT2h2UWJOSGFOcDE5SU85S1hjNUNUVmEwR05tTFZNN3g3U3Y0M1ZJ?= =?utf-8?B?YUJoQjY1a3BCN1l5VXhFUFJrMzlINWxoUWhzNUVQaHZjVml1cDZmVkpHR3VB?= =?utf-8?B?RndjcFZGenB3b0dTQWZGUkNUZHMxZHUxMFU2TTBuWkR4MUhzY2RJZGd5ZEk1?= =?utf-8?B?RmREWEpXcnlxZi82Qk56YmVybm1TaTMram1Fc05OOVNOOEtWNHpxR0lLNVN6?= =?utf-8?B?a0x3cVptb0pHUEVpRFhpMEVuM1NDTHdSSUNUclZCZmdxZWdLWHFnNnBuTlE3?= =?utf-8?B?ckRLaVBEK0JZeEpicFFJUXJmU2E4QlBxbjJtZGF5OGNnaUR2VlFCZHNOYThL?= =?utf-8?Q?U7C2GKbjD/U=3D?= MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-1ff67.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 90285b2f-8b81-4d56-27a4-08da4b3c95bf X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jun 2022 23:54:40.8174 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3P223MB0193 Subject: Re: [FFmpeg-devel] [PATCH] avcodec/qsvenc: make QSV encoder encode VAAPI and D3D11 frames directly X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: "Wu, Tong1" , "Chen, Wenbin" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: > -----Original Message----- > From: ffmpeg-devel On Behalf Of > Xiang, Haihao > Sent: Thursday, June 9, 2022 8:48 AM > To: ffmpeg-devel@ffmpeg.org > Cc: Wu, Tong1 ; Chen, Wenbin > > Subject: Re: [FFmpeg-devel] [PATCH] avcodec/qsvenc: make QSV encoder > encode VAAPI and D3D11 frames directly > > On Wed, 2022-06-08 at 11:13 +0000, Soft Works wrote: > > > -----Original Message----- > > > From: ffmpeg-devel On Behalf Of > Xiang, > > > Haihao > > > Sent: Wednesday, June 8, 2022 10:42 AM > > > To: ffmpeg-devel@ffmpeg.org > > > Cc: Wu, Tong1 ; Chen, Wenbin > > > > Subject: Re: [FFmpeg-devel] [PATCH] avcodec/qsvenc: make QSV > encoder encode > > > VAAPI and D3D11 frames directly > > > > > > On Wed, 2022-06-08 at 05:08 +0000, Soft Works wrote: > > > > > -----Original Message----- > > > > > From: ffmpeg-devel On > Behalf Of Tong > > > > > Wu > > > > > Sent: Tuesday, June 7, 2022 11:22 AM > > > > > To: ffmpeg-devel@ffmpeg.org > > > > > Cc: Tong Wu ; Wenbin Chen > > > > > > Subject: [FFmpeg-devel] [PATCH] avcodec/qsvenc: make QSV > encoder encode > > > > > VAAPI > > > > > and D3D11 frames directly > > > > [..] > > > > > > > 2.35.1.windows.2 > > > > > > > > Hi, > > > > > > > > thanks for submitting this patch. Though, I'm afraid, but this > > > > > > > > - fundamentally contradicts the logic of ffmpeg's handling of > hw > > > > > > acceleration, > > > > hw device and hw frames contexts > > > > - adds code to an encoder, doing things an encoder is not > supposed to do- > > > > > > qsv > > > > encoders and decoders have their own context => QSV > > > > > > nvdec and nvenc have CUDA but nvenc can also support D3D11va, it > sounds make > > > sense for me to support D3D11va/vaapi in qsvenc too as > d3d11va/vaapi are > > > used > > > internally in MediaSDK. > > > > Can you please post a command line showing nvenc working with input > > from a D3D11VA decoder and without using any > hwmap/hwupload/hwdownload > > filters? > > > > According to the code below, nvenc may accept d3d11 frames directly, > > https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/nvenc.c#L46- > L72 > > so the command below should work > > $> ffmpeg -y -hwaccel_output_format d3d11 -hwaccel d3d11va -i > input.mp4 -c:v > hevc_nvenc out.mp4 Right, it does work. Thanks for the command, I had tried like that before, but in a "wrong" branch. Now I took a bit of a deeper look into it and the ability of NVENC to encode from plain D3D11 frames. There are quite a few differences between NVENC and QSVENC. HW Frames Contexts ------------------ QSVENV MSDK cannot work with VAAPI frames, D3D9 frames or D3D11 frames directly. An application is always required to wrap such frames via mfxSurface and manage a collection of mfxSurface descriptions. It's an abstraction that allows coding against the MSDK API independent from the underlying technology. The technical representation of this in ffmpeg is the QSVFramesContext. When there's an input of plain VAAPI or D3D11 frames (hw frames context), then it is required to derive a new QSVFramesContext from the input hw frames context (e.g. via hwmap filter) where the procedure of deriving means to set up a new QSVFramesContext which does the required wrapping (or "mapping" as ffmpeg calls it). I think that the way how this logic is reflected in ffmpeg is thought out very well and provides a high degree of flexibility. NVENC The situation is very different here. Nvidia provides platform independency not by wrapping platform-specific GPU frame types, but instead uses its own custom type - CUDA memory/frames. This is what decoders are outputting, filters are using for input/output and encoders take as input. What I do not know, is whether it would be possible to map D3D11 frames to CUDA frames and vice versa. In case, that would be the preferable way IMO to deal with different hw frame types. At least this isn't implemented at this time. The only possible frames derivation/mapping is from and to Vulkan. Hence I can't say whether the NVENC implementation to take D3D11 frames directly has been done out of laziness or whether it was the only possible way. In case when it wouldn't be possible to map D3D11 frames to CUDA frames, and only NVENC encoders would be able to process D3D11 frames, then it would have been the only option of course. But in any way, it's not the same as with QSVENC, because NVENC can take D3D11 frames as input directly without wrapping/mapping first. ---------------- There are more differences, but I don't want to drive it too far. What stands at the bottom line is: - NVENC can take D3D11 frames context directly - QSVENC can't - it needs to map it to a QSVFramesContext first Concluding opinion: An encoder should not include (duplicated) code for creating a derived frames context. The same goal (= getting those command lines working that Haihao has posted) could be achieved by auto-inserting a hwmap filter in those cases, which would probably take a few lines of code only. We don't have a precedence for auto-insertion of a hwmap filter, but we do that in many other cases, so that would seem to me at least an option to think about. I'm curious about other opinions... Thanks, softworkz _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".