From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id D472B4381C for ; Mon, 1 Aug 2022 13:45:42 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 990F168B9E5; Mon, 1 Aug 2022 16:45:39 +0300 (EEST) Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A20BD68B7DB for ; Mon, 1 Aug 2022 16:45:33 +0300 (EEST) Received: by mail-lj1-f169.google.com with SMTP id s14so12362484ljh.0 for ; Mon, 01 Aug 2022 06:45:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=an7WbqqH7gFwPhHkqs92OulLk3U3fWkIS0Hl7Yg+zd8=; b=gWru8izoZx43J6Q1EYHek+XpYlDzSNcOhzFlMYo8prznwdqmHItrCwjM3ldwPzBhyC O6/KK+8wdKTo3i9hnuQeBUjxg/qWvvfq1i2zpgeoj7UcFVOQlzMw6WGQ590ezuAlgswy NWdVkzo/NudlmrDtOEV1Iqp0LlYWj7uz+B+RwTkwXDY7a4gWPurk8jpp6Iq24tw+Fz8M Rrj7YSevwsYQXVbqqtEV7RbgqdAlnrMgWr75y8sBYnobPxwY/3zu0EJk9eaBshaqiwDU d6KQ8ob5x9jl9A69DawRbK1ZDRFPB2Z6B67EiB+T2QiteqEiiS4kA5IgK1K26/Iw9KTD jd0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=an7WbqqH7gFwPhHkqs92OulLk3U3fWkIS0Hl7Yg+zd8=; b=TDb6mBFz5Hcjr5svh5pHSQmbs/NrEDn9xa/2WqwrBJcC1nAORHNYpVeb4dmbpVJLSZ EFJMTY9LkoZ37s/AAi1f2ZFXcCZJUYsRlNJO3GeTbyzUUkr81RBeAMuDaDZ77FR8ni3p qotFoFG8IV1+nYfl1R9KVjqtr8sSccs+IMg8hea9aCDqZtjgzAbBwu8THSKxI21QwY00 t3C1U2k/IANxgABUFXxJYZtbK6Iut4335QPJRk2et5ATYqbkFJhM4ldzLjS+oSJyx1Pb DzZ0QnXRe+FbQnoTsv+VxJztffC1t9SWW09wpDxax9qYdT2P3jOctZwCDh7o8oQkJ6LL zjxQ== X-Gm-Message-State: AJIora8bBaf2L7XAwlozpGrbQVGxN6g1KOzyu4+aSTC/jufEGXmsPLtY PcDWzIFKsjFx9Axxr/IGv1/pNGyB6mIviLBB+z2xML9b X-Google-Smtp-Source: AGRyM1tGhmMQRDLixNo7PYCX2XYJnAOfWybsYET3c4twm8XU66JuzSb+esQrHreMaM5hmq9sX7uyE96x3SD11ZCJago= X-Received: by 2002:a2e:a78c:0:b0:25e:8a8:6d39 with SMTP id c12-20020a2ea78c000000b0025e08a86d39mr5311959ljf.44.1659361531853; Mon, 01 Aug 2022 06:45:31 -0700 (PDT) MIME-Version: 1.0 References: <20220801132452.GB41618@haasn.xyz> In-Reply-To: <20220801132452.GB41618@haasn.xyz> From: Hendrik Leppkes Date: Mon, 1 Aug 2022 15:45:19 +0200 Message-ID: To: FFmpeg development discussions and patches Subject: Re: [FFmpeg-devel] Enhancement layers in FFmpeg X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: On Mon, Aug 1, 2022 at 1:25 PM Niklas Haas wrote: > > Hey, > > We need to think about possible ways to implement reasonably-transparent > support for enhancement layers in FFmpeg. (SVC, Dolby Vision, ...). > There are more open questions than answers here. > > From what I can tell, these are basically separate bitstreams that carry > some amount of auxiliary information needed to reconstruct the > high-quality bitstream. That is, they are not independent, but need to > be merged with the original bitstream somehow. > > How do we architecturally fit this into FFmpeg? Do we define a new codec > ID for each (common/relevant) combination of base codec and enhancement > layer, e.g. HEVC+DoVi, H.264+SVC, ..., or do we transparently handle it > for the base codec ID and control it via a flag? Do the enhancement > layer packets already make their way to the codec, and if not, how do we > ensure that this is the case? EL on Blu-rays are a separate stream, so that would need to be handled in some fashion. Unless it wouldn't. See below. > > Can the decoder itself recursively initialize a sub-decoder for the > second bitstream? And if so, does the decoder apply the actual > transformation, or does it merely attach the EL data to the AVFrame > somehow in a way that can be used by further filters or end users? My main question is, how closely related are those streams? I know that Dolby EL can be decoded basically entirely separately from the main video stream. But EL might be the special case here. I have no experience with SVC. If the enhancement layer is entirely independent, like Dolby EL, should avcodec need to do anything? It _can_ decode the stream today, a user-application could write code that decodes both the main stream and the EL stream and links them together, without any changes in avcodec. Do we need to complicate this situation by forcing this into avcodec? Decoding them in entirely separate decoder instances has the advantage of being able to use Hardware for the main one, software for the EL, or both in hardware, or whatever one prefers. Of course this applies to the special situation of Dolby EL which is entirely independent, at least in its primary source - Blu-ray. I think MKV might mix both into one stream, which is an unfortunate design decision on their part. avfilter for example is already setup to synchronize two incoming streams (for eg. overlay), so the same mechanic could be used to pass it to a processing filter. > > (What about the case of Dolby Vision, which iirc requires handling the > DoVi RPU metadata before the EL can be applied? What about instances > where the user wants the DoVi/EL application to happen on GPU, e.g. via > libplacebo in mpv/vlc?) > Yes, processing should be left to dedicated filters. > How does this metadata need to be attached? A second AVFrame reference > inside the AVFrame? Raw data in a big side data struct? For Dolby EL, no attachment is necessary if we follow the above concept of just not having avcodec care. - Hendrik _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".