From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id E0EBA43BCA for ; Sun, 20 Nov 2022 16:44:40 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 25C9468BA23; Sun, 20 Nov 2022 18:44:38 +0200 (EET) Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2442E68A791 for ; Sun, 20 Nov 2022 18:44:31 +0200 (EET) Received: by mail-wm1-f46.google.com with SMTP id l39-20020a05600c1d2700b003cf93c8156dso7499198wms.4 for ; Sun, 20 Nov 2022 08:44:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jkqxz-net.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=K1cQeQMoQzBo/XtLvbX36/tD8MQClcjubWlgc4dbpgU=; b=4cm9SRsrSr7aYtCvq18nzXf8Qj8DCsn+5yV/oWpYS/1aWsv/mJ3cmm/4rJHPEuxIzA m0FANcwofJ1dDUHIHEK2yuVVtGJ+STJwtNJGuHCDvqOd1Ta0TKzcnimqr3A4mjHSS0fn 584MSDARz9j9ALYZhR4dGp8TMItAl4ZbjMvkwcdzH7U6GocL+1y2Ow1Fa46KqGplqC15 0UkeP7LLKqVO/pPeJbAmsAZeC0Uh7z5GKXsTn8+IR2N/TnBy26Jtt4ojw1CDXlfkP9xR OssKU8FT4B0lVg5VhwZDVXC8fM6SnrNkLhgsxpBP/3SOjVbkqgh9aJ93AwClA/4AaWYY LQPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=K1cQeQMoQzBo/XtLvbX36/tD8MQClcjubWlgc4dbpgU=; b=6xw/tzm5ZRcWcJqTW6ghD7yhDgeQVuvy9kg3qMp1TnZgw4NC3FsBEa4/NEnHVoEQSp pe4Ve8mLzxdCi6GooWDXqynT0IhkZcfZbLefQreEwqBLyheDs0mDmNhBOB4DwOoOcLXt NMgkp8zIVBMcqfX8MTFkf7wU5yT3mk0G0zY8CM4mB7LbeymE2h1MtbQMq8pBFypHCPnp fJHZ1vS0DzzhLoClsA0TlPDw/AbUmqMlcgNlPXZLiJ95vlkAwBRi4X5bBN8cMnGWhEK2 C/XneN+lTdV6MCtnJfG0av0BZDETdfVYBP5zgbOal3URPE3eUwig3F8RtgDtz0mJUXGe WlRA== X-Gm-Message-State: ANoB5ply6OTjo6H+LnwJauj03RTLUy849aesl2zlsakzD8lOGriXy1bW SO8BRVJwf+GphDVJsGCKfKxyJLUZLOlwxw== X-Google-Smtp-Source: AA0mqf695rnwWT1pvsdy22hryQA5uoqWigA7apKmZpvYMXLTS/QS/C3XtvxAil+t0oFPM6iszqqg7Q== X-Received: by 2002:a1c:770a:0:b0:3cf:ab80:b558 with SMTP id t10-20020a1c770a000000b003cfab80b558mr13614169wmi.155.1668962670328; Sun, 20 Nov 2022 08:44:30 -0800 (PST) Received: from [192.168.0.14] (cpc91224-cmbg18-2-0-cust209.5-4.cable.virginm.net. [81.106.228.210]) by smtp.gmail.com with ESMTPSA id o9-20020a056000010900b00228692033dcsm8977373wrx.91.2022.11.20.08.44.29 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 20 Nov 2022 08:44:29 -0800 (PST) Message-ID: <4776acec-357b-5c36-1292-7e5204c4759f@jkqxz.net> Date: Sun, 20 Nov 2022 16:44:06 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.1 Content-Language: en-US To: ffmpeg-devel@ffmpeg.org References: <20221118153422.67632-1-ruijing.dong@amd.com> <20221120025914.39732-1-ruijing.dong@amd.com> From: Mark Thompson In-Reply-To: <20221120025914.39732-1-ruijing.dong@amd.com> Subject: Re: [FFmpeg-devel] [PATCH v2] avcodec/av1_vaapi: add direct film grain mode X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: On 20/11/2022 02:59, Ruijing Dong wrote: > Adding direct film grain mode for av1 decoder, which > outputs alongside film grain. > > AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN is the new flag > introduced to enable this path. > > issue: > By using AMD av1 decoder via VAAPI, when used with film > grain content, the output displays black screen with > incorrect frame order. > > The issue being discussed in here: > https://gitlab.freedesktop.org/mesa/mesa/-/issues/6903#note_1613807 > > example: > This flag can be used in ffmpeg command: > > ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 > -hwaccel_flags 8 > -i input_with_film_gram.obu > output_with_film_grain.yuv > > Signed-off-by: Ruijing Dong > --- > update: add new option direct_film_grain in optons_table.h > > libavcodec/avcodec.h | 17 +++++++++++++++++ > libavcodec/options_table.h | 1 + > libavcodec/vaapi_av1.c | 6 ++++-- > 3 files changed, 22 insertions(+), 2 deletions(-) My understanding of this is as follows, please correct me if any of this is wrong: * For AV1 with film grain enabled, VAAPI decode is specified with two output surfaces: one for pre-grain (reference) output and one for post-grain (display) output. * The current driver in Mesa always writes to the pre-grain surface and ignores the post-grain surface entirely. * To fix this, you intend to modify the VAAPI code in libavcodec to allow the user to manually override the expected VAAPI behaviour and instead assume that the post-grain output has been written to the pre-grain surface. Is that right? If it is, could you perhaps explain why this manual option is preferable to the more obvious solution of Mesa being fixed to write the post-grain output to the post-grain surface? Thanks, - Mark > diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h > index 3edd8e2636..9420e7820d 100644 > --- a/libavcodec/avcodec.h > +++ b/libavcodec/avcodec.h > @@ -2253,6 +2253,23 @@ typedef struct AVHWAccel { > */ > #define AV_HWACCEL_FLAG_ALLOW_PROFILE_MISMATCH (1 << 2) > > +/** > + * The film grain synthesis could be seperate from decoding process. > + * The downstream device would apply the film grain parameters seperately. > + * The desired film grain parameters can be found in SEI section in H264 > + * or H265 bitstreams. > + * > + * In AV1, film grain is mandatorily specified, av1 decoders like AMD > + * av1 decoder process film grain content internally, and the output > + * includes applied film grain. For the purpose of supporting these av1 > + * decoders, this flag needs to be utilized. > + * > + * @warning If the stream has no film grain content, this flag will > + * be ignored in the supported av1 decoders. It is advised > + * that this flag should only be used in av1 decoders > + * that support it. > + */ > +#define AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN (1 << 3) > /** > * @} > */ > diff --git a/libavcodec/options_table.h b/libavcodec/options_table.h > index cd02f5096f..0302f89280 100644 > --- a/libavcodec/options_table.h > +++ b/libavcodec/options_table.h > @@ -399,6 +399,7 @@ static const AVOption avcodec_options[] = { > {"ignore_level", "ignore level even if the codec level used is unknown or higher than the maximum supported level reported by the hardware driver", 0, AV_OPT_TYPE_CONST, { .i64 = AV_HWACCEL_FLAG_IGNORE_LEVEL }, INT_MIN, INT_MAX, V | D, "hwaccel_flags" }, > {"allow_high_depth", "allow to output YUV pixel formats with a different chroma sampling than 4:2:0 and/or other than 8 bits per component", 0, AV_OPT_TYPE_CONST, {.i64 = AV_HWACCEL_FLAG_ALLOW_HIGH_DEPTH }, INT_MIN, INT_MAX, V | D, "hwaccel_flags"}, > {"allow_profile_mismatch", "attempt to decode anyway if HW accelerated decoder's supported profiles do not exactly match the stream", 0, AV_OPT_TYPE_CONST, {.i64 = AV_HWACCEL_FLAG_ALLOW_PROFILE_MISMATCH }, INT_MIN, INT_MAX, V | D, "hwaccel_flags"}, > +{"direct_film_grain", "allow decoder to directly apply film grain to the output", 0, AV_OPT_TYPE_CONST, {.i64 = AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN }, INT_MIN, INT_MAX, V | D, "hwaccel_flags"}, > {"extra_hw_frames", "Number of extra hardware frames to allocate for the user", OFFSET(extra_hw_frames), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, INT_MAX, V|D }, > {"discard_damaged_percentage", "Percentage of damaged samples to discard a frame", OFFSET(discard_damaged_percentage), AV_OPT_TYPE_INT, {.i64 = 95 }, 0, 100, V|D }, > {NULL}, > diff --git a/libavcodec/vaapi_av1.c b/libavcodec/vaapi_av1.c > index d0339b2705..6db910f2bf 100644 > --- a/libavcodec/vaapi_av1.c > +++ b/libavcodec/vaapi_av1.c > @@ -127,6 +127,7 @@ static int vaapi_av1_start_frame(AVCodecContext *avctx, > int8_t bit_depth_idx; > int err = 0; > int apply_grain = !(avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN) && film_grain->apply_grain; > + int direct_film_grain = avctx->hwaccel_flags & AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN; > uint8_t remap_lr_type[4] = {AV1_RESTORE_NONE, AV1_RESTORE_SWITCHABLE, AV1_RESTORE_WIENER, AV1_RESTORE_SGRPROJ}; > uint8_t segmentation_feature_signed[AV1_SEG_LVL_MAX] = {1, 1, 1, 1, 1, 0, 0, 0}; > uint8_t segmentation_feature_max[AV1_SEG_LVL_MAX] = {255, AV1_MAX_LOOP_FILTER, > @@ -136,7 +137,7 @@ static int vaapi_av1_start_frame(AVCodecContext *avctx, > if (bit_depth_idx < 0) > goto fail; > > - if (apply_grain) { > + if (apply_grain && !direct_film_grain) { > if (ctx->tmp_frame->buf[0]) > ff_thread_release_buffer(avctx, ctx->tmp_frame); > err = ff_thread_get_buffer(avctx, ctx->tmp_frame, AV_GET_BUFFER_FLAG_REF); > @@ -375,6 +376,7 @@ static int vaapi_av1_end_frame(AVCodecContext *avctx) > VAAPIAV1DecContext *ctx = avctx->internal->hwaccel_priv_data; > > int apply_grain = !(avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN) && film_grain->apply_grain; > + int direct_film_grain = avctx->hwaccel_flags & AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN; > int ret; > ret = ff_vaapi_decode_issue(avctx, pic); > if (ret < 0) > @@ -385,7 +387,7 @@ static int vaapi_av1_end_frame(AVCodecContext *avctx) > if (ctx->ref_tab[i].frame->buf[0]) > ff_thread_release_buffer(avctx, ctx->ref_tab[i].frame); > > - if (apply_grain) { > + if (apply_grain && !direct_film_grain) { > ret = av_frame_ref(ctx->ref_tab[i].frame, ctx->tmp_frame); > if (ret < 0) > return ret; > -- > 2.25.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".