From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 0A80845A77 for ; Tue, 14 Mar 2023 06:33:54 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8035568BC49; Tue, 14 Mar 2023 08:33:51 +0200 (EET) Received: from w4.tutanota.de (w4.tutanota.de [81.3.6.165]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7332068BC08 for ; Tue, 14 Mar 2023 08:33:44 +0200 (EET) Received: from tutadb.w10.tutanota.de (unknown [192.168.1.10]) by w4.tutanota.de (Postfix) with ESMTP id 9AF60106016F for ; Tue, 14 Mar 2023 06:33:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1678775623; s=s1; d=lynne.ee; h=From:From:To:To:Subject:Subject:Content-Description:Content-ID:Content-Type:Content-Type:Content-Transfer-Encoding:Cc:Date:Date:In-Reply-To:MIME-Version:MIME-Version:Message-ID:Message-ID:Reply-To:References:Sender; bh=7Nip7FFxNjFPJGGG1k1qcFHhuhto3axww3ACe2bJdkk=; b=AHQ8NeXe+8civ/vf3z9YP4mUEGb7yNg8RFncIun3cTp2coJC7v+cP3rlm1PbXh68 ZUFr0z/Pg57zgjfg6CLyzQ+19szBoja3cYarOHraFk6NTwuy4z6LhtooIY8UClHu5tm XFtYBYcL7rMpmBwiQueUcXCLweaCC99yy8iDfhfeUP4cAFM4yVn+2oS3v5GQ1MGpJD1 0+6Yd5aBwSjhnZ5Ns0FnQFFFwEERsfhKdVnK1/cV8A6tlr2tZ+v85O7HHpyS9OCdSRv MpMZIywBuXcvNv6Hu4pvhctmImM45RIGPiSHOVY2RxCmCTEQCWPAT94CwFpqE5n4ihk TqW72LyZdg== Date: Tue, 14 Mar 2023 07:33:43 +0100 (CET) From: Lynne To: Ffmpeg Devel Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_55579_1396114304.1678775623438" Subject: [FFmpeg-devel] [PATCH 31/92] Vulkan patchset part 1 - common code changes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: ------=_Part_55579_1396114304.1678775623438 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit The attached patchset is all the common code changes that my Vulkan patchset needs. In total lines of code, this part has 425 additions and 131 deletions. Most of that is additions to HEVC parsing. Excluding them, the patchset is 200 lines of code added, which is manageable. Apart from the parser changes, the following other changes have been made to the API: AVHWAccel.free_frame_priv exists due to Vulkan's way of using VkImageView objects to wrap VkImage objects, which we need to free once they're no longer in use. Every other API uses the direct objects in decoding, but with Vulkan, they have to be represented by other objects. We also use it to free the slice offsets buffer. AVHWAccel.flush exists due to Vulkan keeping decoder state, despite being stateless in theory. The decoder has to be notified of flushes in order to reset decoding slots and other data it needs, such as motion vectors and reference lists for AV1. Otherwise, inferring whether a flush has happened can be codec dependent, and hacky. hwaccel_params_buf exists due to Vulkan's way of compiling SPS/PPS data into objects, making updating expensive. The change allows for hardware to only upload new parameters if they have been changed. It's insignificant for H264 and AV1, but HEVC's structures can reach 114 megabytes of data that has to be uploaded, for a specially crafted input, which is enough to DDOS an ingest. The data is set and managed by the hwaccel, but does need to be synchronized between different decoding threads, which this patch performs. Finally, the HWACCEL_CAP_THREAD_SAFE flag is added due to Vulkan being actually threadsafe, and requiring no serialization. It does work and it does actually make a difference, on average, it can increase performance by 20% for an average B-frame using HEVC stream, depending on the number of threads and the number of decode queues. While hardware decoders are fast in general, certain vendors such as AMD can choke up while playing 8k video, and this patch can significantly help increase throughput. In context, the changes can be viewed here: https://github.com/cyanreg/FFmpeg/tree/vulkan The rest of the whole patchset is either rewrites, filter code, or the actual hardware accel code. The patchset will not be pushed standalone, but as part of the greater Vulkan patchset. 31 patches attached. ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0001-h2645_vui-expose-aspect_ratio_idc.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0001-h2645_vui-expose-aspect_ratio_idc.patch >From 0a8637a2301397cb8ff207c4a7a5193b4ce13503 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 14 Dec 2022 00:02:11 +0100 Subject: [PATCH 01/92] h2645_vui: expose aspect_ratio_idc --- libavcodec/h2645_vui.c | 10 +++++----- libavcodec/h2645_vui.h | 1 + 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/libavcodec/h2645_vui.c b/libavcodec/h2645_vui.c index 0633fcbddd..93e83a9e1f 100644 --- a/libavcodec/h2645_vui.c +++ b/libavcodec/h2645_vui.c @@ -42,15 +42,15 @@ void ff_h2645_decode_common_vui_params(GetBitContext *gb, H2645VUI *vui, void *l aspect_ratio_info_present_flag = get_bits1(gb); if (aspect_ratio_info_present_flag) { - uint8_t aspect_ratio_idc = get_bits(gb, 8); - if (aspect_ratio_idc < FF_ARRAY_ELEMS(ff_h2645_pixel_aspect)) - vui->sar = ff_h2645_pixel_aspect[aspect_ratio_idc]; - else if (aspect_ratio_idc == EXTENDED_SAR) { + vui->aspect_ratio_idc = get_bits(gb, 8); + if (vui->aspect_ratio_idc < FF_ARRAY_ELEMS(ff_h2645_pixel_aspect)) + vui->sar = ff_h2645_pixel_aspect[vui->aspect_ratio_idc]; + else if (vui->aspect_ratio_idc == EXTENDED_SAR) { vui->sar.num = get_bits(gb, 16); vui->sar.den = get_bits(gb, 16); } else av_log(logctx, AV_LOG_WARNING, - "Unknown SAR index: %u.\n", aspect_ratio_idc); + "Unknown SAR index: %u.\n", vui->aspect_ratio_idc); } else vui->sar = (AVRational){ 0, 1 }; diff --git a/libavcodec/h2645_vui.h b/libavcodec/h2645_vui.h index 638da7c366..f1aeab7758 100644 --- a/libavcodec/h2645_vui.h +++ b/libavcodec/h2645_vui.h @@ -26,6 +26,7 @@ typedef struct H2645VUI { AVRational sar; + int aspect_ratio_idc; int overscan_info_present_flag; int overscan_appropriate_flag; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0002-h2645_vui-expose-aspect_ratio_info_present_flag.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0002-h2645_vui-expose-aspect_ratio_info_present_flag.patch >From aac126d0f52006d5429b3ae168e51f31a79fc278 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 14 Dec 2022 00:03:44 +0100 Subject: [PATCH 02/92] h2645_vui: expose aspect_ratio_info_present_flag --- libavcodec/h2645_vui.c | 6 ++---- libavcodec/h2645_vui.h | 1 + 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/libavcodec/h2645_vui.c b/libavcodec/h2645_vui.c index 93e83a9e1f..e5c7bf46f9 100644 --- a/libavcodec/h2645_vui.c +++ b/libavcodec/h2645_vui.c @@ -36,12 +36,10 @@ void ff_h2645_decode_common_vui_params(GetBitContext *gb, H2645VUI *vui, void *logctx) { - int aspect_ratio_info_present_flag; - av_log(logctx, AV_LOG_DEBUG, "Decoding VUI\n"); - aspect_ratio_info_present_flag = get_bits1(gb); - if (aspect_ratio_info_present_flag) { + vui->aspect_ratio_info_present_flag = get_bits1(gb); + if (vui->aspect_ratio_info_present_flag) { vui->aspect_ratio_idc = get_bits(gb, 8); if (vui->aspect_ratio_idc < FF_ARRAY_ELEMS(ff_h2645_pixel_aspect)) vui->sar = ff_h2645_pixel_aspect[vui->aspect_ratio_idc]; diff --git a/libavcodec/h2645_vui.h b/libavcodec/h2645_vui.h index f1aeab7758..2c839f4b01 100644 --- a/libavcodec/h2645_vui.h +++ b/libavcodec/h2645_vui.h @@ -27,6 +27,7 @@ typedef struct H2645VUI { AVRational sar; int aspect_ratio_idc; + int aspect_ratio_info_present_flag; int overscan_info_present_flag; int overscan_appropriate_flag; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0003-h264_ps-expose-pps_id.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0003-h264_ps-expose-pps_id.patch >From bc615733da705ee198a9c96eb9847027ff601783 Mon Sep 17 00:00:00 2001 From: Lynne Date: Fri, 18 Mar 2022 15:11:02 +0100 Subject: [PATCH 03/92] h264_ps: expose pps_id --- libavcodec/h264_ps.c | 1 + libavcodec/h264_ps.h | 1 + 2 files changed, 2 insertions(+) diff --git a/libavcodec/h264_ps.c b/libavcodec/h264_ps.c index d0d1e65903..4ec5bd4e80 100644 --- a/libavcodec/h264_ps.c +++ b/libavcodec/h264_ps.c @@ -731,6 +731,7 @@ int ff_h264_decode_picture_parameter_set(GetBitContext *gb, AVCodecContext *avct if (!(bit_length & 7) && pps->data_size < sizeof(pps->data)) pps->data[pps->data_size++] = 0x80; + pps->pps_id = pps_id; pps->sps_id = get_ue_golomb_31(gb); if ((unsigned)pps->sps_id >= MAX_SPS_COUNT || !ps->sps_list[pps->sps_id]) { diff --git a/libavcodec/h264_ps.h b/libavcodec/h264_ps.h index 5c35761fbc..c3f0888f24 100644 --- a/libavcodec/h264_ps.h +++ b/libavcodec/h264_ps.h @@ -103,6 +103,7 @@ typedef struct SPS { * Picture parameter set */ typedef struct PPS { + unsigned int pps_id; unsigned int sps_id; int cabac; ///< entropy_coding_mode_flag int pic_order_present; ///< pic_order_present_flag -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0004-h264_ps-set-pic_scaling_matrix_present_flag.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0004-h264_ps-set-pic_scaling_matrix_present_flag.patch >From be97efa7b1834ce127741f55286a5ba1b6649341 Mon Sep 17 00:00:00 2001 From: Lynne Date: Fri, 18 Mar 2022 16:17:33 +0100 Subject: [PATCH 04/92] h264_ps: set pic_scaling_matrix_present_flag --- libavcodec/h264_ps.c | 7 +++++-- libavcodec/h264_ps.h | 1 + 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/libavcodec/h264_ps.c b/libavcodec/h264_ps.c index 4ec5bd4e80..a94f5350c4 100644 --- a/libavcodec/h264_ps.c +++ b/libavcodec/h264_ps.c @@ -226,6 +226,7 @@ static int decode_scaling_list(GetBitContext *gb, uint8_t *factors, int size, /* returns non zero if the provided SPS scaling matrix has been filled */ static int decode_scaling_matrices(GetBitContext *gb, const SPS *sps, const PPS *pps, int is_sps, + int present_flag, uint8_t(*scaling_matrix4)[16], uint8_t(*scaling_matrix8)[64]) { @@ -237,7 +238,7 @@ static int decode_scaling_matrices(GetBitContext *gb, const SPS *sps, fallback_sps ? sps->scaling_matrix8[3] : default_scaling8[1] }; int ret = 0; - if (get_bits1(gb)) { + if (present_flag) { ret |= decode_scaling_list(gb, scaling_matrix4[0], 16, default_scaling4[0], fallback[0]); // Intra, Y ret |= decode_scaling_list(gb, scaling_matrix4[1], 16, default_scaling4[0], scaling_matrix4[0]); // Intra, Cr ret |= decode_scaling_list(gb, scaling_matrix4[2], 16, default_scaling4[0], scaling_matrix4[1]); // Intra, Cb @@ -368,7 +369,7 @@ int ff_h264_decode_seq_parameter_set(GetBitContext *gb, AVCodecContext *avctx, goto fail; } sps->transform_bypass = get_bits1(gb); - ret = decode_scaling_matrices(gb, sps, NULL, 1, + ret = decode_scaling_matrices(gb, sps, NULL, 1, get_bits1(gb), sps->scaling_matrix4, sps->scaling_matrix8); if (ret < 0) goto fail; @@ -803,7 +804,9 @@ int ff_h264_decode_picture_parameter_set(GetBitContext *gb, AVCodecContext *avct bits_left = bit_length - get_bits_count(gb); if (bits_left > 0 && more_rbsp_data_in_pps(sps, avctx)) { pps->transform_8x8_mode = get_bits1(gb); + pps->pic_scaling_matrix_present_flag = get_bits1(gb); ret = decode_scaling_matrices(gb, sps, pps, 0, + pps->pic_scaling_matrix_present_flag, pps->scaling_matrix4, pps->scaling_matrix8); if (ret < 0) goto fail; diff --git a/libavcodec/h264_ps.h b/libavcodec/h264_ps.h index c3f0888f24..d2413ae0f8 100644 --- a/libavcodec/h264_ps.h +++ b/libavcodec/h264_ps.h @@ -119,6 +119,7 @@ typedef struct PPS { int constrained_intra_pred; ///< constrained_intra_pred_flag int redundant_pic_cnt_present; ///< redundant_pic_cnt_present_flag int transform_8x8_mode; ///< transform_8x8_mode_flag + int pic_scaling_matrix_present_flag; uint8_t scaling_matrix4[6][16]; uint8_t scaling_matrix8[6][64]; uint8_t chroma_qp_table[2][QP_MAX_NUM+1]; ///< pre-scaled (with chroma_qp_index_offset) version of qp_table -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0005-h264_parser-expose-idr_pic_id.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0005-h264_parser-expose-idr_pic_id.patch >From 56ebb0bc2fd4b856196e6436b4c18af8e4d1a86a Mon Sep 17 00:00:00 2001 From: Lynne Date: Thu, 10 Mar 2022 18:08:53 +0100 Subject: [PATCH 05/92] h264_parser: expose idr_pic_id Vulkan needs it. --- libavcodec/h264_parse.h | 1 + libavcodec/h264_parser.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/libavcodec/h264_parse.h b/libavcodec/h264_parse.h index 4ee863df66..4ba0add4f2 100644 --- a/libavcodec/h264_parse.h +++ b/libavcodec/h264_parse.h @@ -85,6 +85,7 @@ typedef struct H264POCContext { int delta_poc_bottom; int delta_poc[2]; int frame_num; + int idr_pic_id; int prev_poc_msb; ///< poc_msb of the last reference pic for POC type 0 int prev_poc_lsb; ///< poc_lsb of the last reference pic for POC type 0 int frame_num_offset; ///< for POC type 2 diff --git a/libavcodec/h264_parser.c b/libavcodec/h264_parser.c index 46134a1c48..1c330484c1 100644 --- a/libavcodec/h264_parser.c +++ b/libavcodec/h264_parser.c @@ -432,7 +432,7 @@ static inline int parse_nal_units(AVCodecParserContext *s, } if (nal.type == H264_NAL_IDR_SLICE) - get_ue_golomb_long(&nal.gb); /* idr_pic_id */ + p->poc.idr_pic_id = get_ue_golomb_long(&nal.gb); /* idr_pic_id */ if (sps->poc_type == 0) { p->poc.poc_lsb = get_bits(&nal.gb, sps->log2_max_poc_lsb); -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0006-h264_ps-comment-pic_order_present-better.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0006-h264_ps-comment-pic_order_present-better.patch >From 806d8cc5ae732a605406ee4e3c7dbe9acfad9da7 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 23 Nov 2022 15:59:23 +0100 Subject: [PATCH 06/92] h264_ps: comment pic_order_present better The official name which CBS uses is bottom_field_pic_order_in_frame_present_flag. --- libavcodec/h264_ps.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavcodec/h264_ps.h b/libavcodec/h264_ps.h index d2413ae0f8..de4529b353 100644 --- a/libavcodec/h264_ps.h +++ b/libavcodec/h264_ps.h @@ -106,7 +106,7 @@ typedef struct PPS { unsigned int pps_id; unsigned int sps_id; int cabac; ///< entropy_coding_mode_flag - int pic_order_present; ///< pic_order_present_flag + int pic_order_present; ///< bottom_field_pic_order_in_frame_present_flag int slice_group_count; ///< num_slice_groups_minus1 + 1 int mb_slice_group_map_type; unsigned int ref_count[2]; ///< num_ref_idx_l0/1_active_minus1 + 1 -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0007-h264_ps-expose-max_dec_frame_buffering.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0007-h264_ps-expose-max_dec_frame_buffering.patch >From b5adb92d6d5bdfe7f1129b897500664827ea5533 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 14 Dec 2022 00:06:04 +0100 Subject: [PATCH 07/92] h264_ps: expose max_dec_frame_buffering --- libavcodec/h264_ps.c | 2 +- libavcodec/h264_ps.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/libavcodec/h264_ps.c b/libavcodec/h264_ps.c index a94f5350c4..d9df570718 100644 --- a/libavcodec/h264_ps.c +++ b/libavcodec/h264_ps.c @@ -176,7 +176,7 @@ static inline int decode_vui_parameters(GetBitContext *gb, void *logctx, get_ue_golomb_31(gb); /* log2_max_mv_length_horizontal */ get_ue_golomb_31(gb); /* log2_max_mv_length_vertical */ sps->num_reorder_frames = get_ue_golomb_31(gb); - get_ue_golomb_31(gb); /*max_dec_frame_buffering*/ + sps->max_dec_frame_buffering = get_ue_golomb_31(gb); if (get_bits_left(gb) < 0) { sps->num_reorder_frames = 0; diff --git a/libavcodec/h264_ps.h b/libavcodec/h264_ps.h index de4529b353..906bab7214 100644 --- a/libavcodec/h264_ps.h +++ b/libavcodec/h264_ps.h @@ -80,6 +80,7 @@ typedef struct SPS { int32_t offset_for_ref_frame[256]; int bitstream_restriction_flag; int num_reorder_frames; + int max_dec_frame_buffering; int scaling_matrix_present; uint8_t scaling_matrix4[6][16]; uint8_t scaling_matrix8[6][64]; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0008-h264_ps-expose-bit-rate-and-CPB-size-fields.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0008-h264_ps-expose-bit-rate-and-CPB-size-fields.patch >From d465e05bc33d74f2469a6c80ba60a696dd2b7835 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 14 Dec 2022 00:09:08 +0100 Subject: [PATCH 08/92] h264_ps: expose bit rate and CPB size fields --- libavcodec/h264_ps.c | 9 +++++---- libavcodec/h264_ps.h | 4 ++++ 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/libavcodec/h264_ps.c b/libavcodec/h264_ps.c index d9df570718..c71330089d 100644 --- a/libavcodec/h264_ps.c +++ b/libavcodec/h264_ps.c @@ -113,12 +113,13 @@ static inline int decode_hrd_parameters(GetBitContext *gb, void *logctx, return AVERROR_INVALIDDATA; } - get_bits(gb, 4); /* bit_rate_scale */ + sps->cpr_flag = 0x0; + sps->bit_rate_scale = get_bits(gb, 4); get_bits(gb, 4); /* cpb_size_scale */ for (i = 0; i < cpb_count; i++) { - get_ue_golomb_long(gb); /* bit_rate_value_minus1 */ - get_ue_golomb_long(gb); /* cpb_size_value_minus1 */ - get_bits1(gb); /* cbr_flag */ + sps->bit_rate_value[i] = get_ue_golomb_long(gb) + 1; /* bit_rate_value_minus1 + 1 */ + sps->cpb_size_value[i] = get_ue_golomb_long(gb) + 1; /* cpb_size_value_minus1 + 1 */ + sps->cpr_flag |= get_bits1(gb) << i; } sps->initial_cpb_removal_delay_length = get_bits(gb, 5) + 1; sps->cpb_removal_delay_length = get_bits(gb, 5) + 1; diff --git a/libavcodec/h264_ps.h b/libavcodec/h264_ps.h index 906bab7214..8adfbd710f 100644 --- a/libavcodec/h264_ps.h +++ b/libavcodec/h264_ps.h @@ -89,6 +89,10 @@ typedef struct SPS { int pic_struct_present_flag; int time_offset_length; int cpb_cnt; ///< See H.264 E.1.2 + int bit_rate_scale; + uint32_t bit_rate_value[32]; ///< bit_rate_value_minus1 + 1 + uint32_t cpb_size_value[32]; ///< cpb_size_value_minus1 + 1 + uint32_t cpr_flag; int initial_cpb_removal_delay_length; ///< initial_cpb_removal_delay_length_minus1 + 1 int cpb_removal_delay_length; ///< cpb_removal_delay_length_minus1 + 1 int dpb_output_delay_length; ///< dpb_output_delay_length_minus1 + 1 -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0009-h264_ps-expose-scaling_matrix_present_mask.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0009-h264_ps-expose-scaling_matrix_present_mask.patch >From a813407e6aebef72f4d4c3be45befbb3dd48e8ab Mon Sep 17 00:00:00 2001 From: Lynne Date: Thu, 15 Dec 2022 17:05:35 +0100 Subject: [PATCH 09/92] h264_ps: expose scaling_matrix_present_mask Vulkan requires it. It technically also requires use_default_scaling_matrix_mask, but we can just be explicit and give it the matrix we fill in as-non default. --- libavcodec/h264_ps.c | 37 +++++++++++++++++++++---------------- libavcodec/h264_ps.h | 2 ++ 2 files changed, 23 insertions(+), 16 deletions(-) diff --git a/libavcodec/h264_ps.c b/libavcodec/h264_ps.c index c71330089d..53446e9aab 100644 --- a/libavcodec/h264_ps.c +++ b/libavcodec/h264_ps.c @@ -198,12 +198,14 @@ static inline int decode_vui_parameters(GetBitContext *gb, void *logctx, } static int decode_scaling_list(GetBitContext *gb, uint8_t *factors, int size, - const uint8_t *jvt_list, - const uint8_t *fallback_list) + const uint8_t *jvt_list, const uint8_t *fallback_list, + uint16_t *mask, int pos) { int i, last = 8, next = 8; const uint8_t *scan = size == 16 ? ff_zigzag_scan : ff_zigzag_direct; - if (!get_bits1(gb)) /* matrix not written, we use the predicted one */ + uint16_t seq_scaling_list_present_flag = get_bits1(gb); + *mask |= (seq_scaling_list_present_flag << pos); + if (!seq_scaling_list_present_flag) /* matrix not written, we use the predicted one */ memcpy(factors, fallback_list, size * sizeof(uint8_t)); else for (i = 0; i < size; i++) { @@ -227,7 +229,7 @@ static int decode_scaling_list(GetBitContext *gb, uint8_t *factors, int size, /* returns non zero if the provided SPS scaling matrix has been filled */ static int decode_scaling_matrices(GetBitContext *gb, const SPS *sps, const PPS *pps, int is_sps, - int present_flag, + int present_flag, uint16_t *mask, uint8_t(*scaling_matrix4)[16], uint8_t(*scaling_matrix8)[64]) { @@ -239,21 +241,22 @@ static int decode_scaling_matrices(GetBitContext *gb, const SPS *sps, fallback_sps ? sps->scaling_matrix8[3] : default_scaling8[1] }; int ret = 0; + *mask = 0x0; if (present_flag) { - ret |= decode_scaling_list(gb, scaling_matrix4[0], 16, default_scaling4[0], fallback[0]); // Intra, Y - ret |= decode_scaling_list(gb, scaling_matrix4[1], 16, default_scaling4[0], scaling_matrix4[0]); // Intra, Cr - ret |= decode_scaling_list(gb, scaling_matrix4[2], 16, default_scaling4[0], scaling_matrix4[1]); // Intra, Cb - ret |= decode_scaling_list(gb, scaling_matrix4[3], 16, default_scaling4[1], fallback[1]); // Inter, Y - ret |= decode_scaling_list(gb, scaling_matrix4[4], 16, default_scaling4[1], scaling_matrix4[3]); // Inter, Cr - ret |= decode_scaling_list(gb, scaling_matrix4[5], 16, default_scaling4[1], scaling_matrix4[4]); // Inter, Cb + ret |= decode_scaling_list(gb, scaling_matrix4[0], 16, default_scaling4[0], fallback[0], mask, 0); // Intra, Y + ret |= decode_scaling_list(gb, scaling_matrix4[1], 16, default_scaling4[0], scaling_matrix4[0], mask, 1); // Intra, Cr + ret |= decode_scaling_list(gb, scaling_matrix4[2], 16, default_scaling4[0], scaling_matrix4[1], mask, 2); // Intra, Cb + ret |= decode_scaling_list(gb, scaling_matrix4[3], 16, default_scaling4[1], fallback[1], mask, 3); // Inter, Y + ret |= decode_scaling_list(gb, scaling_matrix4[4], 16, default_scaling4[1], scaling_matrix4[3], mask, 4); // Inter, Cr + ret |= decode_scaling_list(gb, scaling_matrix4[5], 16, default_scaling4[1], scaling_matrix4[4], mask, 5); // Inter, Cb if (is_sps || pps->transform_8x8_mode) { - ret |= decode_scaling_list(gb, scaling_matrix8[0], 64, default_scaling8[0], fallback[2]); // Intra, Y - ret |= decode_scaling_list(gb, scaling_matrix8[3], 64, default_scaling8[1], fallback[3]); // Inter, Y + ret |= decode_scaling_list(gb, scaling_matrix8[0], 64, default_scaling8[0], fallback[2], mask, 6); // Intra, Y + ret |= decode_scaling_list(gb, scaling_matrix8[3], 64, default_scaling8[1], fallback[3], mask, 7); // Inter, Y if (sps->chroma_format_idc == 3) { - ret |= decode_scaling_list(gb, scaling_matrix8[1], 64, default_scaling8[0], scaling_matrix8[0]); // Intra, Cr - ret |= decode_scaling_list(gb, scaling_matrix8[4], 64, default_scaling8[1], scaling_matrix8[3]); // Inter, Cr - ret |= decode_scaling_list(gb, scaling_matrix8[2], 64, default_scaling8[0], scaling_matrix8[1]); // Intra, Cb - ret |= decode_scaling_list(gb, scaling_matrix8[5], 64, default_scaling8[1], scaling_matrix8[4]); // Inter, Cb + ret |= decode_scaling_list(gb, scaling_matrix8[1], 64, default_scaling8[0], scaling_matrix8[0], mask, 8); // Intra, Cr + ret |= decode_scaling_list(gb, scaling_matrix8[4], 64, default_scaling8[1], scaling_matrix8[3], mask, 9); // Inter, Cr + ret |= decode_scaling_list(gb, scaling_matrix8[2], 64, default_scaling8[0], scaling_matrix8[1], mask, 10); // Intra, Cb + ret |= decode_scaling_list(gb, scaling_matrix8[5], 64, default_scaling8[1], scaling_matrix8[4], mask, 11); // Inter, Cb } } if (!ret) @@ -371,6 +374,7 @@ int ff_h264_decode_seq_parameter_set(GetBitContext *gb, AVCodecContext *avctx, } sps->transform_bypass = get_bits1(gb); ret = decode_scaling_matrices(gb, sps, NULL, 1, get_bits1(gb), + &sps->scaling_matrix_present_mask, sps->scaling_matrix4, sps->scaling_matrix8); if (ret < 0) goto fail; @@ -808,6 +812,7 @@ int ff_h264_decode_picture_parameter_set(GetBitContext *gb, AVCodecContext *avct pps->pic_scaling_matrix_present_flag = get_bits1(gb); ret = decode_scaling_matrices(gb, sps, pps, 0, pps->pic_scaling_matrix_present_flag, + &pps->pic_scaling_matrix_present_mask, pps->scaling_matrix4, pps->scaling_matrix8); if (ret < 0) goto fail; diff --git a/libavcodec/h264_ps.h b/libavcodec/h264_ps.h index 8adfbd710f..e675619635 100644 --- a/libavcodec/h264_ps.h +++ b/libavcodec/h264_ps.h @@ -82,6 +82,7 @@ typedef struct SPS { int num_reorder_frames; int max_dec_frame_buffering; int scaling_matrix_present; + uint16_t scaling_matrix_present_mask; uint8_t scaling_matrix4[6][16]; uint8_t scaling_matrix8[6][64]; int nal_hrd_parameters_present_flag; @@ -125,6 +126,7 @@ typedef struct PPS { int redundant_pic_cnt_present; ///< redundant_pic_cnt_present_flag int transform_8x8_mode; ///< transform_8x8_mode_flag int pic_scaling_matrix_present_flag; + uint16_t pic_scaling_matrix_present_mask; uint8_t scaling_matrix4[6][16]; uint8_t scaling_matrix8[6][64]; uint8_t chroma_qp_table[2][QP_MAX_NUM+1]; ///< pre-scaled (with chroma_qp_index_offset) version of qp_table -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0010-hevc_ps-expose-SPS-and-VPS-headers.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0010-hevc_ps-expose-SPS-and-VPS-headers.patch >From 095753e78bae900460bc39fac55749b9d1bf7baa Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 7 Dec 2022 01:29:57 +0100 Subject: [PATCH 10/92] hevc_ps: expose SPS and VPS headers --- libavcodec/hevc_ps.c | 100 ++++++++++++++++++++++--------------------- libavcodec/hevc_ps.h | 41 ++++++++++++++++++ 2 files changed, 93 insertions(+), 48 deletions(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index 348e4d8de2..d1574fdfcf 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -357,81 +357,84 @@ static int parse_ptl(GetBitContext *gb, AVCodecContext *avctx, } static void decode_sublayer_hrd(GetBitContext *gb, unsigned int nb_cpb, - int subpic_params_present) + HEVCSublayerHdrParams *par, int subpic_params_present) { int i; for (i = 0; i < nb_cpb; i++) { - get_ue_golomb_long(gb); // bit_rate_value_minus1 - get_ue_golomb_long(gb); // cpb_size_value_minus1 + par->bit_rate_value_minus1[i] = get_ue_golomb_long(gb); + par->cpb_size_value_minus1[i] = get_ue_golomb_long(gb); if (subpic_params_present) { - get_ue_golomb_long(gb); // cpb_size_du_value_minus1 - get_ue_golomb_long(gb); // bit_rate_du_value_minus1 + par->cpb_size_du_value_minus1[i] = get_ue_golomb_long(gb); + par->bit_rate_du_value_minus1[i] = get_ue_golomb_long(gb); } - skip_bits1(gb); // cbr_flag + + par->cbr_flag = get_bits1(gb); } } static int decode_hrd(GetBitContext *gb, int common_inf_present, - int max_sublayers) + HEVCHdrParams *hdr, int max_sublayers) { - int nal_params_present = 0, vcl_params_present = 0; - int subpic_params_present = 0; - int i; - if (common_inf_present) { - nal_params_present = get_bits1(gb); - vcl_params_present = get_bits1(gb); - - if (nal_params_present || vcl_params_present) { - subpic_params_present = get_bits1(gb); - - if (subpic_params_present) { - skip_bits(gb, 8); // tick_divisor_minus2 - skip_bits(gb, 5); // du_cpb_removal_delay_increment_length_minus1 - skip_bits(gb, 1); // sub_pic_cpb_params_in_pic_timing_sei_flag - skip_bits(gb, 5); // dpb_output_delay_du_length_minus1 + hdr->flags.nal_hrd_parameters_present_flag = get_bits1(gb); + hdr->flags.vcl_hrd_parameters_present_flag = get_bits1(gb); + + if (hdr->flags.nal_hrd_parameters_present_flag || + hdr->flags.vcl_hrd_parameters_present_flag) { + hdr->flags.sub_pic_hrd_params_present_flag = get_bits1(gb); + + if (hdr->flags.sub_pic_hrd_params_present_flag) { + hdr->tick_divisor_minus2 = get_bits(gb, 8); + hdr->du_cpb_removal_delay_increment_length_minus1 = get_bits(gb, 5); + hdr->flags.sub_pic_cpb_params_in_pic_timing_sei_flag = get_bits1(gb); + hdr->dpb_output_delay_du_length_minus1 = get_bits(gb, 5); } - skip_bits(gb, 4); // bit_rate_scale - skip_bits(gb, 4); // cpb_size_scale + hdr->bit_rate_scale = get_bits(gb, 4); + hdr->cpb_size_scale = get_bits(gb, 4); - if (subpic_params_present) - skip_bits(gb, 4); // cpb_size_du_scale + if (hdr->flags.sub_pic_hrd_params_present_flag) + hdr->cpb_size_du_scale = get_bits(gb, 4); - skip_bits(gb, 5); // initial_cpb_removal_delay_length_minus1 - skip_bits(gb, 5); // au_cpb_removal_delay_length_minus1 - skip_bits(gb, 5); // dpb_output_delay_length_minus1 + hdr->initial_cpb_removal_delay_length_minus1 = get_bits(gb, 5); + hdr->au_cpb_removal_delay_length_minus1 = get_bits(gb, 5); + hdr->dpb_output_delay_length_minus1 = get_bits(gb, 5); } } - for (i = 0; i < max_sublayers; i++) { - int low_delay = 0; - unsigned int nb_cpb = 1; - int fixed_rate = get_bits1(gb); + for (int i = 0; i < max_sublayers; i++) { + hdr->flags.fixed_pic_rate_general_flag = get_bits1(gb); + + hdr->cpb_cnt_minus1[i] = 1; - if (!fixed_rate) - fixed_rate = get_bits1(gb); + if (!hdr->flags.fixed_pic_rate_general_flag) + hdr->flags.fixed_pic_rate_within_cvs_flag = get_bits1(gb); - if (fixed_rate) - get_ue_golomb_long(gb); // elemental_duration_in_tc_minus1 + if (hdr->flags.fixed_pic_rate_within_cvs_flag) + hdr->elemental_duration_in_tc_minus1[i] = get_ue_golomb_long(gb); else - low_delay = get_bits1(gb); + hdr->flags.low_delay_hrd_flag = get_bits1(gb); - if (!low_delay) { - nb_cpb = get_ue_golomb_long(gb) + 1; - if (nb_cpb < 1 || nb_cpb > 32) { - av_log(NULL, AV_LOG_ERROR, "nb_cpb %d invalid\n", nb_cpb); + if (!hdr->flags.low_delay_hrd_flag) { + hdr->cpb_cnt_minus1[i] = get_ue_golomb_long(gb); + if (hdr->cpb_cnt_minus1[i] > 31) { + av_log(NULL, AV_LOG_ERROR, "nb_cpb %d invalid\n", + hdr->cpb_cnt_minus1[i]); return AVERROR_INVALIDDATA; } } - if (nal_params_present) - decode_sublayer_hrd(gb, nb_cpb, subpic_params_present); - if (vcl_params_present) - decode_sublayer_hrd(gb, nb_cpb, subpic_params_present); + if (hdr->flags.nal_hrd_parameters_present_flag) + decode_sublayer_hrd(gb, hdr->cpb_cnt_minus1[i], &hdr->nal_params[i], + hdr->flags.sub_pic_hrd_params_present_flag); + + if (hdr->flags.vcl_hrd_parameters_present_flag) + decode_sublayer_hrd(gb, hdr->cpb_cnt_minus1[i], &hdr->vcl_params[i], + hdr->flags.sub_pic_hrd_params_present_flag); } + return 0; } @@ -538,7 +541,8 @@ int ff_hevc_decode_nal_vps(GetBitContext *gb, AVCodecContext *avctx, get_ue_golomb_long(gb); // hrd_layer_set_idx if (i) common_inf_present = get_bits1(gb); - decode_hrd(gb, common_inf_present, vps->vps_max_sub_layers); + decode_hrd(gb, common_inf_present, &vps->hdr[i], + vps->vps_max_sub_layers); } } get_bits1(gb); /* vps_extension_flag */ @@ -657,7 +661,7 @@ timing_info: vui->vui_num_ticks_poc_diff_one_minus1 = get_ue_golomb_long(gb); vui->vui_hrd_parameters_present_flag = get_bits1(gb); if (vui->vui_hrd_parameters_present_flag) - decode_hrd(gb, 1, sps->max_sub_layers); + decode_hrd(gb, 1, &sps->hdr, sps->max_sub_layers); } vui->bitstream_restriction_flag = get_bits1(gb); diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index 1f704108e3..0024a8ed2e 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -32,6 +32,43 @@ #include "h2645_vui.h" #include "hevc.h" +typedef struct HEVCSublayerHdrParams { + uint32_t bit_rate_value_minus1[HEVC_MAX_CPB_CNT]; + uint32_t cpb_size_value_minus1[HEVC_MAX_CPB_CNT]; + uint32_t cpb_size_du_value_minus1[HEVC_MAX_CPB_CNT]; + uint32_t bit_rate_du_value_minus1[HEVC_MAX_CPB_CNT]; + uint32_t cbr_flag; +} HEVCSublayerHdrParams; + +typedef struct HEVCHdrFlagParams { + uint32_t nal_hrd_parameters_present_flag; + uint32_t vcl_hrd_parameters_present_flag; + uint32_t sub_pic_hrd_params_present_flag; + uint32_t sub_pic_cpb_params_in_pic_timing_sei_flag; + uint32_t fixed_pic_rate_general_flag; + uint32_t fixed_pic_rate_within_cvs_flag; + uint32_t low_delay_hrd_flag; +} HEVCHdrFlagParams; + +typedef struct HEVCHdrParams { + HEVCHdrFlagParams flags; + + uint8_t tick_divisor_minus2; + uint8_t du_cpb_removal_delay_increment_length_minus1; + uint8_t dpb_output_delay_du_length_minus1; + uint8_t bit_rate_scale; + uint8_t cpb_size_scale; + uint8_t cpb_size_du_scale; + uint8_t initial_cpb_removal_delay_length_minus1; + uint8_t au_cpb_removal_delay_length_minus1; + uint8_t dpb_output_delay_length_minus1; + uint8_t cpb_cnt_minus1[HEVC_MAX_SUB_LAYERS]; + uint16_t elemental_duration_in_tc_minus1[HEVC_MAX_SUB_LAYERS]; + + HEVCSublayerHdrParams nal_params[HEVC_MAX_SUB_LAYERS]; + HEVCSublayerHdrParams vcl_params[HEVC_MAX_SUB_LAYERS]; +} HEVCHdrParams; + typedef struct ShortTermRPS { unsigned int num_negative_pics; int num_delta_pocs; @@ -108,6 +145,8 @@ typedef struct PTL { } PTL; typedef struct HEVCVPS { + HEVCHdrParams hdr[HEVC_MAX_LAYER_SETS]; + uint8_t vps_temporal_id_nesting_flag; int vps_max_layers; int vps_max_sub_layers; ///< vps_max_temporal_layers_minus1 + 1 @@ -146,6 +185,8 @@ typedef struct HEVCSPS { HEVCWindow pic_conf_win; + HEVCHdrParams hdr; + int bit_depth; int bit_depth_chroma; int pixel_shift; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0011-hevc_ps-expose-pps_id.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0011-hevc_ps-expose-pps_id.patch >From a2a13fb436ac2f8f7135bd14f5f0c8937bf9edd7 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 7 Dec 2022 05:33:29 +0100 Subject: [PATCH 11/92] hevc_ps: expose pps_id --- libavcodec/hevc_ps.c | 2 +- libavcodec/hevc_ps.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index d1574fdfcf..9627b11d5a 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -1735,7 +1735,7 @@ int ff_hevc_decode_nal_pps(GetBitContext *gb, AVCodecContext *avctx, pps->log2_max_transform_skip_block_size = 2; // Coded parameters - pps_id = get_ue_golomb_long(gb); + pps_id = pps->pps_id = get_ue_golomb_long(gb); if (pps_id >= HEVC_MAX_PPS_COUNT) { av_log(avctx, AV_LOG_ERROR, "PPS id out of range: %d\n", pps_id); ret = AVERROR_INVALIDDATA; diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index 0024a8ed2e..2924c19038 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -289,6 +289,7 @@ typedef struct HEVCSPS { } HEVCSPS; typedef struct HEVCPPS { + unsigned int pps_id; unsigned int sps_id; ///< seq_parameter_set_id uint8_t sign_data_hiding_flag; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0012-hevc_ps-expose-vps_id.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0012-hevc_ps-expose-vps_id.patch >From c62aaeb2f736a087f8f7fa28ee1d9a499e250afe Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 7 Dec 2022 06:42:44 +0100 Subject: [PATCH 12/92] hevc_ps: expose vps_id --- libavcodec/hevc_ps.c | 2 +- libavcodec/hevc_ps.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index 9627b11d5a..333ea41ff1 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -464,7 +464,7 @@ int ff_hevc_decode_nal_vps(GetBitContext *gb, AVCodecContext *avctx, } memcpy(vps->data, gb->buffer, vps->data_size); - vps_id = get_bits(gb, 4); + vps_id = vps->vps_id = get_bits(gb, 4); if (get_bits(gb, 2) != 3) { // vps_reserved_three_2bits av_log(avctx, AV_LOG_ERROR, "vps_reserved_three_2bits is not three\n"); diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index 2924c19038..a37e2d294e 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -145,6 +145,7 @@ typedef struct PTL { } PTL; typedef struct HEVCVPS { + unsigned int vps_id; HEVCHdrParams hdr[HEVC_MAX_LAYER_SETS]; uint8_t vps_temporal_id_nesting_flag; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0013-hevc_ps-expose-pps_extension_present_flag.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0013-hevc_ps-expose-pps_extension_present_flag.patch >From db87bcba0f4bbac4afc45eb464d702675a3e42bc Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 7 Dec 2022 12:49:45 +0100 Subject: [PATCH 13/92] hevc_ps: expose pps_extension_present_flag --- libavcodec/hevc_ps.c | 3 ++- libavcodec/hevc_ps.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index 333ea41ff1..084f6f7da5 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -1908,7 +1908,8 @@ int ff_hevc_decode_nal_pps(GetBitContext *gb, AVCodecContext *avctx, pps->slice_header_extension_present_flag = get_bits1(gb); - if (get_bits1(gb)) { // pps_extension_present_flag + pps->pps_extension_present_flag = get_bits1(gb); + if (pps->pps_extension_present_flag) { pps->pps_range_extensions_flag = get_bits1(gb); pps->pps_multilayer_extension_flag = get_bits1(gb); pps->pps_3d_extension_flag = get_bits1(gb); diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index a37e2d294e..4648c20724 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -340,6 +340,7 @@ typedef struct HEVCPPS { int num_extra_slice_header_bits; uint8_t slice_header_extension_present_flag; uint8_t log2_max_transform_skip_block_size; + uint8_t pps_extension_present_flag; uint8_t pps_range_extensions_flag; uint8_t pps_multilayer_extension_flag; uint8_t pps_3d_extension_flag; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0014-hevcdec-expose-bits_used_for_short_term_rps.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0014-hevcdec-expose-bits_used_for_short_term_rps.patch >From 571cd8e9e72936f7c6e8ddad8177c375123258b8 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 7 Dec 2022 17:11:36 +0100 Subject: [PATCH 14/92] hevcdec: expose bits_used_for_short_term_rps --- libavcodec/hevcdec.c | 1 + libavcodec/hevcdec.h | 1 + 2 files changed, 2 insertions(+) diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 121ceb4e75..0230a41311 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -703,6 +703,7 @@ static int hls_slice_header(HEVCContext *s) if (ret < 0) return ret; + sh->bits_used_for_short_term_rps = pos - get_bits_left(gb); sh->short_term_rps = &sh->slice_rps; } else { int numbits, rps_idx; diff --git a/libavcodec/hevcdec.h b/libavcodec/hevcdec.h index aab816791e..ddb6e2b5d9 100644 --- a/libavcodec/hevcdec.h +++ b/libavcodec/hevcdec.h @@ -268,6 +268,7 @@ typedef struct SliceHeader { ///< RPS coded in the slice header itself is stored here int short_term_ref_pic_set_sps_flag; + int bits_used_for_short_term_rps; int short_term_ref_pic_set_size; ShortTermRPS slice_rps; const ShortTermRPS *short_term_rps; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0015-hevc_ps-expose-log2_diff_max_min_transform_block_siz.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0015-hevc_ps-expose-log2_diff_max_min_transform_block_siz.patch >From a4be62cdebb44367c024439ac264d790de7a0463 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 7 Dec 2022 04:30:46 +0100 Subject: [PATCH 15/92] hevc_ps: expose log2_diff_max_min_transform_block_size --- libavcodec/hevc_ps.c | 20 ++++++++++---------- libavcodec/hevc_ps.h | 1 + 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index 084f6f7da5..3d982bcbd4 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -856,9 +856,8 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, { HEVCWindow *ow; int ret = 0; - int log2_diff_max_min_transform_block_size; int bit_depth_chroma, start, vui_present, sublayer_ordering_info, num_comps; - int i; + int i, j; // Coded parameters @@ -983,12 +982,12 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, } } - sps->log2_min_cb_size = get_ue_golomb_long(gb) + 3; - sps->log2_diff_max_min_coding_block_size = get_ue_golomb_long(gb); - sps->log2_min_tb_size = get_ue_golomb_long(gb) + 2; - log2_diff_max_min_transform_block_size = get_ue_golomb_long(gb); - sps->log2_max_trafo_size = log2_diff_max_min_transform_block_size + - sps->log2_min_tb_size; + sps->log2_min_cb_size = get_ue_golomb_long(gb) + 3; + sps->log2_diff_max_min_coding_block_size = get_ue_golomb_long(gb); + sps->log2_min_tb_size = get_ue_golomb_long(gb) + 2; + sps->log2_diff_max_min_transform_block_size = get_ue_golomb_long(gb); + sps->log2_max_trafo_size = sps->log2_diff_max_min_transform_block_size + + sps->log2_min_tb_size; if (sps->log2_min_cb_size < 3 || sps->log2_min_cb_size > 30) { av_log(avctx, AV_LOG_ERROR, "Invalid value %d for log2_min_cb_size", sps->log2_min_cb_size); @@ -1005,8 +1004,9 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, return AVERROR_INVALIDDATA; } - if (log2_diff_max_min_transform_block_size < 0 || log2_diff_max_min_transform_block_size > 30) { - av_log(avctx, AV_LOG_ERROR, "Invalid value %d for log2_diff_max_min_transform_block_size", log2_diff_max_min_transform_block_size); + if (sps->log2_diff_max_min_transform_block_size > 30) { + av_log(avctx, AV_LOG_ERROR, "Invalid value %d for log2_diff_max_min_transform_block_size", + sps->log2_diff_max_min_transform_block_size); return AVERROR_INVALIDDATA; } diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index 4648c20724..cf28571745 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -237,6 +237,7 @@ typedef struct HEVCSPS { unsigned int log2_max_trafo_size; unsigned int log2_ctb_size; unsigned int log2_min_pu_size; + unsigned int log2_diff_max_min_transform_block_size; int max_transform_hierarchy_depth_inter; int max_transform_hierarchy_depth_intra; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0016-hevc_ps-expose-rps-fields.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0016-hevc_ps-expose-rps-fields.patch >From a360a6db336eeb7330c0f5f579b5ead5441dc836 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 14 Dec 2022 00:25:48 +0100 Subject: [PATCH 16/92] hevc_ps: expose rps fields --- libavcodec/hevc_ps.c | 37 ++++++++++++++++++------------------- libavcodec/hevc_ps.h | 7 +++++++ 2 files changed, 25 insertions(+), 19 deletions(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index 3d982bcbd4..d60d50ca48 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -100,51 +100,50 @@ static void remove_vps(HEVCParamSets *s, int id) int ff_hevc_decode_short_term_rps(GetBitContext *gb, AVCodecContext *avctx, ShortTermRPS *rps, const HEVCSPS *sps, int is_slice_header) { - uint8_t rps_predict = 0; int delta_poc; int k0 = 0; int k = 0; int i; + rps->rps_predict = 0; + if (rps != sps->st_rps && sps->nb_st_rps) - rps_predict = get_bits1(gb); + rps->rps_predict = get_bits1(gb); - if (rps_predict) { + if (rps->rps_predict) { const ShortTermRPS *rps_ridx; int delta_rps; - unsigned abs_delta_rps; - uint8_t use_delta_flag = 0; - uint8_t delta_rps_sign; if (is_slice_header) { - unsigned int delta_idx = get_ue_golomb_long(gb) + 1; - if (delta_idx > sps->nb_st_rps) { + rps->delta_idx = get_ue_golomb_long(gb) + 1; + if (rps->delta_idx > sps->nb_st_rps) { av_log(avctx, AV_LOG_ERROR, "Invalid value of delta_idx in slice header RPS: %d > %d.\n", - delta_idx, sps->nb_st_rps); + rps->delta_idx, sps->nb_st_rps); return AVERROR_INVALIDDATA; } - rps_ridx = &sps->st_rps[sps->nb_st_rps - delta_idx]; + rps_ridx = &sps->st_rps[sps->nb_st_rps - rps->delta_idx]; rps->rps_idx_num_delta_pocs = rps_ridx->num_delta_pocs; } else rps_ridx = &sps->st_rps[rps - sps->st_rps - 1]; - delta_rps_sign = get_bits1(gb); - abs_delta_rps = get_ue_golomb_long(gb) + 1; - if (abs_delta_rps < 1 || abs_delta_rps > 32768) { + rps->delta_rps_sign = get_bits1(gb); + rps->abs_delta_rps = get_ue_golomb_long(gb) + 1; + if (rps->abs_delta_rps > 32768) { av_log(avctx, AV_LOG_ERROR, "Invalid value of abs_delta_rps: %d\n", - abs_delta_rps); + rps->abs_delta_rps); return AVERROR_INVALIDDATA; } - delta_rps = (1 - (delta_rps_sign << 1)) * abs_delta_rps; + delta_rps = (1 - (rps->delta_rps_sign << 1)) * rps->abs_delta_rps; for (i = 0; i <= rps_ridx->num_delta_pocs; i++) { int used = rps->used[k] = get_bits1(gb); + rps->use_delta_flag = 0; if (!used) - use_delta_flag = get_bits1(gb); + rps->use_delta_flag = get_bits1(gb); - if (used || use_delta_flag) { + if (used || rps->use_delta_flag) { if (i < rps_ridx->num_delta_pocs) delta_poc = delta_rps + rps_ridx->delta_poc[i]; else @@ -210,7 +209,7 @@ int ff_hevc_decode_short_term_rps(GetBitContext *gb, AVCodecContext *avctx, if (rps->num_delta_pocs) { prev = 0; for (i = 0; i < rps->num_negative_pics; i++) { - delta_poc = get_ue_golomb_long(gb) + 1; + delta_poc = rps->delta_poc_s0[i] = get_ue_golomb_long(gb) + 1; if (delta_poc < 1 || delta_poc > 32768) { av_log(avctx, AV_LOG_ERROR, "Invalid value of delta_poc: %d\n", @@ -223,7 +222,7 @@ int ff_hevc_decode_short_term_rps(GetBitContext *gb, AVCodecContext *avctx, } prev = 0; for (i = 0; i < nb_positive_pics; i++) { - delta_poc = get_ue_golomb_long(gb) + 1; + delta_poc = rps->delta_poc_s1[i] = get_ue_golomb_long(gb) + 1; if (delta_poc < 1 || delta_poc > 32768) { av_log(avctx, AV_LOG_ERROR, "Invalid value of delta_poc: %d\n", diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index cf28571745..2f94c03a4a 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -70,9 +70,16 @@ typedef struct HEVCHdrParams { } HEVCHdrParams; typedef struct ShortTermRPS { + uint8_t rps_predict; + unsigned int delta_idx; + uint8_t use_delta_flag; + uint8_t delta_rps_sign; + unsigned int abs_delta_rps; unsigned int num_negative_pics; int num_delta_pocs; int rps_idx_num_delta_pocs; + int32_t delta_poc_s0[32]; + int32_t delta_poc_s1[32]; int32_t delta_poc[32]; uint8_t used[32]; } ShortTermRPS; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0017-hevc_ps-expose-vui_present-flag.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0017-hevc_ps-expose-vui_present-flag.patch >From c190825b1c02ecfd4ff54fcdfcb222d5bcbc22f0 Mon Sep 17 00:00:00 2001 From: Lynne Date: Tue, 28 Feb 2023 20:44:06 +0100 Subject: [PATCH 17/92] hevc_ps: expose vui_present flag --- libavcodec/hevc_ps.c | 6 +++--- libavcodec/hevc_ps.h | 1 + 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index d60d50ca48..cd1bec63b2 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -855,7 +855,7 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, { HEVCWindow *ow; int ret = 0; - int bit_depth_chroma, start, vui_present, sublayer_ordering_info, num_comps; + int bit_depth_chroma, start, sublayer_ordering_info, num_comps; int i, j; // Coded parameters @@ -1072,8 +1072,8 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, sps->sps_temporal_mvp_enabled_flag = get_bits1(gb); sps->sps_strong_intra_smoothing_enable_flag = get_bits1(gb); sps->vui.common.sar = (AVRational){0, 1}; - vui_present = get_bits1(gb); - if (vui_present) + sps->vui_present = get_bits1(gb); + if (sps->vui_present) decode_vui(gb, avctx, apply_defdispwin, sps); if (get_bits1(gb)) { // sps_extension_flag diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index 2f94c03a4a..dbc42397ac 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -211,6 +211,7 @@ typedef struct HEVCSPS { } temporal_layer[HEVC_MAX_SUB_LAYERS]; uint8_t temporal_id_nesting_flag; + int vui_present; VUI vui; PTL ptl; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0018-hevc_ps-expose-sublayer_ordering_info_flag.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0018-hevc_ps-expose-sublayer_ordering_info_flag.patch >From f860295d5b48dc214eaab6f9f407bc6af1326f8f Mon Sep 17 00:00:00 2001 From: Lynne Date: Tue, 28 Feb 2023 20:44:38 +0100 Subject: [PATCH 18/92] hevc_ps: expose sublayer_ordering_info_flag --- libavcodec/hevc_ps.c | 8 ++++---- libavcodec/hevc_ps.h | 1 + 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index cd1bec63b2..268a963e05 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -855,7 +855,7 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, { HEVCWindow *ow; int ret = 0; - int bit_depth_chroma, start, sublayer_ordering_info, num_comps; + int bit_depth_chroma, start, num_comps; int i, j; // Coded parameters @@ -951,8 +951,8 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, return AVERROR_INVALIDDATA; } - sublayer_ordering_info = get_bits1(gb); - start = sublayer_ordering_info ? 0 : sps->max_sub_layers - 1; + sps->sublayer_ordering_info_flag = get_bits1(gb); + start = sps->sublayer_ordering_info_flag ? 0 : sps->max_sub_layers - 1; for (i = start; i < sps->max_sub_layers; i++) { sps->temporal_layer[i].max_dec_pic_buffering = get_ue_golomb_long(gb) + 1; sps->temporal_layer[i].num_reorder_pics = get_ue_golomb_long(gb); @@ -973,7 +973,7 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, } } - if (!sublayer_ordering_info) { + if (!sps->sublayer_ordering_info_flag) { for (i = 0; i < start; i++) { sps->temporal_layer[i].max_dec_pic_buffering = sps->temporal_layer[start].max_dec_pic_buffering; sps->temporal_layer[i].num_reorder_pics = sps->temporal_layer[start].num_reorder_pics; diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index dbc42397ac..c8cf12e82d 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -203,6 +203,7 @@ typedef struct HEVCSPS { unsigned int log2_max_poc_lsb; int pcm_enabled_flag; + uint8_t sublayer_ordering_info_flag; int max_sub_layers; struct { int max_dec_pic_buffering; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0019-hevc_ps-expose-conformance_window_flag.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0019-hevc_ps-expose-conformance_window_flag.patch >From b04efa5eea1d5ec81c85500cf6af3c798b6a97a6 Mon Sep 17 00:00:00 2001 From: Lynne Date: Tue, 28 Feb 2023 20:45:05 +0100 Subject: [PATCH 19/92] hevc_ps: expose conformance_window_flag --- libavcodec/hevc_ps.c | 3 ++- libavcodec/hevc_ps.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index 268a963e05..fffff170ca 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -904,7 +904,8 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, sps->height, 0, avctx)) < 0) return ret; - if (get_bits1(gb)) { // pic_conformance_flag + sps->conformance_window_flag = get_bits1(gb); + if (sps->conformance_window_flag) { int vert_mult = hevc_sub_height_c[sps->chroma_format_idc]; int horiz_mult = hevc_sub_width_c[sps->chroma_format_idc]; sps->pic_conf_win.left_offset = get_ue_golomb_long(gb) * horiz_mult; diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index c8cf12e82d..2708c1f848 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -191,6 +191,7 @@ typedef struct HEVCSPS { HEVCWindow output_window; + uint8_t conformance_window_flag; HEVCWindow pic_conf_win; HEVCHdrParams hdr; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0020-hevc_ps-expose-sps_extension_present_flag.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0020-hevc_ps-expose-sps_extension_present_flag.patch >From 288b10dec3d20dbacbd2409a75bfa3f64e72e31b Mon Sep 17 00:00:00 2001 From: Lynne Date: Tue, 28 Feb 2023 20:45:24 +0100 Subject: [PATCH 20/92] hevc_ps: expose sps_extension_present_flag --- libavcodec/hevc_ps.c | 3 ++- libavcodec/hevc_ps.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/libavcodec/hevc_ps.c b/libavcodec/hevc_ps.c index fffff170ca..9cc616e452 100644 --- a/libavcodec/hevc_ps.c +++ b/libavcodec/hevc_ps.c @@ -1077,7 +1077,8 @@ int ff_hevc_parse_sps(HEVCSPS *sps, GetBitContext *gb, unsigned int *sps_id, if (sps->vui_present) decode_vui(gb, avctx, apply_defdispwin, sps); - if (get_bits1(gb)) { // sps_extension_flag + sps->sps_extension_present_flag = get_bits1(gb); + if (sps->sps_extension_present_flag) { sps->sps_range_extension_flag = get_bits1(gb); sps->sps_multilayer_extension_flag = get_bits1(gb); sps->sps_3d_extension_flag = get_bits1(gb); diff --git a/libavcodec/hevc_ps.h b/libavcodec/hevc_ps.h index 2708c1f848..c2f5e3ec96 100644 --- a/libavcodec/hevc_ps.h +++ b/libavcodec/hevc_ps.h @@ -217,6 +217,7 @@ typedef struct HEVCSPS { VUI vui; PTL ptl; + uint8_t sps_extension_present_flag; uint8_t scaling_list_enable_flag; ScalingList scaling_list; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0021-av1dec-expose-ref_order_hint.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0021-av1dec-expose-ref_order_hint.patch >From 3ff54622533627493643c8e4a5e982fc99a7df47 Mon Sep 17 00:00:00 2001 From: Lynne Date: Wed, 22 Feb 2023 22:30:36 +0100 Subject: [PATCH 21/92] av1dec: expose ref_order_hint Commit by Dave Airlie --- libavcodec/av1dec.c | 9 +++++++++ libavcodec/av1dec.h | 1 + 2 files changed, 10 insertions(+) diff --git a/libavcodec/av1dec.c b/libavcodec/av1dec.c index a80e37e33f..5a3c51e94a 100644 --- a/libavcodec/av1dec.c +++ b/libavcodec/av1dec.c @@ -591,6 +591,8 @@ static void av1_frame_unref(AVCodecContext *avctx, AV1Frame *f) f->spatial_id = f->temporal_id = 0; memset(f->skip_mode_frame_idx, 0, 2 * sizeof(uint8_t)); + memset(f->ref_order_hint, 0, + 7 * sizeof(uint8_t)); memset(&f->film_grain, 0, sizeof(f->film_grain)); f->coded_lossless = 0; } @@ -633,6 +635,9 @@ static int av1_frame_ref(AVCodecContext *avctx, AV1Frame *dst, const AV1Frame *s memcpy(dst->skip_mode_frame_idx, src->skip_mode_frame_idx, 2 * sizeof(uint8_t)); + memcpy(dst->ref_order_hint, + src->ref_order_hint, + 7 * sizeof(uint8_t)); memcpy(&dst->film_grain, &src->film_grain, sizeof(dst->film_grain)); @@ -1267,6 +1272,10 @@ static int av1_decode_frame(AVCodecContext *avctx, AVFrame *frame, s->cur_frame.spatial_id = header->spatial_id; s->cur_frame.temporal_id = header->temporal_id; + for (int i = 0; i < 7; i++) + s->cur_frame.ref_order_hint[i] = + s->raw_frame_header->ref_order_hint[s->raw_frame_header->ref_frame_idx[i]]; + if (avctx->hwaccel && s->cur_frame.f->buf[0]) { ret = avctx->hwaccel->start_frame(avctx, unit->data, unit->data_size); diff --git a/libavcodec/av1dec.h b/libavcodec/av1dec.h index cef899f81f..2be28e7469 100644 --- a/libavcodec/av1dec.h +++ b/libavcodec/av1dec.h @@ -48,6 +48,7 @@ typedef struct AV1Frame { int32_t gm_params[AV1_NUM_REF_FRAMES][6]; uint8_t skip_mode_frame_idx[2]; + uint8_t ref_order_hint[7]; AV1RawFilmGrainParams film_grain; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0022-av1-set-skip-mode-frames-properly.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0022-av1-set-skip-mode-frames-properly.patch >From 20e5ce81eb2c4248cde356a425851f9b8b91ce29 Mon Sep 17 00:00:00 2001 From: Dave Airlie Date: Mon, 27 Feb 2023 11:57:54 +1000 Subject: [PATCH 22/92] av1: set skip mode frames properly There are circumstances where the flag isn't set but the skip mode frames are. So don't use the inferred bit which has other inputs when deciding to pass the skip mode frames to the device. This fixes some decoding bugs on intel av1 --- libavcodec/av1dec.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/libavcodec/av1dec.c b/libavcodec/av1dec.c index 5a3c51e94a..1956666cb8 100644 --- a/libavcodec/av1dec.c +++ b/libavcodec/av1dec.c @@ -267,8 +267,10 @@ static void skip_mode_params(AV1DecContext *s) int second_forward_idx, second_forward_hint; int ref_hint, dist, i; - if (!header->skip_mode_present) - return; + if (header->frame_type == AV1_FRAME_KEY || + header->frame_type == AV1_FRAME_INTRA_ONLY || + !header->reference_select || !seq->enable_order_hint) + return; forward_idx = -1; backward_idx = -1; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0023-cbs_av1-expose-tile-col-row-starts-in-SBs.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0023-cbs_av1-expose-tile-col-row-starts-in-SBs.patch >From 570d67eba36f6edda61dd4b963347094d1486500 Mon Sep 17 00:00:00 2001 From: Lynne Date: Thu, 23 Feb 2023 02:46:13 +0100 Subject: [PATCH 23/92] cbs_av1: expose tile col/row starts in SBs --- libavcodec/cbs_av1.h | 2 ++ libavcodec/cbs_av1_syntax_template.c | 10 ++++++++++ 2 files changed, 12 insertions(+) diff --git a/libavcodec/cbs_av1.h b/libavcodec/cbs_av1.h index 1fc80dcfa0..77227650b1 100644 --- a/libavcodec/cbs_av1.h +++ b/libavcodec/cbs_av1.h @@ -215,6 +215,8 @@ typedef struct AV1RawFrameHeader { uint8_t uniform_tile_spacing_flag; uint8_t tile_cols_log2; uint8_t tile_rows_log2; + uint8_t tile_start_col_sb[AV1_MAX_TILE_COLS]; + uint8_t tile_start_row_sb[AV1_MAX_TILE_COLS]; uint8_t width_in_sbs_minus_1[AV1_MAX_TILE_COLS]; uint8_t height_in_sbs_minus_1[AV1_MAX_TILE_ROWS]; uint16_t context_update_tile_id; diff --git a/libavcodec/cbs_av1_syntax_template.c b/libavcodec/cbs_av1_syntax_template.c index e95925a493..1e5858d3ce 100644 --- a/libavcodec/cbs_av1_syntax_template.c +++ b/libavcodec/cbs_av1_syntax_template.c @@ -626,6 +626,10 @@ static int FUNC(tile_info)(CodedBitstreamContext *ctx, RWContext *rw, tile_width_sb = (sb_cols + (1 << current->tile_cols_log2) - 1) >> current->tile_cols_log2; + + for (int off = 0, i = 0; off < sb_cols; off += tile_width_sb) + current->tile_start_col_sb[i++] = off; + current->tile_cols = (sb_cols + tile_width_sb - 1) / tile_width_sb; min_log2_tile_rows = FFMAX(min_log2_tiles - current->tile_cols_log2, 0); @@ -634,6 +638,10 @@ static int FUNC(tile_info)(CodedBitstreamContext *ctx, RWContext *rw, tile_height_sb = (sb_rows + (1 << current->tile_rows_log2) - 1) >> current->tile_rows_log2; + + for (int off = 0, i = 0; off < sb_rows; off += tile_height_sb) + current->tile_start_row_sb[i++] = off; + current->tile_rows = (sb_rows + tile_height_sb - 1) / tile_height_sb; for (i = 0; i < current->tile_cols - 1; i++) @@ -652,6 +660,7 @@ static int FUNC(tile_info)(CodedBitstreamContext *ctx, RWContext *rw, start_sb = 0; for (i = 0; start_sb < sb_cols && i < AV1_MAX_TILE_COLS; i++) { + current->tile_start_col_sb[i] = start_sb; max_width = FFMIN(sb_cols - start_sb, max_tile_width_sb); ns(max_width, width_in_sbs_minus_1[i], 1, i); size_sb = current->width_in_sbs_minus_1[i] + 1; @@ -669,6 +678,7 @@ static int FUNC(tile_info)(CodedBitstreamContext *ctx, RWContext *rw, start_sb = 0; for (i = 0; start_sb < sb_rows && i < AV1_MAX_TILE_ROWS; i++) { + current->tile_start_row_sb[i] = start_sb; max_height = FFMIN(sb_rows - start_sb, max_tile_height_sb); ns(max_height, height_in_sbs_minus_1[i], 1, i); size_sb = current->height_in_sbs_minus_1[i] + 1; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0024-hwconfig-add-a-new-HWACCEL_CAP_THREAD_SAFE-for-threa.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0024-hwconfig-add-a-new-HWACCEL_CAP_THREAD_SAFE-for-threa.patch >From 6e2dfc44e50798264eb16bc2dcabfdbf2fbac2d7 Mon Sep 17 00:00:00 2001 From: Lynne Date: Thu, 15 Dec 2022 01:06:52 +0100 Subject: [PATCH 24/92] hwconfig: add a new HWACCEL_CAP_THREAD_SAFE for threadsafe hwaccels Vulkan is fully threadsafe and stateless, so we can benefit from this. --- libavcodec/hwconfig.h | 1 + libavcodec/pthread_frame.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/libavcodec/hwconfig.h b/libavcodec/hwconfig.h index 721424912c..e6b78f0160 100644 --- a/libavcodec/hwconfig.h +++ b/libavcodec/hwconfig.h @@ -24,6 +24,7 @@ #define HWACCEL_CAP_ASYNC_SAFE (1 << 0) +#define HWACCEL_CAP_THREAD_SAFE (1 << 1) typedef struct AVCodecHWConfigInternal { diff --git a/libavcodec/pthread_frame.c b/libavcodec/pthread_frame.c index 74864e19c5..c096287233 100644 --- a/libavcodec/pthread_frame.c +++ b/libavcodec/pthread_frame.c @@ -204,7 +204,7 @@ static attribute_align_arg void *frame_worker_thread(void *arg) /* if the previous thread uses hwaccel then we take the lock to ensure * the threads don't run concurrently */ - if (avctx->hwaccel) { + if (avctx->hwaccel && !(avctx->hwaccel->caps_internal & HWACCEL_CAP_THREAD_SAFE)) { pthread_mutex_lock(&p->parent->hwaccel_mutex); p->hwaccel_serializing = 1; } -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0025-avcodec-add-AVHWAccel.free_frame_priv-callback.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0025-avcodec-add-AVHWAccel.free_frame_priv-callback.patch >From e37330cf6a1f768c733ca6d0b23b9e5847a5fe4f Mon Sep 17 00:00:00 2001 From: Lynne Date: Thu, 10 Mar 2022 18:03:05 +0100 Subject: [PATCH 25/92] avcodec: add AVHWAccel.free_frame_priv callback --- libavcodec/av1dec.c | 4 ++-- libavcodec/avcodec.h | 8 ++++++++ libavcodec/decode.c | 19 +++++++++++++++++++ libavcodec/decode.h | 11 +++++++++++ libavcodec/h264_slice.c | 3 ++- libavcodec/hevc_refs.c | 3 ++- libavcodec/mpegpicture.c | 4 +++- libavcodec/vp8.c | 2 +- libavcodec/vp9.c | 2 +- 9 files changed, 49 insertions(+), 7 deletions(-) diff --git a/libavcodec/av1dec.c b/libavcodec/av1dec.c index 1956666cb8..fb558b4900 100644 --- a/libavcodec/av1dec.c +++ b/libavcodec/av1dec.c @@ -25,6 +25,7 @@ #include "libavutil/pixdesc.h" #include "libavutil/opt.h" #include "avcodec.h" +#include "decode.h" #include "av1dec.h" #include "atsc_a53.h" #include "bytestream.h" @@ -871,8 +872,7 @@ static int av1_frame_alloc(AVCodecContext *avctx, AV1Frame *f) if (avctx->hwaccel) { const AVHWAccel *hwaccel = avctx->hwaccel; if (hwaccel->frame_priv_data_size) { - f->hwaccel_priv_buf = - av_buffer_allocz(hwaccel->frame_priv_data_size); + f->hwaccel_priv_buf = ff_alloc_hwaccel_frame_priv_data(avctx, hwaccel); if (!f->hwaccel_priv_buf) { ret = AVERROR(ENOMEM); goto fail; diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h index 30f1d312f4..b43e79a2a6 100644 --- a/libavcodec/avcodec.h +++ b/libavcodec/avcodec.h @@ -2233,6 +2233,14 @@ typedef struct AVHWAccel { * that avctx->hwaccel_priv_data is invalid. */ int (*frame_params)(AVCodecContext *avctx, AVBufferRef *hw_frames_ctx); + + /** + * Callback to free the hwaccel-specific frame data. + * + * @param avctx the codec context + * @param data the per-frame hardware accelerator private data to be freed. + */ + void (*free_frame_priv)(AVCodecContext *avctx, void *data); } AVHWAccel; /** diff --git a/libavcodec/decode.c b/libavcodec/decode.c index d1ba7f167f..c4630ce275 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -1685,3 +1685,22 @@ int ff_copy_palette(void *dst, const AVPacket *src, void *logctx) } return 0; } + +AVBufferRef *ff_alloc_hwaccel_frame_priv_data(AVCodecContext *avctx, + const AVHWAccel *hwaccel) +{ + AVBufferRef *ref; + uint8_t *data = av_mallocz(hwaccel->frame_priv_data_size); + if (!data) + return NULL; + + ref = av_buffer_create(data, hwaccel->frame_priv_data_size, + (void (*)(void *, uint8_t *))hwaccel->free_frame_priv, + avctx, 0); + if (!ref) { + av_free(data); + return NULL; + } + + return ref; +} diff --git a/libavcodec/decode.h b/libavcodec/decode.h index 8430ffbd66..aa40baafc0 100644 --- a/libavcodec/decode.h +++ b/libavcodec/decode.h @@ -150,4 +150,15 @@ int ff_reget_buffer(AVCodecContext *avctx, AVFrame *frame, int flags); int ff_side_data_update_matrix_encoding(AVFrame *frame, enum AVMatrixEncoding matrix_encoding); +/** + * Allocate a hwaccel frame private data and create an AVBufferRef + * from it. + * + * @param avctx The codec context which to attach as an opaque value + * @param hwaccel The hwaccel for which to allocate + * @return The allocated buffer + */ +AVBufferRef *ff_alloc_hwaccel_frame_priv_data(AVCodecContext *avctx, + const AVHWAccel *hwaccel); + #endif /* AVCODEC_DECODE_H */ diff --git a/libavcodec/h264_slice.c b/libavcodec/h264_slice.c index 7767e16cf1..579f074c42 100644 --- a/libavcodec/h264_slice.c +++ b/libavcodec/h264_slice.c @@ -33,6 +33,7 @@ #include "libavutil/pixdesc.h" #include "libavutil/timecode.h" #include "internal.h" +#include "decode.h" #include "cabac.h" #include "cabac_functions.h" #include "decode.h" @@ -212,7 +213,7 @@ static int alloc_picture(H264Context *h, H264Picture *pic) const AVHWAccel *hwaccel = h->avctx->hwaccel; av_assert0(!pic->hwaccel_picture_private); if (hwaccel->frame_priv_data_size) { - pic->hwaccel_priv_buf = av_buffer_allocz(hwaccel->frame_priv_data_size); + pic->hwaccel_priv_buf = ff_alloc_hwaccel_frame_priv_data(h->avctx, hwaccel); if (!pic->hwaccel_priv_buf) return AVERROR(ENOMEM); pic->hwaccel_picture_private = pic->hwaccel_priv_buf->data; diff --git a/libavcodec/hevc_refs.c b/libavcodec/hevc_refs.c index 96153a2459..e68a7bc014 100644 --- a/libavcodec/hevc_refs.c +++ b/libavcodec/hevc_refs.c @@ -23,6 +23,7 @@ #include "libavutil/avassert.h" +#include "decode.h" #include "thread.h" #include "hevc.h" #include "hevcdec.h" @@ -118,7 +119,7 @@ static HEVCFrame *alloc_frame(HEVCContext *s) const AVHWAccel *hwaccel = s->avctx->hwaccel; av_assert0(!frame->hwaccel_picture_private); if (hwaccel->frame_priv_data_size) { - frame->hwaccel_priv_buf = av_buffer_allocz(hwaccel->frame_priv_data_size); + frame->hwaccel_priv_buf = ff_alloc_hwaccel_frame_priv_data(s->avctx, hwaccel); if (!frame->hwaccel_priv_buf) goto fail; frame->hwaccel_picture_private = frame->hwaccel_priv_buf->data; diff --git a/libavcodec/mpegpicture.c b/libavcodec/mpegpicture.c index 3204a70578..654e796e38 100644 --- a/libavcodec/mpegpicture.c +++ b/libavcodec/mpegpicture.c @@ -27,6 +27,8 @@ #include "avcodec.h" #include "encode.h" +#include "internal.h" +#include "decode.h" #include "motion_est.h" #include "mpegpicture.h" #include "mpegutils.h" @@ -172,7 +174,7 @@ static int alloc_frame_buffer(AVCodecContext *avctx, Picture *pic, if (avctx->hwaccel) { assert(!pic->hwaccel_picture_private); if (avctx->hwaccel->frame_priv_data_size) { - pic->hwaccel_priv_buf = av_buffer_allocz(avctx->hwaccel->frame_priv_data_size); + pic->hwaccel_priv_buf = ff_alloc_hwaccel_frame_priv_data(avctx, avctx->hwaccel); if (!pic->hwaccel_priv_buf) { av_log(avctx, AV_LOG_ERROR, "alloc_frame_buffer() failed (hwaccel private data allocation)\n"); return -1; diff --git a/libavcodec/vp8.c b/libavcodec/vp8.c index db2419deaf..4c23eb5672 100644 --- a/libavcodec/vp8.c +++ b/libavcodec/vp8.c @@ -109,7 +109,7 @@ static int vp8_alloc_frame(VP8Context *s, VP8Frame *f, int ref) if (s->avctx->hwaccel) { const AVHWAccel *hwaccel = s->avctx->hwaccel; if (hwaccel->frame_priv_data_size) { - f->hwaccel_priv_buf = av_buffer_allocz(hwaccel->frame_priv_data_size); + f->hwaccel_priv_buf = ff_alloc_hwaccel_frame_priv_data(s->avctx, hwaccel); if (!f->hwaccel_priv_buf) goto fail; f->hwaccel_picture_private = f->hwaccel_priv_buf->data; diff --git a/libavcodec/vp9.c b/libavcodec/vp9.c index 7c0a246446..4f345f18db 100644 --- a/libavcodec/vp9.c +++ b/libavcodec/vp9.c @@ -136,7 +136,7 @@ static int vp9_frame_alloc(AVCodecContext *avctx, VP9Frame *f) const AVHWAccel *hwaccel = avctx->hwaccel; av_assert0(!f->hwaccel_picture_private); if (hwaccel->frame_priv_data_size) { - f->hwaccel_priv_buf = av_buffer_allocz(hwaccel->frame_priv_data_size); + f->hwaccel_priv_buf = ff_alloc_hwaccel_frame_priv_data(avctx, hwaccel); if (!f->hwaccel_priv_buf) goto fail; f->hwaccel_picture_private = f->hwaccel_priv_buf->data; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0026-avcodec-add-AVHWAccel.flush-callback.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0026-avcodec-add-AVHWAccel.flush-callback.patch >From ad702a1fce9aaef80fcc72fcdd820f7ce96eb1b1 Mon Sep 17 00:00:00 2001 From: Lynne Date: Fri, 6 Jan 2023 03:32:56 +0100 Subject: [PATCH 26/92] avcodec: add AVHWAccel.flush callback --- libavcodec/av1dec.c | 3 +++ libavcodec/avcodec.h | 5 +++++ libavcodec/h264dec.c | 3 +++ libavcodec/hevcdec.c | 3 +++ libavcodec/vp8.c | 3 +++ libavcodec/vp9.c | 3 +++ 6 files changed, 20 insertions(+) diff --git a/libavcodec/av1dec.c b/libavcodec/av1dec.c index fb558b4900..6fa8b25435 100644 --- a/libavcodec/av1dec.c +++ b/libavcodec/av1dec.c @@ -1419,6 +1419,9 @@ static void av1_decode_flush(AVCodecContext *avctx) av_buffer_unref(&itut_t35.payload_ref); ff_cbs_flush(s->cbc); + + if (avctx->hwaccel && avctx->hwaccel->flush) + avctx->hwaccel->flush(avctx); } #define OFFSET(x) offsetof(AV1DecContext, x) diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h index b43e79a2a6..4ffd6faad2 100644 --- a/libavcodec/avcodec.h +++ b/libavcodec/avcodec.h @@ -2241,6 +2241,11 @@ typedef struct AVHWAccel { * @param data the per-frame hardware accelerator private data to be freed. */ void (*free_frame_priv)(AVCodecContext *avctx, void *data); + + /** + * Callback to flush the hwaccel state. + */ + void (*flush)(AVCodecContext *avctx); } AVHWAccel; /** diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c index 2d691731c5..0aee724c0d 100644 --- a/libavcodec/h264dec.c +++ b/libavcodec/h264dec.c @@ -480,6 +480,9 @@ static void h264_decode_flush(AVCodecContext *avctx) ff_h264_free_tables(h); h->context_initialized = 0; + + if (avctx->hwaccel && avctx->hwaccel->flush) + avctx->hwaccel->flush(avctx); } static int get_last_needed_nal(H264Context *h) diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 0230a41311..d120aaeae1 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -3704,6 +3704,9 @@ static void hevc_decode_flush(AVCodecContext *avctx) av_buffer_unref(&s->rpu_buf); s->max_ra = INT_MAX; s->eos = 1; + + if (avctx->hwaccel && avctx->hwaccel->flush) + avctx->hwaccel->flush(avctx); } #define OFFSET(x) offsetof(HEVCContext, x) diff --git a/libavcodec/vp8.c b/libavcodec/vp8.c index 4c23eb5672..86e7f432bc 100644 --- a/libavcodec/vp8.c +++ b/libavcodec/vp8.c @@ -167,6 +167,9 @@ static void vp8_decode_flush_impl(AVCodecContext *avctx, int free_mem) if (free_mem) free_buffers(s); + + if (avctx->hwaccel && avctx->hwaccel->flush) + avctx->hwaccel->flush(avctx); } static void vp8_decode_flush(AVCodecContext *avctx) diff --git a/libavcodec/vp9.c b/libavcodec/vp9.c index 4f345f18db..54fa08bbe5 100644 --- a/libavcodec/vp9.c +++ b/libavcodec/vp9.c @@ -1791,6 +1791,9 @@ static void vp9_decode_flush(AVCodecContext *avctx) vp9_frame_unref(avctx, &s->s.frames[i]); for (i = 0; i < 8; i++) ff_thread_release_ext_buffer(avctx, &s->s.refs[i]); + + if (avctx->hwaccel && avctx->hwaccel->flush) + avctx->hwaccel->flush(avctx); } static av_cold int vp9_decode_init(AVCodecContext *avctx) -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0027-h264dec-add-hwaccel_params_buf.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0027-h264dec-add-hwaccel_params_buf.patch >From 61f412eea3fbcb1e2a8625796760c0e24fa3fb83 Mon Sep 17 00:00:00 2001 From: Lynne Date: Tue, 17 Jan 2023 05:01:45 +0100 Subject: [PATCH 27/92] h264dec: add hwaccel_params_buf --- libavcodec/h264_slice.c | 4 ++++ libavcodec/h264dec.c | 4 ++++ libavcodec/h264dec.h | 2 ++ 3 files changed, 10 insertions(+) diff --git a/libavcodec/h264_slice.c b/libavcodec/h264_slice.c index 579f074c42..19c10915d1 100644 --- a/libavcodec/h264_slice.c +++ b/libavcodec/h264_slice.c @@ -347,6 +347,10 @@ int ff_h264_update_thread_context(AVCodecContext *dst, return ret; } + ret = av_buffer_replace(&h->hwaccel_params_buf, h1->hwaccel_params_buf); + if (ret < 0) + return ret; + ret = av_buffer_replace(&h->ps.pps_ref, h1->ps.pps_ref); if (ret < 0) return ret; diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c index 0aee724c0d..6559593195 100644 --- a/libavcodec/h264dec.c +++ b/libavcodec/h264dec.c @@ -341,6 +341,7 @@ static av_cold int h264_decode_end(AVCodecContext *avctx) H264Context *h = avctx->priv_data; int i; + av_buffer_unref(&h->hwaccel_params_buf); ff_h264_remove_all_refs(h); ff_h264_free_tables(h); @@ -470,6 +471,7 @@ static void h264_decode_flush(AVCodecContext *avctx) ff_h264_flush_change(h); ff_h264_sei_uninit(&h->sei); + av_buffer_unref(&h->hwaccel_params_buf); for (i = 0; i < H264_MAX_PICTURE_COUNT; i++) ff_h264_unref_picture(h, &h->DPB[i]); @@ -669,6 +671,7 @@ static int decode_nal_units(H264Context *h, const uint8_t *buf, int buf_size) avpriv_request_sample(avctx, "data partitioning"); break; case H264_NAL_SEI: + av_buffer_unref(&h->hwaccel_params_buf); if (h->setup_finished) { avpriv_request_sample(avctx, "Late SEI"); break; @@ -682,6 +685,7 @@ static int decode_nal_units(H264Context *h, const uint8_t *buf, int buf_size) break; case H264_NAL_SPS: { GetBitContext tmp_gb = nal->gb; + av_buffer_unref(&h->hwaccel_params_buf); if (avctx->hwaccel && avctx->hwaccel->decode_params) { ret = avctx->hwaccel->decode_params(avctx, nal->type, diff --git a/libavcodec/h264dec.h b/libavcodec/h264dec.h index 9a1ec1bace..d7352c15d1 100644 --- a/libavcodec/h264dec.h +++ b/libavcodec/h264dec.h @@ -341,6 +341,8 @@ typedef struct H264Context { H264Picture cur_pic; H264Picture last_pic_for_ec; + AVBufferRef *hwaccel_params_buf; + H264SliceContext *slice_ctx; int nb_slice_ctx; int nb_slice_ctx_queued; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0028-hevcdec-add-hwaccel_params_buf.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0028-hevcdec-add-hwaccel_params_buf.patch >From d9f39a790e2977f01ea020678dde27391cafff7f Mon Sep 17 00:00:00 2001 From: Lynne Date: Tue, 17 Jan 2023 05:02:02 +0100 Subject: [PATCH 28/92] hevcdec: add hwaccel_params_buf --- libavcodec/hevcdec.c | 9 +++++++++ libavcodec/hevcdec.h | 2 ++ 2 files changed, 11 insertions(+) diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index d120aaeae1..a18dc6cc79 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -2984,6 +2984,7 @@ static int decode_nal_unit(HEVCContext *s, const H2645NAL *nal) switch (s->nal_unit_type) { case HEVC_NAL_VPS: + av_buffer_unref(&s->hwaccel_params_buf); if (s->avctx->hwaccel && s->avctx->hwaccel->decode_params) { ret = s->avctx->hwaccel->decode_params(s->avctx, nal->type, @@ -2997,6 +2998,7 @@ static int decode_nal_unit(HEVCContext *s, const H2645NAL *nal) goto fail; break; case HEVC_NAL_SPS: + av_buffer_unref(&s->hwaccel_params_buf); if (s->avctx->hwaccel && s->avctx->hwaccel->decode_params) { ret = s->avctx->hwaccel->decode_params(s->avctx, nal->type, @@ -3011,6 +3013,7 @@ static int decode_nal_unit(HEVCContext *s, const H2645NAL *nal) goto fail; break; case HEVC_NAL_PPS: + av_buffer_unref(&s->hwaccel_params_buf); if (s->avctx->hwaccel && s->avctx->hwaccel->decode_params) { ret = s->avctx->hwaccel->decode_params(s->avctx, nal->type, @@ -3477,6 +3480,7 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx) ff_dovi_ctx_unref(&s->dovi_ctx); av_buffer_unref(&s->rpu_buf); + av_buffer_unref(&s->hwaccel_params_buf); av_freep(&s->md5_ctx); @@ -3628,6 +3632,10 @@ static int hevc_update_thread_context(AVCodecContext *dst, if (ret < 0) return ret; + ret = av_buffer_replace(&s->hwaccel_params_buf, s0->hwaccel_params_buf); + if (ret < 0) + return ret; + ret = av_buffer_replace(&s->rpu_buf, s0->rpu_buf); if (ret < 0) return ret; @@ -3705,6 +3713,7 @@ static void hevc_decode_flush(AVCodecContext *avctx) s->max_ra = INT_MAX; s->eos = 1; + av_buffer_unref(&s->hwaccel_params_buf); if (avctx->hwaccel && avctx->hwaccel->flush) avctx->hwaccel->flush(avctx); } diff --git a/libavcodec/hevcdec.h b/libavcodec/hevcdec.h index ddb6e2b5d9..514fc4daa4 100644 --- a/libavcodec/hevcdec.h +++ b/libavcodec/hevcdec.h @@ -514,6 +514,8 @@ typedef struct HEVCContext { uint8_t *sao_pixel_buffer_h[3]; uint8_t *sao_pixel_buffer_v[3]; + AVBufferRef *hwaccel_params_buf; + HEVCParamSets ps; HEVCSEI sei; struct AVMD5 *md5_ctx; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0029-av1dec-add-hwaccel_params_buf.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0029-av1dec-add-hwaccel_params_buf.patch >From f16f425e0a66fd54b5ad4f04df24bae402440641 Mon Sep 17 00:00:00 2001 From: Lynne Date: Tue, 17 Jan 2023 05:07:53 +0100 Subject: [PATCH 29/92] av1dec: add hwaccel_params_buf --- libavcodec/av1dec.c | 2 ++ libavcodec/av1dec.h | 2 ++ 2 files changed, 4 insertions(+) diff --git a/libavcodec/av1dec.c b/libavcodec/av1dec.c index 6fa8b25435..bb38dfb41e 100644 --- a/libavcodec/av1dec.c +++ b/libavcodec/av1dec.c @@ -1177,6 +1177,7 @@ static int av1_decode_frame(AVCodecContext *avctx, AVFrame *frame, switch (unit->type) { case AV1_OBU_SEQUENCE_HEADER: av_buffer_unref(&s->seq_ref); + av_buffer_unref(&s->hwaccel_params_buf); s->seq_ref = av_buffer_ref(unit->content_ref); if (!s->seq_ref) { ret = AVERROR(ENOMEM); @@ -1420,6 +1421,7 @@ static void av1_decode_flush(AVCodecContext *avctx) ff_cbs_flush(s->cbc); + av_buffer_unref(&s->hwaccel_params_buf); if (avctx->hwaccel && avctx->hwaccel->flush) avctx->hwaccel->flush(avctx); } diff --git a/libavcodec/av1dec.h b/libavcodec/av1dec.h index 2be28e7469..7b110ab7db 100644 --- a/libavcodec/av1dec.h +++ b/libavcodec/av1dec.h @@ -66,6 +66,8 @@ typedef struct AV1DecContext { const AVClass *class; AVCodecContext *avctx; + AVBufferRef *hwaccel_params_buf; + enum AVPixelFormat pix_fmt; CodedBitstreamContext *cbc; CodedBitstreamFragment current_obu; -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0030-lavu-add-12-bit-2-plane-422-and-444-pixel-formats.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0030-lavu-add-12-bit-2-plane-422-and-444-pixel-formats.patch >From 530800d9777d20748d95fe5bee5c24e919de1e4f Mon Sep 17 00:00:00 2001 From: Lynne Date: Sat, 25 Feb 2023 09:34:54 +0100 Subject: [PATCH 30/92] lavu: add 12-bit 2-plane 422 and 444 pixel formats --- libavutil/pixdesc.c | 48 +++++++++++++++++++++++++++++++++++++++++ libavutil/pixfmt.h | 8 +++++++ tests/ref/fate/imgutils | 4 ++++ 3 files changed, 60 insertions(+) diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c index 62a2ae08d9..e1e0dd2a9e 100644 --- a/libavutil/pixdesc.c +++ b/libavutil/pixdesc.c @@ -2717,6 +2717,54 @@ static const AVPixFmtDescriptor av_pix_fmt_descriptors[AV_PIX_FMT_NB] = { .flags = AV_PIX_FMT_FLAG_RGB | AV_PIX_FMT_FLAG_FLOAT | AV_PIX_FMT_FLAG_ALPHA, }, + [AV_PIX_FMT_P212BE] = { + .name = "p212be", + .nb_components = 3, + .log2_chroma_w = 1, + .log2_chroma_h = 0, + .comp = { + { 0, 2, 0, 4, 12 }, /* Y */ + { 1, 4, 0, 4, 12 }, /* U */ + { 1, 4, 2, 4, 12 }, /* V */ + }, + .flags = AV_PIX_FMT_FLAG_PLANAR | AV_PIX_FMT_FLAG_BE, + }, + [AV_PIX_FMT_P212LE] = { + .name = "p212le", + .nb_components = 3, + .log2_chroma_w = 1, + .log2_chroma_h = 0, + .comp = { + { 0, 2, 0, 4, 12 }, /* Y */ + { 1, 4, 0, 4, 12 }, /* U */ + { 1, 4, 2, 4, 12 }, /* V */ + }, + .flags = AV_PIX_FMT_FLAG_PLANAR, + }, + [AV_PIX_FMT_P412BE] = { + .name = "p412be", + .nb_components = 3, + .log2_chroma_w = 0, + .log2_chroma_h = 0, + .comp = { + { 0, 2, 0, 4, 12 }, /* Y */ + { 1, 4, 0, 4, 12 }, /* U */ + { 1, 4, 2, 4, 12 }, /* V */ + }, + .flags = AV_PIX_FMT_FLAG_PLANAR | AV_PIX_FMT_FLAG_BE, + }, + [AV_PIX_FMT_P412LE] = { + .name = "p412le", + .nb_components = 3, + .log2_chroma_w = 0, + .log2_chroma_h = 0, + .comp = { + { 0, 2, 0, 4, 12 }, /* Y */ + { 1, 4, 0, 4, 12 }, /* U */ + { 1, 4, 2, 4, 12 }, /* V */ + }, + .flags = AV_PIX_FMT_FLAG_PLANAR, + }, }; static const char * const color_range_names[] = { diff --git a/libavutil/pixfmt.h b/libavutil/pixfmt.h index 37c2c79e01..63e07ba64f 100644 --- a/libavutil/pixfmt.h +++ b/libavutil/pixfmt.h @@ -420,6 +420,12 @@ enum AVPixelFormat { AV_PIX_FMT_RGBAF32BE, ///< IEEE-754 single precision packed RGBA 32:32:32:32, 128bpp, RGBARGBA..., big-endian AV_PIX_FMT_RGBAF32LE, ///< IEEE-754 single precision packed RGBA 32:32:32:32, 128bpp, RGBARGBA..., little-endian + AV_PIX_FMT_P212BE, ///< interleaved chroma YUV 4:2:2, 24bpp, data in the high bits, big-endian + AV_PIX_FMT_P212LE, ///< interleaved chroma YUV 4:2:2, 24bpp, data in the high bits, little-endian + + AV_PIX_FMT_P412BE, ///< interleaved chroma YUV 4:4:4, 36bpp, data in the high bits, big-endian + AV_PIX_FMT_P412LE, ///< interleaved chroma YUV 4:4:4, 36bpp, data in the high bits, little-endian + AV_PIX_FMT_NB ///< number of pixel formats, DO NOT USE THIS if you want to link with shared libav* because the number of formats might differ between versions }; @@ -518,6 +524,8 @@ enum AVPixelFormat { #define AV_PIX_FMT_P210 AV_PIX_FMT_NE(P210BE, P210LE) #define AV_PIX_FMT_P410 AV_PIX_FMT_NE(P410BE, P410LE) +#define AV_PIX_FMT_P212 AV_PIX_FMT_NE(P212BE, P212LE) +#define AV_PIX_FMT_P412 AV_PIX_FMT_NE(P412BE, P412LE) #define AV_PIX_FMT_P216 AV_PIX_FMT_NE(P216BE, P216LE) #define AV_PIX_FMT_P416 AV_PIX_FMT_NE(P416BE, P416LE) diff --git a/tests/ref/fate/imgutils b/tests/ref/fate/imgutils index e79ec7e4b3..02a755f2b7 100644 --- a/tests/ref/fate/imgutils +++ b/tests/ref/fate/imgutils @@ -262,3 +262,7 @@ rgbf32be planes: 1, linesizes: 768 0 0 0, plane_sizes: 36864 0 rgbf32le planes: 1, linesizes: 768 0 0 0, plane_sizes: 36864 0 0 0, plane_offsets: 0 0 0, total_size: 36864 rgbaf32be planes: 1, linesizes: 1024 0 0 0, plane_sizes: 49152 0 0 0, plane_offsets: 0 0 0, total_size: 49152 rgbaf32le planes: 1, linesizes: 1024 0 0 0, plane_sizes: 49152 0 0 0, plane_offsets: 0 0 0, total_size: 49152 +p212be planes: 2, linesizes: 128 128 0 0, plane_sizes: 6144 6144 0 0, plane_offsets: 6144 0 0, total_size: 12288 +p212le planes: 2, linesizes: 128 128 0 0, plane_sizes: 6144 6144 0 0, plane_offsets: 6144 0 0, total_size: 12288 +p412be planes: 2, linesizes: 128 256 0 0, plane_sizes: 6144 12288 0 0, plane_offsets: 6144 0 0, total_size: 18432 +p412le planes: 2, linesizes: 128 256 0 0, plane_sizes: 6144 12288 0 0, plane_offsets: 6144 0 0, total_size: 18432 -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/x-diff; charset=us-ascii; name=0031-lsws-add-in-out-support-for-the-new-12-bit-2-plane-4.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0031-lsws-add-in-out-support-for-the-new-12-bit-2-plane-4.patch >From 26136684812c3e94a35dea25cb5edfe32601a518 Mon Sep 17 00:00:00 2001 From: Lynne Date: Sat, 25 Feb 2023 09:36:58 +0100 Subject: [PATCH 31/92] lsws: add in/out support for the new 12-bit 2-plane 422 and 444 pixfmts --- libswscale/input.c | 8 ++++++++ libswscale/utils.c | 4 ++++ tests/ref/fate/sws-pixdesc-query | 26 ++++++++++++++++++++++++++ 3 files changed, 38 insertions(+) diff --git a/libswscale/input.c b/libswscale/input.c index d5676062a2..41795c636e 100644 --- a/libswscale/input.c +++ b/libswscale/input.c @@ -1452,9 +1452,13 @@ av_cold void ff_sws_init_input_funcs(SwsContext *c) c->chrToYV12 = p010BEToUV_c; break; case AV_PIX_FMT_P012LE: + case AV_PIX_FMT_P212LE: + case AV_PIX_FMT_P412LE: c->chrToYV12 = p012LEToUV_c; break; case AV_PIX_FMT_P012BE: + case AV_PIX_FMT_P212BE: + case AV_PIX_FMT_P412BE: c->chrToYV12 = p012BEToUV_c; break; case AV_PIX_FMT_P016LE: @@ -1944,9 +1948,13 @@ av_cold void ff_sws_init_input_funcs(SwsContext *c) c->lumToYV12 = p010BEToY_c; break; case AV_PIX_FMT_P012LE: + case AV_PIX_FMT_P212LE: + case AV_PIX_FMT_P412LE: c->lumToYV12 = p012LEToY_c; break; case AV_PIX_FMT_P012BE: + case AV_PIX_FMT_P212BE: + case AV_PIX_FMT_P412BE: c->lumToYV12 = p012BEToY_c; break; case AV_PIX_FMT_GRAYF32LE: diff --git a/libswscale/utils.c b/libswscale/utils.c index 925c536bf1..a3a7a40750 100644 --- a/libswscale/utils.c +++ b/libswscale/utils.c @@ -248,8 +248,12 @@ static const FormatEntry format_entries[] = { [AV_PIX_FMT_X2BGR10LE] = { 1, 1 }, [AV_PIX_FMT_P210BE] = { 1, 1 }, [AV_PIX_FMT_P210LE] = { 1, 1 }, + [AV_PIX_FMT_P212BE] = { 1, 1 }, + [AV_PIX_FMT_P212LE] = { 1, 1 }, [AV_PIX_FMT_P410BE] = { 1, 1 }, [AV_PIX_FMT_P410LE] = { 1, 1 }, + [AV_PIX_FMT_P412BE] = { 1, 1 }, + [AV_PIX_FMT_P412LE] = { 1, 1 }, [AV_PIX_FMT_P216BE] = { 1, 1 }, [AV_PIX_FMT_P216LE] = { 1, 1 }, [AV_PIX_FMT_P416BE] = { 1, 1 }, diff --git a/tests/ref/fate/sws-pixdesc-query b/tests/ref/fate/sws-pixdesc-query index 14156a383c..fd7f2aefc0 100644 --- a/tests/ref/fate/sws-pixdesc-query +++ b/tests/ref/fate/sws-pixdesc-query @@ -67,8 +67,12 @@ isNBPS: p012le p210be p210le + p212be + p212le p410be p410le + p412be + p412le x2bgr10be x2bgr10le x2rgb10be @@ -160,8 +164,10 @@ isBE: p012be p016be p210be + p212be p216be p410be + p412be p416be rgb444be rgb48be @@ -226,10 +232,14 @@ isYUV: p016le p210be p210le + p212be + p212le p216be p216le p410be p410le + p412be + p412le p416be p416le uyvy422 @@ -338,10 +348,14 @@ isPlanarYUV: p016le p210be p210le + p212be + p212le p216be p216le p410be p410le + p412be + p412le p416be p416le yuv410p @@ -431,10 +445,14 @@ isSemiPlanarYUV: p016le p210be p210le + p212be + p212le p216be p216le p410be p410le + p412be + p412le p416be p416le @@ -853,10 +871,14 @@ Planar: p016le p210be p210le + p212be + p212le p216be p216le p410be p410le + p412be + p412le p416be p416le yuv410p @@ -1029,8 +1051,12 @@ DataInHighBits: p012le p210be p210le + p212be + p212le p410be p410le + p412be + p412le xv36be xv36le xyz12be -- 2.39.2 ------=_Part_55579_1396114304.1678775623438 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ------=_Part_55579_1396114304.1678775623438--