From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 0007D46529 for ; Sun, 21 May 2023 13:04:00 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3F1F268C109; Sun, 21 May 2023 16:03:38 +0300 (EEST) Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1458668C025 for ; Sun, 21 May 2023 16:03:31 +0300 (EEST) Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-64d3fdcadb8so1271011b3a.3 for ; Sun, 21 May 2023 06:03:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684674209; x=1687266209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JjuMB4SC/J84J3tkmFEtRV5VD0oabMKksHce9VY/oP8=; b=DSK7fx3O+oBq++JNmgorr6jI8yWPZKKYReIsiIKgg7qe5m3bKqyOFZOCdFiSWwpJjH K55anLp0NRDktf7s/bxIQyjo2HyZl+1C4SDW4mpoSDvkZLZMFzW+MFG/ffrxuuAwsFmJ nlY4iyYvPoppfrXpuNieCrO0J4a2D1sEPHxtY6Mt83uk8h0BZ3bwpp8h8nRDJPr0lo/a xOfP1ycxOY6syehn6YamA+VY8oahQ4dwhIbnkkiLOBzB9Gp8wBcRoEUEFf62QmLqG3IY 8PSDF9UeCqOwQVy5HdOEDBLe+z280Pmcv6pI88QYPSpg8uxYnFV//VSgaVU9IH/cjXsX jYsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684674209; x=1687266209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JjuMB4SC/J84J3tkmFEtRV5VD0oabMKksHce9VY/oP8=; b=XCpSju88rrKibdtflJmjhFG8icCR+OdftHpkIkY5B4+iXA7w/dZzBjmYnL6rFQrE7z I2WRn+Zs9RyMqbzsXb6uS5DDfNBhmcJE/AjCht9IjHsqm8kjC4+XBc6TenWd3biwljFx eB+r3If548hNnPpxyL2N7t64LTnQJpvFfWIDQwI1epieYVLCJZ6b4iZTKRAmnP0yi2TS vaEKJIb+f6woWd3anBQle/vrDI2nuYxQjq70RrJHeYhHyZXxjlgeyMF65JMkNc8b3FeX tUZ/ITQ9n9c34Cs05h3XfLH1ijuVYN0E/cxIWhpojJ1l1HWn0/YpAdia6PRVdLYI4Ryo Hviw== X-Gm-Message-State: AC+VfDxJtukO4wC/vjpm31Dj5MmKOBjBiotpcyTdWhcK1ldeXjKnGAOW mdGQoTKnN1hsn4Cd9pZW6jTWiKe6hKZKgg== X-Google-Smtp-Source: ACHHUZ5nR/a9dl3GaiCPnPXhrCvtpLKCR8Op9jXFxzvxP8vegccpG/9B76wMu2vF0E55bIdLHvfCdg== X-Received: by 2002:a05:6a00:cc9:b0:64d:22db:1018 with SMTP id b9-20020a056a000cc900b0064d22db1018mr10248113pfv.25.1684674208797; Sun, 21 May 2023 06:03:28 -0700 (PDT) Received: from NuoMi.localdomain ([112.64.8.97]) by smtp.gmail.com with ESMTPSA id t9-20020aa79389000000b0064d45eea573sm1671872pfe.41.2023.05.21.06.03.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 06:03:28 -0700 (PDT) From: Nuo Mi To: ffmpeg-devel@ffmpeg.org Date: Sun, 21 May 2023 21:03:07 +0800 Message-Id: <20230521130319.13813-3-nuomi2021@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230521130319.13813-1-nuomi2021@gmail.com> References: <20230521130319.13813-1-nuomi2021@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 02/14] vvcdec: add vvc decoder stub X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Nuo Mi Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: --- configure | 1 + libavcodec/allcodecs.c | 1 + libavcodec/vvc/Makefile | 3 +- libavcodec/vvc/vvcdec.c | 84 +++++++++++ libavcodec/vvc/vvcdec.h | 304 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 392 insertions(+), 1 deletion(-) create mode 100644 libavcodec/vvc/vvcdec.c create mode 100644 libavcodec/vvc/vvcdec.h diff --git a/configure b/configure index c5de9a1114..cc5c75f1c9 100755 --- a/configure +++ b/configure @@ -2968,6 +2968,7 @@ vp6f_decoder_select="vp6_decoder" vp7_decoder_select="h264pred videodsp vp8dsp" vp8_decoder_select="h264pred videodsp vp8dsp" vp9_decoder_select="videodsp vp9_parser vp9_superframe_split_bsf" +vvc_decoder_select="cabac golomb videodsp" wcmv_decoder_select="inflate_wrapper" webp_decoder_select="vp8_decoder exif" wmalossless_decoder_select="llauddsp" diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c index b009848a44..16255a5794 100644 --- a/libavcodec/allcodecs.c +++ b/libavcodec/allcodecs.c @@ -383,6 +383,7 @@ extern const FFCodec ff_vp9_rkmpp_decoder; extern const FFCodec ff_vp9_v4l2m2m_decoder; extern const FFCodec ff_vqa_decoder; extern const FFCodec ff_vqc_decoder; +extern const FFCodec ff_vvc_decoder; extern const FFCodec ff_wbmp_decoder; extern const FFCodec ff_wbmp_encoder; extern const FFCodec ff_webp_decoder; diff --git a/libavcodec/vvc/Makefile b/libavcodec/vvc/Makefile index c4b93e0389..06d611ca6d 100644 --- a/libavcodec/vvc/Makefile +++ b/libavcodec/vvc/Makefile @@ -1,4 +1,5 @@ clean:: $(RM) $(CLEANSUFFIXES:%=libavcodec/vvc/%) -OBJS-$(CONFIG_VVC_DECODER) += vvc/vvc_executor.o +OBJS-$(CONFIG_VVC_DECODER) += vvc/vvcdec.o \ + vvc/vvc_executor.o diff --git a/libavcodec/vvc/vvcdec.c b/libavcodec/vvc/vvcdec.c new file mode 100644 index 0000000000..8d027af0b9 --- /dev/null +++ b/libavcodec/vvc/vvcdec.c @@ -0,0 +1,84 @@ +/* + * VVC video decoder + * + * Copyright (C) 2021 Nuo Mi + * Copyright (C) 2022 Xu Mu + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ +#include "config_components.h" + +#include "libavcodec/codec_internal.h" +#include "libavcodec/decode.h" +#include "libavcodec/golomb.h" +#include "libavcodec/profiles.h" +#include "libavcodec/vvc.h" + +#include "libavutil/cpu.h" + +#include "vvcdec.h" + +static int vvc_decode_frame(AVCodecContext *avctx, AVFrame *output, + int *got_output, AVPacket *avpkt) +{ + return avpkt->size; +} + +static void vvc_decode_flush(AVCodecContext *avctx) +{ +} + +static av_cold int vvc_decode_free(AVCodecContext *avctx) +{ + return 0; +} + +static av_cold int vvc_decode_init(AVCodecContext *avctx) +{ + return 0; +} + +#define OFFSET(x) offsetof(VVCContext, x) +#define PAR (AV_OPT_FLAG_DECODING_PARAM | AV_OPT_FLAG_VIDEO_PARAM) + +static const AVOption options[] = { + { NULL }, +}; + +static const AVClass vvc_decoder_class = { + .class_name = "vvc decoder", + .item_name = av_default_item_name, + .option = options, + .version = LIBAVUTIL_VERSION_INT, +}; + +const FFCodec ff_vvc_decoder = { + .p.name = "vvc", + .p.long_name = NULL_IF_CONFIG_SMALL("VVC (Versatile Video Coding)"), + .p.type = AVMEDIA_TYPE_VIDEO, + .p.id = AV_CODEC_ID_VVC, + .priv_data_size = sizeof(VVCContext), + .p.priv_class = &vvc_decoder_class, + .init = vvc_decode_init, + .close = vvc_decode_free, + FF_CODEC_DECODE_CB(vvc_decode_frame), + .flush = vvc_decode_flush, + .p.capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY | AV_CODEC_CAP_OTHER_THREADS, + .caps_internal = FF_CODEC_CAP_EXPORTS_CROPPING | FF_CODEC_CAP_INIT_CLEANUP | + FF_CODEC_CAP_AUTO_THREADS, + .p.profiles = NULL_IF_CONFIG_SMALL(ff_vvc_profiles), +}; diff --git a/libavcodec/vvc/vvcdec.h b/libavcodec/vvc/vvcdec.h new file mode 100644 index 0000000000..7097293af8 --- /dev/null +++ b/libavcodec/vvc/vvcdec.h @@ -0,0 +1,304 @@ +/* + * VVC video decoder + * + * Copyright (C) 2021 Nuo Mi + * Copyright (C) 2022 Xu Mu + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_VVCDEC_H +#define AVCODEC_VVCDEC_H + +#include "libavcodec/h2645_parse.h" +#include "libavcodec/threadframe.h" +#include "libavcodec/videodsp.h" +#include "libavcodec/vvc.h" + +#include "vvc_executor.h" + +#define LUMA 0 +#define CHROMA 1 +#define CB 1 +#define CR 2 +#define JCBCR 3 + +#define MAX_CTU_SIZE 128 + +#define MAX_CU_SIZE MAX_CTU_SIZE +#define MIN_CU_SIZE 4 +#define MIN_CU_LOG2 2 +#define MAX_CU_DEPTH 7 + +#define MIN_PU_SIZE 4 +#define MIN_PU_LOG2 2 + +#define MAX_TB_SIZE 64 +#define MIN_TU_LOG2 2 ///< MinTbLog2SizeY +#define MIN_TU_SIZE 4 +#define MAX_TUS_IN_CU 64 + +#define MAX_PARTS_IN_CTU ((MAX_CTU_SIZE >> MIN_CU_LOG2) * (MAX_CTU_SIZE >> MIN_CU_LOG2)) + +#define MAX_CONTROL_POINTS 3 + +#define MRG_MAX_NUM_CANDS 6 +#define MAX_NUM_HMVP_CANDS 5 + +#define L0 0 +#define L1 1 + +#define CHROMA_EXTRA_BEFORE 1 +#define CHROMA_EXTRA_AFTER 2 +#define CHROMA_EXTRA 3 +#define LUMA_EXTRA_BEFORE 3 +#define LUMA_EXTRA_AFTER 4 +#define LUMA_EXTRA 7 +#define BILINEAR_EXTRA_BEFORE 0 +#define BILINEAR_EXTRA_AFTER 1 +#define BILINEAR_EXTRA 1 + +#define MAX_QP 63 +#define DEFAULT_INTRA_TC_OFFSET 2 + +#define SAO_PADDING_SIZE 1 + +#define ALF_PADDING_SIZE 8 +#define ALF_BLOCK_SIZE 4 + +#define ALF_BORDER_LUMA 3 +#define ALF_BORDER_CHROMA 2 + +#define ALF_VB_POS_ABOVE_LUMA 4 +#define ALF_VB_POS_ABOVE_CHROMA 2 + +#define ALF_GRADIENT_STEP 2 +#define ALF_GRADIENT_BORDER 2 +#define ALF_GRADIENT_SIZE ((MAX_CU_SIZE + ALF_GRADIENT_BORDER * 2) / ALF_GRADIENT_STEP) +#define ALF_NUM_DIR 4 + +#define MAX_PB_SIZE 128 +#define EDGE_EMU_BUFFER_STRIDE (MAX_PB_SIZE + 32) + +#define AFFINE_MIN_BLOCK_SIZE 4 +#define PROF_BORDER_EXT 1 +#define PROF_BLOCK_SIZE (AFFINE_MIN_BLOCK_SIZE + PROF_BORDER_EXT * 2) +#define BDOF_BORDER_EXT 1 + +#define BDOF_PADDED_SIZE (16 + BDOF_BORDER_EXT * 2) +#define BDOF_BLOCK_SIZE 4 +#define BDOF_GRADIENT_SIZE (BDOF_BLOCK_SIZE + BDOF_BORDER_EXT * 2) + +/** + * Value of the luma sample at position (x, y) in the 2D array tab. + */ +#define SAMPLE(tab, x, y) ((tab)[(y) * s->sps->width + (x)]) +#define SAMPLE_CTB(tab, x, y) ((tab)[(y) * min_cb_width + (x)]) +#define CTB(tab, x, y) ((tab)[(y) * fc->ps.pps->ctb_width + (x)]) + +typedef struct VVCLocalContext VVCLocalContext; +typedef struct SliceContext SliceContext; +typedef struct VVCFrameContext VVCFrameContext; +typedef struct VVCFrameThread VVCFrameThread; +typedef struct EntryPoint EntryPoint; +typedef struct VVCTask VVCTask; +typedef struct Mv Mv; +typedef struct MvField MvField; +typedef struct DMVRInfo DMVRInfo; +typedef struct CTU CTU; +typedef struct SAOParams SAOParams; +typedef struct ALFParams ALFParams; + +typedef struct RefPicList { + struct VVCFrame *ref[VVC_MAX_REF_ENTRIES]; + int list[VVC_MAX_REF_ENTRIES]; + int isLongTerm[VVC_MAX_REF_ENTRIES]; + int nb_refs; +} RefPicList; + +typedef struct RefPicListTab { + RefPicList refPicList[2]; +} RefPicListTab; + +typedef struct VVCFrame { + AVFrame *frame; + ThreadFrame tf; + + MvField *tab_mvf; + RefPicList *refPicList; + RefPicListTab **rpl_tab; + + int ctb_count; + + int poc; + + struct VVCFrame *collocated_ref; + + AVBufferRef *tab_mvf_buf; + AVBufferRef *rpl_tab_buf; + AVBufferRef *rpl_buf; + AVBufferRef *progress_buf; + + /** + * A sequence counter, so that old frames are output first + * after a POC reset + */ + uint16_t sequence; + /** + * A combination of VVC_FRAME_FLAG_* + */ + uint8_t flags; +} VVCFrame; + +struct SliceContext { + int slice_idx; + EntryPoint *eps; + int nb_eps; +}; + +struct VVCFrameContext { + AVCodecContext *avctx; + + // +1 for the current frame + VVCFrame DPB[VVC_MAX_DPB_SIZE + 1]; + + AVFrame *frame; + AVFrame *output_frame; + + SliceContext **slices; + int nb_slices; + int nb_slices_allocated; + + VVCFrame *ref; + + VideoDSPContext vdsp; + + VVCFrameThread *frame_thread; + + uint64_t decode_order; + + AVPacket *avpkt; + H2645Packet pkt; + + AVBufferPool *tab_mvf_pool; + AVBufferPool *rpl_tab_pool; + + AVBufferPool *cu_pool; + + struct { + int16_t *slice_idx; + + DMVRInfo *dmvr; + + int *cb_pos_x[2]; ///< CbPosX[][][] + int *cb_pos_y[2]; ///< CbPosY[][][] + uint8_t *cb_width[2]; ///< CbWidth[][][] + uint8_t *cb_height[2]; ///< CbHeight[][][] + uint8_t *cqt_depth[2]; ///< CqtDepth[][][] + int8_t *qp[VVC_MAX_SAMPLE_ARRAYS]; + + uint8_t *skip; ///< CuSkipFlag[][] + uint8_t *ispmf; ///< intra_sub_partitions_mode_flag + uint8_t *msm[2]; ///< MttSplitMode[][][] in 32 pixels + uint8_t *imf; ///< IntraMipFlag[][] + uint8_t *imtf; ///< intra_mip_transposed_flag[][] + uint8_t *imm; ///< intra_mip_mode[][] + uint8_t *ipm; ///< IntraPredModeY[][] + uint8_t *cpm[2]; ///< CuPredMode[][][] + uint8_t *msf; ///< MergeSubblockFlag[][] + uint8_t *iaf; ///< InterAffineFlag[][] + uint8_t *mmi; ///< MotionModelIdc[][] + Mv *cp_mv[2]; ///< CpMvLX[][][][MAX_CONTROL_POINTS]; + + uint8_t *tu_coded_flag[VVC_MAX_SAMPLE_ARRAYS]; ///< tu_y_coded_flag[][], tu_cb_coded_flag[][], tu_cr_coded_flag[][] + uint8_t *tu_joint_cbcr_residual_flag; ///< tu_joint_cbcr_residual_flag[][] + int *tb_pos_x0[2]; + int *tb_pos_y0[2]; + uint8_t *tb_width[2]; + uint8_t *tb_height[2]; + uint8_t *pcmf[2]; + + uint8_t *horizontal_bs[VVC_MAX_SAMPLE_ARRAYS]; + uint8_t *vertical_bs[VVC_MAX_SAMPLE_ARRAYS]; + uint8_t *horizontal_p; ///< horizontal maxFilterLengthPs for luma + uint8_t *horizontal_q; ///< horizontal maxFilterLengthPs for luma + uint8_t *vertical_p; ///< vertical maxFilterLengthQs for luma + uint8_t *vertical_q; ///< vertical maxFilterLengthQs for luma + + uint8_t *sao_pixel_buffer_h[VVC_MAX_SAMPLE_ARRAYS]; + uint8_t *sao_pixel_buffer_v[VVC_MAX_SAMPLE_ARRAYS]; + uint8_t *alf_pixel_buffer_h[VVC_MAX_SAMPLE_ARRAYS][2]; + uint8_t *alf_pixel_buffer_v[VVC_MAX_SAMPLE_ARRAYS][2]; + + int *coeffs; + CTU *ctus; + + //used in arrays_init only + int ctu_count; + int ctu_size; + int pic_size_in_min_cb; + int pic_size_in_min_pu; + int pic_size_in_min_tu; + int ctu_width; + int ctu_height; + int width; + int height; + int chroma_format_idc; + int pixel_shift; + int bs_width; + int bs_height; + } tab; +} ; + +typedef struct VVCContext { + const AVClass *c; // needed by private avoptions + AVCodecContext *avctx; + + int temporal_id; ///< temporal_id_plus1 - 1 + int pocTid0; + + int eos; ///< current packet contains an EOS/EOB NAL + int last_eos; ///< last packet contains an EOS/EOB NAL + + + enum VVCNALUnitType vcl_unit_type; + int no_output_before_recovery_flag; ///< NoOutputBeforeRecoveryFlag + int gdr_recovery_point_poc; ///< recoveryPointPocVal + + /** + * Sequence counters for decoded and output frames, so that old + * frames are output first after a POC reset + */ + uint16_t seq_decode; + uint16_t seq_output; + + int is_nalff; ///< this flag is != 0 if bitstream is encapsulated + ///< as a format defined in 14496-15 + + int apply_defdispwin; + int nal_length_size; ///< Number of bytes used for nal length (1, 2 or 4) + + VVCExecutor *executor; + + VVCFrameContext *fcs; + int nb_fcs; + + uint64_t nb_frames; ///< processed frames + int nb_delayed; ///< delayed frames +} VVCContext ; + +#endif /* AVCODEC_VVCDEC_H */ -- 2.25.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".