From: Kieran Kunhya <kierank@obe.tv>
To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Subject: Re: [FFmpeg-devel] [PATCH] fate: add VVC decoder tests
Date: Sat, 6 Jan 2024 02:41:28 +0000
Message-ID: <CAK+ULv4282QDBVdO4m_NT=H5e6-7+bHo-zsb-7sXx_PomP+FMw@mail.gmail.com> (raw)
In-Reply-To: <CAFXK13epCnppjQdeL5=49kOfMca9f0gaQzCTq4D_VTdrZULtBQ@mail.gmail.com>
On Sat, 6 Jan 2024 at 02:35, Nuo Mi <nuomi2021@gmail.com> wrote:
> On Sat, Jan 6, 2024 at 9:13 AM James Almer <jamrial@gmail.com> wrote:
>
> > On 1/5/2024 10:09 PM, Nuo Mi wrote:
> > > On Sat, Jan 6, 2024 at 5:09 AM James Almer <jamrial@gmail.com> wrote:
> > >
> > >> On 1/3/2024 9:19 PM, James Almer wrote:
> > >>> Signed-off-by: James Almer <jamrial@gmail.com>
> > >>> ---
> > >>> tests/Makefile | 1 +
> > >>> tests/fate/vvc.mak | 50 +++++++
> > >>> tests/ref/fate/vvc-conformance-APSALF_A_2 | 13 ++
> > >>> tests/ref/fate/vvc-conformance-APSLMCS_D_1 | 37 +++++
> > >>> tests/ref/fate/vvc-conformance-APSMULT_A_4 | 53 +++++++
> > >>> tests/ref/fate/vvc-conformance-AUD_A_3 | 35 +++++
> > >>> tests/ref/fate/vvc-conformance-BUMP_A_2 | 45 ++++++
> > >>> tests/ref/fate/vvc-conformance-CROP_B_4 | 105 ++++++++++++++
> > >>> .../fate/vvc-conformance-CodingToolsSets_A_2 | 7 +
> > >>> tests/ref/fate/vvc-conformance-DCI_A_3 | 7 +
> > >>> tests/ref/fate/vvc-conformance-HRD_A_3 | 65 +++++++++
> > >>> tests/ref/fate/vvc-conformance-OPI_B_3 | 6 +
> > >>> tests/ref/fate/vvc-conformance-PHSH_B_1 | 11 ++
> > >>> tests/ref/fate/vvc-conformance-POC_A_1 | 25 ++++
> > >>> tests/ref/fate/vvc-conformance-PPS_B_1 | 69 +++++++++
> > >>> tests/ref/fate/vvc-conformance-RAP_A_1 | 6 +
> > >>> tests/ref/fate/vvc-conformance-SAO_A_3 | 65 +++++++++
> > >>> tests/ref/fate/vvc-conformance-SCALING_A_1 | 69 +++++++++
> > >>> tests/ref/fate/vvc-conformance-SLICES_A_3 | 30 ++++
> > >>> tests/ref/fate/vvc-conformance-SPS_B_1 | 133
> > ++++++++++++++++++
> > >>> tests/ref/fate/vvc-conformance-STILL_B_1 | 10 ++
> > >>> tests/ref/fate/vvc-conformance-SUBPIC_A_3 | 9 ++
> > >>> tests/ref/fate/vvc-conformance-TILE_A_2 | 14 ++
> > >>> tests/ref/fate/vvc-conformance-VPS_A_3 | 6 +
> > >>> tests/ref/fate/vvc-conformance-WPP_A_3 | 54 +++++++
> > >>> tests/ref/fate/vvc-conformance-WP_A_3 | 22 +++
> > >>> tests/ref/fate/vvc-conformance-WRAP_A_4 | 14 ++
> > >>> 27 files changed, 961 insertions(+)
> > >>> create mode 100644 tests/fate/vvc.mak
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-APSALF_A_2
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-APSLMCS_D_1
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-APSMULT_A_4
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-AUD_A_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-BUMP_A_2
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-CROP_B_4
> > >>> create mode 100644
> > tests/ref/fate/vvc-conformance-CodingToolsSets_A_2
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-DCI_A_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-HRD_A_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-OPI_B_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-PHSH_B_1
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-POC_A_1
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-PPS_B_1
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-RAP_A_1
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-SAO_A_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-SCALING_A_1
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-SLICES_A_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-SPS_B_1
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-STILL_B_1
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-SUBPIC_A_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-TILE_A_2
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-VPS_A_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-WPP_A_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-WP_A_3
> > >>> create mode 100644 tests/ref/fate/vvc-conformance-WRAP_A_4
> > >>
> > >> This is using the samples already in the fate suite. I don't know if
> > >> they give good coverage (I think i chose them to cover as much CBS
> > >> methods as possible, not caring about actual decode paths, and not
> even
> > >> that is complete), so I'd like someone more familiar with the
> > >> conformance suite to suggest better samples to upload and use, too.
> > >>
> > >
> > > Here are the clips and their sources:
> > > https://github.com/ffvvc/tests/tree/main
> > > passed v1 is about 194M , and each clip may have different features.
> It's
> > > better to add them all to vvc-conformance, without name change
> > > passed v2 is about 871M , we can pick some smaller clips that cover all
> > bit
> > > depths and color formats.
> >
> > Yeah, it's not so much about the size of the samples than it is about
> > runtime. That many tests would make a fate run take way too much, even
> > after we add assembly.
> > We need to choose a subset that cover as many decoding cases as possible.
> >
> Good point.
> If I disable the asm code on my machine, "make fate-hevc" takes about 1
> minute and 26 seconds. We can use this as the baseline.
> If we remove the largest 5 files in v1, we can decode it in 5 minutes and
> 13 seconds. We can remove more if we find it's still slow.
> From a feature perspective, TREE_A_HHI_3.bit may cover major features in
> TREE_B_HHI_3.bit, TREE_C_HHI_3.bit. It's reasonable to remove them
>
> Largest 5 files:
> TRANS_B_Chipsnmedia_2.bit
> TRANS_A_Chipsnmedia_2.bit
> TREE_B_HHI_3.bit
> LFNST_D_HHI_3.bit
> TREE_C_HHI_3.bit
>
Could we have an option in the FATE config file to do a full run and allow
admins to turn it on at will?
I appreciate there are a lot of embedded devices where running all the VVC
decodes will take forever. But at the same time there are powerful devices
like M1, NUC etc where this isn't a big deal and I'd like to see VVC tested
in full.
Kieran
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
next prev parent reply other threads:[~2024-01-06 2:41 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-04 0:19 James Almer
2024-01-04 11:30 ` Nuo Mi
2024-01-05 21:09 ` James Almer
2024-01-06 1:09 ` Nuo Mi
2024-01-06 1:13 ` James Almer
2024-01-06 2:34 ` Nuo Mi
2024-01-06 2:41 ` Kieran Kunhya [this message]
2024-01-06 10:10 ` Martin Storsjö
2024-01-07 4:33 ` Kieran Kunhya
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAK+ULv4282QDBVdO4m_NT=H5e6-7+bHo-zsb-7sXx_PomP+FMw@mail.gmail.com' \
--to=kierank@obe.tv \
--cc=ffmpeg-devel@ffmpeg.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
This inbox may be cloned and mirrored by anyone:
git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \
ffmpegdev@gitmailbox.com
public-inbox-index ffmpegdev
Example config snippet for mirrors.
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git