Re: [FFmpeg-devel] [PATCH] fate.sh: Allow overriding what targets to make for running the tests

From: Alexander Strasser <eclipse7@gmx.net>
To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Subject: Re: [FFmpeg-devel] [PATCH] fate.sh: Allow overriding what targets to make for running the tests
Date: Mon, 27 Nov 2023 23:10:57 +0100
Message-ID: <ZWUT8Wjehl4jdgZ5@metallschleimette> (raw)
In-Reply-To: <12696030.Brfu6Qr6Id@basile.remlab.net>

Hi Martin,

patch looks technically good to me.

On 2023-11-27 17:46 +0200, Rémi Denis-Courmont wrote:
> Le maanantaina 27. marraskuuta 2023, 14.31.18 EET Martin Storsjö a écrit :
> > This can be useful if doing testing of uncommon CPU extensions by
> > running tests with QEMU (by configuring with e.g.
> > "target_exec=qemu-aarch64"), by only running the checkasm tests,
> > to get a reasonable test coverage without excessive test runtime.
>
> For the purpose of testing future or bleeding-edge CPU extensions on emulator,
> you would normally want to be able to actually filter those in. That is more of
> a matter of patching checkasm than FATE.
>
> Considering the poor coverage of checkasm, I fear that this just gives the
> wrong impression, not to say a false sense of security. It feels misleading to
> encourage or support that paradigm into FATE, in light of that poor coverage.
> Afterall, if it's just about running checkasm, anybody can just run
> `make tests/checkasm/checkasm && tests/checkasm/checkasm`.

Agree, the practice should not be encouraged over cases where it is
necessary to be practical, but having fate clients run the tests and
submit the results seems still worthwhile to me.

We shouldn't get a false sense of security and it can be a problem.
Improving checkasm and qemu and maybe getting actual bleeding edge
hardware into fate could help with that. Not saying it is easy or
effortless...

Still having tests run and some failures detected seems better
than nothing to me. It would be only problematic if the false
positives or negatives weigh out the cases where it is helpful.

@Remi: Please take no offense. This does not look like a black
and white issue to me , thus I wanted to voice what I thought
when initially saw the patch and your response.

Best regards,
  Alexander

> Either way, this feels like a case of cart before horse.
>
> Also FWIW, RV broke due to misaligned accesses and illegal vector types that
> QEMU tolerated. That is rather an argument against QEMU than against this MR
> but still.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".