On Tue, Nov 15, 2022 at 02:30:23PM -0500, Jeremy Dorfman wrote:
> Large rgb2yuv tables and high pixel values cause the intermediate
> int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined
> behavior. This causes libswscale built with LLVM -fsanitize=undefined to
> assert. Using unsigned integers instead has defined behavior and
> produces identical results, and makes rgb64ToUV_c_template match
> rgb64ToY_c_template.
> 
> Fixes: signed integer overflow
> ---
>  libswscale/input.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

There are more cases which should be affected by this too.
I sent a more complete patchset

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Observe your enemies, for they first find out your faults. -- Antisthenes