[FFmpeg-devel] [PATCH] avfilter/x86/vf_colordetect: fix alpha detect tail handling (PR #20387)

[Niklas Haas via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

PR #20387 opened by Niklas Haas (haasn)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20387
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20387.patch

This wrapping logic still considered any nonzero return from the ASM function
to be the overall result, but this is not true since the addition of
FF_ALPHA_TRANSPARENT.

Fix it by only early returning if FF_ALPHA_STRAIGHT is detected.

Fixes: 9b8b78a8153f9ca94b7eb9cba87442954787912a
See-Also: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20301#issuecomment-4802


>From 9931f416a09941ca5f415f5a5626847760ca7850 Mon Sep 17 00:00:00 2001
From: Niklas Haas <git@haasn.dev>
Date: Mon, 1 Sep 2025 16:39:00 +0200
Subject: [PATCH] avfilter/x86/vf_colordetect: fix alpha detect tail handling

This wrapping logic still considered any nonzero return from the ASM function
to be the overall result, but this is not true since the addition of
FF_ALPHA_TRANSPARENT.

Fix it by only early returning if FF_ALPHA_STRAIGHT is detected.

Fixes: 9b8b78a8153f9ca94b7eb9cba87442954787912a
See-Also: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20301#issuecomment-4802
---
 libavfilter/x86/vf_colordetect_init.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/libavfilter/x86/vf_colordetect_init.c b/libavfilter/x86/vf_colordetect_init.c
index 828f01937d..72fa021bf2 100644
--- a/libavfilter/x86/vf_colordetect_init.c
+++ b/libavfilter/x86/vf_colordetect_init.c
@@ -31,11 +31,11 @@ static int FUNC_NAME(const uint8_t *src, ptrdiff_t stride,
 {                                                                               \
     ptrdiff_t bytes = (width << SHIFT) & ~(MMSIZE - 1);                         \
     int ret = ASM_FUNC_NAME(src, stride, bytes, height, min, max);              \
-    if (ret)                                                                    \
+    if (ret == FF_ALPHA_STRAIGHT)                                               \
         return ret;                                                             \
                                                                                 \
-    return C_FUNC_NAME(src + bytes, stride, width - (bytes >> SHIFT),           \
-                       height, min, max);                                       \
+    return ret | C_FUNC_NAME(src + bytes, stride, width - (bytes >> SHIFT),     \
+                             height, min, max);                                 \
 }
 
 #define DETECT_ALPHA_FUNC(FUNC_NAME, ASM_FUNC_NAME, C_FUNC_NAME, SHIFT, MMSIZE) \
@@ -50,11 +50,12 @@ static int FUNC_NAME(const uint8_t *color, ptrdiff_t color_stride,
     ptrdiff_t bytes = (width << SHIFT) & ~(MMSIZE - 1);                         \
     int ret = ASM_FUNC_NAME(color, color_stride, alpha, alpha_stride,           \
                             bytes, height, p, q, k);                            \
-    if (ret)                                                                    \
+    if (ret == FF_ALPHA_STRAIGHT)                                               \
         return ret;                                                             \
                                                                                 \
-    return C_FUNC_NAME(color + bytes, color_stride, alpha + bytes, alpha_stride,\
-                       width - (bytes >> SHIFT), height, p, q, k);              \
+    return ret | C_FUNC_NAME(color + bytes, color_stride, alpha + bytes,        \
+                             alpha_stride, width - (bytes >> SHIFT), height,    \
+                             p, q, k);                                          \
 }
 
 #if HAVE_X86ASM
-- 
2.49.1

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org