[FFmpeg-devel] [PATCH] swscale/ops_internal: fix ff_sws_pack_op_decode() (PR #21263)

[Niklas Haas via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

PR #21263 opened by Niklas Haas (haasn)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21263
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21263.patch

This function was assuming that the bits are MSB-aligned, but they are
LSB-aligned in both practice (and in the actual backend).

Also update the documentation of SwsPackOp to make this clearer.

Fixes an incorrect omission of a clamp after decoding e.g. rgb4, since
the max value range was incorrectly determined as 0 as a result of unpacking
the MSB bits instead of the LSB bits:

 bgr4 -> gray:
   [ u8 XXXX -> +XXX] SWS_OP_READ         : 1 elem(s) packed >> 1
   [ u8 .XXX -> +++X] SWS_OP_UNPACK       : {1 2 1 0}
   [ u8 ...X -> +++X] SWS_OP_SWIZZLE      : 2103
   [ u8 ...X -> +++X] SWS_OP_CONVERT      : u8 -> f32
   [f32 ...X -> .++X] SWS_OP_LINEAR       : dot3 [...]
   [f32 .XXX -> .++X] SWS_OP_DITHER       : 16x16 matrix + {0 3 2 5}
+  [f32 .XXX -> .++X] SWS_OP_MIN          : x <= {255 _ _ _}
   [f32 .XXX -> +++X] SWS_OP_CONVERT      : f32 -> u8
   [ u8 .XXX -> +++X] SWS_OP_WRITE        : 1 elem(s) planar >> 0
     (X = unused, + = exact, 0 = zero)


>From d93ed0e7ada69c32541a0688d50527cb62a68637 Mon Sep 17 00:00:00 2001
From: Niklas Haas <git@haasn.dev>
Date: Mon, 22 Dec 2025 15:31:01 +0100
Subject: [PATCH] swscale/ops_internal: fix ff_sws_pack_op_decode()

This function was assuming that the bits are MSB-aligned, but they are
LSB-aligned in both practice (and in the actual backend).

Also update the documentation of SwsPackOp to make this clearer.

Fixes an incorrect omission of a clamp after decoding e.g. rgb4, since
the max value range was incorrectly determined as 0 as a result of unpacking
the MSB bits instead of the LSB bits:

 bgr4 -> gray:
   [ u8 XXXX -> +XXX] SWS_OP_READ         : 1 elem(s) packed >> 1
   [ u8 .XXX -> +++X] SWS_OP_UNPACK       : {1 2 1 0}
   [ u8 ...X -> +++X] SWS_OP_SWIZZLE      : 2103
   [ u8 ...X -> +++X] SWS_OP_CONVERT      : u8 -> f32
   [f32 ...X -> .++X] SWS_OP_LINEAR       : dot3 [...]
   [f32 .XXX -> .++X] SWS_OP_DITHER       : 16x16 matrix + {0 3 2 5}
+  [f32 .XXX -> .++X] SWS_OP_MIN          : x <= {255 _ _ _}
   [f32 .XXX -> +++X] SWS_OP_CONVERT      : f32 -> u8
   [ u8 .XXX -> +++X] SWS_OP_WRITE        : 1 elem(s) planar >> 0
     (X = unused, + = exact, 0 = zero)
---
 libswscale/ops.h            | 4 ++++
 libswscale/ops_internal.h   | 4 +++-
 tests/ref/fate/sws-ops-list | 2 +-
 3 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/libswscale/ops.h b/libswscale/ops.h
index 6392d0ffdf..6fc7e60a02 100644
--- a/libswscale/ops.h
+++ b/libswscale/ops.h
@@ -109,6 +109,10 @@ typedef struct SwsReadWriteOp {
 } SwsReadWriteOp;
 
 typedef struct SwsPackOp {
+    /**
+     * Packed bits are assumed to be LSB-aligned within the underlying
+     * integer type; i.e. (msb) 0 ... X Y Z W (lsb).
+     */
     uint8_t pattern[4]; /* bit depth pattern, from MSB to LSB */
 } SwsPackOp;
 
diff --git a/libswscale/ops_internal.h b/libswscale/ops_internal.h
index 0071f78558..ba4c9b39da 100644
--- a/libswscale/ops_internal.h
+++ b/libswscale/ops_internal.h
@@ -39,7 +39,9 @@ static inline AVRational ff_sws_pixel_expand(SwsPixelType from, SwsPixelType to)
 
 static inline void ff_sws_pack_op_decode(const SwsOp *op, uint64_t mask[4], int shift[4])
 {
-    const int size = ff_sws_pixel_type_size(op->type) * 8;
+    int size = 0;
+    for (int i = 0; i < 4; i++)
+        size += op->pack.pattern[i];
     for (int i = 0; i < 4; i++) {
         const int bits = op->pack.pattern[i];
         mask[i] = (UINT64_C(1) << bits) - 1;
diff --git a/tests/ref/fate/sws-ops-list b/tests/ref/fate/sws-ops-list
index b49f944794..a7d6149d8b 100644
--- a/tests/ref/fate/sws-ops-list
+++ b/tests/ref/fate/sws-ops-list
@@ -1 +1 @@
-e910ff7ceaeb64bfdbac3f652b67403f
+ef1dd10af970984495f6008e43d0fe1b
-- 
2.49.1

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org