[FFmpeg-devel] [PATCH] lavc/vvc: Only read split_cu_flag if a split is allowed

[Frank Plowman <post@frankplowman.com>]

Add a check to ensure some split is possible before reading the
split_cu_flag.  This is present in the spec, in VVCv3 section 7.3.11.4.
Its omission could lead to infinite loops and ultimately crashing due to
stack overflow.
---
 libavcodec/vvc/vvc_ctu.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/libavcodec/vvc/vvc_ctu.c b/libavcodec/vvc/vvc_ctu.c
index 8ba12c8d9f..32d8bc8f5c 100644
--- a/libavcodec/vvc/vvc_ctu.c
+++ b/libavcodec/vvc/vvc_ctu.c
@@ -2095,6 +2095,7 @@ static int hls_coding_tree(VVCLocalContext *lc,
     const int ch_type               = tree_type_curr == DUAL_TREE_CHROMA;
     int ret;
     VVCAllowedSplit allowed;
+    int split_cu_flag;
 
     if (pps->r->pps_cu_qp_delta_enabled_flag && qg_on_y && cb_sub_div <= sh->cu_qp_delta_subdiv) {
         lc->parse.is_cu_qp_delta_coded = 0;
@@ -2109,7 +2110,11 @@ static int hls_coding_tree(VVCLocalContext *lc,
 
     can_split(lc, x0, y0, cb_width, cb_height, mtt_depth, depth_offset, part_idx,
         last_split_mode, tree_type_curr, mode_type_curr, &allowed);
-    if (ff_vvc_split_cu_flag(lc, x0, y0, cb_width, cb_height, ch_type, &allowed)) {
+    if (allowed.btv || allowed.bth || allowed.ttv || allowed.tth || allowed.qt)
+        split_cu_flag = ff_vvc_split_cu_flag(lc, x0, y0, cb_width, cb_height, ch_type, &allowed);
+    else
+        split_cu_flag = 0;
+    if (split_cu_flag) {
         VVCSplitMode split      = ff_vvc_split_mode(lc, x0, y0, cb_width, cb_height, cqt_depth, mtt_depth, ch_type, &allowed);
         VVCModeType mode_type   = mode_type_decode(lc, x0, y0, cb_width, cb_height, split, ch_type, mode_type_curr);
 
-- 
2.44.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".