[FFmpeg-devel] [PATCH 0/5] replace scale2ref by scale=rw:rh

[FFmpeg-devel] [PATCH 0/5] replace scale2ref by scale=rw:rh

[Niklas Haas]

As discussed in my previous series for fixing scale2ref[1], this filter
is fundamentally broken, and the only real fix would be to switch to
activate(), or ideally FFFrameSync.

[1] https://ffmpeg.org//pipermail/ffmpeg-devel/2024-March/323382.html

The main thing making this difficult is the fact that scale2ref also
wants to output ref frames to its secondary output, which FFFrameSync
does not support, and which is ultimately at least part of the root
cause of trac #10795.

Since this is in principle completely unnecessary (users can just
'split' the ref input and have it be consumed by vf_scale), and to make
the design of this filter a bit more robust and maintainable, switch to
an approach where vf_scale itself gains the ability to reference
a secondary input stream, using the "ref_*" series of variables.

This makes the current [i][ri]scale2ref[o][ro] equivalent to the only
slightly more verbose [ri]split[t][ro]; [i][t]scale=rw:rh[o]. (And
conversely, it is no longer necessary to use nullsink to consume an
unused [ro])

Incidentally, I think it would be nice if lavfi could *automatically*
split filter links referenced multiple times, so we could just write
e.g. [i][ri]scale=rw:rh[o]; [ri][o]overlay[out] and have it work. Maybe
I'll try getting that to work, next.

Either way, after the deprecation period has elapsed, we can delete
scale2ref from the codebase in order to make vf_scale.c IMHO
significantly simpler versus the status quo.

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 1/5] avfilter/framesync: make framesync_class un-static

[Niklas Haas]

From: Niklas Haas <git@haasn.dev>

And rename to ff_framesync_class. More convenient for downstream users.
---
 libavfilter/framesync.c | 6 +++---
 libavfilter/framesync.h | 1 +
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/libavfilter/framesync.c b/libavfilter/framesync.c
index 1afd70ab21..a691136f34 100644
--- a/libavfilter/framesync.c
+++ b/libavfilter/framesync.c
@@ -51,7 +51,7 @@ static const AVOption framesync_options[] = {
             0, AV_OPT_TYPE_CONST, { .i64 = TS_NEAREST }, .flags = FLAGS, .unit = "ts_sync_mode" },
     { NULL }
 };
-static const AVClass framesync_class = {
+const AVClass ff_framesync_class = {
     .version                   = LIBAVUTIL_VERSION_INT,
     .class_name                = "framesync",
     .item_name                 = framesync_name,
@@ -62,7 +62,7 @@ static const AVClass framesync_class = {
 
 const AVClass *ff_framesync_child_class_iterate(void **iter)
 {
-    const AVClass *c = *iter ? NULL : &framesync_class;
+    const AVClass *c = *iter ? NULL : &ff_framesync_class;
     *iter = (void *)(uintptr_t)c;
     return c;
 }
@@ -79,7 +79,7 @@ void ff_framesync_preinit(FFFrameSync *fs)
 {
     if (fs->class)
         return;
-    fs->class  = &framesync_class;
+    fs->class  = &ff_framesync_class;
     av_opt_set_defaults(fs);
 }
 
diff --git a/libavfilter/framesync.h b/libavfilter/framesync.h
index 233f50a0eb..130d067bae 100644
--- a/libavfilter/framesync.h
+++ b/libavfilter/framesync.h
@@ -316,6 +316,7 @@ int ff_framesync_dualinput_get(FFFrameSync *fs, AVFrame **f0, AVFrame **f1);
 int ff_framesync_dualinput_get_writable(FFFrameSync *fs, AVFrame **f0, AVFrame **f1);
 
 const AVClass *ff_framesync_child_class_iterate(void **iter);
+extern const AVClass ff_framesync_class;
 
 #define FRAMESYNC_DEFINE_PURE_CLASS(name, desc, func_prefix, options) \
 static const AVClass name##_class = {                                 \
-- 
2.44.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 2/5] avfilter/vf_scale: switch to FFFrameSync

[Niklas Haas]

From: Niklas Haas <git@haasn.dev>

Preliminary commit, in anticipation of adding support for multiple
inputs (with proper synchronization and activate() callback).
---
 doc/filters.texi       |  4 +--
 libavfilter/vf_scale.c | 65 +++++++++++++++++++++++++++++++++++++++---
 2 files changed, 63 insertions(+), 6 deletions(-)

diff --git a/doc/filters.texi b/doc/filters.texi
index fc813f12c1..f20b72ab96 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -20989,8 +20989,8 @@ the next filter, the scale filter will convert the input to the
 requested format.
 
 @subsection Options
-The filter accepts the following options, or any of the options
-supported by the libswscale scaler.
+The filter accepts the following options, any of the options supported
+by the libswscale scaler, as well as any of the @ref{framesync} options.
 
 See @ref{scaler_options,,the ffmpeg-scaler manual,ffmpeg-scaler} for
 the complete list of scaler options.
diff --git a/libavfilter/vf_scale.c b/libavfilter/vf_scale.c
index 1c07daeddf..a986dc97ae 100644
--- a/libavfilter/vf_scale.c
+++ b/libavfilter/vf_scale.c
@@ -29,6 +29,7 @@
 
 #include "avfilter.h"
 #include "formats.h"
+#include "framesync.h"
 #include "internal.h"
 #include "scale_eval.h"
 #include "video.h"
@@ -113,6 +114,7 @@ typedef struct ScaleContext {
     struct SwsContext *isws[2]; ///< software scaler context for interlaced material
     // context used for forwarding options to sws
     struct SwsContext *sws_opts;
+    FFFrameSync fs;
 
     /**
      * New dimensions. Special values are:
@@ -287,6 +289,8 @@ static av_cold int preinit(AVFilterContext *ctx)
     if (ret < 0)
         return ret;
 
+    ff_framesync_preinit(&scale->fs);
+
     return 0;
 }
 
@@ -302,6 +306,8 @@ static const int sws_colorspaces[] = {
     -1
 };
 
+static int do_scale(FFFrameSync *fs);
+
 static av_cold int init(AVFilterContext *ctx)
 {
     ScaleContext *scale = ctx->priv;
@@ -388,6 +394,7 @@ static av_cold void uninit(AVFilterContext *ctx)
     av_expr_free(scale->w_pexpr);
     av_expr_free(scale->h_pexpr);
     scale->w_pexpr = scale->h_pexpr = NULL;
+    ff_framesync_uninit(&scale->fs);
     sws_freeContext(scale->sws_opts);
     sws_freeContext(scale->sws);
     sws_freeContext(scale->isws[0]);
@@ -677,6 +684,21 @@ static int config_props(AVFilterLink *outlink)
            flags_val);
     av_freep(&flags_val);
 
+    if (ctx->filter != &ff_vf_scale2ref) {
+        ret = ff_framesync_init(&scale->fs, ctx, ctx->nb_inputs);
+        if (ret < 0)
+            return ret;
+        scale->fs.on_event        = do_scale;
+        scale->fs.in[0].time_base = ctx->inputs[0]->time_base;
+        scale->fs.in[0].sync      = 1;
+        scale->fs.in[0].before    = EXT_STOP;
+        scale->fs.in[0].after     = EXT_STOP;
+
+        ret = ff_framesync_configure(&scale->fs);
+        if (ret < 0)
+            return ret;
+    }
+
     return 0;
 
 fail:
@@ -894,6 +916,26 @@ scale:
     return ret;
 }
 
+static int do_scale(FFFrameSync *fs)
+{
+    AVFilterContext *ctx = fs->parent;
+    AVFilterLink *outlink = ctx->outputs[0];
+    AVFrame *in, *out;
+    int ret;
+
+    ret = ff_framesync_get_frame(fs, 0, &in, 1);
+    if (ret < 0)
+        return ret;
+
+    ret = scale_frame(ctx->inputs[0], in, &out);
+    if (out) {
+        out->pts = av_rescale_q(fs->pts, fs->time_base, outlink->time_base);
+        return ff_filter_frame(outlink, out);
+    }
+
+    return ret;
+}
+
 static int filter_frame(AVFilterLink *link, AVFrame *in)
 {
     AVFilterContext *ctx = link->dst;
@@ -972,11 +1014,24 @@ static int process_command(AVFilterContext *ctx, const char *cmd, const char *ar
     return ret;
 }
 
+static int activate(AVFilterContext *ctx)
+{
+    ScaleContext *scale = ctx->priv;
+    return ff_framesync_activate(&scale->fs);
+}
+
 static const AVClass *child_class_iterate(void **iter)
 {
-    const AVClass *c = *iter ? NULL : sws_get_class();
-    *iter = (void*)(uintptr_t)c;
-    return c;
+    switch ((uintptr_t) *iter) {
+    case 0:
+        *iter = (void*)(uintptr_t) 1;
+        return sws_get_class();
+    case 1:
+        *iter = (void*)(uintptr_t) 2;
+        return &ff_framesync_class;
+    }
+
+    return NULL;
 }
 
 static void *child_next(void *obj, void *prev)
@@ -984,6 +1039,8 @@ static void *child_next(void *obj, void *prev)
     ScaleContext *s = obj;
     if (!prev)
         return s->sws_opts;
+    if (prev == s->sws_opts)
+        return &s->fs;
     return NULL;
 }
 
@@ -1051,7 +1108,6 @@ static const AVFilterPad avfilter_vf_scale_inputs[] = {
     {
         .name         = "default",
         .type         = AVMEDIA_TYPE_VIDEO,
-        .filter_frame = filter_frame,
     },
 };
 
@@ -1074,6 +1130,7 @@ const AVFilter ff_vf_scale = {
     FILTER_INPUTS(avfilter_vf_scale_inputs),
     FILTER_OUTPUTS(avfilter_vf_scale_outputs),
     FILTER_QUERY_FUNC(query_formats),
+    .activate        = activate,
     .process_command = process_command,
 };
 
-- 
2.44.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 3/5] avfilter/vf_scale: add optional "ref" input

[Niklas Haas]

From: Niklas Haas <git@haasn.dev>

This is automatically enabled if the width/height expressions reference
any ref_* variable. This will ultimately serve as a more principled
replacement for the fundamentally broken scale2ref.

See-Also: https://trac.ffmpeg.org/ticket/10795
---
 Changelog              |   1 +
 doc/filters.texi       |  26 ++++++++
 libavfilter/vf_scale.c | 132 ++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 156 insertions(+), 3 deletions(-)

diff --git a/Changelog b/Changelog
index 8db14f02b4..e821e5ac74 100644
--- a/Changelog
+++ b/Changelog
@@ -7,6 +7,7 @@ version <next>:
 - ffmpeg CLI filtergraph chaining
 - LC3/LC3plus demuxer and muxer
 - pad_vaapi, drawbox_vaapi filters
+- vf_scale supports secondary ref input and framesync options
 
 
 version 7.0:
diff --git a/doc/filters.texi b/doc/filters.texi
index f20b72ab96..cf884568b0 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -21562,8 +21562,34 @@ The position (byte offset) of the frame in the input stream, or NaN if
 this information is unavailable and/or meaningless (for example in case of synthetic video).
 Only available with @code{eval=frame}.
 Deprecated, do not use.
+
+@item ref_w, rw
+@item ref_h, rh
+@item ref_a
+@item ref_dar, rdar
+@item ref_n
+@item ref_t
+@item ref_pos
+Eqvuialent to the above, but for a second reference input. If any of these
+variables are present, this filter accepts two inputs.
 @end table
 
+@subsection Examples
+
+@itemize
+@item
+Scale a subtitle stream (sub) to match the main video (main) in size before overlaying
+@example
+'[main]split[a][b]; [ref][a]scale=rw:rh[c]; [b][c]overlay'
+@end example
+
+@item
+Scale a logo to 1/10th the height of a video, while preserving its display aspect ratio.
+@example
+[logo-in][video-in]scale=w=oh*dar:h=rh/10[logo-out]
+@end example
+@end itemize
+
 @section scale2ref
 
 Scale (resize) the input video, based on a reference video.
diff --git a/libavfilter/vf_scale.c b/libavfilter/vf_scale.c
index a986dc97ae..f174651333 100644
--- a/libavfilter/vf_scale.c
+++ b/libavfilter/vf_scale.c
@@ -59,6 +59,17 @@ static const char *const var_names[] = {
 #if FF_API_FRAME_PKT
     "pos",
 #endif
+    "ref_w", "rw",
+    "ref_h", "rh",
+    "ref_a",
+    "ref_sar",
+    "ref_dar", "rdar",
+    "ref_hsub",
+    "ref_vsub",
+    "ref_n",
+    "ref_t",
+    "ref_pos",
+    /* Legacy variables for scale2ref */
     "main_w",
     "main_h",
     "main_a",
@@ -89,6 +100,16 @@ enum var_name {
 #if FF_API_FRAME_PKT
     VAR_POS,
 #endif
+    VAR_REF_W, VAR_RW,
+    VAR_REF_H, VAR_RH,
+    VAR_REF_A,
+    VAR_REF_SAR,
+    VAR_REF_DAR, VAR_RDAR,
+    VAR_REF_HSUB,
+    VAR_REF_VSUB,
+    VAR_REF_N,
+    VAR_REF_T,
+    VAR_REF_POS,
     VAR_S2R_MAIN_W,
     VAR_S2R_MAIN_H,
     VAR_S2R_MAIN_A,
@@ -131,6 +152,7 @@ typedef struct ScaleContext {
     int input_is_pal;           ///< set to 1 if the input format is paletted
     int output_is_pal;          ///< set to 1 if the output format is paletted
     int interlaced;
+    int uses_ref;
 
     char *w_expr;               ///< width  expression string
     char *h_expr;               ///< height expression string
@@ -190,6 +212,38 @@ static int check_exprs(AVFilterContext *ctx)
         av_log(ctx, AV_LOG_WARNING, "Circular references detected for width '%s' and height '%s' - possibly invalid.\n", scale->w_expr, scale->h_expr);
     }
 
+    if (vars_w[VAR_REF_W]    || vars_h[VAR_REF_W]    ||
+        vars_w[VAR_RW]       || vars_h[VAR_RW]       ||
+        vars_w[VAR_REF_H]    || vars_h[VAR_REF_H]    ||
+        vars_w[VAR_RH]       || vars_h[VAR_RH]       ||
+        vars_w[VAR_REF_A]    || vars_h[VAR_REF_A]    ||
+        vars_w[VAR_REF_SAR]  || vars_h[VAR_REF_SAR]  ||
+        vars_w[VAR_REF_DAR]  || vars_h[VAR_REF_DAR]  ||
+        vars_w[VAR_RDAR]     || vars_h[VAR_RDAR]     ||
+        vars_w[VAR_REF_HSUB] || vars_h[VAR_REF_HSUB] ||
+        vars_w[VAR_REF_VSUB] || vars_h[VAR_REF_VSUB] ||
+        vars_w[VAR_REF_N]    || vars_h[VAR_REF_N]    ||
+        vars_w[VAR_REF_T]    || vars_h[VAR_REF_T]    ||
+        vars_w[VAR_REF_POS]  || vars_h[VAR_REF_POS]) {
+        scale->uses_ref = 1;
+    }
+
+    if (ctx->filter != &ff_vf_scale2ref &&
+        (vars_w[VAR_S2R_MAIN_W]    || vars_h[VAR_S2R_MAIN_W]    ||
+         vars_w[VAR_S2R_MAIN_H]    || vars_h[VAR_S2R_MAIN_H]    ||
+         vars_w[VAR_S2R_MAIN_A]    || vars_h[VAR_S2R_MAIN_A]    ||
+         vars_w[VAR_S2R_MAIN_SAR]  || vars_h[VAR_S2R_MAIN_SAR]  ||
+         vars_w[VAR_S2R_MAIN_DAR]  || vars_h[VAR_S2R_MAIN_DAR]  ||
+         vars_w[VAR_S2R_MDAR]      || vars_h[VAR_S2R_MDAR]      ||
+         vars_w[VAR_S2R_MAIN_HSUB] || vars_h[VAR_S2R_MAIN_HSUB] ||
+         vars_w[VAR_S2R_MAIN_VSUB] || vars_h[VAR_S2R_MAIN_VSUB] ||
+         vars_w[VAR_S2R_MAIN_N]    || vars_h[VAR_S2R_MAIN_N]    ||
+         vars_w[VAR_S2R_MAIN_T]    || vars_h[VAR_S2R_MAIN_T]    ||
+         vars_w[VAR_S2R_MAIN_POS]  || vars_h[VAR_S2R_MAIN_POS]) ) {
+        av_log(ctx, AV_LOG_ERROR, "Expressions with scale2ref variables are not valid in scale filter.\n");
+        return AVERROR(EINVAL);
+    }
+
     if (ctx->filter != &ff_vf_scale2ref &&
         (vars_w[VAR_S2R_MAIN_W]    || vars_h[VAR_S2R_MAIN_W]    ||
          vars_w[VAR_S2R_MAIN_H]    || vars_h[VAR_S2R_MAIN_H]    ||
@@ -385,6 +439,9 @@ static av_cold int init(AVFilterContext *ctx)
     if (!threads)
         av_opt_set_int(scale->sws_opts, "threads", ff_filter_get_nb_threads(ctx), 0);
 
+    if (ctx->filter != &ff_vf_scale2ref)
+        ctx->nb_inputs = scale->uses_ref ? 2 : 1;
+
     return 0;
 }
 
@@ -506,6 +563,20 @@ static int scale_eval_dimensions(AVFilterContext *ctx)
         scale->var_values[VAR_S2R_MAIN_VSUB] = 1 << main_desc->log2_chroma_h;
     }
 
+    if (scale->uses_ref) {
+        const AVFilterLink *reflink = ctx->inputs[1];
+        const AVPixFmtDescriptor *ref_desc = av_pix_fmt_desc_get(reflink->format);
+        scale->var_values[VAR_REF_W] = scale->var_values[VAR_RW] = reflink->w;
+        scale->var_values[VAR_REF_H] = scale->var_values[VAR_RH] = reflink->h;
+        scale->var_values[VAR_REF_A] = (double) reflink->w / reflink->h;
+        scale->var_values[VAR_REF_SAR] = reflink->sample_aspect_ratio.num ?
+            (double) reflink->sample_aspect_ratio.num / reflink->sample_aspect_ratio.den : 1;
+        scale->var_values[VAR_REF_DAR] = scale->var_values[VAR_RDAR] =
+            scale->var_values[VAR_REF_A] * scale->var_values[VAR_REF_SAR];
+        scale->var_values[VAR_REF_HSUB] = 1 << ref_desc->log2_chroma_w;
+        scale->var_values[VAR_REF_VSUB] = 1 << ref_desc->log2_chroma_h;
+    }
+
     res = av_expr_eval(scale->w_pexpr, scale->var_values, NULL);
     eval_w = scale->var_values[VAR_OUT_W] = scale->var_values[VAR_OW] = (int) res == 0 ? inlink->w : (int) res;
 
@@ -693,6 +764,13 @@ static int config_props(AVFilterLink *outlink)
         scale->fs.in[0].sync      = 1;
         scale->fs.in[0].before    = EXT_STOP;
         scale->fs.in[0].after     = EXT_STOP;
+        if (scale->uses_ref) {
+            av_assert0(ctx->nb_inputs == 2);
+            scale->fs.in[1].time_base = ctx->inputs[1]->time_base;
+            scale->fs.in[1].sync      = 0;
+            scale->fs.in[1].before    = EXT_NULL;
+            scale->fs.in[1].after     = EXT_INFINITY;
+        }
 
         ret = ff_framesync_configure(&scale->fs);
         if (ret < 0)
@@ -919,13 +997,55 @@ scale:
 static int do_scale(FFFrameSync *fs)
 {
     AVFilterContext *ctx = fs->parent;
+    ScaleContext *scale = ctx->priv;
     AVFilterLink *outlink = ctx->outputs[0];
-    AVFrame *in, *out;
-    int ret;
+    AVFrame *out, *in = NULL, *ref = NULL;
+    int ret = 0, frame_changed;
 
     ret = ff_framesync_get_frame(fs, 0, &in, 1);
     if (ret < 0)
-        return ret;
+        goto err;
+
+    if (scale->uses_ref) {
+        ret = ff_framesync_get_frame(fs, 1, &ref, 0);
+        if (ret < 0)
+            goto err;
+    }
+
+    if (ref) {
+        AVFilterLink *reflink = ctx->inputs[1];
+        frame_changed = ref->width  != reflink->w ||
+                        ref->height != reflink->h ||
+                        ref->format != reflink->format ||
+                        ref->sample_aspect_ratio.den != reflink->sample_aspect_ratio.den ||
+                        ref->sample_aspect_ratio.num != reflink->sample_aspect_ratio.num ||
+                        ref->colorspace != reflink->colorspace ||
+                        ref->color_range != reflink->color_range;
+
+        if (frame_changed) {
+            reflink->format = ref->format;
+            reflink->w = ref->width;
+            reflink->h = ref->height;
+            reflink->sample_aspect_ratio.num = ref->sample_aspect_ratio.num;
+            reflink->sample_aspect_ratio.den = ref->sample_aspect_ratio.den;
+            reflink->colorspace = ref->colorspace;
+            reflink->color_range = ref->color_range;
+
+            ret = config_props(outlink);
+            if (ret < 0)
+                goto err;
+        }
+
+        if (scale->eval_mode == EVAL_MODE_FRAME) {
+            scale->var_values[VAR_REF_N] = reflink->frame_count_out;
+            scale->var_values[VAR_REF_T] = TS2T(ref->pts, reflink->time_base);
+#if FF_API_FRAME_PKT
+FF_DISABLE_DEPRECATION_WARNINGS
+            scale->var_values[VAR_REF_POS] = ref->pkt_pos == -1 ? NAN : ref->pkt_pos;
+FF_ENABLE_DEPRECATION_WARNINGS
+#endif
+        }
+    }
 
     ret = scale_frame(ctx->inputs[0], in, &out);
     if (out) {
@@ -933,6 +1053,9 @@ static int do_scale(FFFrameSync *fs)
         return ff_filter_frame(outlink, out);
     }
 
+err:
+    if (ret < 0)
+        av_frame_free(&in);
     return ret;
 }
 
@@ -1108,6 +1231,9 @@ static const AVFilterPad avfilter_vf_scale_inputs[] = {
     {
         .name         = "default",
         .type         = AVMEDIA_TYPE_VIDEO,
+    }, {
+        .name         = "ref",
+        .type         = AVMEDIA_TYPE_VIDEO,
     },
 };
 
-- 
2.44.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 4/5] fate/scale2ref_keep_aspect: switch to vf_scale ref_*

[Niklas Haas]

From: Niklas Haas <git@haasn.dev>

---
 tests/filtergraphs/scale2ref_keep_aspect | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/tests/filtergraphs/scale2ref_keep_aspect b/tests/filtergraphs/scale2ref_keep_aspect
index f407460ec7..00b04fc3d1 100644
--- a/tests/filtergraphs/scale2ref_keep_aspect
+++ b/tests/filtergraphs/scale2ref_keep_aspect
@@ -1,5 +1,4 @@
 sws_flags=+accurate_rnd+bitexact;
 testsrc=size=320x240 [main];
 testsrc=size=640x360 [ref];
-[main][ref] scale2ref=iw/4:ow/mdar [main][ref];
-[ref] nullsink
+[main][ref] scale=rw/4:ow/dar [main]
-- 
2.44.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 5/5] avfilter/scale2ref: deprecate in favor of scale=rw:rh

[Niklas Haas]

From: Niklas Haas <git@haasn.dev>

And remove it from the documentation.
---
 Changelog              |  1 +
 doc/filters.texi       | 73 ------------------------------------------
 libavfilter/vf_scale.c |  3 ++
 3 files changed, 4 insertions(+), 73 deletions(-)

diff --git a/Changelog b/Changelog
index e821e5ac74..e827208439 100644
--- a/Changelog
+++ b/Changelog
@@ -8,6 +8,7 @@ version <next>:
 - LC3/LC3plus demuxer and muxer
 - pad_vaapi, drawbox_vaapi filters
 - vf_scale supports secondary ref input and framesync options
+- vf_scale2ref deprecated
 
 
 version 7.0:
diff --git a/doc/filters.texi b/doc/filters.texi
index cf884568b0..5198f14f28 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -21590,79 +21590,6 @@ Scale a logo to 1/10th the height of a video, while preserving its display aspec
 @end example
 @end itemize
 
-@section scale2ref
-
-Scale (resize) the input video, based on a reference video.
-
-See the scale filter for available options, scale2ref supports the same but
-uses the reference video instead of the main input as basis. scale2ref also
-supports the following additional constants for the @option{w} and
-@option{h} options:
-
-@table @var
-@item main_w
-@item main_h
-The main input video's width and height
-
-@item main_a
-The same as @var{main_w} / @var{main_h}
-
-@item main_sar
-The main input video's sample aspect ratio
-
-@item main_dar, mdar
-The main input video's display aspect ratio. Calculated from
-@code{(main_w / main_h) * main_sar}.
-
-@item main_hsub
-@item main_vsub
-The main input video's horizontal and vertical chroma subsample values.
-For example for the pixel format "yuv422p" @var{hsub} is 2 and @var{vsub}
-is 1.
-
-@item main_n
-The (sequential) number of the main input frame, starting from 0.
-Only available with @code{eval=frame}.
-
-@item main_t
-The presentation timestamp of the main input frame, expressed as a number of
-seconds. Only available with @code{eval=frame}.
-
-@item main_pos
-The position (byte offset) of the frame in the main input stream, or NaN if
-this information is unavailable and/or meaningless (for example in case of synthetic video).
-Only available with @code{eval=frame}.
-@end table
-
-@subsection Examples
-
-@itemize
-@item
-Scale a subtitle stream (b) to match the main video (a) in size before overlaying
-@example
-'scale2ref[b][a];[a][b]overlay'
-@end example
-
-@item
-Scale a logo to 1/10th the height of a video, while preserving its display aspect ratio.
-@example
-[logo-in][video-in]scale2ref=w=oh*mdar:h=ih/10[logo-out][video-out]
-@end example
-@end itemize
-
-@subsection Commands
-
-This filter supports the following commands:
-@table @option
-@item width, w
-@item height, h
-Set the output video dimension expression.
-The command accepts the same syntax of the corresponding option.
-
-If the specified expression is not valid, it is kept at its current
-value.
-@end table
-
 @section scale2ref_npp
 
 Use the NVIDIA Performance Primitives (libnpp) to scale (resize) the input
diff --git a/libavfilter/vf_scale.c b/libavfilter/vf_scale.c
index f174651333..bc53571c1c 100644
--- a/libavfilter/vf_scale.c
+++ b/libavfilter/vf_scale.c
@@ -368,6 +368,9 @@ static av_cold int init(AVFilterContext *ctx)
     int64_t threads;
     int ret;
 
+    if (ctx->filter == &ff_vf_scale2ref)
+        av_log(ctx, AV_LOG_WARNING, "scale2ref is deprecated, use scale=rw:rh instead\n");
+
     if (scale->size_str && (scale->w_expr || scale->h_expr)) {
         av_log(ctx, AV_LOG_ERROR,
                "Size and width/height expressions cannot be set at the same time.\n");
-- 
2.44.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 0/5] replace scale2ref by scale=rw:rh

[Gyan Doshi]

On 2024-04-24 04:21 pm, Niklas Haas wrote:
> As discussed in my previous series for fixing scale2ref[1], this filter
> is fundamentally broken, and the only real fix would be to switch to
> activate(), or ideally FFFrameSync.
>
> [1] https://ffmpeg.org//pipermail/ffmpeg-devel/2024-March/323382.html
>
> The main thing making this difficult is the fact that scale2ref also
> wants to output ref frames to its secondary output, which FFFrameSync
> does not support, and which is ultimately at least part of the root
> cause of trac #10795.
>
> Since this is in principle completely unnecessary (users can just
> 'split' the ref input and have it be consumed by vf_scale), and to make
> the design of this filter a bit more robust and maintainable, switch to
> an approach where vf_scale itself gains the ability to reference
> a secondary input stream, using the "ref_*" series of variables.
>
> This makes the current [i][ri]scale2ref[o][ro] equivalent to the only
> slightly more verbose [ri]split[t][ro]; [i][t]scale=rw:rh[o]. (And
> conversely, it is no longer necessary to use nullsink to consume an
> unused [ro])

In principle, a good idea, but how does this impact memory use and speed 
in the not-so-uncommon scenario where multiple overlay targets are 
scaled 2 ref and then overlaid
e.g.

in current flow:

[a][base]scale2ref[a][ref];
[b][ref]scale2ref[b][ref[;
[c][ref]scale2ref[c][ref[;
[d][ref]scale2ref[d][ref[;
[ref][a]overlay[ref];
[ref][b]overlay[ref];
[ref][c]overlay[ref];
[ref][d]overlay[ref];

in new flow:

[base]split=5[base][refa][refb][refc][refd];
[a][refa]scale[a];
[b][refb]scale[b];
[c][refc]scale[c];
[d][refd]scale[d];
[base][a]overlay[outa];
[outa][b]overlay[outb];
[outb][c]overlay[outc];
[outc][d]overlay[out];


Regards,
Gyan

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 0/5] replace scale2ref by scale=rw:rh

[Timo Rothenpieler]

On 24/04/2024 13:18, Gyan Doshi wrote:
> 
> 
> On 2024-04-24 04:21 pm, Niklas Haas wrote:
>> As discussed in my previous series for fixing scale2ref[1], this filter
>> is fundamentally broken, and the only real fix would be to switch to
>> activate(), or ideally FFFrameSync.
>>
>> [1] https://ffmpeg.org//pipermail/ffmpeg-devel/2024-March/323382.html
>>
>> The main thing making this difficult is the fact that scale2ref also
>> wants to output ref frames to its secondary output, which FFFrameSync
>> does not support, and which is ultimately at least part of the root
>> cause of trac #10795.
>>
>> Since this is in principle completely unnecessary (users can just
>> 'split' the ref input and have it be consumed by vf_scale), and to make
>> the design of this filter a bit more robust and maintainable, switch to
>> an approach where vf_scale itself gains the ability to reference
>> a secondary input stream, using the "ref_*" series of variables.
>>
>> This makes the current [i][ri]scale2ref[o][ro] equivalent to the only
>> slightly more verbose [ri]split[t][ro]; [i][t]scale=rw:rh[o]. (And
>> conversely, it is no longer necessary to use nullsink to consume an
>> unused [ro])
> 
> In principle, a good idea, but how does this impact memory use and speed 
> in the not-so-uncommon scenario where multiple overlay targets are 
> scaled 2 ref and then overlaid
> e.g.
> 
> in current flow:
> 
> [a][base]scale2ref[a][ref];
> [b][ref]scale2ref[b][ref[;
> [c][ref]scale2ref[c][ref[;
> [d][ref]scale2ref[d][ref[;
> [ref][a]overlay[ref];
> [ref][b]overlay[ref];
> [ref][c]overlay[ref];
> [ref][d]overlay[ref];
> 
> in new flow:
> 
> [base]split=5[base][refa][refb][refc][refd];
> [a][refa]scale[a];
> [b][refb]scale[b];
> [c][refc]scale[c];
> [d][refd]scale[d];
> [base][a]overlay[outa];
> [outa][b]overlay[outb];
> [outb][c]overlay[outc];
> [outc][d]overlay[out];

Given that scale never modifies the reference, it just ups the refcount 
of those frames in the split filter, and will thus not change the 
memory-use whatsoever.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 0/5] replace scale2ref by scale=rw:rh

[Niklas Haas]

On Wed, 24 Apr 2024 16:48:54 +0530 Gyan Doshi <ffmpeg@gyani.pro> wrote:
> 
> 
> On 2024-04-24 04:21 pm, Niklas Haas wrote:
> > As discussed in my previous series for fixing scale2ref[1], this filter
> > is fundamentally broken, and the only real fix would be to switch to
> > activate(), or ideally FFFrameSync.
> >
> > [1] https://ffmpeg.org//pipermail/ffmpeg-devel/2024-March/323382.html
> >
> > The main thing making this difficult is the fact that scale2ref also
> > wants to output ref frames to its secondary output, which FFFrameSync
> > does not support, and which is ultimately at least part of the root
> > cause of trac #10795.
> >
> > Since this is in principle completely unnecessary (users can just
> > 'split' the ref input and have it be consumed by vf_scale), and to make
> > the design of this filter a bit more robust and maintainable, switch to
> > an approach where vf_scale itself gains the ability to reference
> > a secondary input stream, using the "ref_*" series of variables.
> >
> > This makes the current [i][ri]scale2ref[o][ro] equivalent to the only
> > slightly more verbose [ri]split[t][ro]; [i][t]scale=rw:rh[o]. (And
> > conversely, it is no longer necessary to use nullsink to consume an
> > unused [ro])
> 
> In principle, a good idea, but how does this impact memory use and speed 
> in the not-so-uncommon scenario where multiple overlay targets are 
> scaled 2 ref and then overlaid
> e.g.
> 
> in current flow:
> 
> [a][base]scale2ref[a][ref];
> [b][ref]scale2ref[b][ref[;
> [c][ref]scale2ref[c][ref[;
> [d][ref]scale2ref[d][ref[;
> [ref][a]overlay[ref];
> [ref][b]overlay[ref];
> [ref][c]overlay[ref];
> [ref][d]overlay[ref];
> 
> in new flow:
> 
> [base]split=5[base][refa][refb][refc][refd];
> [a][refa]scale[a];
> [b][refb]scale[b];
> [c][refc]scale[c];
> [d][refd]scale[d];
> [base][a]overlay[outa];
> [outa][b]overlay[outb];
> [outb][c]overlay[outc];
> [outc][d]overlay[out];
> 
> 
> Regards,
> Gyan

I have not tested it exactly, but based on my understanding of
libavfilter it shouldn't affect memory usage at all. `split` does not
duplicate frame data, it merely creates more references. And since all
of the `overlay` filters are upstream of [out], they all consume both of
their inputs before any forward progress can be made. So there is no
filter in this graph that can buffer more than 1 frame.

Actually, I would suspect memory usage to be slightly *lower* on
average, because ff_filter_activate_default() first consumes all
possible frames from input 1, then all possible frames from input 2,
etc.; whereas FFFrameSync consumes from both inputs simultaneously.
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 0/5] replace scale2ref by scale=rw:rh

[Niklas Haas]

On Wed, 24 Apr 2024 12:51:55 +0200 Niklas Haas <ffmpeg@haasn.xyz> wrote:
> As discussed in my previous series for fixing scale2ref[1], this filter
> is fundamentally broken, and the only real fix would be to switch to
> activate(), or ideally FFFrameSync.
> 
> [1] https://ffmpeg.org//pipermail/ffmpeg-devel/2024-March/323382.html
> 
> The main thing making this difficult is the fact that scale2ref also
> wants to output ref frames to its secondary output, which FFFrameSync
> does not support, and which is ultimately at least part of the root
> cause of trac #10795.
> 
> Since this is in principle completely unnecessary (users can just
> 'split' the ref input and have it be consumed by vf_scale), and to make
> the design of this filter a bit more robust and maintainable, switch to
> an approach where vf_scale itself gains the ability to reference
> a secondary input stream, using the "ref_*" series of variables.
> 
> This makes the current [i][ri]scale2ref[o][ro] equivalent to the only
> slightly more verbose [ri]split[t][ro]; [i][t]scale=rw:rh[o]. (And
> conversely, it is no longer necessary to use nullsink to consume an
> unused [ro])
> 
> Incidentally, I think it would be nice if lavfi could *automatically*
> split filter links referenced multiple times, so we could just write
> e.g. [i][ri]scale=rw:rh[o]; [ri][o]overlay[out] and have it work. Maybe
> I'll try getting that to work, next.
> 
> Either way, after the deprecation period has elapsed, we can delete
> scale2ref from the codebase in order to make vf_scale.c IMHO
> significantly simpler versus the status quo.
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Will merge in around 24h if there is no objection, as this is fixing
a bug marked important.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 0/5] replace scale2ref by scale=rw:rh

[Anton Khirnov]

Quoting Niklas Haas (2024-04-24 12:51:55)
> Incidentally, I think it would be nice if lavfi could *automatically*
> split filter links referenced multiple times, so we could just write
> e.g. [i][ri]scale=rw:rh[o]; [ri][o]overlay[out] and have it work. Maybe
> I'll try getting that to work, next.

While I agree that it would be nice, using duplicate link labels
currently does not break, and I've seen multiple people using those. So
we'll probably need to deprecate this behaviour first.

-- 
Anton Khirnov
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".