[FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

[-- Attachment #1: Type: text/plain, Size: 122 bytes --]

Hi,

Adds the new avfilter `libvmaf_cuda`, integrating the new vmaf_cuda_*
apis in libvmaf. Patch attached.

Thanks,
Kyle

[-- Attachment #2: 0001-avfilter-add-libvmaf_cuda.patch --]
[-- Type: application/octet-stream, Size: 10761 bytes --]

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Mon, Aug 7, 2023 at 10:50 AM Kyle Swanson <k@ylo.ph> wrote:
> Adds the new avfilter `libvmaf_cuda`, integrating the new vmaf_cuda_*
> apis in libvmaf. Patch attached.

Anyone interested in reviewing this? Would be appreciated.

Thanks,
Kyle
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Timo Rothenpieler]

> From 2665b7ddaefe6739bfeef4573183981582bdb995 Mon Sep 17 00:00:00 2001
> From: Kyle Swanson <kswanson@netflix.com>
> Date: Mon, 7 Aug 2023 10:38:12 -0700
> Subject: [PATCH] avfilter: add libvmaf_cuda
> 
> ---
>  configure                |   4 +
>  libavfilter/Makefile     |   1 +
>  libavfilter/allfilters.c |   1 +
>  libavfilter/vf_libvmaf.c | 211 ++++++++++++++++++++++++++++++++++++++-
>  4 files changed, 216 insertions(+), 1 deletion(-)
> 
> diff --git a/configure b/configure
> index d9372c5be6..387221c314 100755
> --- a/configure
> +++ b/configure
> @@ -286,6 +286,7 @@ External library support:
>    --enable-libv4l2         enable libv4l2/v4l-utils [no]
>    --enable-libvidstab      enable video stabilization using vid.stab [no]
>    --enable-libvmaf         enable vmaf filter via libvmaf [no]
> +  --enable-libvmaf-cuda    enable cuda vmaf filter via libvmaf [no]
>    --enable-libvo-amrwbenc  enable AMR-WB encoding via libvo-amrwbenc [no]
>    --enable-libvorbis       enable Vorbis en/decoding via libvorbis,
>                             native implementation exists [no]
> @@ -1902,6 +1903,7 @@ EXTERNAL_LIBRARY_LIST="
>      libuavs3d
>      libv4l2
>      libvmaf
> +    libvmaf_cuda
>      libvorbis
>      libvpx
>      libwebp
> @@ -3830,6 +3832,7 @@ vflip_vulkan_filter_deps="vulkan spirv_compiler"
>  vidstabdetect_filter_deps="libvidstab"
>  vidstabtransform_filter_deps="libvidstab"
>  libvmaf_filter_deps="libvmaf"
> +libvmaf_cuda_filter_deps="libvmaf"

This is missing dependencies on at least ffnvcodec.

>  zmq_filter_deps="libzmq"
>  zoompan_filter_deps="swscale"
>  zscale_filter_deps="libzimg const_nan"
> @@ -6806,6 +6809,7 @@ enabled libuavs3d         && require_pkg_config libuavs3d "uavs3d >= 1.1.41" uav
>  enabled libv4l2           && require_pkg_config libv4l2 libv4l2 libv4l2.h v4l2_ioctl
>  enabled libvidstab        && require_pkg_config libvidstab "vidstab >= 0.98" vid.stab/libvidstab.h vsMotionDetectInit
>  enabled libvmaf           && require_pkg_config libvmaf "libvmaf >= 2.0.0" libvmaf.h vmaf_init
> +enabled libvmaf_cuda      && require_pkg_config libvmaf "libvmaf >= 2.0.0" libvmaf.h vmaf_init

Why is this a separate library if it checks for the literal same thing?
Shouldn't this check for the vmaf cuda header being there at the very least?
Or a specific version since which it's included?

>  enabled libvo_amrwbenc    && require libvo_amrwbenc vo-amrwbenc/enc_if.h E_IF_init -lvo-amrwbenc
>  enabled libvorbis         && require_pkg_config libvorbis vorbis vorbis/codec.h vorbis_info_init &&
>                               require_pkg_config libvorbisenc vorbisenc vorbis/vorbisenc.h vorbis_encode_init
> diff --git a/libavfilter/Makefile b/libavfilter/Makefile
> index 30a0e22ef8..c1405ae924 100644
> --- a/libavfilter/Makefile
> +++ b/libavfilter/Makefile
> @@ -361,6 +361,7 @@ OBJS-$(CONFIG_LENSCORRECTION_FILTER)         += vf_lenscorrection.o
>  OBJS-$(CONFIG_LENSFUN_FILTER)                += vf_lensfun.o
>  OBJS-$(CONFIG_LIBPLACEBO_FILTER)             += vf_libplacebo.o vulkan.o vulkan_filter.o
>  OBJS-$(CONFIG_LIBVMAF_FILTER)                += vf_libvmaf.o framesync.o
> +OBJS-$(CONFIG_LIBVMAF_CUDA_FILTER)           += vf_libvmaf.o framesync.o
>  OBJS-$(CONFIG_LIMITDIFF_FILTER)              += vf_limitdiff.o framesync.o
>  OBJS-$(CONFIG_LIMITER_FILTER)                += vf_limiter.o
>  OBJS-$(CONFIG_LOOP_FILTER)                   += f_loop.o
> diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
> index 089ad3a0ed..8349ba9469 100644
> --- a/libavfilter/allfilters.c
> +++ b/libavfilter/allfilters.c
> @@ -337,6 +337,7 @@ extern const AVFilter ff_vf_lenscorrection;
>  extern const AVFilter ff_vf_lensfun;
>  extern const AVFilter ff_vf_libplacebo;
>  extern const AVFilter ff_vf_libvmaf;
> +extern const AVFilter ff_vf_libvmaf_cuda;
>  extern const AVFilter ff_vf_limitdiff;
>  extern const AVFilter ff_vf_limiter;
>  extern const AVFilter ff_vf_loop;
> diff --git a/libavfilter/vf_libvmaf.c b/libavfilter/vf_libvmaf.c
> index 2586f37d99..33fcbcb4ae 100644
> --- a/libavfilter/vf_libvmaf.c
> +++ b/libavfilter/vf_libvmaf.c
> @@ -24,6 +24,8 @@
>   * Calculate the VMAF between two input videos.
>   */
>  
> +#include "config.h"
> +
>  #include <libvmaf.h>
>  
>  #include "libavutil/avstring.h"
> @@ -36,6 +38,13 @@
>  #include "internal.h"
>  #include "video.h"
>  
> +#ifdef CONFIG_LIBVMAF_CUDA
> +#include <libvmaf_cuda.h>
> +
> +#include "libavutil/hwcontext.h"
> +#include "libavutil/hwcontext_cuda_internal.h"
> +#endif
> +
>  typedef struct LIBVMAFContext {
>      const AVClass *class;
>      FFFrameSync fs;
> @@ -58,6 +67,7 @@ typedef struct LIBVMAFContext {
>      unsigned model_cnt;
>      unsigned frame_cnt;
>      unsigned bpc;
> +    VmafCudaState *cu_state;

Looks like it's missing the usual #ifdef around it.

>  } LIBVMAFContext;
>  
>  #define OFFSET(x) offsetof(LIBVMAFContext, x)
> @@ -682,7 +692,8 @@ static const AVFilterPad libvmaf_inputs[] = {
>      {
>          .name         = "main",
>          .type         = AVMEDIA_TYPE_VIDEO,
> -    },{
> +    },
> +    {

Unrelated change.

>          .name         = "reference",
>          .type         = AVMEDIA_TYPE_VIDEO,
>          .config_props = config_input_ref,
> @@ -710,3 +721,201 @@ const AVFilter ff_vf_libvmaf = {
>      FILTER_OUTPUTS(libvmaf_outputs),
>      FILTER_PIXFMTS_ARRAY(pix_fmts),
>  };
> +
> +#ifdef CONFIG_LIBVMAF_CUDA
> +static const enum AVPixelFormat supported_formats[] = {
> +    AV_PIX_FMT_YUV420P,
> +    AV_PIX_FMT_YUV444P16,
> +};
> +
> +static int format_is_supported(enum AVPixelFormat fmt)
> +{
> +    int i;
> +
> +    for (i = 0; i < FF_ARRAY_ELEMS(supported_formats); i++)
> +        if (supported_formats[i] == fmt)
> +            return 1;
> +    return 0;
> +}
> +
> +static int config_props_cuda(AVFilterLink *outlink)
> +{
> +    int err;
> +    AVFilterContext *ctx = outlink->src;
> +    LIBVMAFContext *s = ctx->priv;
> +    AVFilterLink *inlink = ctx->inputs[0];
> +    AVHWFramesContext *frames_ctx = (AVHWFramesContext*) inlink->hw_frames_ctx->data;
> +    AVCUDADeviceContext *device_hwctx = frames_ctx->device_ctx->hwctx;
> +    CUcontext cu_ctx = device_hwctx->cuda_ctx;
> +    const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(frames_ctx->sw_format);
> +
> +    VmafConfiguration cfg = {
> +        .log_level = log_level_map(av_log_get_level()),
> +        .n_subsample = s->n_subsample,
> +        .n_threads = s->n_threads,
> +    };
> +
> +    VmafCudaPictureConfiguration cuda_pic_cfg = {
> +        .pic_params = {
> +            .bpc = desc->comp[0].depth,
> +            .w = inlink->w,
> +            .h = inlink->h,
> +            .pix_fmt = pix_fmt_map(frames_ctx->sw_format),
> +        },
> +        .pic_prealloc_method = VMAF_CUDA_PICTURE_PREALLOCATION_METHOD_DEVICE,
> +    };
> +
> +    VmafCudaConfiguration cuda_cfg = {
> +        .cu_ctx = cu_ctx,
> +    };
> +
> +    if (!format_is_supported(frames_ctx->sw_format)) {
> +        av_log(s, AV_LOG_ERROR,
> +               "Unsupported input format: %s\n", desc->name);
> +        return AVERROR(EINVAL);
> +    }
> +
> +    err = vmaf_init(&s->vmaf, cfg);
> +    if (err)
> +        return AVERROR(EINVAL);
> +
> +    err = vmaf_cuda_state_init(&s->cu_state, cuda_cfg);
> +    if (err)
> +        return AVERROR(EINVAL);
> +
> +    err = vmaf_cuda_import_state(s->vmaf, s->cu_state);
> +    if (err)
> +        return AVERROR(EINVAL);
> +
> +    err = vmaf_cuda_preallocate_pictures(s->vmaf, cuda_pic_cfg);
> +    if (err < 0)
> +        return err;
> +
> +    err = parse_deprecated_options(ctx);
> +    if (err)
> +        return err;
> +
> +    err = parse_models(ctx);
> +    if (err)
> +        return err;
> +
> +    err = parse_features(ctx);
> +    if (err)
> +        return err;
> +
> +    return config_output(outlink);
> +}
> +
> +static int copy_picture_data_cuda(VmafContext* vmaf,
> +                                  AVCUDADeviceContext* device_hwctx,
> +                                  AVFrame* src, VmafPicture* dst,
> +                                  enum AVPixelFormat pix_fmt)
> +{
> +    const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(pix_fmt);
> +    CudaFunctions *cu = device_hwctx->internal->cuda_dl;
> +
> +    CUDA_MEMCPY2D m = {
> +        .srcMemoryType = CU_MEMORYTYPE_DEVICE,
> +        .dstMemoryType = CU_MEMORYTYPE_DEVICE,
> +    };
> +
> +    int err = vmaf_cuda_fetch_preallocated_picture(vmaf, dst);
> +    if (err)
> +        return AVERROR(ENOMEM);
> +
> +    err = cu->cuCtxPushCurrent(device_hwctx->cuda_ctx);
> +    if (err)

We tend to do a bit more checking here for cuda, but for push/pop, it's 
probably fine.
Would still prefer to see this compared against CUDA_SUCCESS, even 
though it's technically 0.

> +        return AVERROR(ENOMEM);

ENOMEM does not sound correct.
Not sure what fits here, we usually just return EXTERNAL.

> +
> +    for (unsigned i = 0; i < pix_desc->nb_components; i++) {
> +        m.srcDevice = (CUdeviceptr) src->data[i];
> +        m.srcPitch = src->linesize[i];
> +        m.dstDevice = (CUdeviceptr) dst->data[i];
> +        m.dstPitch = dst->stride[i];
> +        m.WidthInBytes = dst->w[i] * ((dst->bpc + 7) / 8);
> +        m.Height = dst->h[i];
> +
> +        err = cu->cuMemcpy2D(&m);
> +        if (err)
> +            return AVERROR(ENOMEM);

This is also not a nomem-situation.

> +        break;
> +    }
> +
> +    err = cu->cuCtxPopCurrent(NULL);
> +    if (err)
> +        return AVERROR(ENOMEM);
> +
> +    return 0;
> +}
> +
> +static int do_vmaf_cuda(FFFrameSync* fs)
> +{
> +    AVFilterContext* ctx = fs->parent;
> +    LIBVMAFContext* s = ctx->priv;
> +    AVFilterLink *inlink = ctx->inputs[0];
> +    AVHWFramesContext *frames_ctx = (AVHWFramesContext*) inlink->hw_frames_ctx->data;
> +    AVCUDADeviceContext *device_hwctx = frames_ctx->device_ctx->hwctx;
> +    VmafPicture pic_ref, pic_dist;
> +    AVFrame *ref, *dist;
> +
> +    int err = 0;
> +
> +    err = ff_framesync_dualinput_get(fs, &dist, &ref);
> +    if (err < 0)
> +        return err;
> +    if (ctx->is_disabled || !ref)
> +        return ff_filter_frame(ctx->outputs[0], dist);
> +
> +    err = copy_picture_data_cuda(s->vmaf, device_hwctx, ref, &pic_ref,
> +                                 frames_ctx->sw_format);
> +    if (err) {
> +        av_log(s, AV_LOG_ERROR, "problem during copy_picture_data_cuda.\n");
> +        return AVERROR(ENOMEM);
> +    }
> +
> +    err = copy_picture_data_cuda(s->vmaf, device_hwctx, dist, &pic_dist,
> +                                 frames_ctx->sw_format);
> +    if (err) {
> +        av_log(s, AV_LOG_ERROR, "problem during copy_picture_data_cuda.\n");
> +        return AVERROR(ENOMEM);
> +    }
> +
> +    err = vmaf_read_pictures(s->vmaf, &pic_ref, &pic_dist, s->frame_cnt++);
> +    if (err) {
> +        av_log(s, AV_LOG_ERROR, "problem during vmaf_read_pictures.\n");
> +        return AVERROR(EINVAL);
> +    }
> +
> +    return ff_filter_frame(ctx->outputs[0], dist);
> +}
> +
> +static av_cold int init_cuda(AVFilterContext *ctx)
> +{
> +    LIBVMAFContext *s = ctx->priv;
> +    s->fs.on_event = do_vmaf_cuda;
> +    return 0;
> +}
> +
> +static const AVFilterPad libvmaf_outputs_cuda[] = {
> +    {
> +        .name         = "default",
> +        .type         = AVMEDIA_TYPE_VIDEO,
> +        .config_props = config_props_cuda,
> +    },
> +};
> +
> +const AVFilter ff_vf_libvmaf_cuda = {
> +    .name           = "libvmaf_cuda",
> +    .description    = NULL_IF_CONFIG_SMALL("Calculate the VMAF between two video streams."),
> +    .preinit        = libvmaf_framesync_preinit,
> +    .init           = init_cuda,
> +    .uninit         = uninit,
> +    .activate       = activate,
> +    .priv_size      = sizeof(LIBVMAFContext),
> +    .priv_class     = &libvmaf_class,
> +    FILTER_INPUTS(libvmaf_inputs),
> +    FILTER_OUTPUTS(libvmaf_outputs_cuda),
> +    FILTER_SINGLE_PIXFMT(AV_PIX_FMT_CUDA),
> +    .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE,
> +};
> +#endif
> -- 
> 2.24.3 (Apple Git-128)
> 

rest of the code generally looks sound to me.
copy_picture_data_cuda looks like mostly a copy of 
av_hwframe_transfer_data(), but wrapping the vmaf image into an AVFrame 
is probably just as much boilerplate.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

[-- Attachment #1: Type: text/plain, Size: 97 bytes --]

Hi,

Thank you Timo for the review. Took your suggestions, updated patch attached.

Thanks,
Kyle

[-- Attachment #2: 0001-avfilter-add-libvmaf_cuda.patch --]
[-- Type: application/octet-stream, Size: 10511 bytes --]

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

[-- Attachment #1: Type: text/plain, Size: 136 bytes --]

Hi,

Added filter docs, new patch attached. Unless there are any other
reviews I will plan on pushing this in a few days.

Thanks,
Kyle

[-- Attachment #2: 0001-avfilter-add-libvmaf_cuda.patch --]
[-- Type: application/octet-stream, Size: 11775 bytes --]

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Timo Rothenpieler]

> From f6f0afffadfc5fae97b11b0feb7c1d740b7c86ab Mon Sep 17 00:00:00 2001
> From: Kyle Swanson <kswanson@netflix.com>
> Date: Mon, 28 Aug 2023 11:49:34 -0700
> Subject: [PATCH] avfilter: add libvmaf_cuda
> 
> ---
>  configure                |   4 +
>  doc/filters.texi         |  26 +++++
>  libavfilter/Makefile     |   1 +
>  libavfilter/allfilters.c |   1 +
>  libavfilter/vf_libvmaf.c | 210 +++++++++++++++++++++++++++++++++++++++
>  5 files changed, 242 insertions(+)
> 
> diff --git a/configure b/configure
> index bd7f7697c8..6f6c6aaf22 100755
> --- a/configure
> +++ b/configure
> @@ -286,6 +286,7 @@ External library support:
>    --enable-libv4l2         enable libv4l2/v4l-utils [no]
>    --enable-libvidstab      enable video stabilization using vid.stab [no]
>    --enable-libvmaf         enable vmaf filter via libvmaf [no]
> +  --enable-libvmaf-cuda    enable cuda vmaf filter via libvmaf [no]
>    --enable-libvo-amrwbenc  enable AMR-WB encoding via libvo-amrwbenc [no]
>    --enable-libvorbis       enable Vorbis en/decoding via libvorbis,
>                             native implementation exists [no]
> @@ -1902,6 +1903,7 @@ EXTERNAL_LIBRARY_LIST="
>      libuavs3d
>      libv4l2
>      libvmaf
> +    libvmaf_cuda
>      libvorbis
>      libvpx
>      libwebp
> @@ -3831,6 +3833,7 @@ vflip_vulkan_filter_deps="vulkan spirv_compiler"
>  vidstabdetect_filter_deps="libvidstab"
>  vidstabtransform_filter_deps="libvidstab"
>  libvmaf_filter_deps="libvmaf"
> +libvmaf_cuda_filter_deps="libvmaf cuda_nvcc"

Does this really depend on nvcc?
Does it not work with only ffnvcodec?

>  zmq_filter_deps="libzmq"
>  zoompan_filter_deps="swscale"
>  zscale_filter_deps="libzimg const_nan"
> @@ -6811,6 +6814,7 @@ enabled libuavs3d         && require_pkg_config libuavs3d "uavs3d >= 1.1.41" uav
>  enabled libv4l2           && require_pkg_config libv4l2 libv4l2 libv4l2.h v4l2_ioctl
>  enabled libvidstab        && require_pkg_config libvidstab "vidstab >= 0.98" vid.stab/libvidstab.h vsMotionDetectInit
>  enabled libvmaf           && require_pkg_config libvmaf "libvmaf >= 2.0.0" libvmaf.h vmaf_init
> +enabled libvmaf_cuda      && require_pkg_config libvmaf "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init
>  enabled libvo_amrwbenc    && require libvo_amrwbenc vo-amrwbenc/enc_if.h E_IF_init -lvo-amrwbenc
>  enabled libvorbis         && require_pkg_config libvorbis vorbis vorbis/codec.h vorbis_info_init &&
>                               require_pkg_config libvorbisenc vorbisenc vorbis/vorbisenc.h vorbis_encode_init
> diff --git a/doc/filters.texi b/doc/filters.texi
> index 14a6be49ac..eaff3f1ddc 100644
> --- a/doc/filters.texi
> +++ b/doc/filters.texi
> @@ -16928,6 +16928,32 @@ ffmpeg -i distorted.mpg -i reference.mkv -lavfi "[0:v]settb=AVTB,setpts=PTS-STAR
>  @end example
>  @end itemize
>  
> +@section libvmaf_cuda
> +
> +This is the CUDA variant of the @ref{libvmaf} filter. It only accepts CUDA frames.
> +
> +It requires Netflix's vmaf library (libvmaf) as a pre-requisite.
> +After installing the library it can be enabled using:
> +@code{./configure --enable-nonfree --enable-cuda-nvcc --enable-libvmaf-cuda}.

see above

> +@subsection Examples
> +@itemize
> +
> +@item
> +Basic usage showing CUVID hardware decoding and CUDA scaling with @ref{scale_cuda}:
> +@example
> +ffmpeg \
> +    -hwaccel cuda -hwaccel_output_format cuda -codec:v av1_cuvid -i dis.obu \
> +    -hwaccel cuda -hwaccel_output_format cuda -codec:v av1_cuvid -i ref.obu \
> +    -filter_complex "
> +        [0:v]scale_cuda=format=yuv420p[ref]; \
> +        [1:v]scale_cuda=format=yuv420p[dis]; \
> +        [dis][ref]libvmaf_cuda=log_fmt=json:log_path=output.json
> +    " \
> +    -f null -
> +@end example
> +@end itemize
> +
>  @section limitdiff
>  Apply limited difference filter using second and optionally third video stream.
>  
> diff --git a/libavfilter/Makefile b/libavfilter/Makefile
> index 2fe0033b21..57f5809acb 100644
> --- a/libavfilter/Makefile
> +++ b/libavfilter/Makefile
> @@ -363,6 +363,7 @@ OBJS-$(CONFIG_LENSCORRECTION_FILTER)         += vf_lenscorrection.o
>  OBJS-$(CONFIG_LENSFUN_FILTER)                += vf_lensfun.o
>  OBJS-$(CONFIG_LIBPLACEBO_FILTER)             += vf_libplacebo.o vulkan.o vulkan_filter.o
>  OBJS-$(CONFIG_LIBVMAF_FILTER)                += vf_libvmaf.o framesync.o
> +OBJS-$(CONFIG_LIBVMAF_CUDA_FILTER)           += vf_libvmaf.o framesync.o
>  OBJS-$(CONFIG_LIMITDIFF_FILTER)              += vf_limitdiff.o framesync.o
>  OBJS-$(CONFIG_LIMITER_FILTER)                += vf_limiter.o
>  OBJS-$(CONFIG_LOOP_FILTER)                   += f_loop.o
> diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
> index d4184d6e80..aa49703c6e 100644
> --- a/libavfilter/allfilters.c
> +++ b/libavfilter/allfilters.c
> @@ -339,6 +339,7 @@ extern const AVFilter ff_vf_lenscorrection;
>  extern const AVFilter ff_vf_lensfun;
>  extern const AVFilter ff_vf_libplacebo;
>  extern const AVFilter ff_vf_libvmaf;
> +extern const AVFilter ff_vf_libvmaf_cuda;
>  extern const AVFilter ff_vf_limitdiff;
>  extern const AVFilter ff_vf_limiter;
>  extern const AVFilter ff_vf_loop;
> diff --git a/libavfilter/vf_libvmaf.c b/libavfilter/vf_libvmaf.c
> index 2586f37d99..d7d853ac3e 100644
> --- a/libavfilter/vf_libvmaf.c
> +++ b/libavfilter/vf_libvmaf.c
> @@ -24,6 +24,8 @@
>   * Calculate the VMAF between two input videos.
>   */
>  
> +#include "config.h"
> +
>  #include <libvmaf.h>
>  
>  #include "libavutil/avstring.h"
> @@ -36,6 +38,13 @@
>  #include "internal.h"
>  #include "video.h"
>  
> +#ifdef CONFIG_LIBVMAF_CUDA
> +#include <libvmaf_cuda.h>

Does this include cuda.h or something like that?
If so, it should probably be included after the cuda hwcontext, to avoid 
it doing that.

> +#include "libavutil/hwcontext.h"
> +#include "libavutil/hwcontext_cuda_internal.h"
> +#endif
> +
>  typedef struct LIBVMAFContext {
>      const AVClass *class;
>      FFFrameSync fs;
> @@ -58,6 +67,9 @@ typedef struct LIBVMAFContext {
>      unsigned model_cnt;
>      unsigned frame_cnt;
>      unsigned bpc;
> +#ifdef CONFIG_LIBVMAF_CUDA
> +    VmafCudaState *cu_state;
> +#endif
>  } LIBVMAFContext;
>  
>  #define OFFSET(x) offsetof(LIBVMAFContext, x)
> @@ -710,3 +722,201 @@ const AVFilter ff_vf_libvmaf = {
>      FILTER_OUTPUTS(libvmaf_outputs),
>      FILTER_PIXFMTS_ARRAY(pix_fmts),
>  };
> +
> +#ifdef CONFIG_LIBVMAF_CUDA
> +static const enum AVPixelFormat supported_formats[] = {
> +    AV_PIX_FMT_YUV420P,
> +    AV_PIX_FMT_YUV444P16,
> +};
> +
> +static int format_is_supported(enum AVPixelFormat fmt)
> +{
> +    int i;
> +
> +    for (i = 0; i < FF_ARRAY_ELEMS(supported_formats); i++)
> +        if (supported_formats[i] == fmt)
> +            return 1;
> +    return 0;
> +}
> +
> +static int config_props_cuda(AVFilterLink *outlink)
> +{
> +    int err;
> +    AVFilterContext *ctx = outlink->src;
> +    LIBVMAFContext *s = ctx->priv;
> +    AVFilterLink *inlink = ctx->inputs[0];
> +    AVHWFramesContext *frames_ctx = (AVHWFramesContext*) inlink->hw_frames_ctx->data;
> +    AVCUDADeviceContext *device_hwctx = frames_ctx->device_ctx->hwctx;
> +    CUcontext cu_ctx = device_hwctx->cuda_ctx;
> +    const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(frames_ctx->sw_format);
> +
> +    VmafConfiguration cfg = {
> +        .log_level = log_level_map(av_log_get_level()),
> +        .n_subsample = s->n_subsample,
> +        .n_threads = s->n_threads,
> +    };
> +
> +    VmafCudaPictureConfiguration cuda_pic_cfg = {
> +        .pic_params = {
> +            .bpc = desc->comp[0].depth,
> +            .w = inlink->w,
> +            .h = inlink->h,
> +            .pix_fmt = pix_fmt_map(frames_ctx->sw_format),
> +        },
> +        .pic_prealloc_method = VMAF_CUDA_PICTURE_PREALLOCATION_METHOD_DEVICE,
> +    };
> +
> +    VmafCudaConfiguration cuda_cfg = {
> +        .cu_ctx = cu_ctx,
> +    };
> +
> +    if (!format_is_supported(frames_ctx->sw_format)) {
> +        av_log(s, AV_LOG_ERROR,
> +               "Unsupported input format: %s\n", desc->name);
> +        return AVERROR(EINVAL);
> +    }
> +
> +    err = vmaf_init(&s->vmaf, cfg);
> +    if (err)
> +        return AVERROR(EINVAL);
> +
> +    err = vmaf_cuda_state_init(&s->cu_state, cuda_cfg);
> +    if (err)
> +        return AVERROR(EINVAL);
> +
> +    err = vmaf_cuda_import_state(s->vmaf, s->cu_state);
> +    if (err)
> +        return AVERROR(EINVAL);
> +
> +    err = vmaf_cuda_preallocate_pictures(s->vmaf, cuda_pic_cfg);
> +    if (err < 0)
> +        return err;
> +
> +    err = parse_deprecated_options(ctx);
> +    if (err)
> +        return err;
> +
> +    err = parse_models(ctx);
> +    if (err)
> +        return err;
> +
> +    err = parse_features(ctx);
> +    if (err)
> +        return err;
> +
> +    return config_output(outlink);
> +}
> +
> +static int copy_picture_data_cuda(VmafContext* vmaf,
> +                                  AVCUDADeviceContext* device_hwctx,
> +                                  AVFrame* src, VmafPicture* dst,
> +                                  enum AVPixelFormat pix_fmt)
> +{
> +    const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(pix_fmt);
> +    CudaFunctions *cu = device_hwctx->internal->cuda_dl;
> +
> +    CUDA_MEMCPY2D m = {
> +        .srcMemoryType = CU_MEMORYTYPE_DEVICE,
> +        .dstMemoryType = CU_MEMORYTYPE_DEVICE,
> +    };
> +
> +    int err = vmaf_cuda_fetch_preallocated_picture(vmaf, dst);
> +    if (err)
> +        return AVERROR(ENOMEM);
> +
> +    err = cu->cuCtxPushCurrent(device_hwctx->cuda_ctx);
> +    if (err)
> +        return AVERROR_EXTERNAL;
> +
> +    for (unsigned i = 0; i < pix_desc->nb_components; i++) {
> +        m.srcDevice = (CUdeviceptr) src->data[i];
> +        m.srcPitch = src->linesize[i];
> +        m.dstDevice = (CUdeviceptr) dst->data[i];
> +        m.dstPitch = dst->stride[i];
> +        m.WidthInBytes = dst->w[i] * ((dst->bpc + 7) / 8);
> +        m.Height = dst->h[i];
> +
> +        err = cu->cuMemcpy2D(&m);
> +        if (err)
> +            return AVERROR_EXTERNAL;
> +        break;
> +    }
> +
> +    err = cu->cuCtxPopCurrent(NULL);
> +    if (err)
> +        return AVERROR_EXTERNAL;
> +
> +    return 0;
> +}
> +
> +static int do_vmaf_cuda(FFFrameSync* fs)
> +{
> +    AVFilterContext* ctx = fs->parent;
> +    LIBVMAFContext* s = ctx->priv;
> +    AVFilterLink *inlink = ctx->inputs[0];
> +    AVHWFramesContext *frames_ctx = (AVHWFramesContext*) inlink->hw_frames_ctx->data;
> +    AVCUDADeviceContext *device_hwctx = frames_ctx->device_ctx->hwctx;
> +    VmafPicture pic_ref, pic_dist;
> +    AVFrame *ref, *dist;
> +
> +    int err = 0;
> +
> +    err = ff_framesync_dualinput_get(fs, &dist, &ref);
> +    if (err < 0)
> +        return err;
> +    if (ctx->is_disabled || !ref)
> +        return ff_filter_frame(ctx->outputs[0], dist);
> +
> +    err = copy_picture_data_cuda(s->vmaf, device_hwctx, ref, &pic_ref,
> +                                 frames_ctx->sw_format);
> +    if (err) {
> +        av_log(s, AV_LOG_ERROR, "problem during copy_picture_data_cuda.\n");
> +        return AVERROR(ENOMEM);
> +    }
> +
> +    err = copy_picture_data_cuda(s->vmaf, device_hwctx, dist, &pic_dist,
> +                                 frames_ctx->sw_format);
> +    if (err) {
> +        av_log(s, AV_LOG_ERROR, "problem during copy_picture_data_cuda.\n");
> +        return AVERROR(ENOMEM);
> +    }
> +
> +    err = vmaf_read_pictures(s->vmaf, &pic_ref, &pic_dist, s->frame_cnt++);
> +    if (err) {
> +        av_log(s, AV_LOG_ERROR, "problem during vmaf_read_pictures.\n");
> +        return AVERROR(EINVAL);
> +    }
> +
> +    return ff_filter_frame(ctx->outputs[0], dist);
> +}
> +
> +static av_cold int init_cuda(AVFilterContext *ctx)
> +{
> +    LIBVMAFContext *s = ctx->priv;
> +    s->fs.on_event = do_vmaf_cuda;
> +    return 0;
> +}
> +
> +static const AVFilterPad libvmaf_outputs_cuda[] = {
> +    {
> +        .name         = "default",
> +        .type         = AVMEDIA_TYPE_VIDEO,
> +        .config_props = config_props_cuda,
> +    },
> +};
> +
> +const AVFilter ff_vf_libvmaf_cuda = {
> +    .name           = "libvmaf_cuda",
> +    .description    = NULL_IF_CONFIG_SMALL("Calculate the VMAF between two video streams."),
> +    .preinit        = libvmaf_framesync_preinit,
> +    .init           = init_cuda,
> +    .uninit         = uninit,
> +    .activate       = activate,
> +    .priv_size      = sizeof(LIBVMAFContext),
> +    .priv_class     = &libvmaf_class,
> +    FILTER_INPUTS(libvmaf_inputs),
> +    FILTER_OUTPUTS(libvmaf_outputs_cuda),
> +    FILTER_SINGLE_PIXFMT(AV_PIX_FMT_CUDA),
> +    .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE,
> +};
> +#endif
> -- 
> 2.24.3 (Apple Git-128)
> 

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Paul B Mahol]

On Mon, Aug 28, 2023 at 8:54 PM Kyle Swanson <k@ylo.ph> wrote:

> Hi,
>
> Added filter docs, new patch attached. Unless there are any other
> reviews I will plan on pushing this in a few days.
>

Only after at least two main devs reviews and agree to merge.

New policy.


> Thanks,
> Kyle
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Mon, Aug 28, 2023 at 11:59 AM Timo Rothenpieler
<timo@rothenpieler.org> wrote:
> > +libvmaf_cuda_filter_deps="libvmaf cuda_nvcc"
>
> Does this really depend on nvcc?
> Does it not work with only ffnvcodec?
>

Ah, you're right, I think ffnvcodec is the right option to use, I'll
switch this in the next patch version.

> > +#ifdef CONFIG_LIBVMAF_CUDA
> > +#include <libvmaf_cuda.h>
>
> Does this include cuda.h or something like that?
> If so, it should probably be included after the cuda hwcontext, to avoid
> it doing that.

No, libvmaf doesn't include cuda.h.

Thanks,
Kyle
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

[-- Attachment #1: Type: text/plain, Size: 223 bytes --]

Hi,

On Mon, Aug 28, 2023 at 1:16 PM Kyle Swanson <k@ylo.ph> wrote:
> Ah, you're right, I think ffnvcodec is the right option to use, I'll
> switch this in the next patch version.

Patch attached.

Thanks,
Kyle

[-- Attachment #2: 0001-avfilter-add-libvmaf_cuda.patch --]
[-- Type: application/octet-stream, Size: 11775 bytes --]

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Andreas Rheinhardt]

Kyle Swanson:
> +#ifdef CONFIG_LIBVMAF_CUDA
> +#include <libvmaf_cuda.h>
> +
> +#include "libavutil/hwcontext.h"
> +#include "libavutil/hwcontext_cuda_internal.h"
> +#endif
> +

The CONFIG_* variables are always defined; you need to check via "#if",
not "#ifdef".

- Andreas

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

[-- Attachment #1: Type: text/plain, Size: 243 bytes --]

Hi,

On Tue, Aug 29, 2023 at 3:13 PM Andreas Rheinhardt
<andreas.rheinhardt@outlook.com> wrote:
> The CONFIG_* variables are always defined; you need to check via "#if",
> not "#ifdef".

Fixed in the attached patch.

Thanks,
Kyle

[-- Attachment #2: 0001-avfilter-add-libvmaf_cuda.patch --]
[-- Type: application/octet-stream, Size: 11766 bytes --]

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Wed, Aug 30, 2023 at 9:44 AM Kyle Swanson <k@ylo.ph> wrote:
> Fixed in the attached patch.

Any other reviews, or is this one OK to merge now?

Thanks,
Kyle
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Thu, Aug 31, 2023 at 11:39 AM Kyle Swanson <k@ylo.ph> wrote:
> On Wed, Aug 30, 2023 at 9:44 AM Kyle Swanson <k@ylo.ph> wrote:
> > Fixed in the attached patch.
>
> Any other reviews, or is this one OK to merge now?

I'll merge this in a few days.

Thanks,
Kyle
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Tue, Sep 5, 2023 at 10:16 AM Kyle Swanson <k@ylo.ph> wrote:
> I'll merge this in a few days.

Not merged yet because I noticed a bug in the configure script.
Setting only `--enable-libvmaf` will enable both `libvmaf` and
`libvmaf_cuda`. Anyone familiar with the configure script know why
this is happening?

Thanks,
Kyle
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Timo Rothenpieler]

On 11.09.2023 19:53, Kyle Swanson wrote:
> Hi,
> 
> On Tue, Sep 5, 2023 at 10:16 AM Kyle Swanson <k@ylo.ph> wrote:
>> I'll merge this in a few days.
> 
> Not merged yet because I noticed a bug in the configure script.
> Setting only `--enable-libvmaf` will enable both `libvmaf` and
> `libvmaf_cuda`. Anyone familiar with the configure script know why
> this is happening?

require_pkg_config libvmaf [...]
Should be:
require_pkg_config libvmaf_cuda [...]

And then you want to also depend on libvmaf_cuda, not libvmaf.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Hi,

On Mon, Sep 11, 2023 at 11:51 AM Timo Rothenpieler
<timo@rothenpieler.org> wrote:
>
> On 11.09.2023 19:53, Kyle Swanson wrote:
> > Hi,
> >
> > On Tue, Sep 5, 2023 at 10:16 AM Kyle Swanson <k@ylo.ph> wrote:
> >> I'll merge this in a few days.
> >
> > Not merged yet because I noticed a bug in the configure script.
> > Setting only `--enable-libvmaf` will enable both `libvmaf` and
> > `libvmaf_cuda`. Anyone familiar with the configure script know why
> > this is happening?
>
> require_pkg_config libvmaf [...]
> Should be:
> require_pkg_config libvmaf_cuda [...]
>
> And then you want to also depend on libvmaf_cuda, not libvmaf.

Problem was slightly different, there is no `libvmaf_cuda`, just
`libvmaf` built with the cuda apis optionally enabled. `./configure`
is fixed now and I've tested all the possible combinations, updated
patch attached.

Thanks,
Kyle

[-- Attachment #2: 0001-avfilter-add-libvmaf_cuda.patch --]
[-- Type: application/octet-stream, Size: 11437 bytes --]

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Timo Rothenpieler]

On 14.09.2023 20:59, Kyle Swanson wrote:
> Problem was slightly different, there is no `libvmaf_cuda`, just
> `libvmaf` built with the cuda apis optionally enabled. `./configure`
> is fixed now and I've tested all the possible combinations, updated
> patch attached.

This looks more complicated than necessary to me.

My idea would have been to change, in the previous original patch,

 > libvmaf_cuda_filter_deps="libvmaf ffnvcodec"

to

 > libvmaf_cuda_filter_deps="libvmaf libvmaf_cuda ffnvcodec"


And the pkg_config check would then enable/disable libvmaf_cuda:

> enabled libvmaf_cuda      && require_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init



One unrelated thing I just thought of:

The macro you'll want to check in the filter itself isn't 
CONFIG_LIBVMAF_CUDA, but CONFIG_LIBVMAF_CUDA_FILTER.
Since it's in theory possible for someone to do the slightly silly thing 
of doing --enable-vmaf-cuda but then manually disabling the filter.

Likewise it's possible for someone to ONLY build the libvmaf_cuda 
filter, and disable the non-cuda variant.

So its "const AVFilter ..." part, and everything only used by it, should 
be wrapped in #if CONFIG_LIBVMAF_FILTER
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Thu, Sep 14, 2023 at 12:10 PM Timo Rothenpieler
<timo@rothenpieler.org> wrote:
> This looks more complicated than necessary to me.
>
> My idea would have been to change, in the previous original patch,
>
>  > libvmaf_cuda_filter_deps="libvmaf ffnvcodec"
>
> to
>
>  > libvmaf_cuda_filter_deps="libvmaf libvmaf_cuda ffnvcodec"
>
>
> And the pkg_config check would then enable/disable libvmaf_cuda:
>
> > enabled libvmaf_cuda      && require_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init
>

I'm not sure this works since `libvmaf_cuda` is not a separate
library. `libvmaf_cuda` is the name of the avfilter, which requires a
`libvmaf` configured/built with cuda support.

>
> One unrelated thing I just thought of:
>
> The macro you'll want to check in the filter itself isn't
> CONFIG_LIBVMAF_CUDA, but CONFIG_LIBVMAF_CUDA_FILTER.
> Since it's in theory possible for someone to do the slightly silly thing
> of doing --enable-vmaf-cuda but then manually disabling the filter.
>
> Likewise it's possible for someone to ONLY build the libvmaf_cuda
> filter, and disable the non-cuda variant.
>
> So its "const AVFilter ..." part, and everything only used by it, should
> be wrapped in #if CONFIG_LIBVMAF_FILTER

Yes, I made this macro change in the most recent patch.

Thanks,
Kyle
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Timo Rothenpieler]

On 15.09.2023 22:31, Kyle Swanson wrote:
> Hi,
> 
> On Thu, Sep 14, 2023 at 12:10 PM Timo Rothenpieler
> <timo@rothenpieler.org> wrote:
>> This looks more complicated than necessary to me.
>>
>> My idea would have been to change, in the previous original patch,
>>
>>   > libvmaf_cuda_filter_deps="libvmaf ffnvcodec"
>>
>> to
>>
>>   > libvmaf_cuda_filter_deps="libvmaf libvmaf_cuda ffnvcodec"
>>
>>
>> And the pkg_config check would then enable/disable libvmaf_cuda:
>>
>>> enabled libvmaf_cuda      && require_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init
>>
> 
> I'm not sure this works since `libvmaf_cuda` is not a separate
> library. `libvmaf_cuda` is the name of the avfilter, which requires a
> `libvmaf` configured/built with cuda support.

I don't see why it wouldn't work. It doesn't matter what it is, it's 
just the name of the dependency.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

[-- Attachment #1: Type: text/plain, Size: 235 bytes --]

Hi,

On Fri, Sep 15, 2023 at 3:34 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
> I don't see why it wouldn't work. It doesn't matter what it is, it's
> just the name of the dependency.

Patch attached.

Thanks,
Kyle

[-- Attachment #2: 0001-avfilter-add-libvmaf_cuda.patch --]
[-- Type: application/octet-stream, Size: 11199 bytes --]

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Marvin Scholz]

>  enabled libvmaf           && require_pkg_config libvmaf "libvmaf >= 2.0.0" libvmaf.h vmaf_init
> +enabled libvmaf           && require_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init
>  enabled libvo_amrwbenc    && require libvo_amrwbenc vo-amrwbenc/enc_if.h E_IF_init -lvo-amrwbenc
>  enabled libvorbis         && require_pkg_config libvorbis vorbis vorbis/codec.h vorbis_info_init &&
>                               require_pkg_config libvorbisenc vorbisenc vorbis/vorbisenc.h vorbis_encode_init

I am far from an expert with the configure script but won't that cause --enable-libvmaf to fail when
libvmaf is built without cuda support? Which seems undesirable to me…

On 18 Sep 2023, at 18:42, Kyle Swanson wrote:

> Hi,
>
> On Fri, Sep 15, 2023 at 3:34 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>> I don't see why it wouldn't work. It doesn't matter what it is, it's
>> just the name of the dependency.
>
> Patch attached.
>
> Thanks,
> Kyle
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Timo Rothenpieler]

On 18.09.2023 21:21, Marvin Scholz wrote:
>>   enabled libvmaf           && require_pkg_config libvmaf "libvmaf >= 2.0.0" libvmaf.h vmaf_init
>> +enabled libvmaf           && require_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init
>>   enabled libvo_amrwbenc    && require libvo_amrwbenc vo-amrwbenc/enc_if.h E_IF_init -lvo-amrwbenc
>>   enabled libvorbis         && require_pkg_config libvorbis vorbis vorbis/codec.h vorbis_info_init &&
>>                                require_pkg_config libvorbisenc vorbisenc vorbis/vorbisenc.h vorbis_encode_init
> 
> I am far from an expert with the configure script but won't that cause --enable-libvmaf to fail when
> libvmaf is built without cuda support? Which seems undesirable to me…

Yeah, hence my suggested change of

> enabled libvmaf_cuda      && require_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init 

As far as configure is concerned, those are two entirely separate 
libraries, so they also need to be treated and tested as such.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Mon, Sep 18, 2023 at 12:41 PM Timo Rothenpieler
<timo@rothenpieler.org> wrote:
> On 18.09.2023 21:21, Marvin Scholz wrote:
> > I am far from an expert with the configure script but won't that cause --enable-libvmaf to fail when
> > libvmaf is built without cuda support? Which seems undesirable to me…
>
> Yeah, hence my suggested change of
>
> > enabled libvmaf_cuda      && require_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init

This only works if I add `libvmaf_cuda` to EXTERNAL_LIBRARY_LIST,
otherwise running `./configure --enable-libvmaf-cuda` will fail with
`Unknown option "--enable-libvmaf-cuda".` I'm not sure that's the
right thing to do given there is no such thing as `libvmaf_cuda`. If
you are aware of a different way of handling this, let me know. Most
of the other libraries we link with optional configurations (see
libopus, libvpx, ...) seem to have the pattern I used in this previous
patch [0], which does avoid the problem pointed out by Marvin.

Thanks,
Kyle

[0] http://ffmpeg.org/pipermail/ffmpeg-devel/2023-September/314409.html
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Mon, Sep 18, 2023 at 2:39 PM Kyle Swanson <k@ylo.ph> wrote:
>
> Hi,
>
> On Mon, Sep 18, 2023 at 12:41 PM Timo Rothenpieler
> <timo@rothenpieler.org> wrote:
> > On 18.09.2023 21:21, Marvin Scholz wrote:
> > > I am far from an expert with the configure script but won't that cause --enable-libvmaf to fail when
> > > libvmaf is built without cuda support? Which seems undesirable to me…
> >
> > Yeah, hence my suggested change of
> >
> > > enabled libvmaf_cuda      && require_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init
>
> This only works if I add `libvmaf_cuda` to EXTERNAL_LIBRARY_LIST,
> otherwise running `./configure --enable-libvmaf-cuda` will fail with
> `Unknown option "--enable-libvmaf-cuda".` I'm not sure that's the
> right thing to do given there is no such thing as `libvmaf_cuda`. If
> you are aware of a different way of handling this, let me know. Most
> of the other libraries we link with optional configurations (see
> libopus, libvpx, ...) seem to have the pattern I used in this previous
> patch [0], which does avoid the problem pointed out by Marvin.
>
> Thanks,
> Kyle
>
> [0] http://ffmpeg.org/pipermail/ffmpeg-devel/2023-September/314409.html

Timo, any NAKs regarding this earlier patch? If you're at VDD this
week, maybe we can chat about it if you'd like.

Thanks,
Kyle
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Timo Rothenpieler]

On 20.09.2023 22:06, Kyle Swanson wrote:
> Hi,
> 
> On Mon, Sep 18, 2023 at 2:39 PM Kyle Swanson <k@ylo.ph> wrote:
>>
>> Hi,
>>
>> On Mon, Sep 18, 2023 at 12:41 PM Timo Rothenpieler
>> <timo@rothenpieler.org> wrote:
>>> On 18.09.2023 21:21, Marvin Scholz wrote:
>>>> I am far from an expert with the configure script but won't that cause --enable-libvmaf to fail when
>>>> libvmaf is built without cuda support? Which seems undesirable to me…
>>>
>>> Yeah, hence my suggested change of
>>>
>>>> enabled libvmaf_cuda      && require_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init
>>
>> This only works if I add `libvmaf_cuda` to EXTERNAL_LIBRARY_LIST,
>> otherwise running `./configure --enable-libvmaf-cuda` will fail with
>> `Unknown option "--enable-libvmaf-cuda".` I'm not sure that's the
>> right thing to do given there is no such thing as `libvmaf_cuda`. If
>> you are aware of a different way of handling this, let me know. Most
>> of the other libraries we link with optional configurations (see
>> libopus, libvpx, ...) seem to have the pattern I used in this previous
>> patch [0], which does avoid the problem pointed out by Marvin.
>>
>> Thanks,
>> Kyle
>>
>> [0] http://ffmpeg.org/pipermail/ffmpeg-devel/2023-September/314409.html
> 
> Timo, any NAKs regarding this earlier patch? If you're at VDD this
> week, maybe we can chat about it if you'd like.

The only remaining issue is the configure situation really.
No other obvious issues from my side.

It definitely needs to be sorted out properly.
Either configure needs to treat libvmaf_cuda like it is an entirely 
independent library (it's being added to --help output after all, which 
indicates to me that this is the intended approach).

Or it needs to be a sub-feature of libvmaf, where if libvmaf is enabled, 
support for CUDA is checked for, and then the vmaf_cuda filter needs to 
depend on that check.

All current patches had some kind of mix between the two.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Wed, Sep 20, 2023 at 11:55 PM Timo Rothenpieler <timo@rothenpieler.org>
wrote:
> Or it needs to be a sub-feature of libvmaf, where if libvmaf is enabled,
> support for CUDA is checked for, and then the vmaf_cuda filter needs to
> depend on that check.

This previous patch should do just that [0]. What's missing?

libvmaf: -Denable_cuda=false:
./configure --enable-libvmaf: vf_libvmaf enabled, vf_libvmaf_cuda disabled.

libvmaf: -Denable_cuda=true:
./configure --enable-libvmaf --enable-ffnvcodec: vf_libvmaf enabled,
vf_libvmaf_cuda enabled.
./configure --enable-libvmaf --disable-ffnvcodec: vf_libvmaf enabled,
vf_libvmaf_cuda disabled.
./configure --enable-libvmaf --enable-ffnvcodec: --disable-filter=libvmaf:
vf_libvmaf disabled, vf_libvmaf_cuda enabled.
./configure --enable-libvmaf --enable-ffnvcodec:
--disable-filter=libvmaf_cuda: vf_libvmaf enabled, vf_libvmaf_cuda disabled.

Thanks,
Kyle

 [0] http://ffmpeg.org/pipermail/ffmpeg-devel/2023-September/314409.html
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Timo Rothenpieler]

On 23.09.2023 11:50, Kyle Swanson wrote:
> Hi,
> 
> On Wed, Sep 20, 2023 at 11:55 PM Timo Rothenpieler <timo@rothenpieler.org>
> wrote:
>> Or it needs to be a sub-feature of libvmaf, where if libvmaf is enabled,
>> support for CUDA is checked for, and then the vmaf_cuda filter needs to
>> depend on that check.
> 
> This previous patch should do just that [0]. What's missing?

The checks in there seem both overly complex and also not correct to me.
I've never seen a "*_filter" in a check_pkg_config or similar call.
With this setup, it'd be possible to call configure with 
--enable-libvmaf, and have it success, even though there is no libvmaf 
on the system, as long as the filter is disabled.

If you want to go the implicit route, you'll want the following two things:

For the dependencies

 > libvmaf_cuda_filter_deps="libvmaf libvmaf_cuda ffnvcodec"

And then at some point after the current "require_pkg_config libvmaf" check:

> enabled libvmaf && check_pkg_config libvmaf_cuda "libvmaf >= 2.0.0" libvmaf_cuda.h vmaf_cuda_state_init

I think there's a place in configure where a bunch of such optional 
checks happen.
But I see no issue with throwing it right after the current libvmaf 
check either.



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

[-- Attachment #1: Type: text/plain, Size: 1444 bytes --]

Hi,

On Sat, Sep 23, 2023 at 12:02 PM Timo Rothenpieler <timo@rothenpieler.org>
wrote:

> On 23.09.2023 11:50, Kyle Swanson wrote:
> > Hi,
> >
> > On Wed, Sep 20, 2023 at 11:55 PM Timo Rothenpieler <
> timo@rothenpieler.org>
> > wrote:
> >> Or it needs to be a sub-feature of libvmaf, where if libvmaf is enabled,
> >> support for CUDA is checked for, and then the vmaf_cuda filter needs to
> >> depend on that check.
> >
> > This previous patch should do just that [0]. What's missing?
>
> The checks in there seem both overly complex and also not correct to me.
> I've never seen a "*_filter" in a check_pkg_config or similar call.
> With this setup, it'd be possible to call configure with
> --enable-libvmaf, and have it success, even though there is no libvmaf
> on the system, as long as the filter is disabled.
>
> If you want to go the implicit route, you'll want the following two things:
>
> For the dependencies
>
>  > libvmaf_cuda_filter_deps="libvmaf libvmaf_cuda ffnvcodec"
>
> And then at some point after the current "require_pkg_config libvmaf"
> check:
>
> > enabled libvmaf && check_pkg_config libvmaf_cuda "libvmaf >= 2.0.0"
> libvmaf_cuda.h vmaf_cuda_state_init
>
> I think there's a place in configure where a bunch of such optional
> checks happen.
> But I see no issue with throwing it right after the current libvmaf
> check either.
>

New patch attached.

Thanks,
Kyle

[-- Attachment #2: 0001-avfilter-add-libvmaf_cuda.patch --]
[-- Type: application/octet-stream, Size: 11197 bytes --]

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Timo Rothenpieler]

On 25.09.2023 14:18, Kyle Swanson wrote:
> Hi,
> 
> On Sat, Sep 23, 2023 at 12:02 PM Timo Rothenpieler <timo@rothenpieler.org>
> wrote:
> 
>> On 23.09.2023 11:50, Kyle Swanson wrote:
>>> Hi,
>>>
>>> On Wed, Sep 20, 2023 at 11:55 PM Timo Rothenpieler <
>> timo@rothenpieler.org>
>>> wrote:
>>>> Or it needs to be a sub-feature of libvmaf, where if libvmaf is enabled,
>>>> support for CUDA is checked for, and then the vmaf_cuda filter needs to
>>>> depend on that check.
>>>
>>> This previous patch should do just that [0]. What's missing?
>>
>> The checks in there seem both overly complex and also not correct to me.
>> I've never seen a "*_filter" in a check_pkg_config or similar call.
>> With this setup, it'd be possible to call configure with
>> --enable-libvmaf, and have it success, even though there is no libvmaf
>> on the system, as long as the filter is disabled.
>>
>> If you want to go the implicit route, you'll want the following two things:
>>
>> For the dependencies
>>
>>   > libvmaf_cuda_filter_deps="libvmaf libvmaf_cuda ffnvcodec"
>>
>> And then at some point after the current "require_pkg_config libvmaf"
>> check:
>>
>>> enabled libvmaf && check_pkg_config libvmaf_cuda "libvmaf >= 2.0.0"
>> libvmaf_cuda.h vmaf_cuda_state_init
>>
>> I think there's a place in configure where a bunch of such optional
>> checks happen.
>> But I see no issue with throwing it right after the current libvmaf
>> check either.
>>
> 
> New patch attached.

lgtm

> Thanks,
> Kyle
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avfilter: add libvmaf_cuda

[Kyle Swanson]

Hi,

On Mon, Sep 25, 2023 at 9:10 AM Timo Rothenpieler <timo@rothenpieler.org>
wrote:

> lgtm
>

Pushed.

Thanks,
Kyle
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".