[FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[gnattu via ffmpeg-devel]

There is no device context to be setup, nor devices to be
selected with VideoToolbox. Just a simple return would allow
us to use derived device in filters like
`hwupload=derive_device=videotoolbox`

Signed-off-by: Gnattu OC <gnattuoc@me.com>
---
 libavutil/hwcontext_videotoolbox.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
index fe469dc161..d13199eca7 100644
--- a/libavutil/hwcontext_videotoolbox.c
+++ b/libavutil/hwcontext_videotoolbox.c
@@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
     return 0;
 }
 
+static int vt_device_derive(AVHWDeviceContext *device_ctx,
+                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
+                            int flags)
+{
+    // There is no context to be setup with VT, just return.
+    return 0;
+}
+
 const HWContextType ff_hwcontext_type_videotoolbox = {
     .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
     .name                 = "videotoolbox",
@@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
     .frames_priv_size     = sizeof(VTFramesContext),
 
     .device_create        = vt_device_create,
+    .device_derive        = vt_device_derive,
     .frames_hwctx_size    = sizeof(AVVTFramesContext),
     .frames_init          = vt_frames_init,
     .frames_get_buffer    = vt_get_buffer,
-- 
2.39.3 (Apple Git-145)

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[Mark Thompson]

On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
> There is no device context to be setup, nor devices to be
> selected with VideoToolbox. Just a simple return would allow
> us to use derived device in filters like
> `hwupload=derive_device=videotoolbox`
> 
> Signed-off-by: Gnattu OC <gnattuoc@me.com>
> ---
>   libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 
> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
> index fe469dc161..d13199eca7 100644
> --- a/libavutil/hwcontext_videotoolbox.c
> +++ b/libavutil/hwcontext_videotoolbox.c
> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>       return 0;
>   }
>   
> +static int vt_device_derive(AVHWDeviceContext *device_ctx,
> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
> +                            int flags)
> +{
> +    // There is no context to be setup with VT, just return.
> +    return 0;
> +}
> +
>   const HWContextType ff_hwcontext_type_videotoolbox = {
>       .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>       .name                 = "videotoolbox",
> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>       .frames_priv_size     = sizeof(VTFramesContext),
>   
>       .device_create        = vt_device_create,
> +    .device_derive        = vt_device_derive,
>       .frames_hwctx_size    = sizeof(AVVTFramesContext),
>       .frames_init          = vt_frames_init,
>       .frames_get_buffer    = vt_get_buffer,

This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.

(Also, try -init_hw_device.)

Thanks,

- Mark
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[ChenLiucheng via ffmpeg-devel]

Well this does make sense. 

If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.

> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
> 
> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>> There is no device context to be setup, nor devices to be
>> selected with VideoToolbox. Just a simple return would allow
>> us to use derived device in filters like
>> `hwupload=derive_device=videotoolbox`
>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>> ---
>>  libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>  1 file changed, 9 insertions(+)
>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>> index fe469dc161..d13199eca7 100644
>> --- a/libavutil/hwcontext_videotoolbox.c
>> +++ b/libavutil/hwcontext_videotoolbox.c
>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>      return 0;
>>  }
>>  +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>> +                            int flags)
>> +{
>> +    // There is no context to be setup with VT, just return.
>> +    return 0;
>> +}
>> +
>>  const HWContextType ff_hwcontext_type_videotoolbox = {
>>      .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>      .name                 = "videotoolbox",
>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>      .frames_priv_size     = sizeof(VTFramesContext),
>>        .device_create        = vt_device_create,
>> +    .device_derive        = vt_device_derive,
>>      .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>      .frames_init          = vt_frames_init,
>>      .frames_get_buffer    = vt_get_buffer,
> 
> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
> 
> (Also, try -init_hw_device.)
> 
> Thanks,
> 
> - Mark
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org <mailto:ffmpeg-devel@ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org <mailto:ffmpeg-devel-request@ffmpeg.org> with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[Mark Thompson]

On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
>>
>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>> There is no device context to be setup, nor devices to be
>>> selected with VideoToolbox. Just a simple return would allow
>>> us to use derived device in filters like
>>> `hwupload=derive_device=videotoolbox`
>>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>>> ---
>>>   libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>   1 file changed, 9 insertions(+)
>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>> index fe469dc161..d13199eca7 100644
>>> --- a/libavutil/hwcontext_videotoolbox.c
>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>       return 0;
>>>   }
>>>   +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>> +                            int flags)
>>> +{
>>> +    // There is no context to be setup with VT, just return.
>>> +    return 0;
>>> +}
>>> +
>>>   const HWContextType ff_hwcontext_type_videotoolbox = {
>>>       .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>       .name                 = "videotoolbox",
>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>       .frames_priv_size     = sizeof(VTFramesContext),
>>>         .device_create        = vt_device_create,
>>> +    .device_derive        = vt_device_derive,
>>>       .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>       .frames_init          = vt_frames_init,
>>>       .frames_get_buffer    = vt_get_buffer,
>>
>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>
>> (Also, try -init_hw_device.)
>>
 > Well this does make sense.
 >
 > If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
 >

Yes, it is unfortunate that the ffmpeg utility is missing this feature.

A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.

This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.

Thanks,

- Mark
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[Gnattu OC via ffmpeg-devel]

Actually, if you examine the `hwcontext_videotoolbox.c` file, you will find that the current function for `vt_device_create` is essentially a no-op: it only returns an error if a user attempts to select a device, otherwise it just returns. This is generally how VideoToolbox works. If the `create_device` function is already a no-op, then the derive operation will certainly also be a no-op. All other hardware filters can utilize the derive API, and there are use cases that depend on this API. Thus, it seems inappropriate to me that VideoToolbox users cannot use the derive API just because VideoToolbox does not require any device creation.

> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly

Can't this just be the modification of the `hwupload` filter? Instead of derive, the user just set the which initialized hardware to use, as `-init_hw_device` can already be called multiple times and creating alias for each hardware.

> On Feb 23, 2024, at 03:50, Mark Thompson <sw@jkqxz.net> wrote:
> 
> On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>>> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
>>> 
>>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>>> There is no device context to be setup, nor devices to be
>>>> selected with VideoToolbox. Just a simple return would allow
>>>> us to use derived device in filters like
>>>> `hwupload=derive_device=videotoolbox`
>>>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>>>> ---
>>>>  libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>>  1 file changed, 9 insertions(+)
>>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>>> index fe469dc161..d13199eca7 100644
>>>> --- a/libavutil/hwcontext_videotoolbox.c
>>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>>      return 0;
>>>>  }
>>>>  +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>>> +                            int flags)
>>>> +{
>>>> +    // There is no context to be setup with VT, just return.
>>>> +    return 0;
>>>> +}
>>>> +
>>>>  const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>      .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>>      .name                 = "videotoolbox",
>>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>      .frames_priv_size     = sizeof(VTFramesContext),
>>>>        .device_create        = vt_device_create,
>>>> +    .device_derive        = vt_device_derive,
>>>>      .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>>      .frames_init          = vt_frames_init,
>>>>      .frames_get_buffer    = vt_get_buffer,
>>> 
>>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>> 
>>> (Also, try -init_hw_device.)
>>> 
> > Well this does make sense.
> >
> > If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
> >
> 
> Yes, it is unfortunate that the ffmpeg utility is missing this feature.
> 
> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.
> 
> This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.
> 
> Thanks,
> 
> - Mark
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[Zhao Zhili]

> 在 2024年2月23日，上午4:19，Gnattu OC via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> 写道：
> 
> Actually, if you examine the `hwcontext_videotoolbox.c` file, you will find that the current function for `vt_device_create` is essentially a no-op: it only returns an error if a user attempts to select a device, otherwise it just returns. This is generally how VideoToolbox works. If the `create_device` function is already a no-op, then the derive operation will certainly also be a no-op.

Device derive means inter operation between different hwcontext, which doesn’t supported by videotoolbox yet, so a no-op implementation for now is misleading.

I’m trying to implement opencl and videotoolbox inter operation, the no-op implementation can make sense afterwards.

> All other hardware filters can utilize the derive API, and there are use cases that depend on this API. Thus, it seems inappropriate to me that VideoToolbox users cannot use the derive API just because VideoToolbox does not require any device creation.
> 
>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly
> 
> Can't this just be the modification of the `hwupload` filter? Instead of derive, the user just set the which initialized hardware to use, as `-init_hw_device` can already be called multiple times and creating alias for each hardware.
> 
>> On Feb 23, 2024, at 03:50, Mark Thompson <sw@jkqxz.net> wrote:
>> 
>> On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>>>> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
>>>> 
>>>>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>>>>> There is no device context to be setup, nor devices to be
>>>>>> selected with VideoToolbox. Just a simple return would allow
>>>>>> us to use derived device in filters like
>>>>>> `hwupload=derive_device=videotoolbox`
>>>>>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>>>>>> ---
>>>>>> libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>>>> 1 file changed, 9 insertions(+)
>>>>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>>>>> index fe469dc161..d13199eca7 100644
>>>>>> --- a/libavutil/hwcontext_videotoolbox.c
>>>>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>>>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>>>>     return 0;
>>>>>> }
>>>>>> +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>>>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>>>>> +                            int flags)
>>>>>> +{
>>>>>> +    // There is no context to be setup with VT, just return.
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>     .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>>>>     .name                 = "videotoolbox",
>>>>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>     .frames_priv_size     = sizeof(VTFramesContext),
>>>>>>       .device_create        = vt_device_create,
>>>>>> +    .device_derive        = vt_device_derive,
>>>>>>     .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>>>>     .frames_init          = vt_frames_init,
>>>>>>     .frames_get_buffer    = vt_get_buffer,
>>>>> 
>>>>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>>>> 
>>>>> (Also, try -init_hw_device.)
>>>>> 
>>> Well this does make sense.
>>> 
>>> If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
>>> 
>> 
>> Yes, it is unfortunate that the ffmpeg utility is missing this feature.
>> 
>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.
>> 
>> This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.
>> 
>> Thanks,
>> 
>> - Mark
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[Zhao Zhili]

> 在 2024年2月23日，上午9:20，Zhao Zhili <quinkblack@foxmail.com> 写道：
> 
> 
>> 在 2024年2月23日，上午4:19，Gnattu OC via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> 写道：
>> 
>> Actually, if you examine the `hwcontext_videotoolbox.c` file, you will find that the current function for `vt_device_create` is essentially a no-op: it only returns an error if a user attempts to select a device, otherwise it just returns. This is generally how VideoToolbox works. If the `create_device` function is already a no-op, then the derive operation will certainly also be a no-op.
> 
> Device derive means inter operation between different hwcontext, which doesn’t supported by videotoolbox yet, so a no-op implementation for now is misleading.
> 
> I’m trying to implement opencl and videotoolbox inter operation, the no-op implementation can make sense afterwards.

I mean a real implementation of device derive.

> 
>> All other hardware filters can utilize the derive API, and there are use cases that depend on this API. Thus, it seems inappropriate to me that VideoToolbox users cannot use the derive API just because VideoToolbox does not require any device creation.
>> 
>>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly
>> 
>> Can't this just be the modification of the `hwupload` filter? Instead of derive, the user just set the which initialized hardware to use, as `-init_hw_device` can already be called multiple times and creating alias for each hardware.
>> 
>>>> On Feb 23, 2024, at 03:50, Mark Thompson <sw@jkqxz.net> wrote:
>>> 
>>> On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>>>>> On Feb 23, 2024, at 03:28, Mark Thompson <sw@jkqxz.net> wrote:
>>>>> 
>>>>>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>>>>>> There is no device context to be setup, nor devices to be
>>>>>>> selected with VideoToolbox. Just a simple return would allow
>>>>>>> us to use derived device in filters like
>>>>>>> `hwupload=derive_device=videotoolbox`
>>>>>>> Signed-off-by: Gnattu OC <gnattuoc@me.com>
>>>>>>> ---
>>>>>>> libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>>>>> 1 file changed, 9 insertions(+)
>>>>>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>>>>>> index fe469dc161..d13199eca7 100644
>>>>>>> --- a/libavutil/hwcontext_videotoolbox.c
>>>>>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>>>>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>>>>>    return 0;
>>>>>>> }
>>>>>>> +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>>>>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>>>>>> +                            int flags)
>>>>>>> +{
>>>>>>> +    // There is no context to be setup with VT, just return.
>>>>>>> +    return 0;
>>>>>>> +}
>>>>>>> +
>>>>>>> const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>>    .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>>>>>    .name                 = "videotoolbox",
>>>>>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>>    .frames_priv_size     = sizeof(VTFramesContext),
>>>>>>>      .device_create        = vt_device_create,
>>>>>>> +    .device_derive        = vt_device_derive,
>>>>>>>    .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>>>>>    .frames_init          = vt_frames_init,
>>>>>>>    .frames_get_buffer    = vt_get_buffer,
>>>>>> 
>>>>>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>>>>> 
>>>>>> (Also, try -init_hw_device.)
>>>>>> 
>>>> Well this does make sense.
>>>> 
>>>> If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
>>>> 
>>> 
>>> Yes, it is unfortunate that the ffmpeg utility is missing this feature.
>>> 
>>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.
>>> 
>>> This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.
>>> 
>>> Thanks,
>>> 
>>> - Mark
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel@ffmpeg.org
>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>> 
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>> 
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.o

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[Zhao Zhili]

[-- Attachment #1: favicon.ico --]
[-- Type: image/vnd.microsoft.icon, Size: 32038 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[Gnattu OC via ffmpeg-devel]

I’ve tested and confirmed to work, and `reverse=1` also works for chaining to other videotoolbox filters. 

> On Feb 23, 2024, at 22:25, Zhao Zhili <quinkblack@foxmail.com> wrote:
> 
> <favicon.ico>_______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".