[FFmpeg-devel] backends of libavfilter/dnn module

From: "Guo, Yejun" <yejun.guo-at-intel.com@ffmpeg.org>
To: "ffmpeg-devel@ffmpeg.org" <ffmpeg-devel@ffmpeg.org>
Cc: "Fu, Ting" <ting.fu@intel.com>, Pedro Arthur <bygrandao@gmail.com>
Subject: [FFmpeg-devel] backends of libavfilter/dnn module
Date: Sat, 3 Dec 2022 09:32:37 +0000
Message-ID: <PH7PR11MB5957913426BA091237AD89C9F1169@PH7PR11MB5957.namprd11.prod.outlook.com> (raw)
In-Reply-To: <23dfad2f-db16-cf49-b12e-d43fd0edeae5@gmail.com>

Hi,

There are discussions about dnn module at https://etherpad.mit.edu/p/FF_dev_meeting_20221202.

1) Regarding "Delete the native backend? Yes",
I agree since it is performance sensitive, and I can take this effort. 

And want to get advice on how to delete it. Usually, we'd mark it as deprecated, and then delete after a long time. As for this case, my idea is:
Step1: delete code of native backend under libavfilter/dnn, and also add error message in filters to let end user get a clear message that it is not supported.
Step2:  after a long time (maybe next major release), delete the 'error message' (all code relative to native backend) in filters.

2) Regarding "Move to a separate library? Delete? Move to somewhere else?",
@Pedro Arthur  any comment?

And I have two other opens about the backend.

3) Adding libtorch as a new backend after native backend is deleted.
Deep learning models are usually developed on different frameworks and so the model files are in different format. The current most popular framework is probably pytorch and we see lots of new model files in pytorch format. My idea is to embrace libtorch as a new backend, and @Fu, Ting has finished the code and is willing to upstream it and also adding a new feature in vf_dnn_processing.c with basic VSR (video super resolution) whose model file is now only available in pytorch format.

4) how about many other deep learning frameworks?
There are also many other promising dnn inference frameworks, and how do we support them? One method is that they add a glue layer to implement the interface in libavfilter/dnn_interface.h, and we (ffmpeg) can dlopen the glue layer library. Actually, the implementation can be another framework, or even be a service. Anyway, this can be a long term solution, and can be discussed in more detail when this requirement appears.

Thanks
Yejun
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".