Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

[Olivier Ayache <olivier.ayache@gmail.com>]

Hello there I implemented that few years ago in
https://github.com/olivierayache/xuggle-xuggler/
And when I tried to submit a patch to FFmpeg for add support for NDK I had
received this answer.
I think using NDK directly rather than JNI can be better in case of
developing native application on Android.
If I can help on these subjects I would be really happy

Olivier Ayache



---------- Forwarded message ---------
From: Olivier Ayache <olivier.ayache@gmail.com>
Date: Sun, Jun 28, 2020 at 2:48 PM
Subject: Re: [FFmpeg-devel] [PATCH] add support to Android ndk MediaCodec
for encoding/decoding
To: <ffmpeg-devel@ffmpeg.org>


Thank for your replies I choose the NDK approach in order to  be able to be
independant from the JVM, to maximize performance and to be sure to detect
compatibility issues (using JNI approach can compile and crash at runtime).
Concerning the version of Android it is compatible from API 21 (94% of
devices)
I agree that JNI overhead is not big but in combinaison with Xuggler it
seems to me a little weird because Xuggler (Java part) already use JNI to
interact with Xuggler (C++) and FFmpeg. If FFmpeg uses JNI in the other way
(from c to Java) it seems to me an anti pattern.

If you think it is better to begin to use JNI I could transform my encoder
and we could discuss about NDK after

Olivier



Le sam. 27 juin 2020 à 20:55, Martin Storsjö <martin@martin.st> a écrit :

> On Sat, 27 Jun 2020, Olivier Ayache wrote:
>
> > Hi everyone this is the first time I post on this mailing list. I am
> > working since several years on a fork of Xuggler for manipulating ffmpeg
> > API with Java/Kotlin.
> > This work leads me to develop encoder and decoder based on Android NDK
> > MediaCodec.
> >
> > This work can be found on my Github repository
> >
> > https://github.com/olivierayache/xuggle-xuggler
> >
> >
> > I know that FFmpeg already integrate MediaCodec for decoding via Jni
> > wrappers since version 3.1. I began this work on FFmpeg 2.8.x and I
> choose
> > the NDK in order to achieve optimum performance.
>
> If you mean you used the NDK MediaCodec API, I'd suggest you use the same
> JNI wrappers as ffmpeg has already, for consistency. The overhead of a few
> JNI calls per encoded frame is generally negligible. If it, at a later
> time, is decided to drop support for older versions at some point, it
> should be straightforward to convert it to use the NDK MediaCodec API
> instead.
>
> // Martin
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".


On Sat, Nov 12, 2022 at 6:07 AM Zhao Zhili <quinkblack@foxmail.com> wrote:

>
>
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Chema
> Gonzalez
> > Sent: 2022年11月12日 0:55
> > To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> > Cc: matthieu.bouron@stupeflix.com; aman@tmm1.net
> > Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec
> encoder and NDK MediaCodec support
> >
> > On Thu, Nov 10, 2022 at 7:36 PM "zhilizhao(赵志立)" <quinkblack@foxmail.com>
> wrote:
> > >
> > > Ping for review.
> > >
> > > > On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack@foxmail.com>
> wrote:
> > > >
> > > > From: Zhao Zhili <zhilizhao@tencent.com>
> > > >
> > > > Firstly, some bugs were fixed (patch 1-4).
> > > >
> > > > Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK
> > > > MediaCodec. The use case I'm considering is run FFmpeg on cmdline
> without JVM,
> > > > for example, run FFmpeg inside of termux (an Android terminal
> emulator). It's
> > > > well known that NDK MediaCodec missing some important functions,
> like get the
> > > > list of codecs, but still useable.
> > > >
> > > > Patch 7 add NDK MediaCodec decoder support. It can be enabled via
> options,
> > > > and enabled automatically if no JVM is available.
> > > >
> > > > Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be
> set by
> > > > user, and can be created via AMediaCodec_createPersistentInputSurface
> > > > automatically. This is a preparation for encoder.
> > > >
> > > > Patch 9 makes MediaCodec decoder to support ANativeWindow directly.
> It worth
> > > > to note that AVMediaCodecContext has only surface. Although we
> provided
> > > > av_mediacodec_alloc_context(), we didn't strictly prevent users to
> allocate
> > > > AVMediaCodecContext on stack. I'm not sure if it's OK to add new
> field to
> > > > AVMediaCodecContext.
> > > >
> > > > Patch 10 add MediaCodec encoder support. Frame can be feed to
> encoder via
> > > > buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is
> used, and
> > > > the frames come from our MediaCodec decoder wrapper, we can control
> it's
> > > > 'render' (send to encoder's surface) via
> av_mediacodec_release_buffer(). A DTS
> > > > generation strategy works in this case. However, if frames comes
> from other
> > > > sources, like a camera, there is no way to control the 'render' yet,
> so DTS is
> > > > missing in this case.
> > > >
> > > > Finally, we can do mediacodec transcoding with FFmpeg cmdline on
> Android.
> > > > More importantly, we can do MediaCodec decoder to encoder without
> copy frames,
> > > > although it's very limited since most of avfilters doesn't work. For
> example:
> > > >
> > > > ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i
> /sdcard/test.mp4 -an -c:v h264_mediacodec -y
> > /sdcard/out.mp4
> >
> > How do you build the ffmpeg binary you're using for experiments in the
> > android device?
>
> Just configure with --enable-mediacodec and --enable-jni. Here is the build
> script I used for the development.
>
> https://github.com/quink-black/ffmpeg-ci/blob/master/android_ffmpeg.sh
>
> Here is the Android Studio project which can be used to test the mediacodec
> wrapper implemented via JNI.
>
> >
> >
> >
> >
> > > >
> > > > Since there is no real AVHWFrameContext implementation in
> hwcontext_mediacodec.
> > > > there is no hwframe_ctx for mediacodec and
> av_hwframe_transfer_data() doesn't
> > > > work. So if -hwaccel_output_format isn't being specified like:
> > > >
> > > > ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v
> h264_mediacodec -y /sdcard/out.mp4
> > > >
> > > > It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a
> check on
> > > > hwframe_ctx. Patch 12 set hwaccel_output_format automatically to
> avoid such
> > > > case.
> > > >
> > > >
> > > > Zhao Zhili (12):
> > > >  avcodec/mediacodec: fix incorrect crop info
> > > >  avcodec/mediacodecdec: don't break out if both input and output port
> > > >    return try again
> > > >  avcodec/mediacodecdec_common: fix misuse av_free/av_freep
> > > >  avcodec/mediacodecdec_common: fix useless av_buffer_unref
> > > >  avcodec/mediacodec_wrapper: separate implementation from interface
> > > >  avcodec/mediacodec: add NDK media codec wrapper
> > > >  avcodec/mediacodecdec: enable NDK mediacodec
> > > >  avutil/hwcontext_mediacodec: add ANativeWindow support
> > > >  avcodec/mediacodec: add ANativeWindow support
> > > >  avcodec: add MediaCodec encoder
> > > >  avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
> > > >  fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec
> > > >
> > > > Changelog                         |   2 +
> > > > configure                         |   6 +
> > > > fftools/ffmpeg_opt.c              |   4 +
> > > > libavcodec/Makefile               |   2 +
> > > > libavcodec/allcodecs.c            |   2 +
> > > > libavcodec/mediacodec_surface.c   |  46 +-
> > > > libavcodec/mediacodec_surface.h   |   8 +-
> > > > libavcodec/mediacodec_wrapper.c   | 942
> +++++++++++++++++++++++++++---
> > > > libavcodec/mediacodec_wrapper.h   | 275 +++++++--
> > > > libavcodec/mediacodecdec.c        |  21 +-
> > > > libavcodec/mediacodecdec_common.c |  35 +-
> > > > libavcodec/mediacodecdec_common.h |   1 +
> > > > libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
> > > > libavcodec/version.h              |   2 +-
> > > > libavutil/hwcontext.c             |   6 +-
> > > > libavutil/hwcontext_mediacodec.c  |  56 +-
> > > > libavutil/hwcontext_mediacodec.h  |  11 +
> > > > libavutil/version.h               |   4 +-
> > > > 18 files changed, 1780 insertions(+), 138 deletions(-)
> > > > create mode 100644 libavcodec/mediacodecenc.c
> > > >
> > > > --
> > > > 2.25.1
> > > >
> > > >
> > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".