Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

["zhilizhao(赵志立)"]

Ping for review.

> On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack@foxmail.com> wrote:
> 
> From: Zhao Zhili <zhilizhao@tencent.com>
> 
> Firstly, some bugs were fixed (patch 1-4).
> 
> Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK
> MediaCodec. The use case I'm considering is run FFmpeg on cmdline without JVM,
> for example, run FFmpeg inside of termux (an Android terminal emulator). It's
> well known that NDK MediaCodec missing some important functions, like get the
> list of codecs, but still useable.
> 
> Patch 7 add NDK MediaCodec decoder support. It can be enabled via options,
> and enabled automatically if no JVM is available.
> 
> Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be set by
> user, and can be created via AMediaCodec_createPersistentInputSurface
> automatically. This is a preparation for encoder.
> 
> Patch 9 makes MediaCodec decoder to support ANativeWindow directly. It worth
> to note that AVMediaCodecContext has only surface. Although we provided
> av_mediacodec_alloc_context(), we didn't strictly prevent users to allocate
> AVMediaCodecContext on stack. I'm not sure if it's OK to add new field to
> AVMediaCodecContext.
> 
> Patch 10 add MediaCodec encoder support. Frame can be feed to encoder via
> buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is used, and
> the frames come from our MediaCodec decoder wrapper, we can control it's
> 'render' (send to encoder's surface) via av_mediacodec_release_buffer(). A DTS
> generation strategy works in this case. However, if frames comes from other
> sources, like a camera, there is no way to control the 'render' yet, so DTS is
> missing in this case.
> 
> Finally, we can do mediacodec transcoding with FFmpeg cmdline on Android.
> More importantly, we can do MediaCodec decoder to encoder without copy frames,
> although it's very limited since most of avfilters doesn't work. For example:
> 
> ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4
> 
> Since there is no real AVHWFrameContext implementation in hwcontext_mediacodec.
> there is no hwframe_ctx for mediacodec and av_hwframe_transfer_data() doesn't
> work. So if -hwaccel_output_format isn't being specified like:
> 
> ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4
> 
> It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a check on
> hwframe_ctx. Patch 12 set hwaccel_output_format automatically to avoid such
> case.
> 
> 
> Zhao Zhili (12):
>  avcodec/mediacodec: fix incorrect crop info
>  avcodec/mediacodecdec: don't break out if both input and output port
>    return try again
>  avcodec/mediacodecdec_common: fix misuse av_free/av_freep
>  avcodec/mediacodecdec_common: fix useless av_buffer_unref
>  avcodec/mediacodec_wrapper: separate implementation from interface
>  avcodec/mediacodec: add NDK media codec wrapper
>  avcodec/mediacodecdec: enable NDK mediacodec
>  avutil/hwcontext_mediacodec: add ANativeWindow support
>  avcodec/mediacodec: add ANativeWindow support
>  avcodec: add MediaCodec encoder
>  avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
>  fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec
> 
> Changelog                         |   2 +
> configure                         |   6 +
> fftools/ffmpeg_opt.c              |   4 +
> libavcodec/Makefile               |   2 +
> libavcodec/allcodecs.c            |   2 +
> libavcodec/mediacodec_surface.c   |  46 +-
> libavcodec/mediacodec_surface.h   |   8 +-
> libavcodec/mediacodec_wrapper.c   | 942 +++++++++++++++++++++++++++---
> libavcodec/mediacodec_wrapper.h   | 275 +++++++--
> libavcodec/mediacodecdec.c        |  21 +-
> libavcodec/mediacodecdec_common.c |  35 +-
> libavcodec/mediacodecdec_common.h |   1 +
> libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
> libavcodec/version.h              |   2 +-
> libavutil/hwcontext.c             |   6 +-
> libavutil/hwcontext_mediacodec.c  |  56 +-
> libavutil/hwcontext_mediacodec.h  |  11 +
> libavutil/version.h               |   4 +-
> 18 files changed, 1780 insertions(+), 138 deletions(-)
> create mode 100644 libavcodec/mediacodecenc.c
> 
> -- 
> 2.25.1
> 
> 

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

[Chema Gonzalez]

On Thu, Nov 10, 2022 at 7:36 PM "zhilizhao(赵志立)" <quinkblack@foxmail.com> wrote:
>
> Ping for review.
>
> > On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack@foxmail.com> wrote:
> >
> > From: Zhao Zhili <zhilizhao@tencent.com>
> >
> > Firstly, some bugs were fixed (patch 1-4).
> >
> > Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK
> > MediaCodec. The use case I'm considering is run FFmpeg on cmdline without JVM,
> > for example, run FFmpeg inside of termux (an Android terminal emulator). It's
> > well known that NDK MediaCodec missing some important functions, like get the
> > list of codecs, but still useable.
> >
> > Patch 7 add NDK MediaCodec decoder support. It can be enabled via options,
> > and enabled automatically if no JVM is available.
> >
> > Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be set by
> > user, and can be created via AMediaCodec_createPersistentInputSurface
> > automatically. This is a preparation for encoder.
> >
> > Patch 9 makes MediaCodec decoder to support ANativeWindow directly. It worth
> > to note that AVMediaCodecContext has only surface. Although we provided
> > av_mediacodec_alloc_context(), we didn't strictly prevent users to allocate
> > AVMediaCodecContext on stack. I'm not sure if it's OK to add new field to
> > AVMediaCodecContext.
> >
> > Patch 10 add MediaCodec encoder support. Frame can be feed to encoder via
> > buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is used, and
> > the frames come from our MediaCodec decoder wrapper, we can control it's
> > 'render' (send to encoder's surface) via av_mediacodec_release_buffer(). A DTS
> > generation strategy works in this case. However, if frames comes from other
> > sources, like a camera, there is no way to control the 'render' yet, so DTS is
> > missing in this case.
> >
> > Finally, we can do mediacodec transcoding with FFmpeg cmdline on Android.
> > More importantly, we can do MediaCodec decoder to encoder without copy frames,
> > although it's very limited since most of avfilters doesn't work. For example:
> >
> > ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4

How do you build the ffmpeg binary you're using for experiments in the
android device?




> >
> > Since there is no real AVHWFrameContext implementation in hwcontext_mediacodec.
> > there is no hwframe_ctx for mediacodec and av_hwframe_transfer_data() doesn't
> > work. So if -hwaccel_output_format isn't being specified like:
> >
> > ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4
> >
> > It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a check on
> > hwframe_ctx. Patch 12 set hwaccel_output_format automatically to avoid such
> > case.
> >
> >
> > Zhao Zhili (12):
> >  avcodec/mediacodec: fix incorrect crop info
> >  avcodec/mediacodecdec: don't break out if both input and output port
> >    return try again
> >  avcodec/mediacodecdec_common: fix misuse av_free/av_freep
> >  avcodec/mediacodecdec_common: fix useless av_buffer_unref
> >  avcodec/mediacodec_wrapper: separate implementation from interface
> >  avcodec/mediacodec: add NDK media codec wrapper
> >  avcodec/mediacodecdec: enable NDK mediacodec
> >  avutil/hwcontext_mediacodec: add ANativeWindow support
> >  avcodec/mediacodec: add ANativeWindow support
> >  avcodec: add MediaCodec encoder
> >  avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
> >  fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec
> >
> > Changelog                         |   2 +
> > configure                         |   6 +
> > fftools/ffmpeg_opt.c              |   4 +
> > libavcodec/Makefile               |   2 +
> > libavcodec/allcodecs.c            |   2 +
> > libavcodec/mediacodec_surface.c   |  46 +-
> > libavcodec/mediacodec_surface.h   |   8 +-
> > libavcodec/mediacodec_wrapper.c   | 942 +++++++++++++++++++++++++++---
> > libavcodec/mediacodec_wrapper.h   | 275 +++++++--
> > libavcodec/mediacodecdec.c        |  21 +-
> > libavcodec/mediacodecdec_common.c |  35 +-
> > libavcodec/mediacodecdec_common.h |   1 +
> > libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
> > libavcodec/version.h              |   2 +-
> > libavutil/hwcontext.c             |   6 +-
> > libavutil/hwcontext_mediacodec.c  |  56 +-
> > libavutil/hwcontext_mediacodec.h  |  11 +
> > libavutil/version.h               |   4 +-
> > 18 files changed, 1780 insertions(+), 138 deletions(-)
> > create mode 100644 libavcodec/mediacodecenc.c
> >
> > --
> > 2.25.1
> >
> >
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

[Zhao Zhili]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Chema Gonzalez
> Sent: 2022年11月12日 0:55
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Cc: matthieu.bouron@stupeflix.com; aman@tmm1.net
> Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support
> 
> On Thu, Nov 10, 2022 at 7:36 PM "zhilizhao(赵志立)" <quinkblack@foxmail.com> wrote:
> >
> > Ping for review.
> >
> > > On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack@foxmail.com> wrote:
> > >
> > > From: Zhao Zhili <zhilizhao@tencent.com>
> > >
> > > Firstly, some bugs were fixed (patch 1-4).
> > >
> > > Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK
> > > MediaCodec. The use case I'm considering is run FFmpeg on cmdline without JVM,
> > > for example, run FFmpeg inside of termux (an Android terminal emulator). It's
> > > well known that NDK MediaCodec missing some important functions, like get the
> > > list of codecs, but still useable.
> > >
> > > Patch 7 add NDK MediaCodec decoder support. It can be enabled via options,
> > > and enabled automatically if no JVM is available.
> > >
> > > Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be set by
> > > user, and can be created via AMediaCodec_createPersistentInputSurface
> > > automatically. This is a preparation for encoder.
> > >
> > > Patch 9 makes MediaCodec decoder to support ANativeWindow directly. It worth
> > > to note that AVMediaCodecContext has only surface. Although we provided
> > > av_mediacodec_alloc_context(), we didn't strictly prevent users to allocate
> > > AVMediaCodecContext on stack. I'm not sure if it's OK to add new field to
> > > AVMediaCodecContext.
> > >
> > > Patch 10 add MediaCodec encoder support. Frame can be feed to encoder via
> > > buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is used, and
> > > the frames come from our MediaCodec decoder wrapper, we can control it's
> > > 'render' (send to encoder's surface) via av_mediacodec_release_buffer(). A DTS
> > > generation strategy works in this case. However, if frames comes from other
> > > sources, like a camera, there is no way to control the 'render' yet, so DTS is
> > > missing in this case.
> > >
> > > Finally, we can do mediacodec transcoding with FFmpeg cmdline on Android.
> > > More importantly, we can do MediaCodec decoder to encoder without copy frames,
> > > although it's very limited since most of avfilters doesn't work. For example:
> > >
> > > ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y
> /sdcard/out.mp4
> 
> How do you build the ffmpeg binary you're using for experiments in the
> android device?

Just configure with --enable-mediacodec and --enable-jni. Here is the build
script I used for the development.

https://github.com/quink-black/ffmpeg-ci/blob/master/android_ffmpeg.sh

Here is the Android Studio project which can be used to test the mediacodec
wrapper implemented via JNI.

> 
> 
> 
> 
> > >
> > > Since there is no real AVHWFrameContext implementation in hwcontext_mediacodec.
> > > there is no hwframe_ctx for mediacodec and av_hwframe_transfer_data() doesn't
> > > work. So if -hwaccel_output_format isn't being specified like:
> > >
> > > ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4
> > >
> > > It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a check on
> > > hwframe_ctx. Patch 12 set hwaccel_output_format automatically to avoid such
> > > case.
> > >
> > >
> > > Zhao Zhili (12):
> > >  avcodec/mediacodec: fix incorrect crop info
> > >  avcodec/mediacodecdec: don't break out if both input and output port
> > >    return try again
> > >  avcodec/mediacodecdec_common: fix misuse av_free/av_freep
> > >  avcodec/mediacodecdec_common: fix useless av_buffer_unref
> > >  avcodec/mediacodec_wrapper: separate implementation from interface
> > >  avcodec/mediacodec: add NDK media codec wrapper
> > >  avcodec/mediacodecdec: enable NDK mediacodec
> > >  avutil/hwcontext_mediacodec: add ANativeWindow support
> > >  avcodec/mediacodec: add ANativeWindow support
> > >  avcodec: add MediaCodec encoder
> > >  avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
> > >  fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec
> > >
> > > Changelog                         |   2 +
> > > configure                         |   6 +
> > > fftools/ffmpeg_opt.c              |   4 +
> > > libavcodec/Makefile               |   2 +
> > > libavcodec/allcodecs.c            |   2 +
> > > libavcodec/mediacodec_surface.c   |  46 +-
> > > libavcodec/mediacodec_surface.h   |   8 +-
> > > libavcodec/mediacodec_wrapper.c   | 942 +++++++++++++++++++++++++++---
> > > libavcodec/mediacodec_wrapper.h   | 275 +++++++--
> > > libavcodec/mediacodecdec.c        |  21 +-
> > > libavcodec/mediacodecdec_common.c |  35 +-
> > > libavcodec/mediacodecdec_common.h |   1 +
> > > libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
> > > libavcodec/version.h              |   2 +-
> > > libavutil/hwcontext.c             |   6 +-
> > > libavutil/hwcontext_mediacodec.c  |  56 +-
> > > libavutil/hwcontext_mediacodec.h  |  11 +
> > > libavutil/version.h               |   4 +-
> > > 18 files changed, 1780 insertions(+), 138 deletions(-)
> > > create mode 100644 libavcodec/mediacodecenc.c
> > >
> > > --
> > > 2.25.1
> > >
> > >
> >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

[Olivier Ayache]

Hello there I implemented that few years ago in
https://github.com/olivierayache/xuggle-xuggler/
And when I tried to submit a patch to FFmpeg for add support for NDK I had
received this answer.
I think using NDK directly rather than JNI can be better in case of
developing native application on Android.
If I can help on these subjects I would be really happy

Olivier Ayache



---------- Forwarded message ---------
From: Olivier Ayache <olivier.ayache@gmail.com>
Date: Sun, Jun 28, 2020 at 2:48 PM
Subject: Re: [FFmpeg-devel] [PATCH] add support to Android ndk MediaCodec
for encoding/decoding
To: <ffmpeg-devel@ffmpeg.org>


Thank for your replies I choose the NDK approach in order to  be able to be
independant from the JVM, to maximize performance and to be sure to detect
compatibility issues (using JNI approach can compile and crash at runtime).
Concerning the version of Android it is compatible from API 21 (94% of
devices)
I agree that JNI overhead is not big but in combinaison with Xuggler it
seems to me a little weird because Xuggler (Java part) already use JNI to
interact with Xuggler (C++) and FFmpeg. If FFmpeg uses JNI in the other way
(from c to Java) it seems to me an anti pattern.

If you think it is better to begin to use JNI I could transform my encoder
and we could discuss about NDK after

Olivier



Le sam. 27 juin 2020 à 20:55, Martin Storsjö <martin@martin.st> a écrit :

> On Sat, 27 Jun 2020, Olivier Ayache wrote:
>
> > Hi everyone this is the first time I post on this mailing list. I am
> > working since several years on a fork of Xuggler for manipulating ffmpeg
> > API with Java/Kotlin.
> > This work leads me to develop encoder and decoder based on Android NDK
> > MediaCodec.
> >
> > This work can be found on my Github repository
> >
> > https://github.com/olivierayache/xuggle-xuggler
> >
> >
> > I know that FFmpeg already integrate MediaCodec for decoding via Jni
> > wrappers since version 3.1. I began this work on FFmpeg 2.8.x and I
> choose
> > the NDK in order to achieve optimum performance.
>
> If you mean you used the NDK MediaCodec API, I'd suggest you use the same
> JNI wrappers as ffmpeg has already, for consistency. The overhead of a few
> JNI calls per encoded frame is generally negligible. If it, at a later
> time, is decided to drop support for older versions at some point, it
> should be straightforward to convert it to use the NDK MediaCodec API
> instead.
>
> // Martin
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".


On Sat, Nov 12, 2022 at 6:07 AM Zhao Zhili <quinkblack@foxmail.com> wrote:

>
>
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Chema
> Gonzalez
> > Sent: 2022年11月12日 0:55
> > To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> > Cc: matthieu.bouron@stupeflix.com; aman@tmm1.net
> > Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec
> encoder and NDK MediaCodec support
> >
> > On Thu, Nov 10, 2022 at 7:36 PM "zhilizhao(赵志立)" <quinkblack@foxmail.com>
> wrote:
> > >
> > > Ping for review.
> > >
> > > > On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack@foxmail.com>
> wrote:
> > > >
> > > > From: Zhao Zhili <zhilizhao@tencent.com>
> > > >
> > > > Firstly, some bugs were fixed (patch 1-4).
> > > >
> > > > Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK
> > > > MediaCodec. The use case I'm considering is run FFmpeg on cmdline
> without JVM,
> > > > for example, run FFmpeg inside of termux (an Android terminal
> emulator). It's
> > > > well known that NDK MediaCodec missing some important functions,
> like get the
> > > > list of codecs, but still useable.
> > > >
> > > > Patch 7 add NDK MediaCodec decoder support. It can be enabled via
> options,
> > > > and enabled automatically if no JVM is available.
> > > >
> > > > Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be
> set by
> > > > user, and can be created via AMediaCodec_createPersistentInputSurface
> > > > automatically. This is a preparation for encoder.
> > > >
> > > > Patch 9 makes MediaCodec decoder to support ANativeWindow directly.
> It worth
> > > > to note that AVMediaCodecContext has only surface. Although we
> provided
> > > > av_mediacodec_alloc_context(), we didn't strictly prevent users to
> allocate
> > > > AVMediaCodecContext on stack. I'm not sure if it's OK to add new
> field to
> > > > AVMediaCodecContext.
> > > >
> > > > Patch 10 add MediaCodec encoder support. Frame can be feed to
> encoder via
> > > > buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is
> used, and
> > > > the frames come from our MediaCodec decoder wrapper, we can control
> it's
> > > > 'render' (send to encoder's surface) via
> av_mediacodec_release_buffer(). A DTS
> > > > generation strategy works in this case. However, if frames comes
> from other
> > > > sources, like a camera, there is no way to control the 'render' yet,
> so DTS is
> > > > missing in this case.
> > > >
> > > > Finally, we can do mediacodec transcoding with FFmpeg cmdline on
> Android.
> > > > More importantly, we can do MediaCodec decoder to encoder without
> copy frames,
> > > > although it's very limited since most of avfilters doesn't work. For
> example:
> > > >
> > > > ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i
> /sdcard/test.mp4 -an -c:v h264_mediacodec -y
> > /sdcard/out.mp4
> >
> > How do you build the ffmpeg binary you're using for experiments in the
> > android device?
>
> Just configure with --enable-mediacodec and --enable-jni. Here is the build
> script I used for the development.
>
> https://github.com/quink-black/ffmpeg-ci/blob/master/android_ffmpeg.sh
>
> Here is the Android Studio project which can be used to test the mediacodec
> wrapper implemented via JNI.
>
> >
> >
> >
> >
> > > >
> > > > Since there is no real AVHWFrameContext implementation in
> hwcontext_mediacodec.
> > > > there is no hwframe_ctx for mediacodec and
> av_hwframe_transfer_data() doesn't
> > > > work. So if -hwaccel_output_format isn't being specified like:
> > > >
> > > > ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v
> h264_mediacodec -y /sdcard/out.mp4
> > > >
> > > > It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a
> check on
> > > > hwframe_ctx. Patch 12 set hwaccel_output_format automatically to
> avoid such
> > > > case.
> > > >
> > > >
> > > > Zhao Zhili (12):
> > > >  avcodec/mediacodec: fix incorrect crop info
> > > >  avcodec/mediacodecdec: don't break out if both input and output port
> > > >    return try again
> > > >  avcodec/mediacodecdec_common: fix misuse av_free/av_freep
> > > >  avcodec/mediacodecdec_common: fix useless av_buffer_unref
> > > >  avcodec/mediacodec_wrapper: separate implementation from interface
> > > >  avcodec/mediacodec: add NDK media codec wrapper
> > > >  avcodec/mediacodecdec: enable NDK mediacodec
> > > >  avutil/hwcontext_mediacodec: add ANativeWindow support
> > > >  avcodec/mediacodec: add ANativeWindow support
> > > >  avcodec: add MediaCodec encoder
> > > >  avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
> > > >  fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec
> > > >
> > > > Changelog                         |   2 +
> > > > configure                         |   6 +
> > > > fftools/ffmpeg_opt.c              |   4 +
> > > > libavcodec/Makefile               |   2 +
> > > > libavcodec/allcodecs.c            |   2 +
> > > > libavcodec/mediacodec_surface.c   |  46 +-
> > > > libavcodec/mediacodec_surface.h   |   8 +-
> > > > libavcodec/mediacodec_wrapper.c   | 942
> +++++++++++++++++++++++++++---
> > > > libavcodec/mediacodec_wrapper.h   | 275 +++++++--
> > > > libavcodec/mediacodecdec.c        |  21 +-
> > > > libavcodec/mediacodecdec_common.c |  35 +-
> > > > libavcodec/mediacodecdec_common.h |   1 +
> > > > libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
> > > > libavcodec/version.h              |   2 +-
> > > > libavutil/hwcontext.c             |   6 +-
> > > > libavutil/hwcontext_mediacodec.c  |  56 +-
> > > > libavutil/hwcontext_mediacodec.h  |  11 +
> > > > libavutil/version.h               |   4 +-
> > > > 18 files changed, 1780 insertions(+), 138 deletions(-)
> > > > create mode 100644 libavcodec/mediacodecenc.c
> > > >
> > > > --
> > > > 2.25.1
> > > >
> > > >
> > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

[Zhao Zhili]

> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Olivier Ayache
> Sent: 2022年11月12日 18:00
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Cc: matthieu.bouron@stupeflix.com; aman@tmm1.net
> Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support
> 
> Hello there I implemented that few years ago in
> https://github.com/olivierayache/xuggle-xuggler/

I can't find where is the patch or source code of FFmpeg.

> And when I tried to submit a patch to FFmpeg for add support for NDK I had
> received this answer.
> I think using NDK directly rather than JNI can be better in case of
> developing native application on Android.

NDK is still missing some important features, and some features only supported recently.
So the NDK implementation doesn't meant to be a replacement of the JNI version:
1. JNI version has more features and supports more devices
2. NDK implementation combined with some JNI functions has almost the same feature
As JNI version on new devices.
3. In the environment where JVM is unavailable, the NDK version is the only choice.

> If I can help on these subjects I would be really happy

Welcome to review and/or test the patch.

> 
> Olivier Ayache
> 
> 
> 
> ---------- Forwarded message ---------
> From: Olivier Ayache <olivier.ayache@gmail.com>
> Date: Sun, Jun 28, 2020 at 2:48 PM
> Subject: Re: [FFmpeg-devel] [PATCH] add support to Android ndk MediaCodec
> for encoding/decoding
> To: <ffmpeg-devel@ffmpeg.org>
> 
> 
> Thank for your replies I choose the NDK approach in order to  be able to be
> independant from the JVM, to maximize performance and to be sure to detect
> compatibility issues (using JNI approach can compile and crash at runtime).
> Concerning the version of Android it is compatible from API 21 (94% of
> devices)
> I agree that JNI overhead is not big but in combinaison with Xuggler it
> seems to me a little weird because Xuggler (Java part) already use JNI to
> interact with Xuggler (C++) and FFmpeg. If FFmpeg uses JNI in the other way
> (from c to Java) it seems to me an anti pattern.
> 
> If you think it is better to begin to use JNI I could transform my encoder
> and we could discuss about NDK after
> 
> Olivier
> 
> 
> 
> Le sam. 27 juin 2020 à 20:55, Martin Storsjö <martin@martin.st> a écrit :
> 
> > On Sat, 27 Jun 2020, Olivier Ayache wrote:
> >
> > > Hi everyone this is the first time I post on this mailing list. I am
> > > working since several years on a fork of Xuggler for manipulating ffmpeg
> > > API with Java/Kotlin.
> > > This work leads me to develop encoder and decoder based on Android NDK
> > > MediaCodec.
> > >
> > > This work can be found on my Github repository
> > >
> > > https://github.com/olivierayache/xuggle-xuggler
> > >
> > >
> > > I know that FFmpeg already integrate MediaCodec for decoding via Jni
> > > wrappers since version 3.1. I began this work on FFmpeg 2.8.x and I
> > choose
> > > the NDK in order to achieve optimum performance.
> >
> > If you mean you used the NDK MediaCodec API, I'd suggest you use the same
> > JNI wrappers as ffmpeg has already, for consistency. The overhead of a few
> > JNI calls per encoded frame is generally negligible. If it, at a later
> > time, is decided to drop support for older versions at some point, it
> > should be straightforward to convert it to use the NDK MediaCodec API
> > instead.
> >
> > // Martin
> >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> 
> 
> On Sat, Nov 12, 2022 at 6:07 AM Zhao Zhili <quinkblack@foxmail.com> wrote:
> 
> >
> >
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Chema
> > Gonzalez
> > > Sent: 2022年11月12日 0:55
> > > To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> > > Cc: matthieu.bouron@stupeflix.com; aman@tmm1.net
> > > Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec
> > encoder and NDK MediaCodec support
> > >
> > > On Thu, Nov 10, 2022 at 7:36 PM "zhilizhao(赵志立)" <quinkblack@foxmail.com>
> > wrote:
> > > >
> > > > Ping for review.
> > > >
> > > > > On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack@foxmail.com>
> > wrote:
> > > > >
> > > > > From: Zhao Zhili <zhilizhao@tencent.com>
> > > > >
> > > > > Firstly, some bugs were fixed (patch 1-4).
> > > > >
> > > > > Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK
> > > > > MediaCodec. The use case I'm considering is run FFmpeg on cmdline
> > without JVM,
> > > > > for example, run FFmpeg inside of termux (an Android terminal
> > emulator). It's
> > > > > well known that NDK MediaCodec missing some important functions,
> > like get the
> > > > > list of codecs, but still useable.
> > > > >
> > > > > Patch 7 add NDK MediaCodec decoder support. It can be enabled via
> > options,
> > > > > and enabled automatically if no JVM is available.
> > > > >
> > > > > Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be
> > set by
> > > > > user, and can be created via AMediaCodec_createPersistentInputSurface
> > > > > automatically. This is a preparation for encoder.
> > > > >
> > > > > Patch 9 makes MediaCodec decoder to support ANativeWindow directly.
> > It worth
> > > > > to note that AVMediaCodecContext has only surface. Although we
> > provided
> > > > > av_mediacodec_alloc_context(), we didn't strictly prevent users to
> > allocate
> > > > > AVMediaCodecContext on stack. I'm not sure if it's OK to add new
> > field to
> > > > > AVMediaCodecContext.
> > > > >
> > > > > Patch 10 add MediaCodec encoder support. Frame can be feed to
> > encoder via
> > > > > buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is
> > used, and
> > > > > the frames come from our MediaCodec decoder wrapper, we can control
> > it's
> > > > > 'render' (send to encoder's surface) via
> > av_mediacodec_release_buffer(). A DTS
> > > > > generation strategy works in this case. However, if frames comes
> > from other
> > > > > sources, like a camera, there is no way to control the 'render' yet,
> > so DTS is
> > > > > missing in this case.
> > > > >
> > > > > Finally, we can do mediacodec transcoding with FFmpeg cmdline on
> > Android.
> > > > > More importantly, we can do MediaCodec decoder to encoder without
> > copy frames,
> > > > > although it's very limited since most of avfilters doesn't work. For
> > example:
> > > > >
> > > > > ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i
> > /sdcard/test.mp4 -an -c:v h264_mediacodec -y
> > > /sdcard/out.mp4
> > >
> > > How do you build the ffmpeg binary you're using for experiments in the
> > > android device?
> >
> > Just configure with --enable-mediacodec and --enable-jni. Here is the build
> > script I used for the development.
> >
> > https://github.com/quink-black/ffmpeg-ci/blob/master/android_ffmpeg.sh
> >
> > Here is the Android Studio project which can be used to test the mediacodec
> > wrapper implemented via JNI.
> >
> > >
> > >
> > >
> > >
> > > > >
> > > > > Since there is no real AVHWFrameContext implementation in
> > hwcontext_mediacodec.
> > > > > there is no hwframe_ctx for mediacodec and
> > av_hwframe_transfer_data() doesn't
> > > > > work. So if -hwaccel_output_format isn't being specified like:
> > > > >
> > > > > ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v
> > h264_mediacodec -y /sdcard/out.mp4
> > > > >
> > > > > It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a
> > check on
> > > > > hwframe_ctx. Patch 12 set hwaccel_output_format automatically to
> > avoid such
> > > > > case.
> > > > >
> > > > >
> > > > > Zhao Zhili (12):
> > > > >  avcodec/mediacodec: fix incorrect crop info
> > > > >  avcodec/mediacodecdec: don't break out if both input and output port
> > > > >    return try again
> > > > >  avcodec/mediacodecdec_common: fix misuse av_free/av_freep
> > > > >  avcodec/mediacodecdec_common: fix useless av_buffer_unref
> > > > >  avcodec/mediacodec_wrapper: separate implementation from interface
> > > > >  avcodec/mediacodec: add NDK media codec wrapper
> > > > >  avcodec/mediacodecdec: enable NDK mediacodec
> > > > >  avutil/hwcontext_mediacodec: add ANativeWindow support
> > > > >  avcodec/mediacodec: add ANativeWindow support
> > > > >  avcodec: add MediaCodec encoder
> > > > >  avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
> > > > >  fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec
> > > > >
> > > > > Changelog                         |   2 +
> > > > > configure                         |   6 +
> > > > > fftools/ffmpeg_opt.c              |   4 +
> > > > > libavcodec/Makefile               |   2 +
> > > > > libavcodec/allcodecs.c            |   2 +
> > > > > libavcodec/mediacodec_surface.c   |  46 +-
> > > > > libavcodec/mediacodec_surface.h   |   8 +-
> > > > > libavcodec/mediacodec_wrapper.c   | 942
> > +++++++++++++++++++++++++++---
> > > > > libavcodec/mediacodec_wrapper.h   | 275 +++++++--
> > > > > libavcodec/mediacodecdec.c        |  21 +-
> > > > > libavcodec/mediacodecdec_common.c |  35 +-
> > > > > libavcodec/mediacodecdec_common.h |   1 +
> > > > > libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
> > > > > libavcodec/version.h              |   2 +-
> > > > > libavutil/hwcontext.c             |   6 +-
> > > > > libavutil/hwcontext_mediacodec.c  |  56 +-
> > > > > libavutil/hwcontext_mediacodec.h  |  11 +
> > > > > libavutil/version.h               |   4 +-
> > > > > 18 files changed, 1780 insertions(+), 138 deletions(-)
> > > > > create mode 100644 libavcodec/mediacodecenc.c
> > > > >
> > > > > --
> > > > > 2.25.1
> > > > >
> > > > >
> > > >
> > > > _______________________________________________
> > > > ffmpeg-devel mailing list
> > > > ffmpeg-devel@ffmpeg.org
> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > >
> > > > To unsubscribe, visit link above, or email
> > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> >
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

[Olivier Ayache]

Hello here is the work I have pushed in a fork of FFmpeg integrated in
source code of xuggler (a Java/C++ wrapper of FFmpeg)
https://github.com/olivierayache/xuggle-xuggler/commit/98469ebea59c921f46a114bd954f7a27a7c95a41

For the limitations of NDK I agree with you and that is why I have made
some backport to be able to support all needed features for FFmpeg (when I
worked on that it was FFmpeg 3.4.x version).
This backport allows support to these functions for Android devices since
API 21 (which is more than 99% of devices).

   - AMediaCodec_createInputSurface
   - AMediaCodec_setParameters
   - ANativeWindow_toSurface


For testing reviewing the patch I will make it as soon as I update FFmpeg
to the current version. I will be able to do it this week.

Olivier


Le dim. 13 nov. 2022, 12:49, Zhao Zhili <quinkblack@foxmail.com> a écrit :

>
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Olivier Ayache
> > Sent: 2022年11月12日 18:00
> > To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> > Cc: matthieu.bouron@stupeflix.com; aman@tmm1.net
> > Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec
> encoder and NDK MediaCodec support
> >
> > Hello there I implemented that few years ago in
> > https://github.com/olivierayache/xuggle-xuggler/
>
> I can't find where is the patch or source code of FFmpeg.
>
> > And when I tried to submit a patch to FFmpeg for add support for NDK I
> had
> > received this answer.
> > I think using NDK directly rather than JNI can be better in case of
> > developing native application on Android.
>
> NDK is still missing some important features, and some features only
> supported recently.
> So the NDK implementation doesn't meant to be a replacement of the JNI
> version:
> 1. JNI version has more features and supports more devices
> 2. NDK implementation combined with some JNI functions has almost the same
> feature
> As JNI version on new devices.
> 3. In the environment where JVM is unavailable, the NDK version is the
> only choice.
>
> > If I can help on these subjects I would be really happy
>
> Welcome to review and/or test the patch.
>
> >
> > Olivier Ayache
> >
> >
> >
> > ---------- Forwarded message ---------
> > From: Olivier Ayache <olivier.ayache@gmail.com>
> > Date: Sun, Jun 28, 2020 at 2:48 PM
> > Subject: Re: [FFmpeg-devel] [PATCH] add support to Android ndk MediaCodec
> > for encoding/decoding
> > To: <ffmpeg-devel@ffmpeg.org>
> >
> >
> > Thank for your replies I choose the NDK approach in order to  be able to
> be
> > independant from the JVM, to maximize performance and to be sure to
> detect
> > compatibility issues (using JNI approach can compile and crash at
> runtime).
> > Concerning the version of Android it is compatible from API 21 (94% of
> > devices)
> > I agree that JNI overhead is not big but in combinaison with Xuggler it
> > seems to me a little weird because Xuggler (Java part) already use JNI to
> > interact with Xuggler (C++) and FFmpeg. If FFmpeg uses JNI in the other
> way
> > (from c to Java) it seems to me an anti pattern.
> >
> > If you think it is better to begin to use JNI I could transform my
> encoder
> > and we could discuss about NDK after
> >
> > Olivier
> >
> >
> >
> > Le sam. 27 juin 2020 à 20:55, Martin Storsjö <martin@martin.st> a écrit
> :
> >
> > > On Sat, 27 Jun 2020, Olivier Ayache wrote:
> > >
> > > > Hi everyone this is the first time I post on this mailing list. I am
> > > > working since several years on a fork of Xuggler for manipulating
> ffmpeg
> > > > API with Java/Kotlin.
> > > > This work leads me to develop encoder and decoder based on Android
> NDK
> > > > MediaCodec.
> > > >
> > > > This work can be found on my Github repository
> > > >
> > > > https://github.com/olivierayache/xuggle-xuggler
> > > >
> > > >
> > > > I know that FFmpeg already integrate MediaCodec for decoding via Jni
> > > > wrappers since version 3.1. I began this work on FFmpeg 2.8.x and I
> > > choose
> > > > the NDK in order to achieve optimum performance.
> > >
> > > If you mean you used the NDK MediaCodec API, I'd suggest you use the
> same
> > > JNI wrappers as ffmpeg has already, for consistency. The overhead of a
> few
> > > JNI calls per encoded frame is generally negligible. If it, at a later
> > > time, is decided to drop support for older versions at some point, it
> > > should be straightforward to convert it to use the NDK MediaCodec API
> > > instead.
> > >
> > > // Martin
> > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> >
> >
> > On Sat, Nov 12, 2022 at 6:07 AM Zhao Zhili <quinkblack@foxmail.com>
> wrote:
> >
> > >
> > >
> > > > -----Original Message-----
> > > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Chema
> > > Gonzalez
> > > > Sent: 2022年11月12日 0:55
> > > > To: FFmpeg development discussions and patches <
> ffmpeg-devel@ffmpeg.org>
> > > > Cc: matthieu.bouron@stupeflix.com; aman@tmm1.net
> > > > Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec
> > > encoder and NDK MediaCodec support
> > > >
> > > > On Thu, Nov 10, 2022 at 7:36 PM "zhilizhao(赵志立)" <
> quinkblack@foxmail.com>
> > > wrote:
> > > > >
> > > > > Ping for review.
> > > > >
> > > > > > On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack@foxmail.com>
> > > wrote:
> > > > > >
> > > > > > From: Zhao Zhili <zhilizhao@tencent.com>
> > > > > >
> > > > > > Firstly, some bugs were fixed (patch 1-4).
> > > > > >
> > > > > > Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec
> and NDK
> > > > > > MediaCodec. The use case I'm considering is run FFmpeg on cmdline
> > > without JVM,
> > > > > > for example, run FFmpeg inside of termux (an Android terminal
> > > emulator). It's
> > > > > > well known that NDK MediaCodec missing some important functions,
> > > like get the
> > > > > > list of codecs, but still useable.
> > > > > >
> > > > > > Patch 7 add NDK MediaCodec decoder support. It can be enabled via
> > > options,
> > > > > > and enabled automatically if no JVM is available.
> > > > > >
> > > > > > Patch 8 add ANativeWindow support to hwcontext_mediacodec. It
> can be
> > > set by
> > > > > > user, and can be created via
> AMediaCodec_createPersistentInputSurface
> > > > > > automatically. This is a preparation for encoder.
> > > > > >
> > > > > > Patch 9 makes MediaCodec decoder to support ANativeWindow
> directly.
> > > It worth
> > > > > > to note that AVMediaCodecContext has only surface. Although we
> > > provided
> > > > > > av_mediacodec_alloc_context(), we didn't strictly prevent users
> to
> > > allocate
> > > > > > AVMediaCodecContext on stack. I'm not sure if it's OK to add new
> > > field to
> > > > > > AVMediaCodecContext.
> > > > > >
> > > > > > Patch 10 add MediaCodec encoder support. Frame can be feed to
> > > encoder via
> > > > > > buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is
> > > used, and
> > > > > > the frames come from our MediaCodec decoder wrapper, we can
> control
> > > it's
> > > > > > 'render' (send to encoder's surface) via
> > > av_mediacodec_release_buffer(). A DTS
> > > > > > generation strategy works in this case. However, if frames comes
> > > from other
> > > > > > sources, like a camera, there is no way to control the 'render'
> yet,
> > > so DTS is
> > > > > > missing in this case.
> > > > > >
> > > > > > Finally, we can do mediacodec transcoding with FFmpeg cmdline on
> > > Android.
> > > > > > More importantly, we can do MediaCodec decoder to encoder without
> > > copy frames,
> > > > > > although it's very limited since most of avfilters doesn't work.
> For
> > > example:
> > > > > >
> > > > > > ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i
> > > /sdcard/test.mp4 -an -c:v h264_mediacodec -y
> > > > /sdcard/out.mp4
> > > >
> > > > How do you build the ffmpeg binary you're using for experiments in
> the
> > > > android device?
> > >
> > > Just configure with --enable-mediacodec and --enable-jni. Here is the
> build
> > > script I used for the development.
> > >
> > > https://github.com/quink-black/ffmpeg-ci/blob/master/android_ffmpeg.sh
> > >
> > > Here is the Android Studio project which can be used to test the
> mediacodec
> > > wrapper implemented via JNI.
> > >
> > > >
> > > >
> > > >
> > > >
> > > > > >
> > > > > > Since there is no real AVHWFrameContext implementation in
> > > hwcontext_mediacodec.
> > > > > > there is no hwframe_ctx for mediacodec and
> > > av_hwframe_transfer_data() doesn't
> > > > > > work. So if -hwaccel_output_format isn't being specified like:
> > > > > >
> > > > > > ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v
> > > h264_mediacodec -y /sdcard/out.mp4
> > > > > >
> > > > > > It will trigger a crash in av_hwframe_transfer_data. Patch 11
> add a
> > > check on
> > > > > > hwframe_ctx. Patch 12 set hwaccel_output_format automatically to
> > > avoid such
> > > > > > case.
> > > > > >
> > > > > >
> > > > > > Zhao Zhili (12):
> > > > > >  avcodec/mediacodec: fix incorrect crop info
> > > > > >  avcodec/mediacodecdec: don't break out if both input and output
> port
> > > > > >    return try again
> > > > > >  avcodec/mediacodecdec_common: fix misuse av_free/av_freep
> > > > > >  avcodec/mediacodecdec_common: fix useless av_buffer_unref
> > > > > >  avcodec/mediacodec_wrapper: separate implementation from
> interface
> > > > > >  avcodec/mediacodec: add NDK media codec wrapper
> > > > > >  avcodec/mediacodecdec: enable NDK mediacodec
> > > > > >  avutil/hwcontext_mediacodec: add ANativeWindow support
> > > > > >  avcodec/mediacodec: add ANativeWindow support
> > > > > >  avcodec: add MediaCodec encoder
> > > > > >  avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
> > > > > >  fftools/ffmpeg_opt: set default hwaccel_output_format for
> mediacodec
> > > > > >
> > > > > > Changelog                         |   2 +
> > > > > > configure                         |   6 +
> > > > > > fftools/ffmpeg_opt.c              |   4 +
> > > > > > libavcodec/Makefile               |   2 +
> > > > > > libavcodec/allcodecs.c            |   2 +
> > > > > > libavcodec/mediacodec_surface.c   |  46 +-
> > > > > > libavcodec/mediacodec_surface.h   |   8 +-
> > > > > > libavcodec/mediacodec_wrapper.c   | 942
> > > +++++++++++++++++++++++++++---
> > > > > > libavcodec/mediacodec_wrapper.h   | 275 +++++++--
> > > > > > libavcodec/mediacodecdec.c        |  21 +-
> > > > > > libavcodec/mediacodecdec_common.c |  35 +-
> > > > > > libavcodec/mediacodecdec_common.h |   1 +
> > > > > > libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
> > > > > > libavcodec/version.h              |   2 +-
> > > > > > libavutil/hwcontext.c             |   6 +-
> > > > > > libavutil/hwcontext_mediacodec.c  |  56 +-
> > > > > > libavutil/hwcontext_mediacodec.h  |  11 +
> > > > > > libavutil/version.h               |   4 +-
> > > > > > 18 files changed, 1780 insertions(+), 138 deletions(-)
> > > > > > create mode 100644 libavcodec/mediacodecenc.c
> > > > > >
> > > > > > --
> > > > > > 2.25.1
> > > > > >
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > ffmpeg-devel mailing list
> > > > > ffmpeg-devel@ffmpeg.org
> > > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > > >
> > > > > To unsubscribe, visit link above, or email
> > > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > > _______________________________________________
> > > > ffmpeg-devel mailing list
> > > > ffmpeg-devel@ffmpeg.org
> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > >
> > > > To unsubscribe, visit link above, or email
> > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

["zhilizhao(赵志立)"]

On Nov 11, 2022, at 11:36, zhilizhao(赵志立) <quinkblack@foxmail.com> wrote:
> 
> Ping for review.

I’m planning to push this week if no more comments.

> 
>> On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack@foxmail.com> wrote:
>> 
>> From: Zhao Zhili <zhilizhao@tencent.com>
>> 
>> Firstly, some bugs were fixed (patch 1-4).
>> 
>> Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK
>> MediaCodec. The use case I'm considering is run FFmpeg on cmdline without JVM,
>> for example, run FFmpeg inside of termux (an Android terminal emulator). It's
>> well known that NDK MediaCodec missing some important functions, like get the
>> list of codecs, but still useable.
>> 
>> Patch 7 add NDK MediaCodec decoder support. It can be enabled via options,
>> and enabled automatically if no JVM is available.
>> 
>> Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be set by
>> user, and can be created via AMediaCodec_createPersistentInputSurface
>> automatically. This is a preparation for encoder.
>> 
>> Patch 9 makes MediaCodec decoder to support ANativeWindow directly. It worth
>> to note that AVMediaCodecContext has only surface. Although we provided
>> av_mediacodec_alloc_context(), we didn't strictly prevent users to allocate
>> AVMediaCodecContext on stack. I'm not sure if it's OK to add new field to
>> AVMediaCodecContext.
>> 
>> Patch 10 add MediaCodec encoder support. Frame can be feed to encoder via
>> buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is used, and
>> the frames come from our MediaCodec decoder wrapper, we can control it's
>> 'render' (send to encoder's surface) via av_mediacodec_release_buffer(). A DTS
>> generation strategy works in this case. However, if frames comes from other
>> sources, like a camera, there is no way to control the 'render' yet, so DTS is
>> missing in this case.
>> 
>> Finally, we can do mediacodec transcoding with FFmpeg cmdline on Android.
>> More importantly, we can do MediaCodec decoder to encoder without copy frames,
>> although it's very limited since most of avfilters doesn't work. For example:
>> 
>> ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4
>> 
>> Since there is no real AVHWFrameContext implementation in hwcontext_mediacodec.
>> there is no hwframe_ctx for mediacodec and av_hwframe_transfer_data() doesn't
>> work. So if -hwaccel_output_format isn't being specified like:
>> 
>> ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4
>> 
>> It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a check on
>> hwframe_ctx. Patch 12 set hwaccel_output_format automatically to avoid such
>> case.
>> 
>> 
>> Zhao Zhili (12):
>> avcodec/mediacodec: fix incorrect crop info
>> avcodec/mediacodecdec: don't break out if both input and output port
>>   return try again
>> avcodec/mediacodecdec_common: fix misuse av_free/av_freep
>> avcodec/mediacodecdec_common: fix useless av_buffer_unref
>> avcodec/mediacodec_wrapper: separate implementation from interface
>> avcodec/mediacodec: add NDK media codec wrapper
>> avcodec/mediacodecdec: enable NDK mediacodec
>> avutil/hwcontext_mediacodec: add ANativeWindow support
>> avcodec/mediacodec: add ANativeWindow support
>> avcodec: add MediaCodec encoder
>> avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
>> fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec
>> 
>> Changelog                         |   2 +
>> configure                         |   6 +
>> fftools/ffmpeg_opt.c              |   4 +
>> libavcodec/Makefile               |   2 +
>> libavcodec/allcodecs.c            |   2 +
>> libavcodec/mediacodec_surface.c   |  46 +-
>> libavcodec/mediacodec_surface.h   |   8 +-
>> libavcodec/mediacodec_wrapper.c   | 942 +++++++++++++++++++++++++++---
>> libavcodec/mediacodec_wrapper.h   | 275 +++++++--
>> libavcodec/mediacodecdec.c        |  21 +-
>> libavcodec/mediacodecdec_common.c |  35 +-
>> libavcodec/mediacodecdec_common.h |   1 +
>> libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
>> libavcodec/version.h              |   2 +-
>> libavutil/hwcontext.c             |   6 +-
>> libavutil/hwcontext_mediacodec.c  |  56 +-
>> libavutil/hwcontext_mediacodec.h  |  11 +
>> libavutil/version.h               |   4 +-
>> 18 files changed, 1780 insertions(+), 138 deletions(-)
>> create mode 100644 libavcodec/mediacodecenc.c
>> 
>> -- 
>> 2.25.1
>> 
>> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".