* [FFmpeg-devel] Add Mediacodec audio decoders support
@ 2024-06-12 13:42 Matthieu Bouron
2024-06-12 13:42 ` [FFmpeg-devel] [PATCH 1/2] avcodec/mediacodecdec_common: ensure input buffer timestamp is positive Matthieu Bouron
` (3 more replies)
0 siblings, 4 replies; 23+ messages in thread
From: Matthieu Bouron @ 2024-06-12 13:42 UTC (permalink / raw)
To: ffmpeg-devel
Hello,
This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
MP3, FLAC, VORBIS and OPUS are supported.
This is mainly useful to avoid shipping Android builds of FFmpeg that are
subjects to licensing/patents (due to AAC and AMR).
Best regards,
Matthieu
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* [FFmpeg-devel] [PATCH 1/2] avcodec/mediacodecdec_common: ensure input buffer timestamp is positive
2024-06-12 13:42 [FFmpeg-devel] Add Mediacodec audio decoders support Matthieu Bouron
@ 2024-06-12 13:42 ` Matthieu Bouron
2024-06-12 13:42 ` [FFmpeg-devel] [PATCH 2/2] avcodec: add Mediacodec audio decoders support Matthieu Bouron
` (2 subsequent siblings)
3 siblings, 0 replies; 23+ messages in thread
From: Matthieu Bouron @ 2024-06-12 13:42 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: Matthieu Bouron
Submitting a buffer with a negative timestamp seems to stall the Mediacodec audio
decoders.
---
libavcodec/mediacodecdec_common.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c
index d6f91e6e89..4d22db8ecf 100644
--- a/libavcodec/mediacodecdec_common.c
+++ b/libavcodec/mediacodecdec_common.c
@@ -658,6 +658,8 @@ int ff_mediacodec_dec_send(AVCodecContext *avctx, MediaCodecDecContext *s,
if (pts && avctx->pkt_timebase.num && avctx->pkt_timebase.den) {
pts = av_rescale_q(pts, avctx->pkt_timebase, AV_TIME_BASE_Q);
}
+ if (pts < 0)
+ pts = 0;
if (need_draining) {
uint32_t flags = ff_AMediaCodec_getBufferFlagEndOfStream(codec);
--
2.45.2
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* [FFmpeg-devel] [PATCH 2/2] avcodec: add Mediacodec audio decoders support
2024-06-12 13:42 [FFmpeg-devel] Add Mediacodec audio decoders support Matthieu Bouron
2024-06-12 13:42 ` [FFmpeg-devel] [PATCH 1/2] avcodec/mediacodecdec_common: ensure input buffer timestamp is positive Matthieu Bouron
@ 2024-06-12 13:42 ` Matthieu Bouron
2024-07-10 8:47 ` Matthieu Bouron
2024-07-28 15:23 ` Zhao Zhili
2024-06-12 14:00 ` [FFmpeg-devel] Add " Paul B Mahol
2024-07-10 13:54 ` Zhao Zhili
3 siblings, 2 replies; 23+ messages in thread
From: Matthieu Bouron @ 2024-06-12 13:42 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: Matthieu Bouron
---
configure | 14 ++
libavcodec/Makefile | 7 +
libavcodec/allcodecs.c | 7 +
libavcodec/mediacodecdec.c | 215 ++++++++++++++++++-
libavcodec/mediacodecdec_common.c | 333 +++++++++++++++++++++++++++---
5 files changed, 545 insertions(+), 31 deletions(-)
diff --git a/configure b/configure
index 83284427df..d7de3b73ed 100755
--- a/configure
+++ b/configure
@@ -3321,14 +3321,22 @@ amf_deps_any="libdl LoadLibrary"
nvenc_deps="ffnvcodec"
nvenc_deps_any="libdl LoadLibrary"
+aac_mediacodec_decoder_deps="mediacodec"
+aac_mediacodec_decoder_select="aac_adtstoasc_bsf aac_parser"
aac_mf_encoder_deps="mediafoundation"
ac3_mf_encoder_deps="mediafoundation"
+amrnb_mediacodec_decoder_deps="mediacodec"
+amrnb_mediacodec_decoder_select="amr_parser"
+amrwb_mediacodec_decoder_deps="mediacodec"
+amrwb_mediacodec_decoder_select="amr_parser"
av1_cuvid_decoder_deps="cuvid CUVIDAV1PICPARAMS"
av1_mediacodec_decoder_deps="mediacodec"
av1_mediacodec_encoder_deps="mediacodec"
av1_mediacodec_encoder_select="extract_extradata_bsf"
av1_nvenc_encoder_deps="nvenc NV_ENC_PIC_PARAMS_AV1"
av1_nvenc_encoder_select="atsc_a53"
+flac_mediacodec_decoder_deps="mediacodec"
+flac_mediacodec_decoder_select="flac_parser"
h263_v4l2m2m_decoder_deps="v4l2_m2m h263_v4l2_m2m"
h263_v4l2m2m_encoder_deps="v4l2_m2m h263_v4l2_m2m"
h264_amf_encoder_deps="amf"
@@ -3377,6 +3385,8 @@ mjpeg_qsv_encoder_select="qsvenc"
mjpeg_vaapi_encoder_deps="VAEncPictureParameterBufferJPEG"
mjpeg_vaapi_encoder_select="cbs_jpeg jpegtables vaapi_encode"
mp3_mf_encoder_deps="mediafoundation"
+mp3_mediacodec_decoder_deps="mediacodec"
+mp3_mediacodec_decoder_select="mpegaudioheader"
mpeg1_cuvid_decoder_deps="cuvid"
mpeg1_v4l2m2m_decoder_deps="v4l2_m2m mpeg1_v4l2_m2m"
mpeg2_cuvid_decoder_deps="cuvid"
@@ -3394,10 +3404,14 @@ mpeg4_mmal_decoder_deps="mmal"
mpeg4_omx_encoder_deps="omx"
mpeg4_v4l2m2m_decoder_deps="v4l2_m2m mpeg4_v4l2_m2m"
mpeg4_v4l2m2m_encoder_deps="v4l2_m2m mpeg4_v4l2_m2m"
+opus_mediacodec_decoder_deps="mediacodec"
+opus_mediacodec_decoder_select="opus_parser"
vc1_cuvid_decoder_deps="cuvid"
vc1_mmal_decoder_deps="mmal"
vc1_qsv_decoder_select="qsvdec"
vc1_v4l2m2m_decoder_deps="v4l2_m2m vc1_v4l2_m2m"
+vorbis_mediacodec_decoder_deps="mediacodec"
+vorbis_mediacodec_decoder_select="vorbis_parser"
vp8_cuvid_decoder_deps="cuvid"
vp8_mediacodec_decoder_deps="mediacodec"
vp8_mediacodec_encoder_deps="mediacodec"
diff --git a/libavcodec/Makefile b/libavcodec/Makefile
index 1a44352906..64771b9944 100644
--- a/libavcodec/Makefile
+++ b/libavcodec/Makefile
@@ -196,6 +196,7 @@ OBJS-$(CONFIG_AAC_ENCODER) += aacenc.o aaccoder.o aacenctab.o \
aacenc_pred.o \
psymodel.o kbdwin.o \
mpeg4audio_sample_rates.o
+OBJS-$(CONFIG_AAC_MEDIACODEC_DECODER) += mediacodecdec.o
OBJS-$(CONFIG_AAC_MF_ENCODER) += mfenc.o mf_utils.o
OBJS-$(CONFIG_AASC_DECODER) += aasc.o msrledec.o
OBJS-$(CONFIG_AC3_DECODER) += ac3dec_float.o ac3dec_data.o ac3.o \
@@ -222,6 +223,8 @@ OBJS-$(CONFIG_AMRWB_DECODER) += amrwbdec.o celp_filters.o \
celp_math.o acelp_filters.o \
acelp_vectors.o \
acelp_pitch_delay.o
+OBJS-$(CONFIG_AMRNB_MEDIACODEC_DECODER) += mediacodecdec.o
+OBJS-$(CONFIG_AMRWB_MEDIACODEC_DECODER) += mediacodecdec.o
OBJS-$(CONFIG_AMV_ENCODER) += mjpegenc.o mjpegenc_common.o
OBJS-$(CONFIG_ANM_DECODER) += anm.o
OBJS-$(CONFIG_ANULL_DECODER) += null.o
@@ -367,6 +370,7 @@ OBJS-$(CONFIG_FIC_DECODER) += fic.o
OBJS-$(CONFIG_FITS_DECODER) += fitsdec.o fits.o
OBJS-$(CONFIG_FITS_ENCODER) += fitsenc.o
OBJS-$(CONFIG_FLAC_DECODER) += flacdec.o flacdata.o flacdsp.o flac.o
+OBJS-$(CONFIG_FLAC_MEDIACODEC_DECODER) += mediacodecdec.o
OBJS-$(CONFIG_FLAC_ENCODER) += flacenc.o flacdata.o flacencdsp.o
OBJS-$(CONFIG_FLASHSV_DECODER) += flashsv.o
OBJS-$(CONFIG_FLASHSV_ENCODER) += flashsvenc.o
@@ -518,6 +522,7 @@ OBJS-$(CONFIG_MP2FIXED_ENCODER) += mpegaudioenc_fixed.o mpegaudio.o \
mpegaudiotabs.o
OBJS-$(CONFIG_MP2FLOAT_DECODER) += mpegaudiodec_float.o
OBJS-$(CONFIG_MP3_DECODER) += mpegaudiodec_fixed.o
+OBJS-$(CONFIG_MP3_MEDIACODEC_DECODER) += mediacodecdec.o
OBJS-$(CONFIG_MP3_MF_ENCODER) += mfenc.o mf_utils.o
OBJS-$(CONFIG_MP3ADU_DECODER) += mpegaudiodec_fixed.o
OBJS-$(CONFIG_MP3ADUFLOAT_DECODER) += mpegaudiodec_float.o
@@ -578,6 +583,7 @@ OBJS-$(CONFIG_OPUS_DECODER) += opusdec.o opusdec_celt.o opus_celt.o \
opusdsp.o opus_parse.o opus_rc.o
OBJS-$(CONFIG_OPUS_ENCODER) += opusenc.o opusenc_psy.o opus_celt.o \
opus_pvq.o opus_rc.o opustab.o
+OBJS-$(CONFIG_OPUS_MEDIACODEC_DECODER) += mediacodecdec.o
OBJS-$(CONFIG_OSQ_DECODER) += osq.o
OBJS-$(CONFIG_PAF_AUDIO_DECODER) += pafaudio.o
OBJS-$(CONFIG_PAF_VIDEO_DECODER) += pafvideo.o
@@ -765,6 +771,7 @@ OBJS-$(CONFIG_VORBIS_DECODER) += vorbisdec.o vorbisdsp.o vorbis.o \
vorbis_data.o
OBJS-$(CONFIG_VORBIS_ENCODER) += vorbisenc.o vorbis.o \
vorbis_data.o
+OBJS-$(CONFIG_VORBIS_MEDIACODEC_DECODER) += mediacodecdec.o
OBJS-$(CONFIG_VP3_DECODER) += vp3.o jpegquanttables.o
OBJS-$(CONFIG_VP5_DECODER) += vp5.o vp56.o vp56data.o vpx_rac.o
OBJS-$(CONFIG_VP6_DECODER) += vp6.o vp56.o vp56data.o \
diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
index b102a8069e..e5ab657ef6 100644
--- a/libavcodec/allcodecs.c
+++ b/libavcodec/allcodecs.c
@@ -821,8 +821,11 @@ extern const FFCodec ff_idf_decoder;
/* external libraries, that shouldn't be used by default if one of the
* above is available */
+extern const FFCodec ff_aac_mediacodec_decoder;
extern const FFCodec ff_aac_mf_encoder;
extern const FFCodec ff_ac3_mf_encoder;
+extern const FFCodec ff_amrnb_mediacodec_decoder;
+extern const FFCodec ff_amrwb_mediacodec_decoder;
extern const FFCodec ff_h263_v4l2m2m_encoder;
extern const FFCodec ff_libaom_av1_decoder;
/* hwaccel hooks only, so prefer external decoders */
@@ -835,6 +838,7 @@ extern const FFCodec ff_av1_qsv_decoder;
extern const FFCodec ff_av1_qsv_encoder;
extern const FFCodec ff_av1_amf_encoder;
extern const FFCodec ff_av1_vaapi_encoder;
+extern const FFCodec ff_flac_mediacodec_decoder;
extern const FFCodec ff_libopenh264_encoder;
extern const FFCodec ff_libopenh264_decoder;
extern const FFCodec ff_h264_amf_encoder;
@@ -861,6 +865,7 @@ extern const FFCodec ff_mjpeg_cuvid_decoder;
extern const FFCodec ff_mjpeg_qsv_encoder;
extern const FFCodec ff_mjpeg_qsv_decoder;
extern const FFCodec ff_mjpeg_vaapi_encoder;
+extern const FFCodec ff_mp3_mediacodec_decoder;
extern const FFCodec ff_mp3_mf_encoder;
extern const FFCodec ff_mpeg1_cuvid_decoder;
extern const FFCodec ff_mpeg2_cuvid_decoder;
@@ -871,8 +876,10 @@ extern const FFCodec ff_mpeg4_mediacodec_decoder;
extern const FFCodec ff_mpeg4_mediacodec_encoder;
extern const FFCodec ff_mpeg4_omx_encoder;
extern const FFCodec ff_mpeg4_v4l2m2m_encoder;
+extern const FFCodec ff_opus_mediacodec_decoder;
extern const FFCodec ff_prores_videotoolbox_encoder;
extern const FFCodec ff_vc1_cuvid_decoder;
+extern const FFCodec ff_vorbis_mediacodec_decoder;
extern const FFCodec ff_vp8_cuvid_decoder;
extern const FFCodec ff_vp8_mediacodec_decoder;
extern const FFCodec ff_vp8_mediacodec_encoder;
diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c
index 6d8dc600fe..dabf71cc04 100644
--- a/libavcodec/mediacodecdec.c
+++ b/libavcodec/mediacodecdec.c
@@ -36,6 +36,7 @@
#include "avcodec.h"
#include "codec_internal.h"
#include "decode.h"
+#include "flac_parse.h"
#include "h264_parse.h"
#include "h264_ps.h"
#include "hevc/parse.h"
@@ -44,6 +45,7 @@
#include "jni.h"
#include "mediacodec_wrapper.h"
#include "mediacodecdec_common.h"
+#include "xiph.h"
typedef struct MediaCodecH264DecContext {
@@ -287,11 +289,87 @@ done:
}
#endif
+#if CONFIG_FLAC_MEDIACODEC_DECODER
+static int flac_set_extradata(AVCodecContext *avctx, FFAMediaFormat *format)
+{
+ uint8_t *streaminfo;
+ uint8_t buffer[42];
+
+ if (!avctx->extradata) {
+ return 0;
+ }
+
+ if (!ff_flac_is_extradata_valid(avctx, &streaminfo))
+ return AVERROR_INVALIDDATA;
+
+ buffer[0] = 'f';
+ buffer[1] = 'L';
+ buffer[2] = 'a';
+ buffer[3] = 'C';
+ buffer[4] = 0x80;
+ buffer[5] = 0;
+ buffer[6] = 0;
+ buffer[7] = 0x22;
+ memcpy(buffer + 8, streaminfo, 34);
+
+ /* csd-0: fLaC + streaminfo */
+ ff_AMediaFormat_setBuffer(format, "csd-0", buffer, 42);
+
+ return 0;
+}
+#endif
+
+#if CONFIG_OPUS_MEDIACODEC_DECODER
+static int opus_set_extradata(AVCodecContext *avctx, FFAMediaFormat *format)
+{
+ if (!avctx->extradata_size) {
+ return 0;
+ }
+
+ if (avctx->extradata_size < 19) {
+ return AVERROR_INVALIDDATA;
+ }
+
+ ff_AMediaFormat_setBuffer(format, "csd-0", avctx->extradata, 19);
+
+ return 0;
+}
+#endif
+
+#if CONFIG_VORBIS_MEDIACODEC_DECODER
+static int vorbis_set_extradata(AVCodecContext *avctx, FFAMediaFormat *format)
+{
+ int ret;
+ const uint8_t *header_start[3];
+ int header_len[3];
+
+ if (!avctx->extradata) {
+ return 0;
+ }
+
+ ret = avpriv_split_xiph_headers(avctx->extradata, avctx->extradata_size, 30, header_start, header_len);
+ if (ret < 0) {
+ av_log(avctx, AV_LOG_ERROR, "Could not parse extradata\n");
+ return ret;
+ }
+
+ /* csd-0: identification header, csd-1: setup header */
+ ff_AMediaFormat_setBuffer(format, "csd-0", header_start[0], header_len[0]);
+ ff_AMediaFormat_setBuffer(format, "csd-1", header_start[2], header_len[2]);
+
+ return 0;
+}
+#endif
+
#if CONFIG_MPEG2_MEDIACODEC_DECODER || \
CONFIG_MPEG4_MEDIACODEC_DECODER || \
CONFIG_VP8_MEDIACODEC_DECODER || \
CONFIG_VP9_MEDIACODEC_DECODER || \
- CONFIG_AV1_MEDIACODEC_DECODER
+ CONFIG_AV1_MEDIACODEC_DECODER || \
+ CONFIG_AAC_MEDIACODEC_DECODER || \
+ CONFIG_AMRNB_MEDIACODEC_DECODER || \
+ CONFIG_AMRWB_MEDIACODEC_DECODER || \
+ CONFIG_MP3_MEDIACODEC_DECODER
static int common_set_extradata(AVCodecContext *avctx, FFAMediaFormat *format)
{
int ret = 0;
@@ -387,14 +465,83 @@ static av_cold int mediacodec_decode_init(AVCodecContext *avctx)
if (ret < 0)
goto done;
break;
+#endif
+#if CONFIG_AAC_MEDIACODEC_DECODER
+ case AV_CODEC_ID_AAC:
+ codec_mime = "audio/mp4a-latm";
+
+ ret = common_set_extradata(avctx, format);
+ if (ret < 0)
+ goto done;
+ break;
+#endif
+#if CONFIG_AMRNB_MEDIACODEC_DECODER
+ case AV_CODEC_ID_AMR_NB:
+ codec_mime = "audio/3gpp";
+
+ ret = common_set_extradata(avctx, format);
+ if (ret < 0)
+ goto done;
+ break;
+#endif
+#if CONFIG_AMRWB_MEDIACODEC_DECODER
+ case AV_CODEC_ID_AMR_WB:
+ codec_mime = "audio/amr-wb";
+
+ ret = common_set_extradata(avctx, format);
+ if (ret < 0)
+ goto done;
+ break;
+#endif
+#if CONFIG_FLAC_MEDIACODEC_DECODER
+ case AV_CODEC_ID_FLAC:
+ codec_mime = "audio/flac";
+
+ ret = flac_set_extradata(avctx, format);
+ if (ret < 0)
+ goto done;
+ break;
+#endif
+#if CONFIG_MP3_MEDIACODEC_DECODER
+ case AV_CODEC_ID_MP3:
+ codec_mime = "audio/mpeg";
+
+ ret = common_set_extradata(avctx, format);
+ if (ret < 0)
+ goto done;
+ break;
+#endif
+#if CONFIG_OPUS_MEDIACODEC_DECODER
+ case AV_CODEC_ID_OPUS:
+ codec_mime = "audio/opus";
+
+ ret = opus_set_extradata(avctx, format);
+ if (ret < 0)
+ goto done;
+ break;
+#endif
+#if CONFIG_VORBIS_MEDIACODEC_DECODER
+ case AV_CODEC_ID_VORBIS:
+ codec_mime = "audio/vorbis";
+
+ ret = vorbis_set_extradata(avctx, format);
+ if (ret < 0)
+ goto done;
+ break;
#endif
default:
av_assert0(0);
}
ff_AMediaFormat_setString(format, "mime", codec_mime);
- ff_AMediaFormat_setInt32(format, "width", avctx->width);
- ff_AMediaFormat_setInt32(format, "height", avctx->height);
+
+ if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
+ ff_AMediaFormat_setInt32(format, "width", avctx->width);
+ ff_AMediaFormat_setInt32(format, "height", avctx->height);
+ } else {
+ ff_AMediaFormat_setInt32(format, "channel-count", avctx->ch_layout.nb_channels);
+ ff_AMediaFormat_setInt32(format, "sample-rate", avctx->sample_rate);
+ }
s->ctx = av_mallocz(sizeof(*s->ctx));
if (!s->ctx) {
@@ -611,3 +758,65 @@ DECLARE_MEDIACODEC_VDEC(vp9, "VP9", AV_CODEC_ID_VP9, NULL)
#if CONFIG_AV1_MEDIACODEC_DECODER
DECLARE_MEDIACODEC_VDEC(av1, "AV1", AV_CODEC_ID_AV1, NULL)
#endif
+
+#define AD AV_OPT_FLAG_AUDIO_PARAM | AV_OPT_FLAG_DECODING_PARAM
+static const AVOption ff_mediacodec_adec_options[] = {
+ { "ndk_codec", "Use MediaCodec from NDK",
+ OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 = -1}, -1, 1, AD },
+ { NULL }
+};
+
+#define DECLARE_MEDIACODEC_ACLASS(short_name) \
+static const AVClass ff_##short_name##_mediacodec_dec_class = { \
+ .class_name = #short_name "_mediacodec", \
+ .item_name = av_default_item_name, \
+ .option = ff_mediacodec_adec_options, \
+ .version = LIBAVUTIL_VERSION_INT, \
+};
+
+#define DECLARE_MEDIACODEC_ADEC(short_name, full_name, codec_id, bsf) \
+DECLARE_MEDIACODEC_VCLASS(short_name) \
+const FFCodec ff_ ## short_name ## _mediacodec_decoder = { \
+ .p.name = #short_name "_mediacodec", \
+ CODEC_LONG_NAME(full_name " Android MediaCodec decoder"), \
+ .p.type = AVMEDIA_TYPE_AUDIO, \
+ .p.id = codec_id, \
+ .p.priv_class = &ff_##short_name##_mediacodec_dec_class, \
+ .priv_data_size = sizeof(MediaCodecH264DecContext), \
+ .init = mediacodec_decode_init, \
+ FF_CODEC_RECEIVE_FRAME_CB(mediacodec_receive_frame), \
+ .flush = mediacodec_decode_flush, \
+ .close = mediacodec_decode_close, \
+ .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE, \
+ .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE, \
+ .bsfs = bsf, \
+ .p.wrapper_name = "mediacodec", \
+}; \
+
+#if CONFIG_AAC_MEDIACODEC_DECODER
+DECLARE_MEDIACODEC_ADEC(aac, "AAC", AV_CODEC_ID_AAC, "aac_adtstoasc")
+#endif
+
+#if CONFIG_AMRNB_MEDIACODEC_DECODER
+DECLARE_MEDIACODEC_ADEC(amrnb, "AMR-NB", AV_CODEC_ID_AMR_NB, NULL)
+#endif
+
+#if CONFIG_AMRWB_MEDIACODEC_DECODER
+DECLARE_MEDIACODEC_ADEC(amrwb, "AMR-WB", AV_CODEC_ID_AMR_WB, NULL)
+#endif
+
+#if CONFIG_FLAC_MEDIACODEC_DECODER
+DECLARE_MEDIACODEC_ADEC(flac, "FLAC", AV_CODEC_ID_FLAC, NULL)
+#endif
+
+#if CONFIG_MP3_MEDIACODEC_DECODER
+DECLARE_MEDIACODEC_ADEC(mp3, "MP3", AV_CODEC_ID_MP3, NULL)
+#endif
+
+#if CONFIG_OPUS_MEDIACODEC_DECODER
+DECLARE_MEDIACODEC_ADEC(opus, "OPUS", AV_CODEC_ID_OPUS, NULL)
+#endif
+
+#if CONFIG_VORBIS_MEDIACODEC_DECODER
+DECLARE_MEDIACODEC_ADEC(vorbis, "VORBIS", AV_CODEC_ID_VORBIS, NULL)
+#endif
diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c
index 4d22db8ecf..4bdd0128c2 100644
--- a/libavcodec/mediacodecdec_common.c
+++ b/libavcodec/mediacodecdec_common.c
@@ -23,6 +23,7 @@
#include <string.h>
#include <sys/types.h>
+#include "libavutil/avassert.h"
#include "libavutil/common.h"
#include "libavutil/hwcontext_mediacodec.h"
#include "libavutil/mem.h"
@@ -30,6 +31,7 @@
#include "libavutil/pixfmt.h"
#include "libavutil/time.h"
#include "libavutil/timestamp.h"
+#include "libavutil/channel_layout.h"
#include "avcodec.h"
#include "decode.h"
@@ -85,6 +87,110 @@
#define OUTPUT_DEQUEUE_TIMEOUT_US 8000
#define OUTPUT_DEQUEUE_BLOCK_TIMEOUT_US 1000000
+enum {
+ ENCODING_PCM_16BIT = 0x00000002,
+ ENCODING_PCM_8BIT = 0x00000003,
+ ENCODING_PCM_FLOAT = 0x00000004,
+ ENCODING_PCM_24BIT_PACKED = 0x00000015,
+ ENCODING_PCM_32BIT = 0x00000016,
+};
+
+static const struct {
+
+ int pcm_format;
+ enum AVSampleFormat sample_format;
+
+} sample_formats[] = {
+
+ { ENCODING_PCM_16BIT, AV_SAMPLE_FMT_S16 },
+ { ENCODING_PCM_8BIT, AV_SAMPLE_FMT_U8 },
+ { ENCODING_PCM_FLOAT, AV_SAMPLE_FMT_FLT },
+ { ENCODING_PCM_24BIT_PACKED, AV_SAMPLE_FMT_S32 },
+ { ENCODING_PCM_32BIT, AV_SAMPLE_FMT_S32 },
+ { 0 }
+};
+
+static enum AVSampleFormat mcdec_map_pcm_format(AVCodecContext *avctx,
+ MediaCodecDecContext *s,
+ int pcm_format)
+{
+ int i;
+ enum AVSampleFormat ret = AV_SAMPLE_FMT_NONE;
+
+ for (i = 0; i < FF_ARRAY_ELEMS(sample_formats); i++) {
+ if (sample_formats[i].pcm_format == pcm_format) {
+ return sample_formats[i].sample_format;
+ }
+ }
+
+ av_log(avctx, AV_LOG_ERROR, "Output sample format 0x%x (value=%d) is not supported\n",
+ pcm_format, pcm_format);
+
+ return ret;
+}
+
+enum
+{
+ CHANNEL_OUT_FRONT_LEFT = 0x4,
+ CHANNEL_OUT_FRONT_RIGHT = 0x8,
+ CHANNEL_OUT_FRONT_CENTER = 0x10,
+ CHANNEL_OUT_LOW_FREQUENCY = 0x20,
+ CHANNEL_OUT_BACK_LEFT = 0x40,
+ CHANNEL_OUT_BACK_RIGHT = 0x80,
+ CHANNEL_OUT_FRONT_LEFT_OF_CENTER = 0x100,
+ CHANNEL_OUT_FRONT_RIGHT_OF_CENTER = 0x200,
+ CHANNEL_OUT_BACK_CENTER = 0x400,
+ CHANNEL_OUT_SIDE_LEFT = 0x800,
+ CHANNEL_OUT_SIDE_RIGHT = 0x1000,
+ CHANNEL_OUT_TOP_CENTER = 0x2000,
+ CHANNEL_OUT_TOP_FRONT_LEFT = 0x4000,
+ CHANNEL_OUT_TOP_FRONT_CENTER = 0x8000,
+ CHANNEL_OUT_TOP_FRONT_RIGHT = 0x10000,
+ CHANNEL_OUT_TOP_BACK_LEFT = 0x20000,
+ CHANNEL_OUT_TOP_BACK_CENTER = 0x40000,
+ CHANNEL_OUT_TOP_BACK_RIGHT = 0x80000,
+};
+
+static const struct {
+
+ int mask;
+ uint64_t layout;
+
+} channel_masks[] = {
+ { CHANNEL_OUT_FRONT_LEFT, AV_CH_FRONT_LEFT },
+ { CHANNEL_OUT_FRONT_RIGHT, AV_CH_FRONT_RIGHT },
+ { CHANNEL_OUT_FRONT_CENTER, AV_CH_FRONT_CENTER },
+ { CHANNEL_OUT_LOW_FREQUENCY, AV_CH_LOW_FREQUENCY },
+ { CHANNEL_OUT_BACK_LEFT, AV_CH_BACK_LEFT },
+ { CHANNEL_OUT_BACK_RIGHT, AV_CH_BACK_RIGHT },
+ { CHANNEL_OUT_FRONT_LEFT_OF_CENTER, AV_CH_FRONT_LEFT_OF_CENTER },
+ { CHANNEL_OUT_FRONT_RIGHT_OF_CENTER, AV_CH_FRONT_RIGHT_OF_CENTER },
+ { CHANNEL_OUT_BACK_CENTER, AV_CH_BACK_CENTER },
+ { CHANNEL_OUT_SIDE_LEFT, AV_CH_SIDE_LEFT },
+ { CHANNEL_OUT_SIDE_RIGHT, AV_CH_SIDE_RIGHT },
+ { CHANNEL_OUT_TOP_CENTER, AV_CH_TOP_CENTER },
+ { CHANNEL_OUT_TOP_FRONT_LEFT, AV_CH_TOP_FRONT_LEFT },
+ { CHANNEL_OUT_TOP_FRONT_CENTER, AV_CH_TOP_FRONT_CENTER },
+ { CHANNEL_OUT_TOP_FRONT_RIGHT, AV_CH_TOP_FRONT_RIGHT },
+ { CHANNEL_OUT_TOP_BACK_LEFT, AV_CH_TOP_BACK_LEFT },
+ { CHANNEL_OUT_TOP_BACK_CENTER, AV_CH_TOP_BACK_CENTER },
+ { CHANNEL_OUT_TOP_BACK_RIGHT, AV_CH_TOP_BACK_RIGHT },
+};
+
+static uint64_t mcdec_map_channel_mask(AVCodecContext *avctx,
+ int channel_mask)
+{
+ int i;
+ uint64_t channel_layout = 0;
+
+ for (i = 0; i < FF_ARRAY_ELEMS(channel_masks); i++) {
+ if (channel_mask & channel_masks[i].mask)
+ channel_layout |= channel_masks[i].layout;
+ }
+
+ return channel_layout;
+}
+
enum {
COLOR_FormatYUV420Planar = 0x13,
COLOR_FormatYUV420SemiPlanar = 0x15,
@@ -265,13 +371,77 @@ fail:
return ret;
}
-static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx,
- MediaCodecDecContext *s,
- uint8_t *data,
- size_t size,
- ssize_t index,
- FFAMediaCodecBufferInfo *info,
- AVFrame *frame)
+static int mediacodec_wrap_sw_audio_buffer(AVCodecContext *avctx,
+ MediaCodecDecContext *s,
+ uint8_t *data,
+ size_t size,
+ ssize_t index,
+ FFAMediaCodecBufferInfo *info,
+ AVFrame *frame)
+{
+ int ret = 0;
+ int status = 0;
+ const int sample_size = av_get_bytes_per_sample(avctx->sample_fmt);
+ if (!sample_size) {
+ av_log(avctx, AV_LOG_ERROR, "Could not get bytes per sample\n");
+ goto done;
+ }
+
+ frame->format = avctx->sample_fmt;
+ frame->sample_rate = avctx->sample_rate;
+ frame->nb_samples = info->size / (sample_size * avctx->ch_layout.nb_channels);
+
+ ret = av_channel_layout_copy(&frame->ch_layout, &avctx->ch_layout);
+ if (ret < 0) {
+ av_log(avctx, AV_LOG_ERROR, "Could not copy channel layout\n");
+ goto done;
+ }
+
+ /* MediaCodec buffers needs to be copied to our own refcounted buffers
+ * because the flush command invalidates all input and output buffers.
+ */
+ if ((ret = ff_get_buffer(avctx, frame, 0)) < 0) {
+ av_log(avctx, AV_LOG_ERROR, "Could not allocate buffer\n");
+ goto done;
+ }
+
+ /* Override frame->pts as ff_get_buffer will override its value based
+ * on the last avpacket received which is not in sync with the frame:
+ * * N avpackets can be pushed before 1 frame is actually returned
+ * * 0-sized avpackets are pushed to flush remaining frames at EOS */
+ if (avctx->pkt_timebase.num && avctx->pkt_timebase.den) {
+ frame->pts = av_rescale_q(info->presentationTimeUs,
+ AV_TIME_BASE_Q,
+ avctx->pkt_timebase);
+ } else {
+ frame->pts = info->presentationTimeUs;
+ }
+ frame->pkt_dts = AV_NOPTS_VALUE;
+
+ av_log(avctx, AV_LOG_TRACE,
+ "Frame: format=%d channels=%d sample_rate=%d nb_samples=%d",
+ avctx->sample_fmt, avctx->ch_layout.nb_channels, avctx->sample_rate, frame->nb_samples);
+
+ memcpy(frame->data[0], data, info->size);
+
+ ret = 0;
+done:
+ status = ff_AMediaCodec_releaseOutputBuffer(s->codec, index, 0);
+ if (status < 0) {
+ av_log(avctx, AV_LOG_ERROR, "Failed to release output buffer\n");
+ ret = AVERROR_EXTERNAL;
+ }
+
+ return ret;
+}
+
+static int mediacodec_wrap_sw_video_buffer(AVCodecContext *avctx,
+ MediaCodecDecContext *s,
+ uint8_t *data,
+ size_t size,
+ ssize_t index,
+ FFAMediaCodecBufferInfo *info,
+ AVFrame *frame)
{
int ret = 0;
int status = 0;
@@ -343,6 +513,22 @@ done:
return ret;
}
+static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx,
+ MediaCodecDecContext *s,
+ uint8_t *data,
+ size_t size,
+ ssize_t index,
+ FFAMediaCodecBufferInfo *info,
+ AVFrame *frame)
+{
+ if (avctx->codec_type == AVMEDIA_TYPE_AUDIO)
+ return mediacodec_wrap_sw_audio_buffer(avctx, s, data, size, index, info, frame);
+ else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO)
+ return mediacodec_wrap_sw_video_buffer(avctx, s, data, size, index, info, frame);
+ else
+ av_assert0(0);
+}
+
#define AMEDIAFORMAT_GET_INT32(name, key, mandatory) do { \
int32_t value = 0; \
if (ff_AMediaFormat_getInt32(s->format, key, &value)) { \
@@ -354,7 +540,7 @@ done:
} \
} while (0) \
-static int mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecContext *s)
+static int mediacodec_dec_parse_video_format(AVCodecContext *avctx, MediaCodecDecContext *s)
{
int ret = 0;
int width = 0;
@@ -463,6 +649,63 @@ fail:
return ret;
}
+static int mediacodec_dec_parse_audio_format(AVCodecContext *avctx, MediaCodecDecContext *s)
+{
+ int ret = 0;
+ int sample_rate = 0;
+ int channel_count = 0;
+ int channel_mask = 0;
+ int pcm_encoding = 0;
+ char *format = NULL;
+
+ if (!s->format) {
+ av_log(avctx, AV_LOG_ERROR, "Output MediaFormat is not set\n");
+ return AVERROR(EINVAL);
+ }
+
+ format = ff_AMediaFormat_toString(s->format);
+ if (!format) {
+ return AVERROR_EXTERNAL;
+ }
+ av_log(avctx, AV_LOG_DEBUG, "Parsing MediaFormat %s\n", format);
+
+ /* Mandatory fields */
+ AMEDIAFORMAT_GET_INT32(channel_count, "channel-count", 1);
+ AMEDIAFORMAT_GET_INT32(sample_rate, "sample-rate", 1);
+
+ AMEDIAFORMAT_GET_INT32(pcm_encoding, "pcm-encoding", 0);
+ if (pcm_encoding)
+ avctx->sample_fmt = mcdec_map_pcm_format(avctx, s, pcm_encoding);
+ else
+ avctx->sample_fmt = AV_SAMPLE_FMT_S16;
+
+ avctx->sample_rate = sample_rate;
+
+ AMEDIAFORMAT_GET_INT32(channel_mask, "channel-mask", 0);
+ if (channel_mask)
+ av_channel_layout_from_mask(&avctx->ch_layout, mcdec_map_channel_mask(avctx, channel_mask));
+ else
+ av_channel_layout_default(&avctx->ch_layout, channel_count);
+
+ av_log(avctx, AV_LOG_INFO,
+ "Output parameters channel-count=%d channel-layout=%x sample-rate=%d\n",
+ channel_count, channel_mask, sample_rate);
+
+fail:
+ av_freep(&format);
+ return ret;
+}
+
+static int mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecContext *s)
+{
+ if (avctx->codec_type == AVMEDIA_TYPE_AUDIO)
+ return mediacodec_dec_parse_audio_format(avctx, s);
+ else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO)
+ return mediacodec_dec_parse_video_format(avctx, s);
+ else
+ av_assert0(0);
+}
+
static int mediacodec_dec_flush_codec(AVCodecContext *avctx, MediaCodecDecContext *s)
{
FFAMediaCodec *codec = s->codec;
@@ -486,11 +729,9 @@ static int mediacodec_dec_flush_codec(AVCodecContext *avctx, MediaCodecDecContex
return 0;
}
-int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
- const char *mime, FFAMediaFormat *format)
+static int mediacodec_dec_get_video_codec(AVCodecContext *avctx, MediaCodecDecContext *s,
+ const char *mime, FFAMediaFormat *format)
{
- int ret = 0;
- int status;
int profile;
enum AVPixelFormat pix_fmt;
@@ -499,12 +740,6 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
AV_PIX_FMT_NONE,
};
- s->avctx = avctx;
- atomic_init(&s->refcount, 1);
- atomic_init(&s->hw_buffer_count, 0);
- atomic_init(&s->serial, 1);
- s->current_input_buffer = -1;
-
pix_fmt = ff_get_format(avctx, pix_fmts);
if (pix_fmt == AV_PIX_FMT_MEDIACODEC) {
AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
@@ -536,8 +771,7 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
// getCodecNameByType() can fail due to missing JVM, while NDK
// mediacodec can be used without JVM.
if (!s->use_ndk_codec) {
- ret = AVERROR_EXTERNAL;
- goto fail;
+ return AVERROR_EXTERNAL;
}
av_log(avctx, AV_LOG_INFO, "Failed to getCodecNameByType\n");
} else {
@@ -556,10 +790,51 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
}
if (!s->codec) {
av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for type %s and name %s\n", mime, s->codec_name);
- ret = AVERROR_EXTERNAL;
- goto fail;
+ return AVERROR_EXTERNAL;
+ }
+
+ return 0;
+}
+
+static int mediacodec_dec_get_audio_codec(AVCodecContext *avctx, MediaCodecDecContext *s,
+ const char *mime, FFAMediaFormat *format)
+{
+ s->codec = ff_AMediaCodec_createDecoderByType(mime, s->use_ndk_codec);
+ if (!s->codec) {
+ av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for mime %s\n", mime);
+ return AVERROR_EXTERNAL;
+ }
+
+ s->codec_name = ff_AMediaCodec_getName(s->codec);
+ if (!s->codec_name) {
+ av_log(avctx, AV_LOG_ERROR, "Failed to get codec name");
+ return AVERROR_EXTERNAL;
}
+ return 0;
+}
+
+int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
+ const char *mime, FFAMediaFormat *format)
+{
+ int ret;
+ int status;
+
+ s->avctx = avctx;
+ atomic_init(&s->refcount, 1);
+ atomic_init(&s->hw_buffer_count, 0);
+ atomic_init(&s->serial, 1);
+ s->current_input_buffer = -1;
+
+ if (avctx->codec_type == AVMEDIA_TYPE_AUDIO)
+ ret = mediacodec_dec_get_audio_codec(avctx, s, mime, format);
+ else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO)
+ ret = mediacodec_dec_get_video_codec(avctx, s, mime, format);
+ else
+ av_assert0(0);
+ if (ret < 0)
+ goto fail;
+
status = ff_AMediaCodec_configure(s->codec, format, s->surface, NULL, 0);
if (status < 0) {
char *desc = ff_AMediaFormat_toString(format);
@@ -583,12 +858,14 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
goto fail;
}
- s->format = ff_AMediaCodec_getOutputFormat(s->codec);
- if (s->format) {
- if ((ret = mediacodec_dec_parse_format(avctx, s)) < 0) {
- av_log(avctx, AV_LOG_ERROR,
- "Failed to configure context\n");
- goto fail;
+ if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
+ s->format = ff_AMediaCodec_getOutputFormat(s->codec);
+ if (s->format) {
+ if ((ret = mediacodec_dec_parse_format(avctx, s)) < 0) {
+ av_log(avctx, AV_LOG_ERROR,
+ "Failed to configure context\n");
+ goto fail;
+ }
}
}
--
2.45.2
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-06-12 13:42 [FFmpeg-devel] Add Mediacodec audio decoders support Matthieu Bouron
2024-06-12 13:42 ` [FFmpeg-devel] [PATCH 1/2] avcodec/mediacodecdec_common: ensure input buffer timestamp is positive Matthieu Bouron
2024-06-12 13:42 ` [FFmpeg-devel] [PATCH 2/2] avcodec: add Mediacodec audio decoders support Matthieu Bouron
@ 2024-06-12 14:00 ` Paul B Mahol
2024-07-10 13:54 ` Zhao Zhili
3 siblings, 0 replies; 23+ messages in thread
From: Paul B Mahol @ 2024-06-12 14:00 UTC (permalink / raw)
To: FFmpeg development discussions and patches
FFmpeg RIP
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] [PATCH 2/2] avcodec: add Mediacodec audio decoders support
2024-06-12 13:42 ` [FFmpeg-devel] [PATCH 2/2] avcodec: add Mediacodec audio decoders support Matthieu Bouron
@ 2024-07-10 8:47 ` Matthieu Bouron
2024-07-28 15:23 ` Zhao Zhili
1 sibling, 0 replies; 23+ messages in thread
From: Matthieu Bouron @ 2024-07-10 8:47 UTC (permalink / raw)
To: FFmpeg development discussions and patches
Le mer. 12 juin 2024, 3:47 PM, Matthieu Bouron <matthieu.bouron@gmail.com>
a écrit :
> ---
> configure | 14 ++
> libavcodec/Makefile | 7 +
> libavcodec/allcodecs.c | 7 +
> libavcodec/mediacodecdec.c | 215 ++++++++++++++++++-
> libavcodec/mediacodecdec_common.c | 333 +++++++++++++++++++++++++++---
> 5 files changed, 545 insertions(+), 31 deletions(-)
>
> diff --git a/configure b/configure
> index 83284427df..d7de3b73ed 100755
> --- a/configure
> +++ b/configure
> @@ -3321,14 +3321,22 @@ amf_deps_any="libdl LoadLibrary"
> nvenc_deps="ffnvcodec"
> nvenc_deps_any="libdl LoadLibrary"
>
> +aac_mediacodec_decoder_deps="mediacodec"
> +aac_mediacodec_decoder_select="aac_adtstoasc_bsf aac_parser"
> aac_mf_encoder_deps="mediafoundation"
> ac3_mf_encoder_deps="mediafoundation"
> +amrnb_mediacodec_decoder_deps="mediacodec"
> +amrnb_mediacodec_decoder_select="amr_parser"
> +amrwb_mediacodec_decoder_deps="mediacodec"
> +amrwb_mediacodec_decoder_select="amr_parser"
> av1_cuvid_decoder_deps="cuvid CUVIDAV1PICPARAMS"
> av1_mediacodec_decoder_deps="mediacodec"
> av1_mediacodec_encoder_deps="mediacodec"
> av1_mediacodec_encoder_select="extract_extradata_bsf"
> av1_nvenc_encoder_deps="nvenc NV_ENC_PIC_PARAMS_AV1"
> av1_nvenc_encoder_select="atsc_a53"
> +flac_mediacodec_decoder_deps="mediacodec"
> +flac_mediacodec_decoder_select="flac_parser"
> h263_v4l2m2m_decoder_deps="v4l2_m2m h263_v4l2_m2m"
> h263_v4l2m2m_encoder_deps="v4l2_m2m h263_v4l2_m2m"
> h264_amf_encoder_deps="amf"
> @@ -3377,6 +3385,8 @@ mjpeg_qsv_encoder_select="qsvenc"
> mjpeg_vaapi_encoder_deps="VAEncPictureParameterBufferJPEG"
> mjpeg_vaapi_encoder_select="cbs_jpeg jpegtables vaapi_encode"
> mp3_mf_encoder_deps="mediafoundation"
> +mp3_mediacodec_decoder_deps="mediacodec"
> +mp3_mediacodec_decoder_select="mpegaudioheader"
> mpeg1_cuvid_decoder_deps="cuvid"
> mpeg1_v4l2m2m_decoder_deps="v4l2_m2m mpeg1_v4l2_m2m"
> mpeg2_cuvid_decoder_deps="cuvid"
> @@ -3394,10 +3404,14 @@ mpeg4_mmal_decoder_deps="mmal"
> mpeg4_omx_encoder_deps="omx"
> mpeg4_v4l2m2m_decoder_deps="v4l2_m2m mpeg4_v4l2_m2m"
> mpeg4_v4l2m2m_encoder_deps="v4l2_m2m mpeg4_v4l2_m2m"
> +opus_mediacodec_decoder_deps="mediacodec"
> +opus_mediacodec_decoder_select="opus_parser"
> vc1_cuvid_decoder_deps="cuvid"
> vc1_mmal_decoder_deps="mmal"
> vc1_qsv_decoder_select="qsvdec"
> vc1_v4l2m2m_decoder_deps="v4l2_m2m vc1_v4l2_m2m"
> +vorbis_mediacodec_decoder_deps="mediacodec"
> +vorbis_mediacodec_decoder_select="vorbis_parser"
> vp8_cuvid_decoder_deps="cuvid"
> vp8_mediacodec_decoder_deps="mediacodec"
> vp8_mediacodec_encoder_deps="mediacodec"
> diff --git a/libavcodec/Makefile b/libavcodec/Makefile
> index 1a44352906..64771b9944 100644
> --- a/libavcodec/Makefile
> +++ b/libavcodec/Makefile
> @@ -196,6 +196,7 @@ OBJS-$(CONFIG_AAC_ENCODER) += aacenc.o
> aaccoder.o aacenctab.o \
> aacenc_pred.o \
> psymodel.o kbdwin.o \
> mpeg4audio_sample_rates.o
> +OBJS-$(CONFIG_AAC_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_AAC_MF_ENCODER) += mfenc.o mf_utils.o
> OBJS-$(CONFIG_AASC_DECODER) += aasc.o msrledec.o
> OBJS-$(CONFIG_AC3_DECODER) += ac3dec_float.o ac3dec_data.o
> ac3.o \
> @@ -222,6 +223,8 @@ OBJS-$(CONFIG_AMRWB_DECODER) += amrwbdec.o
> celp_filters.o \
> celp_math.o acelp_filters.o \
> acelp_vectors.o \
> acelp_pitch_delay.o
> +OBJS-$(CONFIG_AMRNB_MEDIACODEC_DECODER) += mediacodecdec.o
> +OBJS-$(CONFIG_AMRWB_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_AMV_ENCODER) += mjpegenc.o mjpegenc_common.o
> OBJS-$(CONFIG_ANM_DECODER) += anm.o
> OBJS-$(CONFIG_ANULL_DECODER) += null.o
> @@ -367,6 +370,7 @@ OBJS-$(CONFIG_FIC_DECODER) += fic.o
> OBJS-$(CONFIG_FITS_DECODER) += fitsdec.o fits.o
> OBJS-$(CONFIG_FITS_ENCODER) += fitsenc.o
> OBJS-$(CONFIG_FLAC_DECODER) += flacdec.o flacdata.o flacdsp.o
> flac.o
> +OBJS-$(CONFIG_FLAC_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_FLAC_ENCODER) += flacenc.o flacdata.o
> flacencdsp.o
> OBJS-$(CONFIG_FLASHSV_DECODER) += flashsv.o
> OBJS-$(CONFIG_FLASHSV_ENCODER) += flashsvenc.o
> @@ -518,6 +522,7 @@ OBJS-$(CONFIG_MP2FIXED_ENCODER) +=
> mpegaudioenc_fixed.o mpegaudio.o \
> mpegaudiotabs.o
> OBJS-$(CONFIG_MP2FLOAT_DECODER) += mpegaudiodec_float.o
> OBJS-$(CONFIG_MP3_DECODER) += mpegaudiodec_fixed.o
> +OBJS-$(CONFIG_MP3_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_MP3_MF_ENCODER) += mfenc.o mf_utils.o
> OBJS-$(CONFIG_MP3ADU_DECODER) += mpegaudiodec_fixed.o
> OBJS-$(CONFIG_MP3ADUFLOAT_DECODER) += mpegaudiodec_float.o
> @@ -578,6 +583,7 @@ OBJS-$(CONFIG_OPUS_DECODER) += opusdec.o
> opusdec_celt.o opus_celt.o \
> opusdsp.o opus_parse.o opus_rc.o
> OBJS-$(CONFIG_OPUS_ENCODER) += opusenc.o opusenc_psy.o
> opus_celt.o \
> opus_pvq.o opus_rc.o opustab.o
> +OBJS-$(CONFIG_OPUS_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_OSQ_DECODER) += osq.o
> OBJS-$(CONFIG_PAF_AUDIO_DECODER) += pafaudio.o
> OBJS-$(CONFIG_PAF_VIDEO_DECODER) += pafvideo.o
> @@ -765,6 +771,7 @@ OBJS-$(CONFIG_VORBIS_DECODER) += vorbisdec.o
> vorbisdsp.o vorbis.o \
> vorbis_data.o
> OBJS-$(CONFIG_VORBIS_ENCODER) += vorbisenc.o vorbis.o \
> vorbis_data.o
> +OBJS-$(CONFIG_VORBIS_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_VP3_DECODER) += vp3.o jpegquanttables.o
> OBJS-$(CONFIG_VP5_DECODER) += vp5.o vp56.o vp56data.o
> vpx_rac.o
> OBJS-$(CONFIG_VP6_DECODER) += vp6.o vp56.o vp56data.o \
> diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
> index b102a8069e..e5ab657ef6 100644
> --- a/libavcodec/allcodecs.c
> +++ b/libavcodec/allcodecs.c
> @@ -821,8 +821,11 @@ extern const FFCodec ff_idf_decoder;
>
> /* external libraries, that shouldn't be used by default if one of the
> * above is available */
> +extern const FFCodec ff_aac_mediacodec_decoder;
> extern const FFCodec ff_aac_mf_encoder;
> extern const FFCodec ff_ac3_mf_encoder;
> +extern const FFCodec ff_amrnb_mediacodec_decoder;
> +extern const FFCodec ff_amrwb_mediacodec_decoder;
> extern const FFCodec ff_h263_v4l2m2m_encoder;
> extern const FFCodec ff_libaom_av1_decoder;
> /* hwaccel hooks only, so prefer external decoders */
> @@ -835,6 +838,7 @@ extern const FFCodec ff_av1_qsv_decoder;
> extern const FFCodec ff_av1_qsv_encoder;
> extern const FFCodec ff_av1_amf_encoder;
> extern const FFCodec ff_av1_vaapi_encoder;
> +extern const FFCodec ff_flac_mediacodec_decoder;
> extern const FFCodec ff_libopenh264_encoder;
> extern const FFCodec ff_libopenh264_decoder;
> extern const FFCodec ff_h264_amf_encoder;
> @@ -861,6 +865,7 @@ extern const FFCodec ff_mjpeg_cuvid_decoder;
> extern const FFCodec ff_mjpeg_qsv_encoder;
> extern const FFCodec ff_mjpeg_qsv_decoder;
> extern const FFCodec ff_mjpeg_vaapi_encoder;
> +extern const FFCodec ff_mp3_mediacodec_decoder;
> extern const FFCodec ff_mp3_mf_encoder;
> extern const FFCodec ff_mpeg1_cuvid_decoder;
> extern const FFCodec ff_mpeg2_cuvid_decoder;
> @@ -871,8 +876,10 @@ extern const FFCodec ff_mpeg4_mediacodec_decoder;
> extern const FFCodec ff_mpeg4_mediacodec_encoder;
> extern const FFCodec ff_mpeg4_omx_encoder;
> extern const FFCodec ff_mpeg4_v4l2m2m_encoder;
> +extern const FFCodec ff_opus_mediacodec_decoder;
> extern const FFCodec ff_prores_videotoolbox_encoder;
> extern const FFCodec ff_vc1_cuvid_decoder;
> +extern const FFCodec ff_vorbis_mediacodec_decoder;
> extern const FFCodec ff_vp8_cuvid_decoder;
> extern const FFCodec ff_vp8_mediacodec_decoder;
> extern const FFCodec ff_vp8_mediacodec_encoder;
> diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c
> index 6d8dc600fe..dabf71cc04 100644
> --- a/libavcodec/mediacodecdec.c
> +++ b/libavcodec/mediacodecdec.c
> @@ -36,6 +36,7 @@
> #include "avcodec.h"
> #include "codec_internal.h"
> #include "decode.h"
> +#include "flac_parse.h"
> #include "h264_parse.h"
> #include "h264_ps.h"
> #include "hevc/parse.h"
> @@ -44,6 +45,7 @@
> #include "jni.h"
> #include "mediacodec_wrapper.h"
> #include "mediacodecdec_common.h"
> +#include "xiph.h"
>
> typedef struct MediaCodecH264DecContext {
>
> @@ -287,11 +289,87 @@ done:
> }
> #endif
>
> +#if CONFIG_FLAC_MEDIACODEC_DECODER
> +static int flac_set_extradata(AVCodecContext *avctx, FFAMediaFormat
> *format)
> +{
> + uint8_t *streaminfo;
> + uint8_t buffer[42];
> +
> + if (!avctx->extradata) {
> + return 0;
> + }
> +
> + if (!ff_flac_is_extradata_valid(avctx, &streaminfo))
> + return AVERROR_INVALIDDATA;
> +
> + buffer[0] = 'f';
> + buffer[1] = 'L';
> + buffer[2] = 'a';
> + buffer[3] = 'C';
> + buffer[4] = 0x80;
> + buffer[5] = 0;
> + buffer[6] = 0;
> + buffer[7] = 0x22;
> + memcpy(buffer + 8, streaminfo, 34);
> +
> + /* csd-0: fLaC + streaminfo */
> + ff_AMediaFormat_setBuffer(format, "csd-0", buffer, 42);
> +
> + return 0;
> +}
> +#endif
> +
> +#if CONFIG_OPUS_MEDIACODEC_DECODER
> +static int opus_set_extradata(AVCodecContext *avctx, FFAMediaFormat
> *format)
> +{
> + if (!avctx->extradata_size) {
> + return 0;
> + }
> +
> + if (avctx->extradata_size < 19) {
> + return AVERROR_INVALIDDATA;
> + }
> +
> + ff_AMediaFormat_setBuffer(format, "csd-0", avctx->extradata, 19);
> +
> + return 0;
> +}
> +#endif
> +
> +#if CONFIG_VORBIS_MEDIACODEC_DECODER
> +static int vorbis_set_extradata(AVCodecContext *avctx, FFAMediaFormat
> *format)
> +{
> + int ret;
> + const uint8_t *header_start[3];
> + int header_len[3];
> +
> + if (!avctx->extradata) {
> + return 0;
> + }
> +
> + ret = avpriv_split_xiph_headers(avctx->extradata,
> avctx->extradata_size, 30, header_start, header_len);
> + if (ret < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Could not parse extradata\n");
> + return ret;
> + }
> +
> + /* csd-0: identification header, csd-1: setup header */
> + ff_AMediaFormat_setBuffer(format, "csd-0", header_start[0],
> header_len[0]);
> + ff_AMediaFormat_setBuffer(format, "csd-1", header_start[2],
> header_len[2]);
> +
> + return 0;
> +}
> +#endif
> +
> #if CONFIG_MPEG2_MEDIACODEC_DECODER || \
> CONFIG_MPEG4_MEDIACODEC_DECODER || \
> CONFIG_VP8_MEDIACODEC_DECODER || \
> CONFIG_VP9_MEDIACODEC_DECODER || \
> - CONFIG_AV1_MEDIACODEC_DECODER
> + CONFIG_AV1_MEDIACODEC_DECODER || \
> + CONFIG_AAC_MEDIACODEC_DECODER || \
> + CONFIG_AMRNB_MEDIACODEC_DECODER || \
> + CONFIG_AMRWB_MEDIACODEC_DECODER || \
> + CONFIG_MP3_MEDIACODEC_DECODER
> static int common_set_extradata(AVCodecContext *avctx, FFAMediaFormat
> *format)
> {
> int ret = 0;
> @@ -387,14 +465,83 @@ static av_cold int
> mediacodec_decode_init(AVCodecContext *avctx)
> if (ret < 0)
> goto done;
> break;
> +#endif
> +#if CONFIG_AAC_MEDIACODEC_DECODER
> + case AV_CODEC_ID_AAC:
> + codec_mime = "audio/mp4a-latm";
> +
> + ret = common_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_AMRNB_MEDIACODEC_DECODER
> + case AV_CODEC_ID_AMR_NB:
> + codec_mime = "audio/3gpp";
> +
> + ret = common_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_AMRWB_MEDIACODEC_DECODER
> + case AV_CODEC_ID_AMR_WB:
> + codec_mime = "audio/amr-wb";
> +
> + ret = common_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_FLAC_MEDIACODEC_DECODER
> + case AV_CODEC_ID_FLAC:
> + codec_mime = "audio/flac";
> +
> + ret = flac_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_MP3_MEDIACODEC_DECODER
> + case AV_CODEC_ID_MP3:
> + codec_mime = "audio/mpeg";
> +
> + ret = common_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_OPUS_MEDIACODEC_DECODER
> + case AV_CODEC_ID_OPUS:
> + codec_mime = "audio/opus";
> +
> + ret = opus_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_VORBIS_MEDIACODEC_DECODER
> + case AV_CODEC_ID_VORBIS:
> + codec_mime = "audio/vorbis";
> +
> + ret = vorbis_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> #endif
> default:
> av_assert0(0);
> }
>
> ff_AMediaFormat_setString(format, "mime", codec_mime);
> - ff_AMediaFormat_setInt32(format, "width", avctx->width);
> - ff_AMediaFormat_setInt32(format, "height", avctx->height);
> +
> + if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
> + ff_AMediaFormat_setInt32(format, "width", avctx->width);
> + ff_AMediaFormat_setInt32(format, "height", avctx->height);
> + } else {
> + ff_AMediaFormat_setInt32(format, "channel-count",
> avctx->ch_layout.nb_channels);
> + ff_AMediaFormat_setInt32(format, "sample-rate",
> avctx->sample_rate);
> + }
>
> s->ctx = av_mallocz(sizeof(*s->ctx));
> if (!s->ctx) {
> @@ -611,3 +758,65 @@ DECLARE_MEDIACODEC_VDEC(vp9, "VP9", AV_CODEC_ID_VP9,
> NULL)
> #if CONFIG_AV1_MEDIACODEC_DECODER
> DECLARE_MEDIACODEC_VDEC(av1, "AV1", AV_CODEC_ID_AV1, NULL)
> #endif
> +
> +#define AD AV_OPT_FLAG_AUDIO_PARAM | AV_OPT_FLAG_DECODING_PARAM
> +static const AVOption ff_mediacodec_adec_options[] = {
> + { "ndk_codec", "Use MediaCodec from NDK",
> + OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 = -1},
> -1, 1, AD },
> + { NULL }
> +};
> +
> +#define DECLARE_MEDIACODEC_ACLASS(short_name) \
> +static const AVClass ff_##short_name##_mediacodec_dec_class = { \
> + .class_name = #short_name "_mediacodec", \
> + .item_name = av_default_item_name, \
> + .option = ff_mediacodec_adec_options, \
> + .version = LIBAVUTIL_VERSION_INT, \
> +};
> +
> +#define DECLARE_MEDIACODEC_ADEC(short_name, full_name, codec_id, bsf)
> \
> +DECLARE_MEDIACODEC_VCLASS(short_name)
> \
> +const FFCodec ff_ ## short_name ## _mediacodec_decoder = {
> \
> + .p.name = #short_name "_mediacodec",
> \
> + CODEC_LONG_NAME(full_name " Android MediaCodec decoder"),
> \
> + .p.type = AVMEDIA_TYPE_AUDIO,
> \
> + .p.id = codec_id,
> \
> + .p.priv_class = &ff_##short_name##_mediacodec_dec_class,
> \
> + .priv_data_size = sizeof(MediaCodecH264DecContext),
> \
> + .init = mediacodec_decode_init,
> \
> + FF_CODEC_RECEIVE_FRAME_CB(mediacodec_receive_frame),
> \
> + .flush = mediacodec_decode_flush,
> \
> + .close = mediacodec_decode_close,
> \
> + .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE,
> \
> + .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE,
> \
> + .bsfs = bsf,
> \
> + .p.wrapper_name = "mediacodec",
> \
> +};
> \
> +
> +#if CONFIG_AAC_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(aac, "AAC", AV_CODEC_ID_AAC, "aac_adtstoasc")
> +#endif
> +
> +#if CONFIG_AMRNB_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(amrnb, "AMR-NB", AV_CODEC_ID_AMR_NB, NULL)
> +#endif
> +
> +#if CONFIG_AMRWB_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(amrwb, "AMR-WB", AV_CODEC_ID_AMR_WB, NULL)
> +#endif
> +
> +#if CONFIG_FLAC_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(flac, "FLAC", AV_CODEC_ID_FLAC, NULL)
> +#endif
> +
> +#if CONFIG_MP3_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(mp3, "MP3", AV_CODEC_ID_MP3, NULL)
> +#endif
> +
> +#if CONFIG_OPUS_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(opus, "OPUS", AV_CODEC_ID_OPUS, NULL)
> +#endif
> +
> +#if CONFIG_VORBIS_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(vorbis, "VORBIS", AV_CODEC_ID_VORBIS, NULL)
> +#endif
> diff --git a/libavcodec/mediacodecdec_common.c
> b/libavcodec/mediacodecdec_common.c
> index 4d22db8ecf..4bdd0128c2 100644
> --- a/libavcodec/mediacodecdec_common.c
> +++ b/libavcodec/mediacodecdec_common.c
> @@ -23,6 +23,7 @@
> #include <string.h>
> #include <sys/types.h>
>
> +#include "libavutil/avassert.h"
> #include "libavutil/common.h"
> #include "libavutil/hwcontext_mediacodec.h"
> #include "libavutil/mem.h"
> @@ -30,6 +31,7 @@
> #include "libavutil/pixfmt.h"
> #include "libavutil/time.h"
> #include "libavutil/timestamp.h"
> +#include "libavutil/channel_layout.h"
>
> #include "avcodec.h"
> #include "decode.h"
> @@ -85,6 +87,110 @@
> #define OUTPUT_DEQUEUE_TIMEOUT_US 8000
> #define OUTPUT_DEQUEUE_BLOCK_TIMEOUT_US 1000000
>
> +enum {
> + ENCODING_PCM_16BIT = 0x00000002,
> + ENCODING_PCM_8BIT = 0x00000003,
> + ENCODING_PCM_FLOAT = 0x00000004,
> + ENCODING_PCM_24BIT_PACKED = 0x00000015,
> + ENCODING_PCM_32BIT = 0x00000016,
> +};
> +
> +static const struct {
> +
> + int pcm_format;
> + enum AVSampleFormat sample_format;
> +
> +} sample_formats[] = {
> +
> + { ENCODING_PCM_16BIT, AV_SAMPLE_FMT_S16 },
> + { ENCODING_PCM_8BIT, AV_SAMPLE_FMT_U8 },
> + { ENCODING_PCM_FLOAT, AV_SAMPLE_FMT_FLT },
> + { ENCODING_PCM_24BIT_PACKED, AV_SAMPLE_FMT_S32 },
> + { ENCODING_PCM_32BIT, AV_SAMPLE_FMT_S32 },
> + { 0 }
> +};
> +
> +static enum AVSampleFormat mcdec_map_pcm_format(AVCodecContext *avctx,
> + MediaCodecDecContext *s,
> + int pcm_format)
> +{
> + int i;
> + enum AVSampleFormat ret = AV_SAMPLE_FMT_NONE;
> +
> + for (i = 0; i < FF_ARRAY_ELEMS(sample_formats); i++) {
> + if (sample_formats[i].pcm_format == pcm_format) {
> + return sample_formats[i].sample_format;
> + }
> + }
> +
> + av_log(avctx, AV_LOG_ERROR, "Output sample format 0x%x (value=%d) is
> not supported\n",
> + pcm_format, pcm_format);
> +
> + return ret;
> +}
> +
> +enum
> +{
> + CHANNEL_OUT_FRONT_LEFT = 0x4,
> + CHANNEL_OUT_FRONT_RIGHT = 0x8,
> + CHANNEL_OUT_FRONT_CENTER = 0x10,
> + CHANNEL_OUT_LOW_FREQUENCY = 0x20,
> + CHANNEL_OUT_BACK_LEFT = 0x40,
> + CHANNEL_OUT_BACK_RIGHT = 0x80,
> + CHANNEL_OUT_FRONT_LEFT_OF_CENTER = 0x100,
> + CHANNEL_OUT_FRONT_RIGHT_OF_CENTER = 0x200,
> + CHANNEL_OUT_BACK_CENTER = 0x400,
> + CHANNEL_OUT_SIDE_LEFT = 0x800,
> + CHANNEL_OUT_SIDE_RIGHT = 0x1000,
> + CHANNEL_OUT_TOP_CENTER = 0x2000,
> + CHANNEL_OUT_TOP_FRONT_LEFT = 0x4000,
> + CHANNEL_OUT_TOP_FRONT_CENTER = 0x8000,
> + CHANNEL_OUT_TOP_FRONT_RIGHT = 0x10000,
> + CHANNEL_OUT_TOP_BACK_LEFT = 0x20000,
> + CHANNEL_OUT_TOP_BACK_CENTER = 0x40000,
> + CHANNEL_OUT_TOP_BACK_RIGHT = 0x80000,
> +};
> +
> +static const struct {
> +
> + int mask;
> + uint64_t layout;
> +
> +} channel_masks[] = {
> + { CHANNEL_OUT_FRONT_LEFT, AV_CH_FRONT_LEFT },
> + { CHANNEL_OUT_FRONT_RIGHT, AV_CH_FRONT_RIGHT },
> + { CHANNEL_OUT_FRONT_CENTER, AV_CH_FRONT_CENTER },
> + { CHANNEL_OUT_LOW_FREQUENCY, AV_CH_LOW_FREQUENCY },
> + { CHANNEL_OUT_BACK_LEFT, AV_CH_BACK_LEFT },
> + { CHANNEL_OUT_BACK_RIGHT, AV_CH_BACK_RIGHT },
> + { CHANNEL_OUT_FRONT_LEFT_OF_CENTER, AV_CH_FRONT_LEFT_OF_CENTER },
> + { CHANNEL_OUT_FRONT_RIGHT_OF_CENTER, AV_CH_FRONT_RIGHT_OF_CENTER },
> + { CHANNEL_OUT_BACK_CENTER, AV_CH_BACK_CENTER },
> + { CHANNEL_OUT_SIDE_LEFT, AV_CH_SIDE_LEFT },
> + { CHANNEL_OUT_SIDE_RIGHT, AV_CH_SIDE_RIGHT },
> + { CHANNEL_OUT_TOP_CENTER, AV_CH_TOP_CENTER },
> + { CHANNEL_OUT_TOP_FRONT_LEFT, AV_CH_TOP_FRONT_LEFT },
> + { CHANNEL_OUT_TOP_FRONT_CENTER, AV_CH_TOP_FRONT_CENTER },
> + { CHANNEL_OUT_TOP_FRONT_RIGHT, AV_CH_TOP_FRONT_RIGHT },
> + { CHANNEL_OUT_TOP_BACK_LEFT, AV_CH_TOP_BACK_LEFT },
> + { CHANNEL_OUT_TOP_BACK_CENTER, AV_CH_TOP_BACK_CENTER },
> + { CHANNEL_OUT_TOP_BACK_RIGHT, AV_CH_TOP_BACK_RIGHT },
> +};
> +
> +static uint64_t mcdec_map_channel_mask(AVCodecContext *avctx,
> + int channel_mask)
> +{
> + int i;
> + uint64_t channel_layout = 0;
> +
> + for (i = 0; i < FF_ARRAY_ELEMS(channel_masks); i++) {
> + if (channel_mask & channel_masks[i].mask)
> + channel_layout |= channel_masks[i].layout;
> + }
> +
> + return channel_layout;
> +}
> +
> enum {
> COLOR_FormatYUV420Planar = 0x13,
> COLOR_FormatYUV420SemiPlanar = 0x15,
> @@ -265,13 +371,77 @@ fail:
> return ret;
> }
>
> -static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx,
> - MediaCodecDecContext *s,
> - uint8_t *data,
> - size_t size,
> - ssize_t index,
> - FFAMediaCodecBufferInfo *info,
> - AVFrame *frame)
> +static int mediacodec_wrap_sw_audio_buffer(AVCodecContext *avctx,
> + MediaCodecDecContext *s,
> + uint8_t *data,
> + size_t size,
> + ssize_t index,
> + FFAMediaCodecBufferInfo *info,
> + AVFrame *frame)
> +{
> + int ret = 0;
> + int status = 0;
> + const int sample_size = av_get_bytes_per_sample(avctx->sample_fmt);
> + if (!sample_size) {
> + av_log(avctx, AV_LOG_ERROR, "Could not get bytes per sample\n");
> + goto done;
> + }
> +
> + frame->format = avctx->sample_fmt;
> + frame->sample_rate = avctx->sample_rate;
> + frame->nb_samples = info->size / (sample_size *
> avctx->ch_layout.nb_channels);
> +
> + ret = av_channel_layout_copy(&frame->ch_layout, &avctx->ch_layout);
> + if (ret < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Could not copy channel layout\n");
> + goto done;
> + }
> +
> + /* MediaCodec buffers needs to be copied to our own refcounted buffers
> + * because the flush command invalidates all input and output buffers.
> + */
> + if ((ret = ff_get_buffer(avctx, frame, 0)) < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Could not allocate buffer\n");
> + goto done;
> + }
> +
> + /* Override frame->pts as ff_get_buffer will override its value based
> + * on the last avpacket received which is not in sync with the frame:
> + * * N avpackets can be pushed before 1 frame is actually returned
> + * * 0-sized avpackets are pushed to flush remaining frames at EOS
> */
> + if (avctx->pkt_timebase.num && avctx->pkt_timebase.den) {
> + frame->pts = av_rescale_q(info->presentationTimeUs,
> + AV_TIME_BASE_Q,
> + avctx->pkt_timebase);
> + } else {
> + frame->pts = info->presentationTimeUs;
> + }
> + frame->pkt_dts = AV_NOPTS_VALUE;
> +
> + av_log(avctx, AV_LOG_TRACE,
> + "Frame: format=%d channels=%d sample_rate=%d nb_samples=%d",
> + avctx->sample_fmt, avctx->ch_layout.nb_channels,
> avctx->sample_rate, frame->nb_samples);
> +
> + memcpy(frame->data[0], data, info->size);
> +
> + ret = 0;
> +done:
> + status = ff_AMediaCodec_releaseOutputBuffer(s->codec, index, 0);
> + if (status < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Failed to release output buffer\n");
> + ret = AVERROR_EXTERNAL;
> + }
> +
> + return ret;
> +}
> +
> +static int mediacodec_wrap_sw_video_buffer(AVCodecContext *avctx,
> + MediaCodecDecContext *s,
> + uint8_t *data,
> + size_t size,
> + ssize_t index,
> + FFAMediaCodecBufferInfo *info,
> + AVFrame *frame)
> {
> int ret = 0;
> int status = 0;
> @@ -343,6 +513,22 @@ done:
> return ret;
> }
>
> +static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx,
> + MediaCodecDecContext *s,
> + uint8_t *data,
> + size_t size,
> + ssize_t index,
> + FFAMediaCodecBufferInfo *info,
> + AVFrame *frame)
> +{
> + if (avctx->codec_type == AVMEDIA_TYPE_AUDIO)
> + return mediacodec_wrap_sw_audio_buffer(avctx, s, data, size,
> index, info, frame);
> + else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO)
> + return mediacodec_wrap_sw_video_buffer(avctx, s, data, size,
> index, info, frame);
> + else
> + av_assert0(0);
> +}
> +
> #define AMEDIAFORMAT_GET_INT32(name, key, mandatory) do {
> \
> int32_t value = 0;
> \
> if (ff_AMediaFormat_getInt32(s->format, key, &value)) {
> \
> @@ -354,7 +540,7 @@ done:
> }
> \
> } while (0)
> \
>
> -static int mediacodec_dec_parse_format(AVCodecContext *avctx,
> MediaCodecDecContext *s)
> +static int mediacodec_dec_parse_video_format(AVCodecContext *avctx,
> MediaCodecDecContext *s)
> {
> int ret = 0;
> int width = 0;
> @@ -463,6 +649,63 @@ fail:
> return ret;
> }
>
> +static int mediacodec_dec_parse_audio_format(AVCodecContext *avctx,
> MediaCodecDecContext *s)
> +{
> + int ret = 0;
> + int sample_rate = 0;
> + int channel_count = 0;
> + int channel_mask = 0;
> + int pcm_encoding = 0;
> + char *format = NULL;
> +
> + if (!s->format) {
> + av_log(avctx, AV_LOG_ERROR, "Output MediaFormat is not set\n");
> + return AVERROR(EINVAL);
> + }
> +
> + format = ff_AMediaFormat_toString(s->format);
> + if (!format) {
> + return AVERROR_EXTERNAL;
> + }
> + av_log(avctx, AV_LOG_DEBUG, "Parsing MediaFormat %s\n", format);
> +
> + /* Mandatory fields */
> + AMEDIAFORMAT_GET_INT32(channel_count, "channel-count", 1);
> + AMEDIAFORMAT_GET_INT32(sample_rate, "sample-rate", 1);
> +
> + AMEDIAFORMAT_GET_INT32(pcm_encoding, "pcm-encoding", 0);
> + if (pcm_encoding)
> + avctx->sample_fmt = mcdec_map_pcm_format(avctx, s, pcm_encoding);
> + else
> + avctx->sample_fmt = AV_SAMPLE_FMT_S16;
> +
> + avctx->sample_rate = sample_rate;
> +
> + AMEDIAFORMAT_GET_INT32(channel_mask, "channel-mask", 0);
> + if (channel_mask)
> + av_channel_layout_from_mask(&avctx->ch_layout,
> mcdec_map_channel_mask(avctx, channel_mask));
> + else
> + av_channel_layout_default(&avctx->ch_layout, channel_count);
> +
> + av_log(avctx, AV_LOG_INFO,
> + "Output parameters channel-count=%d channel-layout=%x
> sample-rate=%d\n",
> + channel_count, channel_mask, sample_rate);
> +
> +fail:
> + av_freep(&format);
> + return ret;
> +}
> +
> +static int mediacodec_dec_parse_format(AVCodecContext *avctx,
> MediaCodecDecContext *s)
> +{
> + if (avctx->codec_type == AVMEDIA_TYPE_AUDIO)
> + return mediacodec_dec_parse_audio_format(avctx, s);
> + else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO)
> + return mediacodec_dec_parse_video_format(avctx, s);
> + else
> + av_assert0(0);
> +}
> +
> static int mediacodec_dec_flush_codec(AVCodecContext *avctx,
> MediaCodecDecContext *s)
> {
> FFAMediaCodec *codec = s->codec;
> @@ -486,11 +729,9 @@ static int mediacodec_dec_flush_codec(AVCodecContext
> *avctx, MediaCodecDecContex
> return 0;
> }
>
> -int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
> - const char *mime, FFAMediaFormat *format)
> +static int mediacodec_dec_get_video_codec(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> + const char *mime,
> FFAMediaFormat *format)
> {
> - int ret = 0;
> - int status;
> int profile;
>
> enum AVPixelFormat pix_fmt;
> @@ -499,12 +740,6 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> AV_PIX_FMT_NONE,
> };
>
> - s->avctx = avctx;
> - atomic_init(&s->refcount, 1);
> - atomic_init(&s->hw_buffer_count, 0);
> - atomic_init(&s->serial, 1);
> - s->current_input_buffer = -1;
> -
> pix_fmt = ff_get_format(avctx, pix_fmts);
> if (pix_fmt == AV_PIX_FMT_MEDIACODEC) {
> AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
> @@ -536,8 +771,7 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> // getCodecNameByType() can fail due to missing JVM, while NDK
> // mediacodec can be used without JVM.
> if (!s->use_ndk_codec) {
> - ret = AVERROR_EXTERNAL;
> - goto fail;
> + return AVERROR_EXTERNAL;
> }
> av_log(avctx, AV_LOG_INFO, "Failed to getCodecNameByType\n");
> } else {
> @@ -556,10 +790,51 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> }
> if (!s->codec) {
> av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for
> type %s and name %s\n", mime, s->codec_name);
> - ret = AVERROR_EXTERNAL;
> - goto fail;
> + return AVERROR_EXTERNAL;
> + }
> +
> + return 0;
> +}
> +
> +static int mediacodec_dec_get_audio_codec(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> + const char *mime,
> FFAMediaFormat *format)
> +{
> + s->codec = ff_AMediaCodec_createDecoderByType(mime, s->use_ndk_codec);
> + if (!s->codec) {
> + av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for
> mime %s\n", mime);
> + return AVERROR_EXTERNAL;
> + }
> +
> + s->codec_name = ff_AMediaCodec_getName(s->codec);
> + if (!s->codec_name) {
> + av_log(avctx, AV_LOG_ERROR, "Failed to get codec name");
> + return AVERROR_EXTERNAL;
> }
>
> + return 0;
> +}
> +
> +int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
> + const char *mime, FFAMediaFormat *format)
> +{
> + int ret;
> + int status;
> +
> + s->avctx = avctx;
> + atomic_init(&s->refcount, 1);
> + atomic_init(&s->hw_buffer_count, 0);
> + atomic_init(&s->serial, 1);
> + s->current_input_buffer = -1;
> +
> + if (avctx->codec_type == AVMEDIA_TYPE_AUDIO)
> + ret = mediacodec_dec_get_audio_codec(avctx, s, mime, format);
> + else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO)
> + ret = mediacodec_dec_get_video_codec(avctx, s, mime, format);
> + else
> + av_assert0(0);
> + if (ret < 0)
> + goto fail;
> +
> status = ff_AMediaCodec_configure(s->codec, format, s->surface, NULL,
> 0);
> if (status < 0) {
> char *desc = ff_AMediaFormat_toString(format);
> @@ -583,12 +858,14 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> goto fail;
> }
>
> - s->format = ff_AMediaCodec_getOutputFormat(s->codec);
> - if (s->format) {
> - if ((ret = mediacodec_dec_parse_format(avctx, s)) < 0) {
> - av_log(avctx, AV_LOG_ERROR,
> - "Failed to configure context\n");
> - goto fail;
> + if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
> + s->format = ff_AMediaCodec_getOutputFormat(s->codec);
> + if (s->format) {
> + if ((ret = mediacodec_dec_parse_format(avctx, s)) < 0) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to configure context\n");
> + goto fail;
> + }
> }
> }
>
> --
> 2.45.2
>
Ping
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-06-12 13:42 [FFmpeg-devel] Add Mediacodec audio decoders support Matthieu Bouron
` (2 preceding siblings ...)
2024-06-12 14:00 ` [FFmpeg-devel] Add " Paul B Mahol
@ 2024-07-10 13:54 ` Zhao Zhili
2024-07-10 16:31 ` Matthieu Bouron
2024-07-23 11:35 ` Martin Storsjö
3 siblings, 2 replies; 23+ messages in thread
From: Zhao Zhili @ 2024-07-10 13:54 UTC (permalink / raw)
To: FFmpeg development discussions and patches
> On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>
> Hello,
>
> This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
> MP3, FLAC, VORBIS and OPUS are supported.
>
> This is mainly useful to avoid shipping Android builds of FFmpeg that are
> subjects to licensing/patents (due to AAC and AMR).
I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They don’t bring
new features, they don’t improve performance. I know these type of wrapper exist in current
project, but I’m not sure if it’s a good idea to add more.
This is a policy issue rather than technique issue, it requires more people to discuss and
decide.
>
> Best regards,
> Matthieu
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-10 13:54 ` Zhao Zhili
@ 2024-07-10 16:31 ` Matthieu Bouron
2024-07-16 13:20 ` Matthieu Bouron
2024-07-23 11:35 ` Martin Storsjö
1 sibling, 1 reply; 23+ messages in thread
From: Matthieu Bouron @ 2024-07-10 16:31 UTC (permalink / raw)
To: FFmpeg development discussions and patches
On Wed, Jul 10, 2024 at 4:04 PM Zhao Zhili <quinkblack@foxmail.com> wrote:
>
>
> > On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
> >
> > Hello,
> >
> > This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
> > MP3, FLAC, VORBIS and OPUS are supported.
> >
> > This is mainly useful to avoid shipping Android builds of FFmpeg that are
> > subjects to licensing/patents (due to AAC and AMR).
>
> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They don’t bring
> new features, they don’t improve performance. I know these type of wrapper exist in current
> project, but I’m not sure if it’s a good idea to add more.
I agree that on the technical side it doesn't bring new features nor
performance improvements. It's all about avoiding licensing/patents
issues with AAC and AMR here.
In this specific case we already have the wrapper infrastructure, the
audio part only needs small adjustments to work.
Moreover, if that helps, I can reduce the scope of the patch to AAC
and AMR only and get rid of mp3/flac/vorbis/opus support. What do you
think ?
>
> This is a policy issue rather than technique issue, it requires more people to discuss and
> decide.
>
[...]
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-10 16:31 ` Matthieu Bouron
@ 2024-07-16 13:20 ` Matthieu Bouron
2024-07-16 14:58 ` Zhao Zhili
0 siblings, 1 reply; 23+ messages in thread
From: Matthieu Bouron @ 2024-07-16 13:20 UTC (permalink / raw)
To: FFmpeg development discussions and patches
On Wed, Jul 10, 2024 at 6:31 PM Matthieu Bouron
<matthieu.bouron@gmail.com> wrote:
>
> On Wed, Jul 10, 2024 at 4:04 PM Zhao Zhili <quinkblack@foxmail.com> wrote:
> >
> >
> > > On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
> > > MP3, FLAC, VORBIS and OPUS are supported.
> > >
> > > This is mainly useful to avoid shipping Android builds of FFmpeg that are
> > > subjects to licensing/patents (due to AAC and AMR).
> >
> > I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They don’t bring
> > new features, they don’t improve performance. I know these type of wrapper exist in current
> > project, but I’m not sure if it’s a good idea to add more.
>
> I agree that on the technical side it doesn't bring new features nor
> performance improvements. It's all about avoiding licensing/patents
> issues with AAC and AMR here.
> In this specific case we already have the wrapper infrastructure, the
> audio part only needs small adjustments to work.
> Moreover, if that helps, I can reduce the scope of the patch to AAC
> and AMR only and get rid of mp3/flac/vorbis/opus support. What do you
> think ?
Ping.
IMHO, this benefits users wanting to ship an Android app that relies
on FFmpeg upstream in countries that are subject to AAC/AMR licensing.
While I agree that it's not great from a purely technical pov since we
already have better native decoders, it will allow the use of FFmpeg
in such situation without the need to use or create another FFmpeg
fork dedicated to Android. Plus, as I said above, we already have the
wrapper and the additional code to make it work for audio is
relatively small and scoped. Restricting the wrapper to AAC/AMR seems
like a good compromise to me.
[...]
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-16 13:20 ` Matthieu Bouron
@ 2024-07-16 14:58 ` Zhao Zhili
[not found] ` <2009DBEE-686A-42F6-A5DF-35147272EEF0@cosmin.at>
0 siblings, 1 reply; 23+ messages in thread
From: Zhao Zhili @ 2024-07-16 14:58 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: tc
> On Jul 16, 2024, at 21:20, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>
> On Wed, Jul 10, 2024 at 6:31 PM Matthieu Bouron
> <matthieu.bouron@gmail.com> wrote:
>>
>> On Wed, Jul 10, 2024 at 4:04 PM Zhao Zhili <quinkblack@foxmail.com> wrote:
>>>
>>>
>>>> On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
>>>> MP3, FLAC, VORBIS and OPUS are supported.
>>>>
>>>> This is mainly useful to avoid shipping Android builds of FFmpeg that are
>>>> subjects to licensing/patents (due to AAC and AMR).
>>>
>>> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They don’t bring
>>> new features, they don’t improve performance. I know these type of wrapper exist in current
>>> project, but I’m not sure if it’s a good idea to add more.
>>
>> I agree that on the technical side it doesn't bring new features nor
>> performance improvements. It's all about avoiding licensing/patents
>> issues with AAC and AMR here.
>> In this specific case we already have the wrapper infrastructure, the
>> audio part only needs small adjustments to work.
>> Moreover, if that helps, I can reduce the scope of the patch to AAC
>> and AMR only and get rid of mp3/flac/vorbis/opus support. What do you
>> think ?
>
> Ping.
>
> IMHO, this benefits users wanting to ship an Android app that relies
> on FFmpeg upstream in countries that are subject to AAC/AMR licensing.
> While I agree that it's not great from a purely technical pov since we
> already have better native decoders, it will allow the use of FFmpeg
> in such situation without the need to use or create another FFmpeg
> fork dedicated to Android. Plus, as I said above, we already have the
> wrapper and the additional code to make it work for audio is
> relatively small and scoped. Restricting the wrapper to AAC/AMR seems
> like a good compromise to me.
Sounds reasonable, but I’d like to get more opinions. cc TC.
>
> [...]
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
[not found] ` <2009DBEE-686A-42F6-A5DF-35147272EEF0@cosmin.at>
@ 2024-07-16 15:48 ` Cosmin Stejerean via ffmpeg-devel
2024-07-16 16:04 ` Zhao Zhili
` (2 more replies)
0 siblings, 3 replies; 23+ messages in thread
From: Cosmin Stejerean via ffmpeg-devel @ 2024-07-16 15:48 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: Cosmin Stejerean, tc
> On Jul 16, 2024, at 4:58 PM, Zhao Zhili <quinkblack@foxmail.com> wrote:
>
>
>
>> On Jul 16, 2024, at 21:20, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>>
>> On Wed, Jul 10, 2024 at 6:31 PM Matthieu Bouron
>> <matthieu.bouron@gmail.com> wrote:
>>>
>>> On Wed, Jul 10, 2024 at 4:04 PM Zhao Zhili <quinkblack@foxmail.com> wrote:
>>>>
>>>>
>>>>> On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
>>>>> MP3, FLAC, VORBIS and OPUS are supported.
>>>>>
>>>>> This is mainly useful to avoid shipping Android builds of FFmpeg that are
>>>>> subjects to licensing/patents (due to AAC and AMR).
>>>>
>>>> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They don’t bring
>>>> new features, they don’t improve performance. I know these type of wrapper exist in current
>>>> project, but I’m not sure if it’s a good idea to add more.
>>>
>>> I agree that on the technical side it doesn't bring new features nor
>>> performance improvements. It's all about avoiding licensing/patents
>>> issues with AAC and AMR here.
>>> In this specific case we already have the wrapper infrastructure, the
>>> audio part only needs small adjustments to work.
>>> Moreover, if that helps, I can reduce the scope of the patch to AAC
>>> and AMR only and get rid of mp3/flac/vorbis/opus support. What do you
>>> think ?
>>
>> Ping.
>>
>> IMHO, this benefits users wanting to ship an Android app that relies
>> on FFmpeg upstream in countries that are subject to AAC/AMR licensing.
>> While I agree that it's not great from a purely technical pov since we
>> already have better native decoders, it will allow the use of FFmpeg
>> in such situation without the need to use or create another FFmpeg
>> fork dedicated to Android. Plus, as I said above, we already have the
>> wrapper and the additional code to make it work for audio is
>> relatively small and scoped. Restricting the wrapper to AAC/AMR seems
>> like a good compromise to me.
>
> Sounds reasonable, but I’d like to get more opinions. cc TC.
To add another data point, the platform decoders might also be more secure due to sandboxing. I believe as of Android Q the software decoders provided by MediaCodec have been moved to run within a constrained sandbox.
- Cosmin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-16 15:48 ` Cosmin Stejerean via ffmpeg-devel
@ 2024-07-16 16:04 ` Zhao Zhili
[not found] ` <057FD6DF-E71C-49D0-8D11-7A7E4D5FB5CC@cosmin.at>
2024-07-16 18:24 ` Rémi Denis-Courmont
2024-07-17 15:39 ` Paul B Mahol
2 siblings, 1 reply; 23+ messages in thread
From: Zhao Zhili @ 2024-07-16 16:04 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: tc
> On Jul 16, 2024, at 23:48, Cosmin Stejerean via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> wrote:
>
>
>
>> On Jul 16, 2024, at 4:58 PM, Zhao Zhili <quinkblack@foxmail.com> wrote:
>>
>>
>>
>>> On Jul 16, 2024, at 21:20, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>>>
>>> On Wed, Jul 10, 2024 at 6:31 PM Matthieu Bouron
>>> <matthieu.bouron@gmail.com> wrote:
>>>>
>>>> On Wed, Jul 10, 2024 at 4:04 PM Zhao Zhili <quinkblack@foxmail.com> wrote:
>>>>>
>>>>>
>>>>>> On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
>>>>>> MP3, FLAC, VORBIS and OPUS are supported.
>>>>>>
>>>>>> This is mainly useful to avoid shipping Android builds of FFmpeg that are
>>>>>> subjects to licensing/patents (due to AAC and AMR).
>>>>>
>>>>> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They don’t bring
>>>>> new features, they don’t improve performance. I know these type of wrapper exist in current
>>>>> project, but I’m not sure if it’s a good idea to add more.
>>>>
>>>> I agree that on the technical side it doesn't bring new features nor
>>>> performance improvements. It's all about avoiding licensing/patents
>>>> issues with AAC and AMR here.
>>>> In this specific case we already have the wrapper infrastructure, the
>>>> audio part only needs small adjustments to work.
>>>> Moreover, if that helps, I can reduce the scope of the patch to AAC
>>>> and AMR only and get rid of mp3/flac/vorbis/opus support. What do you
>>>> think ?
>>>
>>> Ping.
>>>
>>> IMHO, this benefits users wanting to ship an Android app that relies
>>> on FFmpeg upstream in countries that are subject to AAC/AMR licensing.
>>> While I agree that it's not great from a purely technical pov since we
>>> already have better native decoders, it will allow the use of FFmpeg
>>> in such situation without the need to use or create another FFmpeg
>>> fork dedicated to Android. Plus, as I said above, we already have the
>>> wrapper and the additional code to make it work for audio is
>>> relatively small and scoped. Restricting the wrapper to AAC/AMR seems
>>> like a good compromise to me.
>>
>> Sounds reasonable, but I’d like to get more opinions. cc TC.
>
> To add another data point, the platform decoders might also be more secure due to sandboxing. I believe as of Android Q the software decoders provided by MediaCodec have been moved to run within a constrained sandbox.
I don’t think what Android says “secure” apply to use. The sandbox means not to
break the OS and leak information from one APP to another APP, because mediacodec
service is shared by all APP. An APP with FFmpeg inside and use software decoder/encoder
don’t have such issues since the APP don’t share decoders/encoders with other APP.
>
> - Cosmin
>
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
[not found] ` <057FD6DF-E71C-49D0-8D11-7A7E4D5FB5CC@cosmin.at>
@ 2024-07-16 16:42 ` Cosmin Stejerean via ffmpeg-devel
0 siblings, 0 replies; 23+ messages in thread
From: Cosmin Stejerean via ffmpeg-devel @ 2024-07-16 16:42 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: Cosmin Stejerean, tc
> On Jul 16, 2024, at 6:04 PM, Zhao Zhili <quinkblack@foxmail.com> wrote:
>
>
>
>> On Jul 16, 2024, at 23:48, Cosmin Stejerean via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> wrote:
>>
>>
>>
>>> On Jul 16, 2024, at 4:58 PM, Zhao Zhili <quinkblack@foxmail.com> wrote:
>>>
>>>
>>>
>>>> On Jul 16, 2024, at 21:20, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>>>>
>>>> On Wed, Jul 10, 2024 at 6:31 PM Matthieu Bouron
>>>> <matthieu.bouron@gmail.com> wrote:
>>>>>
>>>>> On Wed, Jul 10, 2024 at 4:04 PM Zhao Zhili <quinkblack@foxmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>> On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
>>>>>>> MP3, FLAC, VORBIS and OPUS are supported.
>>>>>>>
>>>>>>> This is mainly useful to avoid shipping Android builds of FFmpeg that are
>>>>>>> subjects to licensing/patents (due to AAC and AMR).
>>>>>>
>>>>>> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They don’t bring
>>>>>> new features, they don’t improve performance. I know these type of wrapper exist in current
>>>>>> project, but I’m not sure if it’s a good idea to add more.
>>>>>
>>>>> I agree that on the technical side it doesn't bring new features nor
>>>>> performance improvements. It's all about avoiding licensing/patents
>>>>> issues with AAC and AMR here.
>>>>> In this specific case we already have the wrapper infrastructure, the
>>>>> audio part only needs small adjustments to work.
>>>>> Moreover, if that helps, I can reduce the scope of the patch to AAC
>>>>> and AMR only and get rid of mp3/flac/vorbis/opus support. What do you
>>>>> think ?
>>>>
>>>> Ping.
>>>>
>>>> IMHO, this benefits users wanting to ship an Android app that relies
>>>> on FFmpeg upstream in countries that are subject to AAC/AMR licensing.
>>>> While I agree that it's not great from a purely technical pov since we
>>>> already have better native decoders, it will allow the use of FFmpeg
>>>> in such situation without the need to use or create another FFmpeg
>>>> fork dedicated to Android. Plus, as I said above, we already have the
>>>> wrapper and the additional code to make it work for audio is
>>>> relatively small and scoped. Restricting the wrapper to AAC/AMR seems
>>>> like a good compromise to me.
>>>
>>> Sounds reasonable, but I’d like to get more opinions. cc TC.
>>
>> To add another data point, the platform decoders might also be more secure due to sandboxing. I believe as of Android Q the software decoders provided by MediaCodec have been moved to run within a constrained sandbox.
>
> I don’t think what Android says “secure” apply to use. The sandbox means not to
> break the OS and leak information from one APP to another APP, because mediacodec
> service is shared by all APP. An APP with FFmpeg inside and use software decoder/encoder
> don’t have such issues since the APP don’t share decoders/encoders with other APP.
>
It's still an issue if a vulnerability in a software decoder gives an attacker access to the same app's data, even if per app sandboxing keeps a compromised application from accessing data from other apps.
For example consider a messaging app. If you receive a message containing media that exploits a vulnerability in a software decoder and that gives the attacker access to all of your contacts and messages within the app, that's definitely a security issue.
The app itself might also have other permissions to system wide resources, etc.
- Cosmin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-16 15:48 ` Cosmin Stejerean via ffmpeg-devel
2024-07-16 16:04 ` Zhao Zhili
@ 2024-07-16 18:24 ` Rémi Denis-Courmont
[not found] ` <205EB89E-F6F9-438D-A9BC-E76E5439C7C7@cosmin.at>
2024-07-17 15:39 ` Paul B Mahol
2 siblings, 1 reply; 23+ messages in thread
From: Rémi Denis-Courmont @ 2024-07-16 18:24 UTC (permalink / raw)
To: FFmpeg development discussions and patches
Le tiistaina 16. heinäkuuta 2024, 18.48.06 EEST Cosmin Stejerean via ffmpeg-
devel a écrit :
> To add another data point, the platform decoders might also be more secure
> due to sandboxing. I believe as of Android Q the software decoders provided
> by MediaCodec have been moved to run within a constrained sandbox.
Platform decoders are in all likelihood strictly less secure than software
decoders. Software decoders will run in a user-space sandboxed within their
respective application. Platform decoders will run in a more privileged system
service, with direct access to a kernel driver in EL1, through that to the
firmware running on the video DSP.
More performant and energy-efficient. But also way way less secure.
The only viewpoint whence this is more secure, is the content publisher's:
this model enables DRM with hardware pass-through (but that does not even
apply if you use FFmpeg as the front end).
--
雷米‧德尼-库尔蒙
http://www.remlab.net/
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
[not found] ` <205EB89E-F6F9-438D-A9BC-E76E5439C7C7@cosmin.at>
@ 2024-07-16 20:14 ` Cosmin Stejerean via ffmpeg-devel
2024-07-17 9:38 ` Anton Khirnov
0 siblings, 1 reply; 23+ messages in thread
From: Cosmin Stejerean via ffmpeg-devel @ 2024-07-16 20:14 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: Cosmin Stejerean
> On Jul 16, 2024, at 8:24 PM, Rémi Denis-Courmont <remi@remlab.net> wrote:
>
> Le tiistaina 16. heinäkuuta 2024, 18.48.06 EEST Cosmin Stejerean via ffmpeg-
> devel a écrit :
>> To add another data point, the platform decoders might also be more secure
>> due to sandboxing. I believe as of Android Q the software decoders provided
>> by MediaCodec have been moved to run within a constrained sandbox.
>
> Platform decoders are in all likelihood strictly less secure than software
> decoders. Software decoders will run in a user-space sandboxed within their
> respective application. Platform decoders will run in a more privileged system
> service, with direct access to a kernel driver in EL1, through that to the
> firmware running on the video DSP.
>
> More performant and energy-efficient. But also way way less secure.
>
> The only viewpoint whence this is more secure, is the content publisher's:
> this model enables DRM with hardware pass-through (but that does not even
> apply if you use FFmpeg as the front end).
>
Platform provided *software* decoders should be more secure than bundled software decoders due to the sandboxing of software decoders in recent versions of Android.
Hardware decoders might very well not be for the reasons you mentioned.
- Cosmin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-16 20:14 ` Cosmin Stejerean via ffmpeg-devel
@ 2024-07-17 9:38 ` Anton Khirnov
2024-07-17 12:19 ` Michael Niedermayer
2024-07-19 19:59 ` Martin Storsjö
0 siblings, 2 replies; 23+ messages in thread
From: Anton Khirnov @ 2024-07-17 9:38 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: Cosmin Stejerean
Quoting Cosmin Stejerean via ffmpeg-devel (2024-07-16 22:14:19)
>
>
> > On Jul 16, 2024, at 8:24 PM, Rémi Denis-Courmont <remi@remlab.net> wrote:
> >
> > Le tiistaina 16. heinäkuuta 2024, 18.48.06 EEST Cosmin Stejerean via ffmpeg-
> > devel a écrit :
> >> To add another data point, the platform decoders might also be more secure
> >> due to sandboxing. I believe as of Android Q the software decoders provided
> >> by MediaCodec have been moved to run within a constrained sandbox.
> >
> > Platform decoders are in all likelihood strictly less secure than software
> > decoders. Software decoders will run in a user-space sandboxed within their
> > respective application. Platform decoders will run in a more privileged system
> > service, with direct access to a kernel driver in EL1, through that to the
> > firmware running on the video DSP.
> >
> > More performant and energy-efficient. But also way way less secure.
> >
> > The only viewpoint whence this is more secure, is the content publisher's:
> > this model enables DRM with hardware pass-through (but that does not even
> > apply if you use FFmpeg as the front end).
> >
>
> Platform provided *software* decoders should be more secure than bundled software decoders due to the sandboxing of software decoders in recent versions of Android.
If that is such an important feature to someone then it is not
inconceivable to implement some sort of sandboxing inside avcodec.
I'm not a big fan of the argument "we should provide passthrough to
proprietary decoders because they are more secure".
--
Anton Khirnov
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-17 9:38 ` Anton Khirnov
@ 2024-07-17 12:19 ` Michael Niedermayer
2024-07-19 19:59 ` Martin Storsjö
1 sibling, 0 replies; 23+ messages in thread
From: Michael Niedermayer @ 2024-07-17 12:19 UTC (permalink / raw)
To: FFmpeg development discussions and patches, Cosmin Stejerean
[-- Attachment #1.1: Type: text/plain, Size: 2096 bytes --]
On Wed, Jul 17, 2024 at 11:38:33AM +0200, Anton Khirnov wrote:
> Quoting Cosmin Stejerean via ffmpeg-devel (2024-07-16 22:14:19)
> >
> >
> > > On Jul 16, 2024, at 8:24 PM, Rémi Denis-Courmont <remi@remlab.net> wrote:
> > >
> > > Le tiistaina 16. heinäkuuta 2024, 18.48.06 EEST Cosmin Stejerean via ffmpeg-
> > > devel a écrit :
> > >> To add another data point, the platform decoders might also be more secure
> > >> due to sandboxing. I believe as of Android Q the software decoders provided
> > >> by MediaCodec have been moved to run within a constrained sandbox.
> > >
> > > Platform decoders are in all likelihood strictly less secure than software
> > > decoders. Software decoders will run in a user-space sandboxed within their
> > > respective application. Platform decoders will run in a more privileged system
> > > service, with direct access to a kernel driver in EL1, through that to the
> > > firmware running on the video DSP.
> > >
> > > More performant and energy-efficient. But also way way less secure.
> > >
> > > The only viewpoint whence this is more secure, is the content publisher's:
> > > this model enables DRM with hardware pass-through (but that does not even
> > > apply if you use FFmpeg as the front end).
> > >
> >
> > Platform provided *software* decoders should be more secure than bundled software decoders due to the sandboxing of software decoders in recent versions of Android.
>
> If that is such an important feature to someone then it is not
> inconceivable to implement some sort of sandboxing inside avcodec.
>
> I'm not a big fan of the argument "we should provide passthrough to
> proprietary decoders because they are more secure".
+1
thx
[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
Dictatorship: All citizens are under surveillance, all their steps and
actions recorded, for the politicians to enforce control.
Democracy: All politicians are under surveillance, all their steps and
actions recorded, for the citizens to enforce control.
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
[-- Attachment #2: Type: text/plain, Size: 251 bytes --]
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-16 15:48 ` Cosmin Stejerean via ffmpeg-devel
2024-07-16 16:04 ` Zhao Zhili
2024-07-16 18:24 ` Rémi Denis-Courmont
@ 2024-07-17 15:39 ` Paul B Mahol
[not found] ` <4F1B9768-96C8-4708-92ED-AC8E1C9FC46F@cosmin.at>
2 siblings, 1 reply; 23+ messages in thread
From: Paul B Mahol @ 2024-07-17 15:39 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: Cosmin Stejerean, tc
On Tue, Jul 16, 2024 at 5:56 PM Cosmin Stejerean via ffmpeg-devel <
ffmpeg-devel@ffmpeg.org> wrote:
>
>
> > On Jul 16, 2024, at 4:58 PM, Zhao Zhili <quinkblack@foxmail.com> wrote:
> >
> >
> >
> >> On Jul 16, 2024, at 21:20, Matthieu Bouron <matthieu.bouron@gmail.com>
> wrote:
> >>
> >> On Wed, Jul 10, 2024 at 6:31 PM Matthieu Bouron
> >> <matthieu.bouron@gmail.com> wrote:
> >>>
> >>> On Wed, Jul 10, 2024 at 4:04 PM Zhao Zhili <quinkblack@foxmail.com>
> wrote:
> >>>>
> >>>>
> >>>>> On Jun 12, 2024, at 21:42, Matthieu Bouron <
> matthieu.bouron@gmail.com> wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> This patchset adds Mediacodec audio decoders support. Currently,
> only AAC, AMR,
> >>>>> MP3, FLAC, VORBIS and OPUS are supported.
> >>>>>
> >>>>> This is mainly useful to avoid shipping Android builds of FFmpeg
> that are
> >>>>> subjects to licensing/patents (due to AAC and AMR).
> >>>>
> >>>> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg.
> They don’t bring
> >>>> new features, they don’t improve performance. I know these type of
> wrapper exist in current
> >>>> project, but I’m not sure if it’s a good idea to add more.
> >>>
> >>> I agree that on the technical side it doesn't bring new features nor
> >>> performance improvements. It's all about avoiding licensing/patents
> >>> issues with AAC and AMR here.
> >>> In this specific case we already have the wrapper infrastructure, the
> >>> audio part only needs small adjustments to work.
> >>> Moreover, if that helps, I can reduce the scope of the patch to AAC
> >>> and AMR only and get rid of mp3/flac/vorbis/opus support. What do you
> >>> think ?
> >>
> >> Ping.
> >>
> >> IMHO, this benefits users wanting to ship an Android app that relies
> >> on FFmpeg upstream in countries that are subject to AAC/AMR licensing.
> >> While I agree that it's not great from a purely technical pov since we
> >> already have better native decoders, it will allow the use of FFmpeg
> >> in such situation without the need to use or create another FFmpeg
> >> fork dedicated to Android. Plus, as I said above, we already have the
> >> wrapper and the additional code to make it work for audio is
> >> relatively small and scoped. Restricting the wrapper to AAC/AMR seems
> >> like a good compromise to me.
> >
> > Sounds reasonable, but I’d like to get more opinions. cc TC.
>
> To add another data point, the platform decoders might also be more secure
> due to sandboxing. I believe as of Android Q the software decoders provided
> by MediaCodec have been moved to run within a constrained sandbox.
>
"May".
Need citations and exact repeatable scientific proof.
> - Cosmin
>
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
[not found] ` <4F1B9768-96C8-4708-92ED-AC8E1C9FC46F@cosmin.at>
@ 2024-07-17 17:35 ` Cosmin Stejerean via ffmpeg-devel
0 siblings, 0 replies; 23+ messages in thread
From: Cosmin Stejerean via ffmpeg-devel @ 2024-07-17 17:35 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: Cosmin Stejerean, tc
> On Jul 17, 2024, at 5:39 PM, Paul B Mahol <onemda@gmail.com> wrote:
>
> On Tue, Jul 16, 2024 at 5:56 PM Cosmin Stejerean via ffmpeg-devel <
> ffmpeg-devel@ffmpeg.org> wrote:
>
>>
>>
>>> On Jul 16, 2024, at 4:58 PM, Zhao Zhili <quinkblack@foxmail.com> wrote:
>>>
>>>
>>>
>>>> On Jul 16, 2024, at 21:20, Matthieu Bouron <matthieu.bouron@gmail.com>
>> wrote:
>>>>
>>>> On Wed, Jul 10, 2024 at 6:31 PM Matthieu Bouron
>>>> <matthieu.bouron@gmail.com> wrote:
>>>>>
>>>>> On Wed, Jul 10, 2024 at 4:04 PM Zhao Zhili <quinkblack@foxmail.com>
>> wrote:
>>>>>>
>>>>>>
>>>>>>> On Jun 12, 2024, at 21:42, Matthieu Bouron <
>> matthieu.bouron@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> This patchset adds Mediacodec audio decoders support. Currently,
>> only AAC, AMR,
>>>>>>> MP3, FLAC, VORBIS and OPUS are supported.
>>>>>>>
>>>>>>> This is mainly useful to avoid shipping Android builds of FFmpeg
>> that are
>>>>>>> subjects to licensing/patents (due to AAC and AMR).
>>>>>>
>>>>>> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg.
>> They don’t bring
>>>>>> new features, they don’t improve performance. I know these type of
>> wrapper exist in current
>>>>>> project, but I’m not sure if it’s a good idea to add more.
>>>>>
>>>>> I agree that on the technical side it doesn't bring new features nor
>>>>> performance improvements. It's all about avoiding licensing/patents
>>>>> issues with AAC and AMR here.
>>>>> In this specific case we already have the wrapper infrastructure, the
>>>>> audio part only needs small adjustments to work.
>>>>> Moreover, if that helps, I can reduce the scope of the patch to AAC
>>>>> and AMR only and get rid of mp3/flac/vorbis/opus support. What do you
>>>>> think ?
>>>>
>>>> Ping.
>>>>
>>>> IMHO, this benefits users wanting to ship an Android app that relies
>>>> on FFmpeg upstream in countries that are subject to AAC/AMR licensing.
>>>> While I agree that it's not great from a purely technical pov since we
>>>> already have better native decoders, it will allow the use of FFmpeg
>>>> in such situation without the need to use or create another FFmpeg
>>>> fork dedicated to Android. Plus, as I said above, we already have the
>>>> wrapper and the additional code to make it work for audio is
>>>> relatively small and scoped. Restricting the wrapper to AAC/AMR seems
>>>> like a good compromise to me.
>>>
>>> Sounds reasonable, but I’d like to get more opinions. cc TC.
>>
>> To add another data point, the platform decoders might also be more secure
>> due to sandboxing. I believe as of Android Q the software decoders provided
>> by MediaCodec have been moved to run within a constrained sandbox.
>>
>
> "May".
> Need citations and exact repeatable scientific proof.
>
Citation https://android-developers.googleblog.com/2019/05/queue-hardening-enhancements.html#:~:text=In%20Android%20Q%2C%20we%20moved,components%20into%20less%20privileged%20sandboxes.
See the section titled "A Constrained Sandbox for Software Codecs"
- Cosmin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-17 9:38 ` Anton Khirnov
2024-07-17 12:19 ` Michael Niedermayer
@ 2024-07-19 19:59 ` Martin Storsjö
1 sibling, 0 replies; 23+ messages in thread
From: Martin Storsjö @ 2024-07-19 19:59 UTC (permalink / raw)
To: FFmpeg development discussions and patches,
FFmpeg development discussions and patches, Cosmin Stejerean
On Wed, 17 Jul 2024, Anton Khirnov wrote:
> Quoting Cosmin Stejerean via ffmpeg-devel (2024-07-16 22:14:19)
>>
>>
>> > On Jul 16, 2024, at 8:24 PM, Rémi Denis-Courmont <remi@remlab.net> wrote:
>> >
>> > Le tiistaina 16. heinäkuuta 2024, 18.48.06 EEST Cosmin Stejerean via ffmpeg-
>> > devel a écrit :
>> >> To add another data point, the platform decoders might also be more secure
>> >> due to sandboxing. I believe as of Android Q the software decoders provided
>> >> by MediaCodec have been moved to run within a constrained sandbox.
>> >
>> > Platform decoders are in all likelihood strictly less secure than software
>> > decoders. Software decoders will run in a user-space sandboxed within their
>> > respective application. Platform decoders will run in a more privileged system
>> > service, with direct access to a kernel driver in EL1, through that to the
>> > firmware running on the video DSP.
>> >
>> > More performant and energy-efficient. But also way way less secure.
>> >
>> > The only viewpoint whence this is more secure, is the content publisher's:
>> > this model enables DRM with hardware pass-through (but that does not even
>> > apply if you use FFmpeg as the front end).
>> >
>>
>> Platform provided *software* decoders should be more secure than bundled software decoders due to the sandboxing of software decoders in recent versions of Android.
>
> If that is such an important feature to someone then it is not
> inconceivable to implement some sort of sandboxing inside avcodec.
>
> I'm not a big fan of the argument "we should provide passthrough to
> proprietary decoders because they are more secure".
My 2 cents on this matter: I don't care much about the arguments about
more secure or less secure here - I don't see that as affecting the
decision either way.
We generally don't add wrappings of third party proprietary
encoders/decoders.
But for codecs shipped as part of the OS, I don't see an issue with us
allowing accessing those codecs. It's not like we're favouring any
specific third party, we're just facilitating access to whatever is
already there.
And especially in this case, we already have the general code for
accessing the MediaCodec API for codecs on Android - I don't have a
problem with extending this to a few more codecs, as that's not much more
than just providing mappings to the codec identifiers.
Similarly, we already allow decoding and encoding of a long range of
codecs via AudioToolbox on Apple platforms - I don't see MediaCodec on
Android as any different than that.
// Martin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-10 13:54 ` Zhao Zhili
2024-07-10 16:31 ` Matthieu Bouron
@ 2024-07-23 11:35 ` Martin Storsjö
2024-07-23 11:44 ` Zhao Zhili
1 sibling, 1 reply; 23+ messages in thread
From: Martin Storsjö @ 2024-07-23 11:35 UTC (permalink / raw)
To: FFmpeg development discussions and patches
On Wed, 10 Jul 2024, Zhao Zhili wrote:
>
>> On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>>
>> Hello,
>>
>> This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
>> MP3, FLAC, VORBIS and OPUS are supported.
>>
>> This is mainly useful to avoid shipping Android builds of FFmpeg that are
>> subjects to licensing/patents (due to AAC and AMR).
>
> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They
> don’t bring new features, they don’t improve performance. I know these
> type of wrapper exist in current project, but I’m not sure if it’s a
> good idea to add more.
I don't see a problem with it. It doesn't add much extra code, we already
have MediaCodec interfacing in place, it allows users to set up whichever
configuration they want. We have this for other OS codec interfaces as
well, I don't see a problem with adding this one as well - no need for the
further arguments about security and sandboxes.
// Martin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-23 11:35 ` Martin Storsjö
@ 2024-07-23 11:44 ` Zhao Zhili
2024-07-23 13:43 ` Anton Khirnov
0 siblings, 1 reply; 23+ messages in thread
From: Zhao Zhili @ 2024-07-23 11:44 UTC (permalink / raw)
To: FFmpeg development discussions and patches
> On Jul 23, 2024, at 19:35, Martin Storsjö <martin@martin.st> wrote:
>
> On Wed, 10 Jul 2024, Zhao Zhili wrote:
>
>>
>>> On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>>> Hello,
>>> This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
>>> MP3, FLAC, VORBIS and OPUS are supported.
>>> This is mainly useful to avoid shipping Android builds of FFmpeg that are
>>> subjects to licensing/patents (due to AAC and AMR).
>>
>> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They don’t bring new features, they don’t improve performance. I know these type of wrapper exist in current project, but I’m not sure if it’s a good idea to add more.
>
> I don't see a problem with it. It doesn't add much extra code, we already have MediaCodec interfacing in place, it allows users to set up whichever configuration they want. We have this for other OS codec interfaces as well, I don't see a problem with adding this one as well - no need for the further arguments about security and sandboxes.
OK. Security isn’t what I’m concerned. I wasn’t clear on the policy side concern than technique issues.
I will do a review and test as soon as possible.
>
> // Martin
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] Add Mediacodec audio decoders support
2024-07-23 11:44 ` Zhao Zhili
@ 2024-07-23 13:43 ` Anton Khirnov
0 siblings, 0 replies; 23+ messages in thread
From: Anton Khirnov @ 2024-07-23 13:43 UTC (permalink / raw)
To: FFmpeg development discussions and patches
Quoting Zhao Zhili (2024-07-23 13:44:37)
> > On Jul 23, 2024, at 19:35, Martin Storsjö <martin@martin.st> wrote:
> >
> > On Wed, 10 Jul 2024, Zhao Zhili wrote:
> >
> >>
> >>> On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
> >>> Hello,
> >>> This patchset adds Mediacodec audio decoders support. Currently, only AAC, AMR,
> >>> MP3, FLAC, VORBIS and OPUS are supported.
> >>> This is mainly useful to avoid shipping Android builds of FFmpeg that are
> >>> subjects to licensing/patents (due to AAC and AMR).
> >>
> >> I’m not keen on put OS audio decoder/encoder wrapper into FFmpeg. They don’t bring new features, they don’t improve performance. I know these type of wrapper exist in current project, but I’m not sure if it’s a good idea to add more.
> >
> > I don't see a problem with it. It doesn't add much extra code, we already have MediaCodec interfacing in place, it allows users to set up whichever configuration they want. We have this for other OS codec interfaces as well, I don't see a problem with adding this one as well - no need for the further arguments about security and sandboxes.
>
> OK. Security isn’t what I’m concerned. I wasn’t clear on the policy side concern than technique issues.
>
> I will do a review and test as soon as possible.
I don't think we should add wrappers for random decoders just because
they are there, without a good reason. And if the reason here is
patents, then it does not apply to flac, vorbis, or opus.
--
Anton Khirnov
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [FFmpeg-devel] [PATCH 2/2] avcodec: add Mediacodec audio decoders support
2024-06-12 13:42 ` [FFmpeg-devel] [PATCH 2/2] avcodec: add Mediacodec audio decoders support Matthieu Bouron
2024-07-10 8:47 ` Matthieu Bouron
@ 2024-07-28 15:23 ` Zhao Zhili
1 sibling, 0 replies; 23+ messages in thread
From: Zhao Zhili @ 2024-07-28 15:23 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: Matthieu Bouron
> On Jun 12, 2024, at 21:42, Matthieu Bouron <matthieu.bouron@gmail.com> wrote:
>
> ---
> configure | 14 ++
> libavcodec/Makefile | 7 +
> libavcodec/allcodecs.c | 7 +
> libavcodec/mediacodecdec.c | 215 ++++++++++++++++++-
> libavcodec/mediacodecdec_common.c | 333 +++++++++++++++++++++++++++---
> 5 files changed, 545 insertions(+), 31 deletions(-)
>
> diff --git a/configure b/configure
> index 83284427df..d7de3b73ed 100755
> --- a/configure
> +++ b/configure
> @@ -3321,14 +3321,22 @@ amf_deps_any="libdl LoadLibrary"
> nvenc_deps="ffnvcodec"
> nvenc_deps_any="libdl LoadLibrary"
>
> +aac_mediacodec_decoder_deps="mediacodec"
> +aac_mediacodec_decoder_select="aac_adtstoasc_bsf aac_parser"
> aac_mf_encoder_deps="mediafoundation"
> ac3_mf_encoder_deps="mediafoundation"
> +amrnb_mediacodec_decoder_deps="mediacodec"
> +amrnb_mediacodec_decoder_select="amr_parser"
> +amrwb_mediacodec_decoder_deps="mediacodec"
> +amrwb_mediacodec_decoder_select="amr_parser"
> av1_cuvid_decoder_deps="cuvid CUVIDAV1PICPARAMS"
> av1_mediacodec_decoder_deps="mediacodec"
> av1_mediacodec_encoder_deps="mediacodec"
> av1_mediacodec_encoder_select="extract_extradata_bsf"
> av1_nvenc_encoder_deps="nvenc NV_ENC_PIC_PARAMS_AV1"
> av1_nvenc_encoder_select="atsc_a53"
> +flac_mediacodec_decoder_deps="mediacodec"
> +flac_mediacodec_decoder_select="flac_parser"
> h263_v4l2m2m_decoder_deps="v4l2_m2m h263_v4l2_m2m"
> h263_v4l2m2m_encoder_deps="v4l2_m2m h263_v4l2_m2m"
> h264_amf_encoder_deps="amf"
> @@ -3377,6 +3385,8 @@ mjpeg_qsv_encoder_select="qsvenc"
> mjpeg_vaapi_encoder_deps="VAEncPictureParameterBufferJPEG"
> mjpeg_vaapi_encoder_select="cbs_jpeg jpegtables vaapi_encode"
> mp3_mf_encoder_deps="mediafoundation"
> +mp3_mediacodec_decoder_deps="mediacodec"
> +mp3_mediacodec_decoder_select="mpegaudioheader"
> mpeg1_cuvid_decoder_deps="cuvid"
> mpeg1_v4l2m2m_decoder_deps="v4l2_m2m mpeg1_v4l2_m2m"
> mpeg2_cuvid_decoder_deps="cuvid"
> @@ -3394,10 +3404,14 @@ mpeg4_mmal_decoder_deps="mmal"
> mpeg4_omx_encoder_deps="omx"
> mpeg4_v4l2m2m_decoder_deps="v4l2_m2m mpeg4_v4l2_m2m"
> mpeg4_v4l2m2m_encoder_deps="v4l2_m2m mpeg4_v4l2_m2m"
> +opus_mediacodec_decoder_deps="mediacodec"
> +opus_mediacodec_decoder_select="opus_parser"
> vc1_cuvid_decoder_deps="cuvid"
> vc1_mmal_decoder_deps="mmal"
> vc1_qsv_decoder_select="qsvdec"
> vc1_v4l2m2m_decoder_deps="v4l2_m2m vc1_v4l2_m2m"
> +vorbis_mediacodec_decoder_deps="mediacodec"
> +vorbis_mediacodec_decoder_select="vorbis_parser"
> vp8_cuvid_decoder_deps="cuvid"
> vp8_mediacodec_decoder_deps="mediacodec"
> vp8_mediacodec_encoder_deps="mediacodec"
> diff --git a/libavcodec/Makefile b/libavcodec/Makefile
> index 1a44352906..64771b9944 100644
> --- a/libavcodec/Makefile
> +++ b/libavcodec/Makefile
> @@ -196,6 +196,7 @@ OBJS-$(CONFIG_AAC_ENCODER) += aacenc.o aaccoder.o aacenctab.o \
> aacenc_pred.o \
> psymodel.o kbdwin.o \
> mpeg4audio_sample_rates.o
> +OBJS-$(CONFIG_AAC_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_AAC_MF_ENCODER) += mfenc.o mf_utils.o
> OBJS-$(CONFIG_AASC_DECODER) += aasc.o msrledec.o
> OBJS-$(CONFIG_AC3_DECODER) += ac3dec_float.o ac3dec_data.o ac3.o \
> @@ -222,6 +223,8 @@ OBJS-$(CONFIG_AMRWB_DECODER) += amrwbdec.o celp_filters.o \
> celp_math.o acelp_filters.o \
> acelp_vectors.o \
> acelp_pitch_delay.o
> +OBJS-$(CONFIG_AMRNB_MEDIACODEC_DECODER) += mediacodecdec.o
> +OBJS-$(CONFIG_AMRWB_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_AMV_ENCODER) += mjpegenc.o mjpegenc_common.o
> OBJS-$(CONFIG_ANM_DECODER) += anm.o
> OBJS-$(CONFIG_ANULL_DECODER) += null.o
> @@ -367,6 +370,7 @@ OBJS-$(CONFIG_FIC_DECODER) += fic.o
> OBJS-$(CONFIG_FITS_DECODER) += fitsdec.o fits.o
> OBJS-$(CONFIG_FITS_ENCODER) += fitsenc.o
> OBJS-$(CONFIG_FLAC_DECODER) += flacdec.o flacdata.o flacdsp.o flac.o
> +OBJS-$(CONFIG_FLAC_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_FLAC_ENCODER) += flacenc.o flacdata.o flacencdsp.o
> OBJS-$(CONFIG_FLASHSV_DECODER) += flashsv.o
> OBJS-$(CONFIG_FLASHSV_ENCODER) += flashsvenc.o
> @@ -518,6 +522,7 @@ OBJS-$(CONFIG_MP2FIXED_ENCODER) += mpegaudioenc_fixed.o mpegaudio.o \
> mpegaudiotabs.o
> OBJS-$(CONFIG_MP2FLOAT_DECODER) += mpegaudiodec_float.o
> OBJS-$(CONFIG_MP3_DECODER) += mpegaudiodec_fixed.o
> +OBJS-$(CONFIG_MP3_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_MP3_MF_ENCODER) += mfenc.o mf_utils.o
> OBJS-$(CONFIG_MP3ADU_DECODER) += mpegaudiodec_fixed.o
> OBJS-$(CONFIG_MP3ADUFLOAT_DECODER) += mpegaudiodec_float.o
> @@ -578,6 +583,7 @@ OBJS-$(CONFIG_OPUS_DECODER) += opusdec.o opusdec_celt.o opus_celt.o \
> opusdsp.o opus_parse.o opus_rc.o
> OBJS-$(CONFIG_OPUS_ENCODER) += opusenc.o opusenc_psy.o opus_celt.o \
> opus_pvq.o opus_rc.o opustab.o
> +OBJS-$(CONFIG_OPUS_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_OSQ_DECODER) += osq.o
> OBJS-$(CONFIG_PAF_AUDIO_DECODER) += pafaudio.o
> OBJS-$(CONFIG_PAF_VIDEO_DECODER) += pafvideo.o
> @@ -765,6 +771,7 @@ OBJS-$(CONFIG_VORBIS_DECODER) += vorbisdec.o vorbisdsp.o vorbis.o \
> vorbis_data.o
> OBJS-$(CONFIG_VORBIS_ENCODER) += vorbisenc.o vorbis.o \
> vorbis_data.o
> +OBJS-$(CONFIG_VORBIS_MEDIACODEC_DECODER) += mediacodecdec.o
> OBJS-$(CONFIG_VP3_DECODER) += vp3.o jpegquanttables.o
> OBJS-$(CONFIG_VP5_DECODER) += vp5.o vp56.o vp56data.o vpx_rac.o
> OBJS-$(CONFIG_VP6_DECODER) += vp6.o vp56.o vp56data.o \
> diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
> index b102a8069e..e5ab657ef6 100644
> --- a/libavcodec/allcodecs.c
> +++ b/libavcodec/allcodecs.c
> @@ -821,8 +821,11 @@ extern const FFCodec ff_idf_decoder;
>
> /* external libraries, that shouldn't be used by default if one of the
> * above is available */
> +extern const FFCodec ff_aac_mediacodec_decoder;
> extern const FFCodec ff_aac_mf_encoder;
> extern const FFCodec ff_ac3_mf_encoder;
> +extern const FFCodec ff_amrnb_mediacodec_decoder;
> +extern const FFCodec ff_amrwb_mediacodec_decoder;
> extern const FFCodec ff_h263_v4l2m2m_encoder;
> extern const FFCodec ff_libaom_av1_decoder;
> /* hwaccel hooks only, so prefer external decoders */
> @@ -835,6 +838,7 @@ extern const FFCodec ff_av1_qsv_decoder;
> extern const FFCodec ff_av1_qsv_encoder;
> extern const FFCodec ff_av1_amf_encoder;
> extern const FFCodec ff_av1_vaapi_encoder;
> +extern const FFCodec ff_flac_mediacodec_decoder;
> extern const FFCodec ff_libopenh264_encoder;
> extern const FFCodec ff_libopenh264_decoder;
> extern const FFCodec ff_h264_amf_encoder;
> @@ -861,6 +865,7 @@ extern const FFCodec ff_mjpeg_cuvid_decoder;
> extern const FFCodec ff_mjpeg_qsv_encoder;
> extern const FFCodec ff_mjpeg_qsv_decoder;
> extern const FFCodec ff_mjpeg_vaapi_encoder;
> +extern const FFCodec ff_mp3_mediacodec_decoder;
> extern const FFCodec ff_mp3_mf_encoder;
> extern const FFCodec ff_mpeg1_cuvid_decoder;
> extern const FFCodec ff_mpeg2_cuvid_decoder;
> @@ -871,8 +876,10 @@ extern const FFCodec ff_mpeg4_mediacodec_decoder;
> extern const FFCodec ff_mpeg4_mediacodec_encoder;
> extern const FFCodec ff_mpeg4_omx_encoder;
> extern const FFCodec ff_mpeg4_v4l2m2m_encoder;
> +extern const FFCodec ff_opus_mediacodec_decoder;
> extern const FFCodec ff_prores_videotoolbox_encoder;
> extern const FFCodec ff_vc1_cuvid_decoder;
> +extern const FFCodec ff_vorbis_mediacodec_decoder;
> extern const FFCodec ff_vp8_cuvid_decoder;
> extern const FFCodec ff_vp8_mediacodec_decoder;
> extern const FFCodec ff_vp8_mediacodec_encoder;
> diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c
> index 6d8dc600fe..dabf71cc04 100644
> --- a/libavcodec/mediacodecdec.c
> +++ b/libavcodec/mediacodecdec.c
> @@ -36,6 +36,7 @@
> #include "avcodec.h"
> #include "codec_internal.h"
> #include "decode.h"
> +#include "flac_parse.h"
> #include "h264_parse.h"
> #include "h264_ps.h"
> #include "hevc/parse.h"
> @@ -44,6 +45,7 @@
> #include "jni.h"
> #include "mediacodec_wrapper.h"
> #include "mediacodecdec_common.h"
> +#include "xiph.h"
>
> typedef struct MediaCodecH264DecContext {
>
> @@ -287,11 +289,87 @@ done:
> }
> #endif
>
> +#if CONFIG_FLAC_MEDIACODEC_DECODER
> +static int flac_set_extradata(AVCodecContext *avctx, FFAMediaFormat *format)
> +{
> + uint8_t *streaminfo;
> + uint8_t buffer[42];
> +
> + if (!avctx->extradata) {
> + return 0;
> + }
It seems extradata is required for FLAC decoding, so here should return an error code.
Don't use braces here for coding style.
> +
> + if (!ff_flac_is_extradata_valid(avctx, &streaminfo))
> + return AVERROR_INVALIDDATA;
> +
> + buffer[0] = 'f';
> + buffer[1] = 'L';
> + buffer[2] = 'a';
> + buffer[3] = 'C';
> + buffer[4] = 0x80;
> + buffer[5] = 0;
> + buffer[6] = 0;
> + buffer[7] = 0x22;
> + memcpy(buffer + 8, streaminfo, 34);
> +
> + /* csd-0: fLaC + streaminfo */
> + ff_AMediaFormat_setBuffer(format, "csd-0", buffer, 42);
> +
> + return 0;
> +}
> +#endif
> +
> +#if CONFIG_OPUS_MEDIACODEC_DECODER
> +static int opus_set_extradata(AVCodecContext *avctx, FFAMediaFormat *format)
> +{
> + if (!avctx->extradata_size) {
> + return 0;
> + }
Ditto.
> +
> + if (avctx->extradata_size < 19) {
> + return AVERROR_INVALIDDATA;
> + }
> +
> + ff_AMediaFormat_setBuffer(format, "csd-0", avctx->extradata, 19);
> +
> + return 0;
> +}
> +#endif
Not sure why assert failure
./ffmpeg_g -c:a opus_mediacodec -i /sdcard/opus.mp4 -f null -
[opus_mediacodec @ 0xb4000070239c9650] MediaCodec started successfully: codec = c2.android.opus.decoder, ret = 0
Stream mapping:
Stream #0:0 -> #0:0 (opus (opus_mediacodec) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[opus_mediacodec @ 0xb4000070239c9650] Output MediaFormat changed to channel-count: int32(2), mime: string(audio/raw), sample-rate: int32(48000), android._config-pcm-encoding: int32(2)}
[opus_mediacodec @ 0xb4000070239c9650] Output parameters channel-count=2 channel-layout=0 sample-rate=48000
Assertion pkt failed at src/fftools/ffmpeg_dec.c:726
Aborted
> +
> +#if CONFIG_VORBIS_MEDIACODEC_DECODER
> +static int vorbis_set_extradata(AVCodecContext *avctx, FFAMediaFormat *format)
> +{
> + int ret;
> + const uint8_t *header_start[3];
> + int header_len[3];
> +
> + if (!avctx->extradata) {
> + return 0;
> + }
> +
> + ret = avpriv_split_xiph_headers(avctx->extradata, avctx->extradata_size, 30, header_start, header_len);
> + if (ret < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Could not parse extradata\n");
> + return ret;
> + }
> +
> + /* csd-0: identification header, csd-1: setup header */
> + ff_AMediaFormat_setBuffer(format, "csd-0", header_start[0], header_len[0]);
> + ff_AMediaFormat_setBuffer(format, "csd-1", header_start[2], header_len[2]);
> +
> + return 0;
> +}
> +#endif
> +
> #if CONFIG_MPEG2_MEDIACODEC_DECODER || \
> CONFIG_MPEG4_MEDIACODEC_DECODER || \
> CONFIG_VP8_MEDIACODEC_DECODER || \
> CONFIG_VP9_MEDIACODEC_DECODER || \
> - CONFIG_AV1_MEDIACODEC_DECODER
> + CONFIG_AV1_MEDIACODEC_DECODER || \
> + CONFIG_AAC_MEDIACODEC_DECODER || \
> + CONFIG_AMRNB_MEDIACODEC_DECODER || \
> + CONFIG_AMRWB_MEDIACODEC_DECODER || \
> + CONFIG_MP3_MEDIACODEC_DECODER
> static int common_set_extradata(AVCodecContext *avctx, FFAMediaFormat *format)
> {
> int ret = 0;
> @@ -387,14 +465,83 @@ static av_cold int mediacodec_decode_init(AVCodecContext *avctx)
> if (ret < 0)
> goto done;
> break;
> +#endif
> +#if CONFIG_AAC_MEDIACODEC_DECODER
> + case AV_CODEC_ID_AAC:
> + codec_mime = "audio/mp4a-latm";
> +
> + ret = common_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_AMRNB_MEDIACODEC_DECODER
> + case AV_CODEC_ID_AMR_NB:
> + codec_mime = "audio/3gpp";
> +
> + ret = common_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_AMRWB_MEDIACODEC_DECODER
> + case AV_CODEC_ID_AMR_WB:
> + codec_mime = "audio/amr-wb";
> +
> + ret = common_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_FLAC_MEDIACODEC_DECODER
> + case AV_CODEC_ID_FLAC:
> + codec_mime = "audio/flac";
> +
> + ret = flac_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_MP3_MEDIACODEC_DECODER
> + case AV_CODEC_ID_MP3:
> + codec_mime = "audio/mpeg";
> +
> + ret = common_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_OPUS_MEDIACODEC_DECODER
> + case AV_CODEC_ID_OPUS:
> + codec_mime = "audio/opus";
> +
> + ret = opus_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> +#endif
> +#if CONFIG_VORBIS_MEDIACODEC_DECODER
> + case AV_CODEC_ID_VORBIS:
> + codec_mime = "audio/vorbis";
> +
> + ret = vorbis_set_extradata(avctx, format);
> + if (ret < 0)
> + goto done;
> + break;
> #endif
> default:
> av_assert0(0);
> }
>
> ff_AMediaFormat_setString(format, "mime", codec_mime);
> - ff_AMediaFormat_setInt32(format, "width", avctx->width);
> - ff_AMediaFormat_setInt32(format, "height", avctx->height);
> +
> + if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
> + ff_AMediaFormat_setInt32(format, "width", avctx->width);
> + ff_AMediaFormat_setInt32(format, "height", avctx->height);
> + } else {
> + ff_AMediaFormat_setInt32(format, "channel-count", avctx->ch_layout.nb_channels);
> + ff_AMediaFormat_setInt32(format, "sample-rate", avctx->sample_rate);
> + }
>
> s->ctx = av_mallocz(sizeof(*s->ctx));
> if (!s->ctx) {
> @@ -611,3 +758,65 @@ DECLARE_MEDIACODEC_VDEC(vp9, "VP9", AV_CODEC_ID_VP9, NULL)
> #if CONFIG_AV1_MEDIACODEC_DECODER
> DECLARE_MEDIACODEC_VDEC(av1, "AV1", AV_CODEC_ID_AV1, NULL)
> #endif
> +
> +#define AD AV_OPT_FLAG_AUDIO_PARAM | AV_OPT_FLAG_DECODING_PARAM
> +static const AVOption ff_mediacodec_adec_options[] = {
> + { "ndk_codec", "Use MediaCodec from NDK",
> + OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 = -1}, -1, 1, AD },
> + { NULL }
> +};
> +
> +#define DECLARE_MEDIACODEC_ACLASS(short_name) \
> +static const AVClass ff_##short_name##_mediacodec_dec_class = { \
> + .class_name = #short_name "_mediacodec", \
> + .item_name = av_default_item_name, \
> + .option = ff_mediacodec_adec_options, \
> + .version = LIBAVUTIL_VERSION_INT, \
> +};
> +
> +#define DECLARE_MEDIACODEC_ADEC(short_name, full_name, codec_id, bsf) \
> +DECLARE_MEDIACODEC_VCLASS(short_name) \
> +const FFCodec ff_ ## short_name ## _mediacodec_decoder = { \
> + .p.name = #short_name "_mediacodec", \
> + CODEC_LONG_NAME(full_name " Android MediaCodec decoder"), \
> + .p.type = AVMEDIA_TYPE_AUDIO, \
> + .p.id = codec_id, \
> + .p.priv_class = &ff_##short_name##_mediacodec_dec_class, \
> + .priv_data_size = sizeof(MediaCodecH264DecContext), \
> + .init = mediacodec_decode_init, \
> + FF_CODEC_RECEIVE_FRAME_CB(mediacodec_receive_frame), \
> + .flush = mediacodec_decode_flush, \
> + .close = mediacodec_decode_close, \
> + .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE, \
> + .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE, \
> + .bsfs = bsf, \
> + .p.wrapper_name = "mediacodec", \
> +}; \
> +
> +#if CONFIG_AAC_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(aac, "AAC", AV_CODEC_ID_AAC, "aac_adtstoasc")
> +#endif
> +
> +#if CONFIG_AMRNB_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(amrnb, "AMR-NB", AV_CODEC_ID_AMR_NB, NULL)
> +#endif
> +
> +#if CONFIG_AMRWB_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(amrwb, "AMR-WB", AV_CODEC_ID_AMR_WB, NULL)
> +#endif
> +
> +#if CONFIG_FLAC_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(flac, "FLAC", AV_CODEC_ID_FLAC, NULL)
> +#endif
> +
> +#if CONFIG_MP3_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(mp3, "MP3", AV_CODEC_ID_MP3, NULL)
> +#endif
> +
> +#if CONFIG_OPUS_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(opus, "OPUS", AV_CODEC_ID_OPUS, NULL)
> +#endif
> +
> +#if CONFIG_VORBIS_MEDIACODEC_DECODER
> +DECLARE_MEDIACODEC_ADEC(vorbis, "VORBIS", AV_CODEC_ID_VORBIS, NULL)
> +#endif
> diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c
> index 4d22db8ecf..4bdd0128c2 100644
> --- a/libavcodec/mediacodecdec_common.c
> +++ b/libavcodec/mediacodecdec_common.c
> @@ -23,6 +23,7 @@
> #include <string.h>
> #include <sys/types.h>
>
> +#include "libavutil/avassert.h"
> #include "libavutil/common.h"
> #include "libavutil/hwcontext_mediacodec.h"
> #include "libavutil/mem.h"
> @@ -30,6 +31,7 @@
> #include "libavutil/pixfmt.h"
> #include "libavutil/time.h"
> #include "libavutil/timestamp.h"
> +#include "libavutil/channel_layout.h"
>
> #include "avcodec.h"
> #include "decode.h"
> @@ -85,6 +87,110 @@
> #define OUTPUT_DEQUEUE_TIMEOUT_US 8000
> #define OUTPUT_DEQUEUE_BLOCK_TIMEOUT_US 1000000
>
> +enum {
> + ENCODING_PCM_16BIT = 0x00000002,
> + ENCODING_PCM_8BIT = 0x00000003,
> + ENCODING_PCM_FLOAT = 0x00000004,
> + ENCODING_PCM_24BIT_PACKED = 0x00000015,
> + ENCODING_PCM_32BIT = 0x00000016,
> +};
> +
> +static const struct {
> +
> + int pcm_format;
> + enum AVSampleFormat sample_format;
> +
> +} sample_formats[] = {
> +
> + { ENCODING_PCM_16BIT, AV_SAMPLE_FMT_S16 },
> + { ENCODING_PCM_8BIT, AV_SAMPLE_FMT_U8 },
> + { ENCODING_PCM_FLOAT, AV_SAMPLE_FMT_FLT },
> + { ENCODING_PCM_24BIT_PACKED, AV_SAMPLE_FMT_S32 },
> + { ENCODING_PCM_32BIT, AV_SAMPLE_FMT_S32 },
> + { 0 }
> +};
> +
> +static enum AVSampleFormat mcdec_map_pcm_format(AVCodecContext *avctx,
> + MediaCodecDecContext *s,
> + int pcm_format)
> +{
> + int i;
> + enum AVSampleFormat ret = AV_SAMPLE_FMT_NONE;
> +
> + for (i = 0; i < FF_ARRAY_ELEMS(sample_formats); i++) {
for (int i = 0;
> + if (sample_formats[i].pcm_format == pcm_format) {
> + return sample_formats[i].sample_format;
> + }
> + }
> +
> + av_log(avctx, AV_LOG_ERROR, "Output sample format 0x%x (value=%d) is not supported\n",
> + pcm_format, pcm_format);
> +
> + return ret;
> +}
> +
> +enum
> +{
> + CHANNEL_OUT_FRONT_LEFT = 0x4,
> + CHANNEL_OUT_FRONT_RIGHT = 0x8,
> + CHANNEL_OUT_FRONT_CENTER = 0x10,
> + CHANNEL_OUT_LOW_FREQUENCY = 0x20,
> + CHANNEL_OUT_BACK_LEFT = 0x40,
> + CHANNEL_OUT_BACK_RIGHT = 0x80,
> + CHANNEL_OUT_FRONT_LEFT_OF_CENTER = 0x100,
> + CHANNEL_OUT_FRONT_RIGHT_OF_CENTER = 0x200,
> + CHANNEL_OUT_BACK_CENTER = 0x400,
> + CHANNEL_OUT_SIDE_LEFT = 0x800,
> + CHANNEL_OUT_SIDE_RIGHT = 0x1000,
> + CHANNEL_OUT_TOP_CENTER = 0x2000,
> + CHANNEL_OUT_TOP_FRONT_LEFT = 0x4000,
> + CHANNEL_OUT_TOP_FRONT_CENTER = 0x8000,
> + CHANNEL_OUT_TOP_FRONT_RIGHT = 0x10000,
> + CHANNEL_OUT_TOP_BACK_LEFT = 0x20000,
> + CHANNEL_OUT_TOP_BACK_CENTER = 0x40000,
> + CHANNEL_OUT_TOP_BACK_RIGHT = 0x80000,
> +};
> +
> +static const struct {
> +
> + int mask;
> + uint64_t layout;
> +
> +} channel_masks[] = {
> + { CHANNEL_OUT_FRONT_LEFT, AV_CH_FRONT_LEFT },
> + { CHANNEL_OUT_FRONT_RIGHT, AV_CH_FRONT_RIGHT },
> + { CHANNEL_OUT_FRONT_CENTER, AV_CH_FRONT_CENTER },
> + { CHANNEL_OUT_LOW_FREQUENCY, AV_CH_LOW_FREQUENCY },
> + { CHANNEL_OUT_BACK_LEFT, AV_CH_BACK_LEFT },
> + { CHANNEL_OUT_BACK_RIGHT, AV_CH_BACK_RIGHT },
> + { CHANNEL_OUT_FRONT_LEFT_OF_CENTER, AV_CH_FRONT_LEFT_OF_CENTER },
> + { CHANNEL_OUT_FRONT_RIGHT_OF_CENTER, AV_CH_FRONT_RIGHT_OF_CENTER },
> + { CHANNEL_OUT_BACK_CENTER, AV_CH_BACK_CENTER },
> + { CHANNEL_OUT_SIDE_LEFT, AV_CH_SIDE_LEFT },
> + { CHANNEL_OUT_SIDE_RIGHT, AV_CH_SIDE_RIGHT },
> + { CHANNEL_OUT_TOP_CENTER, AV_CH_TOP_CENTER },
> + { CHANNEL_OUT_TOP_FRONT_LEFT, AV_CH_TOP_FRONT_LEFT },
> + { CHANNEL_OUT_TOP_FRONT_CENTER, AV_CH_TOP_FRONT_CENTER },
> + { CHANNEL_OUT_TOP_FRONT_RIGHT, AV_CH_TOP_FRONT_RIGHT },
> + { CHANNEL_OUT_TOP_BACK_LEFT, AV_CH_TOP_BACK_LEFT },
> + { CHANNEL_OUT_TOP_BACK_CENTER, AV_CH_TOP_BACK_CENTER },
> + { CHANNEL_OUT_TOP_BACK_RIGHT, AV_CH_TOP_BACK_RIGHT },
> +};
Two space indents?
> +
> +static uint64_t mcdec_map_channel_mask(AVCodecContext *avctx,
> + int channel_mask)
> +{
> + int i;
> + uint64_t channel_layout = 0;
> +
> + for (i = 0; i < FF_ARRAY_ELEMS(channel_masks); i++) {
> + if (channel_mask & channel_masks[i].mask)
> + channel_layout |= channel_masks[i].layout;
> + }
> +
> + return channel_layout;
> +}
> +
> enum {
> COLOR_FormatYUV420Planar = 0x13,
> COLOR_FormatYUV420SemiPlanar = 0x15,
> @@ -265,13 +371,77 @@ fail:
> return ret;
> }
>
> -static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx,
> - MediaCodecDecContext *s,
> - uint8_t *data,
> - size_t size,
> - ssize_t index,
> - FFAMediaCodecBufferInfo *info,
> - AVFrame *frame)
> +static int mediacodec_wrap_sw_audio_buffer(AVCodecContext *avctx,
> + MediaCodecDecContext *s,
> + uint8_t *data,
> + size_t size,
> + ssize_t index,
> + FFAMediaCodecBufferInfo *info,
> + AVFrame *frame)
> +{
> + int ret = 0;
> + int status = 0;
> + const int sample_size = av_get_bytes_per_sample(avctx->sample_fmt);
> + if (!sample_size) {
> + av_log(avctx, AV_LOG_ERROR, "Could not get bytes per sample\n");
> + goto done;
ret is 0 on error path.
> + }
> +
> + frame->format = avctx->sample_fmt;
> + frame->sample_rate = avctx->sample_rate;
> + frame->nb_samples = info->size / (sample_size * avctx->ch_layout.nb_channels);
Is this right for ENCODING_PCM_24BIT_PACKED?
> +
> + ret = av_channel_layout_copy(&frame->ch_layout, &avctx->ch_layout);
> + if (ret < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Could not copy channel layout\n");
> + goto done;
> + }
> +
> + /* MediaCodec buffers needs to be copied to our own refcounted buffers
> + * because the flush command invalidates all input and output buffers.
> + */
> + if ((ret = ff_get_buffer(avctx, frame, 0)) < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Could not allocate buffer\n");
> + goto done;
> + }
> +
> + /* Override frame->pts as ff_get_buffer will override its value based
> + * on the last avpacket received which is not in sync with the frame:
> + * * N avpackets can be pushed before 1 frame is actually returned
> + * * 0-sized avpackets are pushed to flush remaining frames at EOS */
> + if (avctx->pkt_timebase.num && avctx->pkt_timebase.den) {
> + frame->pts = av_rescale_q(info->presentationTimeUs,
> + AV_TIME_BASE_Q,
> + avctx->pkt_timebase);
> + } else {
> + frame->pts = info->presentationTimeUs;
> + }
> + frame->pkt_dts = AV_NOPTS_VALUE;
> +
> + av_log(avctx, AV_LOG_TRACE,
> + "Frame: format=%d channels=%d sample_rate=%d nb_samples=%d",
> + avctx->sample_fmt, avctx->ch_layout.nb_channels, avctx->sample_rate, frame->nb_samples);
> +
> + memcpy(frame->data[0], data, info->size);
> +
> + ret = 0;
> +done:
> + status = ff_AMediaCodec_releaseOutputBuffer(s->codec, index, 0);
> + if (status < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Failed to release output buffer\n");
> + ret = AVERROR_EXTERNAL;
> + }
> +
> + return ret;
> +}
> +
> +static int mediacodec_wrap_sw_video_buffer(AVCodecContext *avctx,
> + MediaCodecDecContext *s,
> + uint8_t *data,
> + size_t size,
> + ssize_t index,
> + FFAMediaCodecBufferInfo *info,
> + AVFrame *frame)
> {
> int ret = 0;
> int status = 0;
> @@ -343,6 +513,22 @@ done:
> return ret;
> }
>
> +static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx,
> + MediaCodecDecContext *s,
> + uint8_t *data,
> + size_t size,
> + ssize_t index,
> + FFAMediaCodecBufferInfo *info,
> + AVFrame *frame)
> +{
> + if (avctx->codec_type == AVMEDIA_TYPE_AUDIO)
> + return mediacodec_wrap_sw_audio_buffer(avctx, s, data, size, index, info, frame);
> + else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO)
> + return mediacodec_wrap_sw_video_buffer(avctx, s, data, size, index, info, frame);
> + else
> + av_assert0(0);
> +}
> +
> #define AMEDIAFORMAT_GET_INT32(name, key, mandatory) do { \
> int32_t value = 0; \
> if (ff_AMediaFormat_getInt32(s->format, key, &value)) { \
> @@ -354,7 +540,7 @@ done:
> } \
> } while (0) \
>
> -static int mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecContext *s)
> +static int mediacodec_dec_parse_video_format(AVCodecContext *avctx, MediaCodecDecContext *s)
> {
> int ret = 0;
> int width = 0;
> @@ -463,6 +649,63 @@ fail:
> return ret;
> }
>
> +static int mediacodec_dec_parse_audio_format(AVCodecContext *avctx, MediaCodecDecContext *s)
> +{
> + int ret = 0;
> + int sample_rate = 0;
> + int channel_count = 0;
> + int channel_mask = 0;
> + int pcm_encoding = 0;
> + char *format = NULL;
> +
> + if (!s->format) {
> + av_log(avctx, AV_LOG_ERROR, "Output MediaFormat is not set\n");
> + return AVERROR(EINVAL);
> + }
> +
> + format = ff_AMediaFormat_toString(s->format);
> + if (!format) {
> + return AVERROR_EXTERNAL;
> + }
> + av_log(avctx, AV_LOG_DEBUG, "Parsing MediaFormat %s\n", format);
> +
> + /* Mandatory fields */
> + AMEDIAFORMAT_GET_INT32(channel_count, "channel-count", 1);
> + AMEDIAFORMAT_GET_INT32(sample_rate, "sample-rate", 1);
> +
> + AMEDIAFORMAT_GET_INT32(pcm_encoding, "pcm-encoding", 0);
> + if (pcm_encoding)
> + avctx->sample_fmt = mcdec_map_pcm_format(avctx, s, pcm_encoding);
> + else
> + avctx->sample_fmt = AV_SAMPLE_FMT_S16;
> +
> + avctx->sample_rate = sample_rate;
> +
> + AMEDIAFORMAT_GET_INT32(channel_mask, "channel-mask", 0);
> + if (channel_mask)
> + av_channel_layout_from_mask(&avctx->ch_layout, mcdec_map_channel_mask(avctx, channel_mask));
> + else
> + av_channel_layout_default(&avctx->ch_layout, channel_count);
> +
> + av_log(avctx, AV_LOG_INFO,
> + "Output parameters channel-count=%d channel-layout=%x sample-rate=%d\n",
> + channel_count, channel_mask, sample_rate);
> +
> +fail:
> + av_freep(&format);
> + return ret;
> +}
> +
> +static int mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecContext *s)
> +{
> + if (avctx->codec_type == AVMEDIA_TYPE_AUDIO)
> + return mediacodec_dec_parse_audio_format(avctx, s);
> + else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO)
> + return mediacodec_dec_parse_video_format(avctx, s);
> + else
> + av_assert0(0);
> +}
> +
> static int mediacodec_dec_flush_codec(AVCodecContext *avctx, MediaCodecDecContext *s)
> {
> FFAMediaCodec *codec = s->codec;
> @@ -486,11 +729,9 @@ static int mediacodec_dec_flush_codec(AVCodecContext *avctx, MediaCodecDecContex
> return 0;
> }
>
> -int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
> - const char *mime, FFAMediaFormat *format)
> +static int mediacodec_dec_get_video_codec(AVCodecContext *avctx, MediaCodecDecContext *s,
> + const char *mime, FFAMediaFormat *format)
> {
> - int ret = 0;
> - int status;
> int profile;
>
> enum AVPixelFormat pix_fmt;
> @@ -499,12 +740,6 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
> AV_PIX_FMT_NONE,
> };
>
> - s->avctx = avctx;
> - atomic_init(&s->refcount, 1);
> - atomic_init(&s->hw_buffer_count, 0);
> - atomic_init(&s->serial, 1);
> - s->current_input_buffer = -1;
> -
> pix_fmt = ff_get_format(avctx, pix_fmts);
> if (pix_fmt == AV_PIX_FMT_MEDIACODEC) {
> AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
> @@ -536,8 +771,7 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
> // getCodecNameByType() can fail due to missing JVM, while NDK
> // mediacodec can be used without JVM.
> if (!s->use_ndk_codec) {
> - ret = AVERROR_EXTERNAL;
> - goto fail;
> + return AVERROR_EXTERNAL;
> }
> av_log(avctx, AV_LOG_INFO, "Failed to getCodecNameByType\n");
> } else {
> @@ -556,10 +790,51 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
> }
> if (!s->codec) {
> av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for type %s and name %s\n", mime, s->codec_name);
> - ret = AVERROR_EXTERNAL;
> - goto fail;
> + return AVERROR_EXTERNAL;
> + }
> +
> + return 0;
> +}
> +
> +static int mediacodec_dec_get_audio_codec(AVCodecContext *avctx, MediaCodecDecContext *s,
> + const char *mime, FFAMediaFormat *format)
> +{
> + s->codec = ff_AMediaCodec_createDecoderByType(mime, s->use_ndk_codec);
> + if (!s->codec) {
> + av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for mime %s\n", mime);
> + return AVERROR_EXTERNAL;
> + }
> +
> + s->codec_name = ff_AMediaCodec_getName(s->codec);
> + if (!s->codec_name) {
> + av_log(avctx, AV_LOG_ERROR, "Failed to get codec name");
> + return AVERROR_EXTERNAL;
> }
getName failure isn't a fatal error and can happen without JVM.
>
> + return 0;
> +}
> +
> +int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
> + const char *mime, FFAMediaFormat *format)
> +{
> + int ret;
> + int status;
> +
> + s->avctx = avctx;
> + atomic_init(&s->refcount, 1);
> + atomic_init(&s->hw_buffer_count, 0);
> + atomic_init(&s->serial, 1);
> + s->current_input_buffer = -1;
> +
> + if (avctx->codec_type == AVMEDIA_TYPE_AUDIO)
> + ret = mediacodec_dec_get_audio_codec(avctx, s, mime, format);
> + else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO)
> + ret = mediacodec_dec_get_video_codec(avctx, s, mime, format);
> + else
> + av_assert0(0);
> + if (ret < 0)
> + goto fail;
> +
> status = ff_AMediaCodec_configure(s->codec, format, s->surface, NULL, 0);
> if (status < 0) {
> char *desc = ff_AMediaFormat_toString(format);
> @@ -583,12 +858,14 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
> goto fail;
> }
>
> - s->format = ff_AMediaCodec_getOutputFormat(s->codec);
> - if (s->format) {
> - if ((ret = mediacodec_dec_parse_format(avctx, s)) < 0) {
> - av_log(avctx, AV_LOG_ERROR,
> - "Failed to configure context\n");
> - goto fail;
> + if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
> + s->format = ff_AMediaCodec_getOutputFormat(s->codec);
> + if (s->format) {
> + if ((ret = mediacodec_dec_parse_format(avctx, s)) < 0) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to configure context\n");
> + goto fail;
> + }
> }
> }
>
> --
> 2.45.2
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2024-07-28 15:24 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-06-12 13:42 [FFmpeg-devel] Add Mediacodec audio decoders support Matthieu Bouron
2024-06-12 13:42 ` [FFmpeg-devel] [PATCH 1/2] avcodec/mediacodecdec_common: ensure input buffer timestamp is positive Matthieu Bouron
2024-06-12 13:42 ` [FFmpeg-devel] [PATCH 2/2] avcodec: add Mediacodec audio decoders support Matthieu Bouron
2024-07-10 8:47 ` Matthieu Bouron
2024-07-28 15:23 ` Zhao Zhili
2024-06-12 14:00 ` [FFmpeg-devel] Add " Paul B Mahol
2024-07-10 13:54 ` Zhao Zhili
2024-07-10 16:31 ` Matthieu Bouron
2024-07-16 13:20 ` Matthieu Bouron
2024-07-16 14:58 ` Zhao Zhili
[not found] ` <2009DBEE-686A-42F6-A5DF-35147272EEF0@cosmin.at>
2024-07-16 15:48 ` Cosmin Stejerean via ffmpeg-devel
2024-07-16 16:04 ` Zhao Zhili
[not found] ` <057FD6DF-E71C-49D0-8D11-7A7E4D5FB5CC@cosmin.at>
2024-07-16 16:42 ` Cosmin Stejerean via ffmpeg-devel
2024-07-16 18:24 ` Rémi Denis-Courmont
[not found] ` <205EB89E-F6F9-438D-A9BC-E76E5439C7C7@cosmin.at>
2024-07-16 20:14 ` Cosmin Stejerean via ffmpeg-devel
2024-07-17 9:38 ` Anton Khirnov
2024-07-17 12:19 ` Michael Niedermayer
2024-07-19 19:59 ` Martin Storsjö
2024-07-17 15:39 ` Paul B Mahol
[not found] ` <4F1B9768-96C8-4708-92ED-AC8E1C9FC46F@cosmin.at>
2024-07-17 17:35 ` Cosmin Stejerean via ffmpeg-devel
2024-07-23 11:35 ` Martin Storsjö
2024-07-23 11:44 ` Zhao Zhili
2024-07-23 13:43 ` Anton Khirnov
Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
This inbox may be cloned and mirrored by anyone:
git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \
ffmpegdev@gitmailbox.com
public-inbox-index ffmpegdev
Example config snippet for mirrors.
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git