* [FFmpeg-devel] [PATCH 01/12] avcodec/mediacodec: fix incorrect crop info
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
@ 2022-10-24 3:16 ` Zhao Zhili
[not found] ` <CAK=uwuy5YEqEV_k9yVBVHw44iUK_NdprL7vZq8TV63Aczvt-pQ@mail.gmail.com>
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 02/12] avcodec/mediacodecdec: don't break out if both input and output port return try again Zhao Zhili
` (10 subsequent siblings)
11 siblings, 1 reply; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
The crop info is optional, but used unconditionally.
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
libavcodec/mediacodecdec_common.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c
index 2a605e7f5b..c1cbb28488 100644
--- a/libavcodec/mediacodecdec_common.c
+++ b/libavcodec/mediacodecdec_common.c
@@ -487,6 +487,11 @@ static int mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecConte
AMEDIAFORMAT_GET_INT32(s->crop_left, "crop-left", 0);
AMEDIAFORMAT_GET_INT32(s->crop_right, "crop-right", 0);
+ if (s->crop_bottom == 0 || s->crop_right == 0) {
+ s->crop_top = s->crop_left = 0;
+ s->crop_right = s->width - 1;
+ s->crop_bottom = s->height - 1;
+ }
width = s->crop_right + 1 - s->crop_left;
height = s->crop_bottom + 1 - s->crop_top;
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 02/12] avcodec/mediacodecdec: don't break out if both input and output port return try again
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 01/12] avcodec/mediacodec: fix incorrect crop info Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 03/12] avcodec/mediacodecdec_common: fix misuse av_free/av_freep Zhao Zhili
` (9 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
At the beginning of decoding, if we feed mediacodec too fast, the
input port will return try again. It takes some time for mediacodec
to consume bitstream and output frame. So the output port also return
try again. It possible that mediacodec_receive_frame doesn't consume
any AVPacket and no AVFrame is output. Then both avcodec_send_packet()
and avcodec_receive_frame() return EAGAIN, which shouldn't happen.
This bug can be produced with decoding benchmark on Pixel 3.
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
libavcodec/mediacodecdec.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c
index 322b448d27..2c66f38541 100644
--- a/libavcodec/mediacodecdec.c
+++ b/libavcodec/mediacodecdec.c
@@ -444,7 +444,16 @@ static int mediacodec_receive_frame(AVCodecContext *avctx, AVFrame *frame)
index = ff_AMediaCodec_dequeueInputBuffer(s->ctx->codec, 0);
if (index < 0) {
/* no space, block for an output frame to appear */
- return ff_mediacodec_dec_receive(avctx, s->ctx, frame, true);
+ ret = ff_mediacodec_dec_receive(avctx, s->ctx, frame, true);
+ /* Try again if both input port and output port return EAGAIN.
+ * If no data is consumed and no frame in output, it can make
+ * both avcodec_send_packet() and avcodec_receive_frame()
+ * return EAGAIN, which violate the design.
+ */
+ if (ff_AMediaCodec_infoTryAgainLater(s->ctx->codec, index) &&
+ ret == AVERROR(EAGAIN))
+ continue;
+ return ret;
}
s->ctx->current_input_buffer = index;
}
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 03/12] avcodec/mediacodecdec_common: fix misuse av_free/av_freep
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 01/12] avcodec/mediacodec: fix incorrect crop info Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 02/12] avcodec/mediacodecdec: don't break out if both input and output port return try again Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 04/12] avcodec/mediacodecdec_common: fix useless av_buffer_unref Zhao Zhili
` (8 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
libavcodec/mediacodecdec_common.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c
index c1cbb28488..940b4e02d5 100644
--- a/libavcodec/mediacodecdec_common.c
+++ b/libavcodec/mediacodecdec_common.c
@@ -334,7 +334,7 @@ static int mediacodec_wrap_hw_buffer(AVCodecContext *avctx,
return 0;
fail:
- av_freep(buffer);
+ av_freep(&buffer);
av_buffer_unref(&frame->buf[0]);
status = ff_AMediaCodec_releaseOutputBuffer(s->codec, index, 0);
if (status < 0) {
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 04/12] avcodec/mediacodecdec_common: fix useless av_buffer_unref
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
` (2 preceding siblings ...)
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 03/12] avcodec/mediacodecdec_common: fix misuse av_free/av_freep Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 05/12] avcodec/mediacodec_wrapper: separate implementation from interface Zhao Zhili
` (7 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
Since frame->buf[0] is always NULL in this case, av_buffer_unref
has no effect. If it's not NULL, double-free will happen.
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
libavcodec/mediacodecdec_common.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c
index 940b4e02d5..d6ce709dd8 100644
--- a/libavcodec/mediacodecdec_common.c
+++ b/libavcodec/mediacodecdec_common.c
@@ -335,7 +335,6 @@ static int mediacodec_wrap_hw_buffer(AVCodecContext *avctx,
return 0;
fail:
av_freep(&buffer);
- av_buffer_unref(&frame->buf[0]);
status = ff_AMediaCodec_releaseOutputBuffer(s->codec, index, 0);
if (status < 0) {
av_log(avctx, AV_LOG_ERROR, "Failed to release output buffer\n");
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 05/12] avcodec/mediacodec_wrapper: separate implementation from interface
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
` (3 preceding siblings ...)
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 04/12] avcodec/mediacodecdec_common: fix useless av_buffer_unref Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 06/12] avcodec/mediacodec: add NDK media codec wrapper Zhao Zhili
` (6 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
This is in preparation for NDK media codec wrapper.
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
libavcodec/mediacodec_wrapper.c | 244 +++++++++++++++++++++---------
libavcodec/mediacodec_wrapper.h | 255 +++++++++++++++++++++++++++-----
2 files changed, 394 insertions(+), 105 deletions(-)
diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c
index 8ffc58e1d8..e0c614680e 100644
--- a/libavcodec/mediacodec_wrapper.c
+++ b/libavcodec/mediacodec_wrapper.c
@@ -160,12 +160,14 @@ static const AVClass amediaformat_class = {
.version = LIBAVUTIL_VERSION_INT,
};
-struct FFAMediaFormat {
+typedef struct FFAMediaFormatJni {
+ FFAMediaFormat api;
- const AVClass *class;
struct JNIAMediaFormatFields jfields;
jobject object;
-};
+} FFAMediaFormatJni;
+
+static const FFAMediaFormat media_format_jni;
struct JNIAMediaCodecFields {
@@ -272,9 +274,8 @@ static const AVClass amediacodec_class = {
.version = LIBAVUTIL_VERSION_INT,
};
-struct FFAMediaCodec {
-
- const AVClass *class;
+typedef struct FFAMediaCodecJni {
+ FFAMediaCodec api;
struct JNIAMediaCodecFields jfields;
@@ -295,7 +296,9 @@ struct FFAMediaCodec {
int CONFIGURE_FLAG_ENCODE;
int has_get_i_o_buffer;
-};
+} FFAMediaCodecJni;
+
+static const FFAMediaCodec media_codec_jni;
#define JNI_GET_ENV_OR_RETURN(env, log_ctx, ret) do { \
(env) = ff_jni_get_env(log_ctx); \
@@ -622,17 +625,17 @@ done:
return name;
}
-FFAMediaFormat *ff_AMediaFormat_new(void)
+static FFAMediaFormat *mediaformat_jni_new(void)
{
JNIEnv *env = NULL;
- FFAMediaFormat *format = NULL;
+ FFAMediaFormatJni *format = NULL;
jobject object = NULL;
- format = av_mallocz(sizeof(FFAMediaFormat));
+ format = av_mallocz(sizeof(*format));
if (!format) {
return NULL;
}
- format->class = &amediaformat_class;
+ format->api = media_format_jni;
env = ff_jni_get_env(format);
if (!env) {
@@ -664,19 +667,19 @@ fail:
av_freep(&format);
}
- return format;
+ return (FFAMediaFormat *)format;
}
-static FFAMediaFormat *ff_AMediaFormat_newFromObject(void *object)
+static FFAMediaFormat *mediaformat_jni_newFromObject(void *object)
{
JNIEnv *env = NULL;
- FFAMediaFormat *format = NULL;
+ FFAMediaFormatJni *format = NULL;
- format = av_mallocz(sizeof(FFAMediaFormat));
+ format = av_mallocz(sizeof(*format));
if (!format) {
return NULL;
}
- format->class = &amediaformat_class;
+ format->api = media_format_jni;
env = ff_jni_get_env(format);
if (!env) {
@@ -693,7 +696,7 @@ static FFAMediaFormat *ff_AMediaFormat_newFromObject(void *object)
goto fail;
}
- return format;
+ return (FFAMediaFormat *)format;
fail:
ff_jni_reset_jfields(env, &format->jfields, jni_amediaformat_mapping, 1, format);
@@ -702,10 +705,10 @@ fail:
return NULL;
}
-int ff_AMediaFormat_delete(FFAMediaFormat* format)
+static int mediaformat_jni_delete(FFAMediaFormat* ctx)
{
int ret = 0;
-
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
JNIEnv *env = NULL;
if (!format) {
@@ -724,10 +727,10 @@ int ff_AMediaFormat_delete(FFAMediaFormat* format)
return ret;
}
-char* ff_AMediaFormat_toString(FFAMediaFormat* format)
+static char* mediaformat_jni_toString(FFAMediaFormat* ctx)
{
char *ret = NULL;
-
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
JNIEnv *env = NULL;
jstring description = NULL;
@@ -749,10 +752,10 @@ fail:
return ret;
}
-int ff_AMediaFormat_getInt32(FFAMediaFormat* format, const char *name, int32_t *out)
+static int mediaformat_jni_getInt32(FFAMediaFormat* ctx, const char *name, int32_t *out)
{
int ret = 1;
-
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
JNIEnv *env = NULL;
jstring key = NULL;
jboolean contains_key;
@@ -788,10 +791,10 @@ fail:
return ret;
}
-int ff_AMediaFormat_getInt64(FFAMediaFormat* format, const char *name, int64_t *out)
+static int mediaformat_jni_getInt64(FFAMediaFormat* ctx, const char *name, int64_t *out)
{
int ret = 1;
-
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
JNIEnv *env = NULL;
jstring key = NULL;
jboolean contains_key;
@@ -827,10 +830,10 @@ fail:
return ret;
}
-int ff_AMediaFormat_getFloat(FFAMediaFormat* format, const char *name, float *out)
+static int mediaformat_jni_getFloat(FFAMediaFormat* ctx, const char *name, float *out)
{
int ret = 1;
-
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
JNIEnv *env = NULL;
jstring key = NULL;
jboolean contains_key;
@@ -866,10 +869,10 @@ fail:
return ret;
}
-int ff_AMediaFormat_getBuffer(FFAMediaFormat* format, const char *name, void** data, size_t *size)
+static int mediaformat_jni_getBuffer(FFAMediaFormat* ctx, const char *name, void** data, size_t *size)
{
int ret = 1;
-
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
JNIEnv *env = NULL;
jstring key = NULL;
jboolean contains_key;
@@ -924,10 +927,10 @@ fail:
return ret;
}
-int ff_AMediaFormat_getString(FFAMediaFormat* format, const char *name, const char **out)
+static int mediaformat_jni_getString(FFAMediaFormat* ctx, const char *name, const char **out)
{
int ret = 1;
-
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
JNIEnv *env = NULL;
jstring key = NULL;
jboolean contains_key;
@@ -974,10 +977,11 @@ fail:
return ret;
}
-void ff_AMediaFormat_setInt32(FFAMediaFormat* format, const char* name, int32_t value)
+static void mediaformat_jni_setInt32(FFAMediaFormat* ctx, const char* name, int32_t value)
{
JNIEnv *env = NULL;
jstring key = NULL;
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
av_assert0(format != NULL);
@@ -999,10 +1003,11 @@ fail:
}
}
-void ff_AMediaFormat_setInt64(FFAMediaFormat* format, const char* name, int64_t value)
+static void mediaformat_jni_setInt64(FFAMediaFormat* ctx, const char* name, int64_t value)
{
JNIEnv *env = NULL;
jstring key = NULL;
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
av_assert0(format != NULL);
@@ -1024,10 +1029,11 @@ fail:
}
}
-void ff_AMediaFormat_setFloat(FFAMediaFormat* format, const char* name, float value)
+static void mediaformat_jni_setFloat(FFAMediaFormat* ctx, const char* name, float value)
{
JNIEnv *env = NULL;
jstring key = NULL;
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
av_assert0(format != NULL);
@@ -1049,11 +1055,12 @@ fail:
}
}
-void ff_AMediaFormat_setString(FFAMediaFormat* format, const char* name, const char* value)
+static void mediaformat_jni_setString(FFAMediaFormat* ctx, const char* name, const char* value)
{
JNIEnv *env = NULL;
jstring key = NULL;
jstring string = NULL;
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
av_assert0(format != NULL);
@@ -1084,12 +1091,13 @@ fail:
}
}
-void ff_AMediaFormat_setBuffer(FFAMediaFormat* format, const char* name, void* data, size_t size)
+static void mediaformat_jni_setBuffer(FFAMediaFormat* ctx, const char* name, void* data, size_t size)
{
JNIEnv *env = NULL;
jstring key = NULL;
jobject buffer = NULL;
void *buffer_data = NULL;
+ FFAMediaFormatJni *format = (FFAMediaFormatJni *)ctx;
av_assert0(format != NULL);
@@ -1131,7 +1139,7 @@ fail:
}
}
-static int codec_init_static_fields(FFAMediaCodec *codec)
+static int codec_init_static_fields(FFAMediaCodecJni *codec)
{
int ret = 0;
JNIEnv *env = NULL;
@@ -1193,17 +1201,17 @@ static inline FFAMediaCodec *codec_create(int method, const char *arg)
{
int ret = -1;
JNIEnv *env = NULL;
- FFAMediaCodec *codec = NULL;
+ FFAMediaCodecJni *codec = NULL;
jstring jarg = NULL;
jobject object = NULL;
jobject buffer_info = NULL;
jmethodID create_id = NULL;
- codec = av_mallocz(sizeof(FFAMediaCodec));
+ codec = av_mallocz(sizeof(*codec));
if (!codec) {
return NULL;
}
- codec->class = &amediacodec_class;
+ codec->api = media_codec_jni;
env = ff_jni_get_env(codec);
if (!env) {
@@ -1286,11 +1294,11 @@ fail:
av_freep(&codec);
}
- return codec;
+ return (FFAMediaCodec *)codec;
}
#define DECLARE_FF_AMEDIACODEC_CREATE_FUNC(name, method) \
-FFAMediaCodec *ff_AMediaCodec_##name(const char *arg) \
+static FFAMediaCodec *mediacodec_jni_##name(const char *arg) \
{ \
return codec_create(method, arg); \
} \
@@ -1299,10 +1307,10 @@ DECLARE_FF_AMEDIACODEC_CREATE_FUNC(createCodecByName, CREATE_CODEC_BY_NAME)
DECLARE_FF_AMEDIACODEC_CREATE_FUNC(createDecoderByType, CREATE_DECODER_BY_TYPE)
DECLARE_FF_AMEDIACODEC_CREATE_FUNC(createEncoderByType, CREATE_ENCODER_BY_TYPE)
-int ff_AMediaCodec_delete(FFAMediaCodec* codec)
+static int mediacodec_jni_delete(FFAMediaCodec* ctx)
{
int ret = 0;
-
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNIEnv *env = NULL;
if (!codec) {
@@ -1335,11 +1343,12 @@ int ff_AMediaCodec_delete(FFAMediaCodec* codec)
return ret;
}
-char *ff_AMediaCodec_getName(FFAMediaCodec *codec)
+static char *mediacodec_jni_getName(FFAMediaCodec *ctx)
{
char *ret = NULL;
JNIEnv *env = NULL;
jobject *name = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNI_GET_ENV_OR_RETURN(env, codec, NULL);
@@ -1358,10 +1367,12 @@ fail:
return ret;
}
-int ff_AMediaCodec_configure(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags)
+static int mediacodec_jni_configure(FFAMediaCodec* ctx, const FFAMediaFormat* format_ctx, void* surface, void *crypto, uint32_t flags)
{
int ret = 0;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
+ const FFAMediaFormatJni *format = (FFAMediaFormatJni *)format_ctx;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -1375,10 +1386,11 @@ fail:
return ret;
}
-int ff_AMediaCodec_start(FFAMediaCodec* codec)
+static int mediacodec_jni_start(FFAMediaCodec* ctx)
{
int ret = 0;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -1392,10 +1404,11 @@ fail:
return ret;
}
-int ff_AMediaCodec_stop(FFAMediaCodec* codec)
+static int mediacodec_jni_stop(FFAMediaCodec* ctx)
{
int ret = 0;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -1409,10 +1422,11 @@ fail:
return ret;
}
-int ff_AMediaCodec_flush(FFAMediaCodec* codec)
+static int mediacodec_jni_flush(FFAMediaCodec* ctx)
{
int ret = 0;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -1426,10 +1440,11 @@ fail:
return ret;
}
-int ff_AMediaCodec_releaseOutputBuffer(FFAMediaCodec* codec, size_t idx, int render)
+static int mediacodec_jni_releaseOutputBuffer(FFAMediaCodec* ctx, size_t idx, int render)
{
int ret = 0;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -1443,10 +1458,11 @@ fail:
return ret;
}
-int ff_AMediaCodec_releaseOutputBufferAtTime(FFAMediaCodec *codec, size_t idx, int64_t timestampNs)
+static int mediacodec_jni_releaseOutputBufferAtTime(FFAMediaCodec *ctx, size_t idx, int64_t timestampNs)
{
int ret = 0;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -1460,10 +1476,11 @@ fail:
return ret;
}
-ssize_t ff_AMediaCodec_dequeueInputBuffer(FFAMediaCodec* codec, int64_t timeoutUs)
+static ssize_t mediacodec_jni_dequeueInputBuffer(FFAMediaCodec* ctx, int64_t timeoutUs)
{
int ret = 0;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -1477,10 +1494,11 @@ fail:
return ret;
}
-int ff_AMediaCodec_queueInputBuffer(FFAMediaCodec* codec, size_t idx, off_t offset, size_t size, uint64_t time, uint32_t flags)
+static int mediacodec_jni_queueInputBuffer(FFAMediaCodec* ctx, size_t idx, off_t offset, size_t size, uint64_t time, uint32_t flags)
{
int ret = 0;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -1494,10 +1512,11 @@ fail:
return ret;
}
-ssize_t ff_AMediaCodec_dequeueOutputBuffer(FFAMediaCodec* codec, FFAMediaCodecBufferInfo *info, int64_t timeoutUs)
+static ssize_t mediacodec_jni_dequeueOutputBuffer(FFAMediaCodec* ctx, FFAMediaCodecBufferInfo *info, int64_t timeoutUs)
{
int ret = 0;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -1529,11 +1548,11 @@ ssize_t ff_AMediaCodec_dequeueOutputBuffer(FFAMediaCodec* codec, FFAMediaCodecBu
return ret;
}
-uint8_t* ff_AMediaCodec_getInputBuffer(FFAMediaCodec* codec, size_t idx, size_t *out_size)
+static uint8_t* mediacodec_jni_getInputBuffer(FFAMediaCodec* ctx, size_t idx, size_t *out_size)
{
uint8_t *ret = NULL;
JNIEnv *env = NULL;
-
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
jobject buffer = NULL;
jobject input_buffers = NULL;
@@ -1577,11 +1596,11 @@ fail:
return ret;
}
-uint8_t* ff_AMediaCodec_getOutputBuffer(FFAMediaCodec* codec, size_t idx, size_t *out_size)
+static uint8_t* mediacodec_jni_getOutputBuffer(FFAMediaCodec* ctx, size_t idx, size_t *out_size)
{
uint8_t *ret = NULL;
JNIEnv *env = NULL;
-
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
jobject buffer = NULL;
jobject output_buffers = NULL;
@@ -1625,10 +1644,11 @@ fail:
return ret;
}
-FFAMediaFormat* ff_AMediaCodec_getOutputFormat(FFAMediaCodec* codec)
+static FFAMediaFormat* mediacodec_jni_getOutputFormat(FFAMediaCodec* ctx)
{
FFAMediaFormat *ret = NULL;
JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
jobject mediaformat = NULL;
@@ -1639,7 +1659,7 @@ FFAMediaFormat* ff_AMediaCodec_getOutputFormat(FFAMediaCodec* codec)
goto fail;
}
- ret = ff_AMediaFormat_newFromObject(mediaformat);
+ ret = mediaformat_jni_newFromObject(mediaformat);
fail:
if (mediaformat) {
(*env)->DeleteLocalRef(env, mediaformat);
@@ -1648,44 +1668,52 @@ fail:
return ret;
}
-int ff_AMediaCodec_infoTryAgainLater(FFAMediaCodec *codec, ssize_t idx)
+static int mediacodec_jni_infoTryAgainLater(FFAMediaCodec *ctx, ssize_t idx)
{
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
return idx == codec->INFO_TRY_AGAIN_LATER;
}
-int ff_AMediaCodec_infoOutputBuffersChanged(FFAMediaCodec *codec, ssize_t idx)
+static int mediacodec_jni_infoOutputBuffersChanged(FFAMediaCodec *ctx, ssize_t idx)
{
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
return idx == codec->INFO_OUTPUT_BUFFERS_CHANGED;
}
-int ff_AMediaCodec_infoOutputFormatChanged(FFAMediaCodec *codec, ssize_t idx)
+static int mediacodec_jni_infoOutputFormatChanged(FFAMediaCodec *ctx, ssize_t idx)
{
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
return idx == codec->INFO_OUTPUT_FORMAT_CHANGED;
}
-int ff_AMediaCodec_getBufferFlagCodecConfig(FFAMediaCodec *codec)
+static int mediacodec_jni_getBufferFlagCodecConfig(FFAMediaCodec *ctx)
{
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
return codec->BUFFER_FLAG_CODEC_CONFIG;
}
-int ff_AMediaCodec_getBufferFlagEndOfStream(FFAMediaCodec *codec)
+static int mediacodec_jni_getBufferFlagEndOfStream(FFAMediaCodec *ctx)
{
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
return codec->BUFFER_FLAG_END_OF_STREAM;
}
-int ff_AMediaCodec_getBufferFlagKeyFrame(FFAMediaCodec *codec)
+static int mediacodec_jni_getBufferFlagKeyFrame(FFAMediaCodec *ctx)
{
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
return codec->BUFFER_FLAG_KEY_FRAME;
}
-int ff_AMediaCodec_getConfigureFlagEncode(FFAMediaCodec *codec)
+static int mediacodec_jni_getConfigureFlagEncode(FFAMediaCodec *ctx)
{
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
return codec->CONFIGURE_FLAG_ENCODE;
}
-int ff_AMediaCodec_cleanOutputBuffers(FFAMediaCodec *codec)
+static int mediacodec_jni_cleanOutputBuffers(FFAMediaCodec *ctx)
{
int ret = 0;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
if (!codec->has_get_i_o_buffer) {
if (codec->output_buffers) {
@@ -1706,6 +1734,86 @@ fail:
return ret;
}
+static const FFAMediaFormat media_format_jni = {
+ .class = &amediaformat_class,
+
+ .create = mediaformat_jni_new,
+ .delete = mediaformat_jni_delete,
+
+ .toString = mediaformat_jni_toString,
+
+ .getInt32 = mediaformat_jni_getInt32,
+ .getInt64 = mediaformat_jni_getInt64,
+ .getFloat = mediaformat_jni_getFloat,
+ .getBuffer = mediaformat_jni_getBuffer,
+ .getString = mediaformat_jni_getString,
+
+ .setInt32 = mediaformat_jni_setInt32,
+ .setInt64 = mediaformat_jni_setInt64,
+ .setFloat = mediaformat_jni_setFloat,
+ .setString = mediaformat_jni_setString,
+ .setBuffer = mediaformat_jni_setBuffer,
+};
+
+static const FFAMediaCodec media_codec_jni = {
+ .class = &amediacodec_class,
+
+ .getName = mediacodec_jni_getName,
+
+ .createCodecByName = mediacodec_jni_createCodecByName,
+ .createDecoderByType = mediacodec_jni_createDecoderByType,
+ .createEncoderByType = mediacodec_jni_createEncoderByType,
+ .delete = mediacodec_jni_delete,
+
+ .configure = mediacodec_jni_configure,
+ .start = mediacodec_jni_start,
+ .stop = mediacodec_jni_stop,
+ .flush = mediacodec_jni_flush,
+
+ .getInputBuffer = mediacodec_jni_getInputBuffer,
+ .getOutputBuffer = mediacodec_jni_getOutputBuffer,
+
+ .dequeueInputBuffer = mediacodec_jni_dequeueInputBuffer,
+ .queueInputBuffer = mediacodec_jni_queueInputBuffer,
+
+ .dequeueOutputBuffer = mediacodec_jni_dequeueOutputBuffer,
+ .getOutputFormat = mediacodec_jni_getOutputFormat,
+
+ .releaseOutputBuffer = mediacodec_jni_releaseOutputBuffer,
+ .releaseOutputBufferAtTime = mediacodec_jni_releaseOutputBufferAtTime,
+
+ .infoTryAgainLater = mediacodec_jni_infoTryAgainLater,
+ .infoOutputBuffersChanged = mediacodec_jni_infoOutputBuffersChanged,
+ .infoOutputFormatChanged = mediacodec_jni_infoOutputFormatChanged,
+
+ .getBufferFlagCodecConfig = mediacodec_jni_getBufferFlagCodecConfig,
+ .getBufferFlagEndOfStream = mediacodec_jni_getBufferFlagEndOfStream,
+ .getBufferFlagKeyFrame = mediacodec_jni_getBufferFlagKeyFrame,
+
+ .getConfigureFlagEncode = mediacodec_jni_getConfigureFlagEncode,
+ .cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers,
+};
+
+FFAMediaFormat *ff_AMediaFormat_new(void)
+{
+ return media_format_jni.create();
+}
+
+FFAMediaCodec* ff_AMediaCodec_createCodecByName(const char *name)
+{
+ return media_codec_jni.createCodecByName(name);
+}
+
+FFAMediaCodec* ff_AMediaCodec_createDecoderByType(const char *mime_type)
+{
+ return media_codec_jni.createDecoderByType(mime_type);
+}
+
+FFAMediaCodec* ff_AMediaCodec_createEncoderByType(const char *mime_type)
+{
+ return media_codec_jni.createEncoderByType(mime_type);
+}
+
int ff_Build_SDK_INT(AVCodecContext *avctx)
{
int ret = -1;
diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h
index b106ff315a..606fdbede5 100644
--- a/libavcodec/mediacodec_wrapper.h
+++ b/libavcodec/mediacodec_wrapper.h
@@ -58,28 +58,90 @@ int ff_AMediaCodecProfile_getProfileFromAVCodecContext(AVCodecContext *avctx);
char *ff_AMediaCodecList_getCodecNameByType(const char *mime, int profile, int encoder, void *log_ctx);
-struct FFAMediaFormat;
typedef struct FFAMediaFormat FFAMediaFormat;
+struct FFAMediaFormat {
+ const AVClass *class;
+
+ FFAMediaFormat *(*create)(void);
+ int (*delete)(FFAMediaFormat *);
+
+ char* (*toString)(FFAMediaFormat* format);
+
+ int (*getInt32)(FFAMediaFormat* format, const char *name, int32_t *out);
+ int (*getInt64)(FFAMediaFormat* format, const char *name, int64_t *out);
+ int (*getFloat)(FFAMediaFormat* format, const char *name, float *out);
+ int (*getBuffer)(FFAMediaFormat* format, const char *name, void** data, size_t *size);
+ int (*getString)(FFAMediaFormat* format, const char *name, const char **out);
+
+ void (*setInt32)(FFAMediaFormat* format, const char* name, int32_t value);
+ void (*setInt64)(FFAMediaFormat* format, const char* name, int64_t value);
+ void (*setFloat)(FFAMediaFormat* format, const char* name, float value);
+ void (*setString)(FFAMediaFormat* format, const char* name, const char* value);
+ void (*setBuffer)(FFAMediaFormat* format, const char* name, void* data, size_t size);
+};
FFAMediaFormat *ff_AMediaFormat_new(void);
-int ff_AMediaFormat_delete(FFAMediaFormat* format);
-char* ff_AMediaFormat_toString(FFAMediaFormat* format);
+static inline int ff_AMediaFormat_delete(FFAMediaFormat* format)
+{
+ return format->delete(format);
+}
-int ff_AMediaFormat_getInt32(FFAMediaFormat* format, const char *name, int32_t *out);
-int ff_AMediaFormat_getInt64(FFAMediaFormat* format, const char *name, int64_t *out);
-int ff_AMediaFormat_getFloat(FFAMediaFormat* format, const char *name, float *out);
-int ff_AMediaFormat_getBuffer(FFAMediaFormat* format, const char *name, void** data, size_t *size);
-int ff_AMediaFormat_getString(FFAMediaFormat* format, const char *name, const char **out);
+static inline char* ff_AMediaFormat_toString(FFAMediaFormat* format)
+{
+ return format->toString(format);
+}
-void ff_AMediaFormat_setInt32(FFAMediaFormat* format, const char* name, int32_t value);
-void ff_AMediaFormat_setInt64(FFAMediaFormat* format, const char* name, int64_t value);
-void ff_AMediaFormat_setFloat(FFAMediaFormat* format, const char* name, float value);
-void ff_AMediaFormat_setString(FFAMediaFormat* format, const char* name, const char* value);
-void ff_AMediaFormat_setBuffer(FFAMediaFormat* format, const char* name, void* data, size_t size);
+static inline int ff_AMediaFormat_getInt32(FFAMediaFormat* format, const char *name, int32_t *out)
+{
+ return format->getInt32(format, name, out);
+}
+
+static inline int ff_AMediaFormat_getInt64(FFAMediaFormat* format, const char *name, int64_t *out)
+{
+ return format->getInt64(format, name, out);
+}
+
+static inline int ff_AMediaFormat_getFloat(FFAMediaFormat* format, const char *name, float *out)
+{
+ return format->getFloat(format, name, out);
+}
+
+static inline int ff_AMediaFormat_getBuffer(FFAMediaFormat* format, const char *name, void** data, size_t *size)
+{
+ return format->getBuffer(format, name, data, size);
+}
+
+static inline int ff_AMediaFormat_getString(FFAMediaFormat* format, const char *name, const char **out)
+{
+ return format->getString(format, name, out);
+}
+
+static inline void ff_AMediaFormat_setInt32(FFAMediaFormat* format, const char* name, int32_t value)
+{
+ format->setInt32(format, name, value);
+}
+
+static inline void ff_AMediaFormat_setInt64(FFAMediaFormat* format, const char* name, int64_t value)
+{
+ format->setInt64(format, name, value);
+}
+
+static inline void ff_AMediaFormat_setFloat(FFAMediaFormat* format, const char* name, float value)
+{
+ format->setFloat(format, name, value);
+}
+
+static inline void ff_AMediaFormat_setString(FFAMediaFormat* format, const char* name, const char* value)
+{
+ format->setString(format, name, value);
+}
+
+static inline void ff_AMediaFormat_setBuffer(FFAMediaFormat* format, const char* name, void* data, size_t size)
+{
+ format->setBuffer(format, name, data, size);
+}
-struct FFAMediaCodec;
-typedef struct FFAMediaCodec FFAMediaCodec;
typedef struct FFAMediaCodecCryptoInfo FFAMediaCodecCryptoInfo;
struct FFAMediaCodecBufferInfo {
@@ -90,41 +152,160 @@ struct FFAMediaCodecBufferInfo {
};
typedef struct FFAMediaCodecBufferInfo FFAMediaCodecBufferInfo;
-char *ff_AMediaCodec_getName(FFAMediaCodec *codec);
+typedef struct FFAMediaCodec FFAMediaCodec;
+struct FFAMediaCodec {
+ const AVClass *class;
+
+ char *(*getName)(FFAMediaCodec *codec);
+
+ FFAMediaCodec* (*createCodecByName)(const char *name);
+ FFAMediaCodec* (*createDecoderByType)(const char *mime_type);
+ FFAMediaCodec* (*createEncoderByType)(const char *mime_type);
+ int (*delete)(FFAMediaCodec* codec);
+
+ int (*configure)(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags);
+ int (*start)(FFAMediaCodec* codec);
+ int (*stop)(FFAMediaCodec* codec);
+ int (*flush)(FFAMediaCodec* codec);
+
+ uint8_t* (*getInputBuffer)(FFAMediaCodec* codec, size_t idx, size_t *out_size);
+ uint8_t* (*getOutputBuffer)(FFAMediaCodec* codec, size_t idx, size_t *out_size);
+
+ ssize_t (*dequeueInputBuffer)(FFAMediaCodec* codec, int64_t timeoutUs);
+ int (*queueInputBuffer)(FFAMediaCodec* codec, size_t idx, off_t offset, size_t size, uint64_t time, uint32_t flags);
+
+ ssize_t (*dequeueOutputBuffer)(FFAMediaCodec* codec, FFAMediaCodecBufferInfo *info, int64_t timeoutUs);
+ FFAMediaFormat* (*getOutputFormat)(FFAMediaCodec* codec);
+
+ int (*releaseOutputBuffer)(FFAMediaCodec* codec, size_t idx, int render);
+ int (*releaseOutputBufferAtTime)(FFAMediaCodec *codec, size_t idx, int64_t timestampNs);
+
+ int (*infoTryAgainLater)(FFAMediaCodec *codec, ssize_t idx);
+ int (*infoOutputBuffersChanged)(FFAMediaCodec *codec, ssize_t idx);
+ int (*infoOutputFormatChanged)(FFAMediaCodec *codec, ssize_t indx);
+
+ int (*getBufferFlagCodecConfig)(FFAMediaCodec *codec);
+ int (*getBufferFlagEndOfStream)(FFAMediaCodec *codec);
+ int (*getBufferFlagKeyFrame)(FFAMediaCodec *codec);
+
+ int (*getConfigureFlagEncode)(FFAMediaCodec *codec);
+
+ int (*cleanOutputBuffers)(FFAMediaCodec *codec);
+};
+
+static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec)
+{
+ return codec->getName(codec);
+}
FFAMediaCodec* ff_AMediaCodec_createCodecByName(const char *name);
FFAMediaCodec* ff_AMediaCodec_createDecoderByType(const char *mime_type);
FFAMediaCodec* ff_AMediaCodec_createEncoderByType(const char *mime_type);
-int ff_AMediaCodec_configure(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags);
-int ff_AMediaCodec_start(FFAMediaCodec* codec);
-int ff_AMediaCodec_stop(FFAMediaCodec* codec);
-int ff_AMediaCodec_flush(FFAMediaCodec* codec);
-int ff_AMediaCodec_delete(FFAMediaCodec* codec);
+static inline int ff_AMediaCodec_configure(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags)
+{
+ return codec->configure(codec, format, surface, crypto, flags);
+}
+
+static inline int ff_AMediaCodec_start(FFAMediaCodec* codec)
+{
+ return codec->start(codec);
+}
+
+static inline int ff_AMediaCodec_stop(FFAMediaCodec* codec)
+{
+ return codec->stop(codec);
+}
+
+static inline int ff_AMediaCodec_flush(FFAMediaCodec* codec)
+{
+ return codec->flush(codec);
+}
+
+static inline int ff_AMediaCodec_delete(FFAMediaCodec* codec)
+{
+ return codec->delete(codec);
+}
+
+static inline uint8_t* ff_AMediaCodec_getInputBuffer(FFAMediaCodec* codec, size_t idx, size_t *out_size)
+{
+ return codec->getInputBuffer(codec, idx, out_size);
+}
+
+static inline uint8_t* ff_AMediaCodec_getOutputBuffer(FFAMediaCodec* codec, size_t idx, size_t *out_size)
+{
+ return codec->getOutputBuffer(codec, idx, out_size);
+}
+
+static inline ssize_t ff_AMediaCodec_dequeueInputBuffer(FFAMediaCodec* codec, int64_t timeoutUs)
+{
+ return codec->dequeueInputBuffer(codec, timeoutUs);
+}
+
+static inline int ff_AMediaCodec_queueInputBuffer(FFAMediaCodec *codec, size_t idx, off_t offset, size_t size, uint64_t time, uint32_t flags)
+{
+ return codec->queueInputBuffer(codec, idx, offset, size, time, flags);
+}
+
+static inline ssize_t ff_AMediaCodec_dequeueOutputBuffer(FFAMediaCodec* codec, FFAMediaCodecBufferInfo *info, int64_t timeoutUs)
+{
+ return codec->dequeueOutputBuffer(codec, info, timeoutUs);
+}
+
+static inline FFAMediaFormat* ff_AMediaCodec_getOutputFormat(FFAMediaCodec* codec)
+{
+ return codec->getOutputFormat(codec);
+}
+
+static inline int ff_AMediaCodec_releaseOutputBuffer(FFAMediaCodec* codec, size_t idx, int render)
+{
+ return codec->releaseOutputBuffer(codec, idx, render);
+}
+
+static inline int ff_AMediaCodec_releaseOutputBufferAtTime(FFAMediaCodec *codec, size_t idx, int64_t timestampNs)
+{
+ return codec->releaseOutputBufferAtTime(codec, idx, timestampNs);
+}
-uint8_t* ff_AMediaCodec_getInputBuffer(FFAMediaCodec* codec, size_t idx, size_t *out_size);
-uint8_t* ff_AMediaCodec_getOutputBuffer(FFAMediaCodec* codec, size_t idx, size_t *out_size);
+static inline int ff_AMediaCodec_infoTryAgainLater(FFAMediaCodec *codec, ssize_t idx)
+{
+ return codec->infoTryAgainLater(codec, idx);
+}
-ssize_t ff_AMediaCodec_dequeueInputBuffer(FFAMediaCodec* codec, int64_t timeoutUs);
-int ff_AMediaCodec_queueInputBuffer(FFAMediaCodec* codec, size_t idx, off_t offset, size_t size, uint64_t time, uint32_t flags);
+static inline int ff_AMediaCodec_infoOutputBuffersChanged(FFAMediaCodec *codec, ssize_t idx)
+{
+ return codec->infoOutputBuffersChanged(codec, idx);
+}
-ssize_t ff_AMediaCodec_dequeueOutputBuffer(FFAMediaCodec* codec, FFAMediaCodecBufferInfo *info, int64_t timeoutUs);
-FFAMediaFormat* ff_AMediaCodec_getOutputFormat(FFAMediaCodec* codec);
+static inline int ff_AMediaCodec_infoOutputFormatChanged(FFAMediaCodec *codec, ssize_t idx)
+{
+ return codec->infoOutputFormatChanged(codec, idx);
+}
-int ff_AMediaCodec_releaseOutputBuffer(FFAMediaCodec* codec, size_t idx, int render);
-int ff_AMediaCodec_releaseOutputBufferAtTime(FFAMediaCodec *codec, size_t idx, int64_t timestampNs);
+static inline int ff_AMediaCodec_getBufferFlagCodecConfig(FFAMediaCodec *codec)
+{
+ return codec->getBufferFlagCodecConfig(codec);
+}
-int ff_AMediaCodec_infoTryAgainLater(FFAMediaCodec *codec, ssize_t idx);
-int ff_AMediaCodec_infoOutputBuffersChanged(FFAMediaCodec *codec, ssize_t idx);
-int ff_AMediaCodec_infoOutputFormatChanged(FFAMediaCodec *codec, ssize_t indx);
+static inline int ff_AMediaCodec_getBufferFlagEndOfStream(FFAMediaCodec *codec)
+{
+ return codec->getBufferFlagEndOfStream(codec);
+}
-int ff_AMediaCodec_getBufferFlagCodecConfig (FFAMediaCodec *codec);
-int ff_AMediaCodec_getBufferFlagEndOfStream(FFAMediaCodec *codec);
-int ff_AMediaCodec_getBufferFlagKeyFrame(FFAMediaCodec *codec);
+static inline int ff_AMediaCodec_getBufferFlagKeyFrame(FFAMediaCodec *codec)
+{
+ return codec->getBufferFlagKeyFrame(codec);
+}
-int ff_AMediaCodec_getConfigureFlagEncode(FFAMediaCodec *codec);
+static inline int ff_AMediaCodec_getConfigureFlagEncode(FFAMediaCodec *codec)
+{
+ return codec->getConfigureFlagEncode(codec);
+}
-int ff_AMediaCodec_cleanOutputBuffers(FFAMediaCodec *codec);
+static inline int ff_AMediaCodec_cleanOutputBuffers(FFAMediaCodec *codec)
+{
+ return codec->cleanOutputBuffers(codec);
+}
int ff_Build_SDK_INT(AVCodecContext *avctx);
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 06/12] avcodec/mediacodec: add NDK media codec wrapper
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
` (4 preceding siblings ...)
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 05/12] avcodec/mediacodec_wrapper: separate implementation from interface Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 07/12] avcodec/mediacodecdec: enable NDK mediacodec Zhao Zhili
` (5 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
configure | 2 +
libavcodec/mediacodec_wrapper.c | 596 +++++++++++++++++++++++++++++-
libavcodec/mediacodec_wrapper.h | 8 +-
libavcodec/mediacodecdec.c | 2 +-
libavcodec/mediacodecdec_common.c | 2 +-
5 files changed, 600 insertions(+), 10 deletions(-)
diff --git a/configure b/configure
index eefd103414..ee2e3ba6ac 100755
--- a/configure
+++ b/configure
@@ -3191,6 +3191,7 @@ h264_crystalhd_decoder_select="crystalhd h264_mp4toannexb_bsf h264_parser"
h264_cuvid_decoder_deps="cuvid"
h264_cuvid_decoder_select="h264_mp4toannexb_bsf"
h264_mediacodec_decoder_deps="mediacodec"
+h264_mediacodec_decoder_extralibs="-landroid"
h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser"
h264_mf_encoder_deps="mediafoundation"
h264_mmal_decoder_deps="mmal"
@@ -3209,6 +3210,7 @@ hevc_amf_encoder_deps="amf"
hevc_cuvid_decoder_deps="cuvid"
hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf"
hevc_mediacodec_decoder_deps="mediacodec"
+hevc_mediacodec_decoder_extralibs="-landroid"
hevc_mediacodec_decoder_select="hevc_mp4toannexb_bsf hevc_parser"
hevc_mf_encoder_deps="mediafoundation"
hevc_nvenc_encoder_deps="nvenc"
diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c
index e0c614680e..b12aced711 100644
--- a/libavcodec/mediacodec_wrapper.c
+++ b/libavcodec/mediacodec_wrapper.c
@@ -20,7 +20,11 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
+#include <dlfcn.h>
#include <jni.h>
+#include <media/NdkMediaFormat.h>
+#include <media/NdkMediaCodec.h>
+#include <android/native_window_jni.h>
#include "libavutil/avassert.h"
#include "libavutil/mem.h"
@@ -1794,23 +1798,607 @@ static const FFAMediaCodec media_codec_jni = {
.cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers,
};
-FFAMediaFormat *ff_AMediaFormat_new(void)
+typedef struct FFAMediaFormatNdk {
+ FFAMediaFormat api;
+
+ void *libmedia;
+ AMediaFormat *impl;
+
+ AMediaFormat *(*new)(void);
+ media_status_t (*delete)(AMediaFormat*);
+
+ const char* (*toString)(AMediaFormat*);
+
+ bool (*getInt32)(AMediaFormat*, const char *name, int32_t *out);
+ bool (*getInt64)(AMediaFormat*, const char *name, int64_t *out);
+ bool (*getFloat)(AMediaFormat*, const char *name, float *out);
+ bool (*getSize)(AMediaFormat*, const char *name, size_t *out);
+ bool (*getBuffer)(AMediaFormat*, const char *name, void** data, size_t *size);
+ bool (*getString)(AMediaFormat*, const char *name, const char **out);
+
+ void (*setInt32)(AMediaFormat*, const char* name, int32_t value);
+ void (*setInt64)(AMediaFormat*, const char* name, int64_t value);
+ void (*setFloat)(AMediaFormat*, const char* name, float value);
+ void (*setString)(AMediaFormat*, const char* name, const char* value);
+ void (*setBuffer)(AMediaFormat*, const char* name, const void* data, size_t size);
+} FFAMediaFormatNdk;
+
+typedef struct FFAMediaCodecNdk {
+ FFAMediaCodec api;
+
+ void *libmedia;
+ AMediaCodec *impl;
+ ANativeWindow *window;
+
+ AMediaCodec* (*createCodecByName)(const char *name);
+ AMediaCodec* (*createDecoderByType)(const char *mime_type);
+ AMediaCodec* (*createEncoderByType)(const char *mime_type);
+ media_status_t (*delete)(AMediaCodec*);
+
+ media_status_t (*configure)(AMediaCodec *,
+ const AMediaFormat *format,
+ ANativeWindow *surface,
+ AMediaCrypto *crypto,
+ uint32_t flags);
+ media_status_t (*start)(AMediaCodec*);
+ media_status_t (*stop)(AMediaCodec*);
+ media_status_t (*flush)(AMediaCodec*);
+
+ uint8_t* (*getInputBuffer)(AMediaCodec*, size_t idx, size_t *out_size);
+ uint8_t* (*getOutputBuffer)(AMediaCodec*, size_t idx, size_t *out_size);
+
+ ssize_t (*dequeueInputBuffer)(AMediaCodec*, int64_t timeoutUs);
+ media_status_t (*queueInputBuffer)(AMediaCodec*, size_t idx,
+ long offset, size_t size,
+ uint64_t time, uint32_t flags);
+
+ ssize_t (*dequeueOutputBuffer)(AMediaCodec*, AMediaCodecBufferInfo *info, int64_t timeoutUs);
+ AMediaFormat* (*getOutputFormat)(AMediaCodec*);
+
+ media_status_t (*releaseOutputBuffer)(AMediaCodec*, size_t idx, bool render);
+ media_status_t (*releaseOutputBufferAtTime)(AMediaCodec *mData, size_t idx, int64_t timestampNs);
+
+ // Available since API level 28.
+ media_status_t (*getName)(AMediaCodec*, char** out_name);
+ void (*releaseName)(AMediaCodec*, char* name);
+} FFAMediaCodecNdk;
+
+static const FFAMediaFormat media_format_ndk;
+static const FFAMediaCodec media_codec_ndk;
+
+static const AVClass amediaformat_ndk_class = {
+ .class_name = "amediaformat_ndk",
+ .item_name = av_default_item_name,
+ .version = LIBAVUTIL_VERSION_INT,
+};
+
+static const AVClass amediacodec_ndk_class = {
+ .class_name = "amediacodec_ndk",
+ .item_name = av_default_item_name,
+ .version = LIBAVUTIL_VERSION_INT,
+};
+
+static FFAMediaFormat *mediaformat_ndk_create(AMediaFormat *impl)
+{
+ FFAMediaFormatNdk *format = av_mallocz(sizeof(*format));
+ if (!format)
+ return NULL;
+
+ format->api = media_format_ndk;
+
+ format->libmedia = dlopen("libmediandk.so", RTLD_NOW);
+ if (!format->libmedia)
+ goto error;
+
+#define GET_SYMBOL(sym) \
+ format->sym = dlsym(format->libmedia, "AMediaFormat_" #sym); \
+ if (!format->sym) \
+ goto error;
+
+ GET_SYMBOL(new)
+ GET_SYMBOL(delete)
+
+ GET_SYMBOL(toString)
+
+ GET_SYMBOL(getInt32)
+ GET_SYMBOL(getInt64)
+ GET_SYMBOL(getFloat)
+ GET_SYMBOL(getSize)
+ GET_SYMBOL(getBuffer)
+ GET_SYMBOL(getString)
+
+ GET_SYMBOL(setInt32)
+ GET_SYMBOL(setInt64)
+ GET_SYMBOL(setFloat)
+ GET_SYMBOL(setString)
+ GET_SYMBOL(setBuffer)
+
+#undef GET_SYMBOL
+
+ if (impl) {
+ format->impl = impl;
+ } else {
+ format->impl = format->new();
+ if (!format->impl)
+ goto error;
+ }
+
+ return (FFAMediaFormat *)format;
+
+error:
+ if (format->libmedia)
+ dlclose(format->libmedia);
+ av_freep(&format);
+ return NULL;
+}
+
+static FFAMediaFormat *mediaformat_ndk_new(void)
+{
+ return mediaformat_ndk_create(NULL);
+}
+
+static int mediaformat_ndk_delete(FFAMediaFormat* ctx)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ int ret = 0;
+ if (!format)
+ return 0;
+
+ av_assert0(format->api.class == &amediaformat_ndk_class);
+
+ if (format->impl && (format->delete(format->impl) != AMEDIA_OK))
+ ret = AVERROR_EXTERNAL;
+ if (format->libmedia)
+ dlclose(format->libmedia);
+ av_free(format);
+
+ return ret;
+}
+
+static char* mediaformat_ndk_toString(FFAMediaFormat* ctx)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ const char *str = format->toString(format->impl);
+ return av_strdup(str);
+}
+
+static int mediaformat_ndk_getInt32(FFAMediaFormat* ctx, const char *name, int32_t *out)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ return format->getInt32(format->impl, name, out);
+}
+
+static int mediaformat_ndk_getInt64(FFAMediaFormat* ctx, const char *name, int64_t *out)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ return format->getInt64(format->impl, name, out);
+}
+
+static int mediaformat_ndk_getFloat(FFAMediaFormat* ctx, const char *name, float *out)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ return format->getFloat(format->impl, name, out);
+}
+
+static int mediaformat_ndk_getBuffer(FFAMediaFormat* ctx, const char *name, void** data, size_t *size)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ return format->getBuffer(format->impl, name, data, size);
+}
+
+static int mediaformat_ndk_getString(FFAMediaFormat* ctx, const char *name, const char **out)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ const char *tmp = NULL;
+ int ret = format->getString(format->impl, name, &tmp);
+
+ if (tmp)
+ *out = av_strdup(tmp);
+ return ret;
+}
+
+static void mediaformat_ndk_setInt32(FFAMediaFormat* ctx, const char* name, int32_t value)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ format->setInt32(format->impl, name, value);
+}
+
+static void mediaformat_ndk_setInt64(FFAMediaFormat* ctx, const char* name, int64_t value)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ format->setInt64(format->impl, name, value);
+}
+
+static void mediaformat_ndk_setFloat(FFAMediaFormat* ctx, const char* name, float value)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ format->setFloat(format->impl, name, value);
+}
+
+static void mediaformat_ndk_setString(FFAMediaFormat* ctx, const char* name, const char* value)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ format->setString(format->impl, name, value);
+}
+
+static void mediaformat_ndk_setBuffer(FFAMediaFormat* ctx, const char* name, void* data, size_t size)
+{
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)ctx;
+ format->setBuffer(format->impl, name, data, size);
+}
+
+static char *mediacodec_ndk_getName(FFAMediaCodec *ctx)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ char *ret = NULL;
+ char *name = NULL;
+
+ if (!codec->getName || !codec->releaseName) {
+ av_log(ctx, AV_LOG_DEBUG, "getName() unavailable\n");
+ return ret;
+ }
+
+ codec->getName(codec->impl, &name);
+ if (name) {
+ ret = av_strdup(name);
+ codec->releaseName(codec->impl, name);
+ }
+
+ return ret;
+}
+
+static inline FFAMediaCodec *ndk_codec_create(int method, const char *arg) {
+ FFAMediaCodecNdk *codec = av_mallocz(sizeof(*codec));
+ const char *lib_name = "libmediandk.so";
+
+ if (!codec)
+ return NULL;
+
+ codec->api = media_codec_ndk;
+ codec->libmedia = dlopen(lib_name, RTLD_NOW);
+ if (!codec->libmedia)
+ goto error;
+
+#define GET_SYMBOL(sym, required) \
+ codec->sym = dlsym(codec->libmedia, "AMediaCodec_" #sym); \
+ if (!codec->sym) { \
+ av_log(codec, required ? AV_LOG_ERROR : AV_LOG_INFO, \
+ #sym "() unavailable from %s\n", lib_name); \
+ if (required) \
+ goto error; \
+ }
+
+ GET_SYMBOL(createCodecByName, 1)
+ GET_SYMBOL(createDecoderByType, 1)
+ GET_SYMBOL(createEncoderByType, 1)
+ GET_SYMBOL(delete, 1)
+
+ GET_SYMBOL(configure, 1)
+ GET_SYMBOL(start, 1)
+ GET_SYMBOL(stop, 1)
+ GET_SYMBOL(flush, 1)
+
+ GET_SYMBOL(getInputBuffer, 1)
+ GET_SYMBOL(getOutputBuffer, 1)
+
+ GET_SYMBOL(dequeueInputBuffer, 1)
+ GET_SYMBOL(queueInputBuffer, 1)
+
+ GET_SYMBOL(dequeueOutputBuffer, 1)
+ GET_SYMBOL(getOutputFormat, 1)
+
+ GET_SYMBOL(releaseOutputBuffer, 1)
+ GET_SYMBOL(releaseOutputBufferAtTime, 1)
+
+ GET_SYMBOL(getName, 0)
+ GET_SYMBOL(releaseName, 0)
+
+#undef GET_SYMBOL
+
+ switch (method) {
+ case CREATE_CODEC_BY_NAME:
+ codec->impl = codec->createCodecByName(arg);
+ break;
+ case CREATE_DECODER_BY_TYPE:
+ codec->impl = codec->createDecoderByType(arg);
+ break;
+ case CREATE_ENCODER_BY_TYPE:
+ codec->impl = codec->createEncoderByType(arg);
+ break;
+ default:
+ av_assert0(0);
+ }
+ if (!codec->impl)
+ goto error;
+
+ return (FFAMediaCodec *)codec;
+
+error:
+ if (codec->libmedia)
+ dlclose(codec->libmedia);
+ av_freep(&codec);
+ return NULL;
+}
+
+#define DECLARE_NDK_AMEDIACODEC_CREATE_FUNC(name, method) \
+static FFAMediaCodec *mediacodec_ndk_##name(const char *arg) \
+{ \
+ return ndk_codec_create(method, arg); \
+} \
+
+DECLARE_NDK_AMEDIACODEC_CREATE_FUNC(createCodecByName, CREATE_CODEC_BY_NAME)
+DECLARE_NDK_AMEDIACODEC_CREATE_FUNC(createDecoderByType, CREATE_DECODER_BY_TYPE)
+DECLARE_NDK_AMEDIACODEC_CREATE_FUNC(createEncoderByType, CREATE_ENCODER_BY_TYPE)
+
+static int mediacodec_ndk_delete(FFAMediaCodec* ctx)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ int ret = 0;
+
+ if (!codec)
+ return 0;
+
+ av_assert0(codec->api.class == &amediacodec_ndk_class);
+
+ if (codec->impl && (codec->delete(codec->impl) != AMEDIA_OK))
+ ret = AVERROR_EXTERNAL;
+ if (codec->window)
+ ANativeWindow_release(codec->window);
+ if (codec->libmedia)
+ dlclose(codec->libmedia);
+ av_free(codec);
+
+ return ret;
+}
+
+static int mediacodec_ndk_configure(FFAMediaCodec* ctx, const FFAMediaFormat* format_ctx, void* surface, void *crypto, uint32_t flags)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)format_ctx;
+ media_status_t status;
+
+ if (surface) {
+ JNIEnv *env = NULL;
+ JNI_GET_ENV_OR_RETURN(env, ctx, -1);
+ codec->window = ANativeWindow_fromSurface(env, surface);
+ }
+
+ if (format_ctx->class != &amediaformat_ndk_class) {
+ av_log(ctx, AV_LOG_ERROR, "invalid media format\n");
+ return AVERROR(EINVAL);
+ }
+
+ status = codec->configure(codec->impl, format->impl, codec->window, NULL, flags);
+ if (status != AMEDIA_OK) {
+ av_log(codec, AV_LOG_ERROR, "configure failed, %d\n", status);
+ return AVERROR_EXTERNAL;
+ }
+
+ return 0;
+}
+
+#define MEDIACODEC_NDK_WRAPPER(method) \
+static int mediacodec_ndk_ ## method(FFAMediaCodec* ctx) \
+{ \
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx; \
+ media_status_t status = codec->method(codec->impl); \
+ \
+ if (status != AMEDIA_OK) { \
+ av_log(codec, AV_LOG_ERROR, #method " failed, %d\n", status); \
+ return AVERROR_EXTERNAL; \
+ } \
+ \
+ return 0; \
+} \
+
+MEDIACODEC_NDK_WRAPPER(start)
+MEDIACODEC_NDK_WRAPPER(stop)
+MEDIACODEC_NDK_WRAPPER(flush)
+
+static uint8_t* mediacodec_ndk_getInputBuffer(FFAMediaCodec* ctx, size_t idx, size_t *out_size)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ return codec->getInputBuffer(codec->impl, idx, out_size);
+}
+
+static uint8_t* mediacodec_ndk_getOutputBuffer(FFAMediaCodec* ctx, size_t idx, size_t *out_size)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ return codec->getOutputBuffer(codec->impl, idx, out_size);
+}
+
+static ssize_t mediacodec_ndk_dequeueInputBuffer(FFAMediaCodec* ctx, int64_t timeoutUs)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ return codec->dequeueInputBuffer(codec->impl, timeoutUs);
+}
+
+static int mediacodec_ndk_queueInputBuffer(FFAMediaCodec *ctx, size_t idx,
+ off_t offset, size_t size,
+ uint64_t time, uint32_t flags)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ return codec->queueInputBuffer(codec->impl, idx, offset, size, time, flags);
+}
+
+static ssize_t mediacodec_ndk_dequeueOutputBuffer(FFAMediaCodec* ctx, FFAMediaCodecBufferInfo *info, int64_t timeoutUs)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ AMediaCodecBufferInfo buf_info = {0};
+ ssize_t ret;
+
+ ret = codec->dequeueOutputBuffer(codec->impl, &buf_info, timeoutUs);
+ info->offset = buf_info.offset;
+ info->size = buf_info.size;
+ info->presentationTimeUs = buf_info.presentationTimeUs;
+ info->flags = buf_info.flags;
+
+ return ret;
+}
+
+static FFAMediaFormat* mediacodec_ndk_getOutputFormat(FFAMediaCodec* ctx)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ AMediaFormat *format = codec->getOutputFormat(codec->impl);
+
+ if (!format)
+ return NULL;
+ return mediaformat_ndk_create(format);
+}
+
+static int mediacodec_ndk_releaseOutputBuffer(FFAMediaCodec* ctx, size_t idx, int render)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ media_status_t status;
+
+ status = codec->releaseOutputBuffer(codec->impl, idx, render);
+ if (status != AMEDIA_OK) {
+ av_log(codec, AV_LOG_ERROR, "release output buffer failed, %d\n", status);
+ return AVERROR_EXTERNAL;
+ }
+
+ return 0;
+}
+
+static int mediacodec_ndk_releaseOutputBufferAtTime(FFAMediaCodec *ctx, size_t idx, int64_t timestampNs)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ media_status_t status;
+
+ status = codec->releaseOutputBufferAtTime(codec->impl, idx, timestampNs);
+ if (status != AMEDIA_OK) {
+ av_log(codec, AV_LOG_ERROR, "releaseOutputBufferAtTime failed, %d\n", status);
+ return AVERROR_EXTERNAL;
+ }
+
+ return 0;
+}
+
+static int mediacodec_ndk_infoTryAgainLater(FFAMediaCodec *ctx, ssize_t idx)
+{
+ return idx == AMEDIACODEC_INFO_TRY_AGAIN_LATER;
+}
+
+static int mediacodec_ndk_infoOutputBuffersChanged(FFAMediaCodec *ctx, ssize_t idx)
+{
+ return idx == AMEDIACODEC_INFO_OUTPUT_BUFFERS_CHANGED;
+}
+
+static int mediacodec_ndk_infoOutputFormatChanged(FFAMediaCodec *ctx, ssize_t idx)
+{
+ return idx == AMEDIACODEC_INFO_OUTPUT_FORMAT_CHANGED;
+}
+
+static int mediacodec_ndk_getBufferFlagCodecConfig(FFAMediaCodec *ctx)
+{
+ return AMEDIACODEC_BUFFER_FLAG_CODEC_CONFIG;
+}
+
+static int mediacodec_ndk_getBufferFlagEndOfStream(FFAMediaCodec *ctx)
+{
+ return AMEDIACODEC_BUFFER_FLAG_END_OF_STREAM;
+}
+
+static int mediacodec_ndk_getBufferFlagKeyFrame(FFAMediaCodec *ctx)
+{
+ return 1;
+}
+
+static int mediacodec_ndk_getConfigureFlagEncode(FFAMediaCodec *ctx)
+{
+ return AMEDIACODEC_CONFIGURE_FLAG_ENCODE;
+}
+
+static int mediacodec_ndk_cleanOutputBuffers(FFAMediaCodec *ctx)
+{
+ return 0;
+}
+
+static const FFAMediaFormat media_format_ndk = {
+ .class = &amediaformat_ndk_class,
+
+ .create = mediaformat_ndk_new,
+ .delete = mediaformat_ndk_delete,
+
+ .toString = mediaformat_ndk_toString,
+
+ .getInt32 = mediaformat_ndk_getInt32,
+ .getInt64 = mediaformat_ndk_getInt64,
+ .getFloat = mediaformat_ndk_getFloat,
+ .getBuffer = mediaformat_ndk_getBuffer,
+ .getString = mediaformat_ndk_getString,
+
+ .setInt32 = mediaformat_ndk_setInt32,
+ .setInt64 = mediaformat_ndk_setInt64,
+ .setFloat = mediaformat_ndk_setFloat,
+ .setString = mediaformat_ndk_setString,
+ .setBuffer = mediaformat_ndk_setBuffer,
+};
+
+static const FFAMediaCodec media_codec_ndk = {
+ .class = &amediacodec_ndk_class,
+
+ .getName = mediacodec_ndk_getName,
+
+ .createCodecByName = mediacodec_ndk_createCodecByName,
+ .createDecoderByType = mediacodec_ndk_createDecoderByType,
+ .createEncoderByType = mediacodec_ndk_createEncoderByType,
+ .delete = mediacodec_ndk_delete,
+
+ .configure = mediacodec_ndk_configure,
+ .start = mediacodec_ndk_start,
+ .stop = mediacodec_ndk_stop,
+ .flush = mediacodec_ndk_flush,
+
+ .getInputBuffer = mediacodec_ndk_getInputBuffer,
+ .getOutputBuffer = mediacodec_ndk_getOutputBuffer,
+
+ .dequeueInputBuffer = mediacodec_ndk_dequeueInputBuffer,
+ .queueInputBuffer = mediacodec_ndk_queueInputBuffer,
+
+ .dequeueOutputBuffer = mediacodec_ndk_dequeueOutputBuffer,
+ .getOutputFormat = mediacodec_ndk_getOutputFormat,
+
+ .releaseOutputBuffer = mediacodec_ndk_releaseOutputBuffer,
+ .releaseOutputBufferAtTime = mediacodec_ndk_releaseOutputBufferAtTime,
+
+ .infoTryAgainLater = mediacodec_ndk_infoTryAgainLater,
+ .infoOutputBuffersChanged = mediacodec_ndk_infoOutputBuffersChanged,
+ .infoOutputFormatChanged = mediacodec_ndk_infoOutputFormatChanged,
+
+ .getBufferFlagCodecConfig = mediacodec_ndk_getBufferFlagCodecConfig,
+ .getBufferFlagEndOfStream = mediacodec_ndk_getBufferFlagEndOfStream,
+ .getBufferFlagKeyFrame = mediacodec_ndk_getBufferFlagKeyFrame,
+
+ .getConfigureFlagEncode = mediacodec_ndk_getConfigureFlagEncode,
+ .cleanOutputBuffers = mediacodec_ndk_cleanOutputBuffers,
+};
+
+FFAMediaFormat *ff_AMediaFormat_new(int ndk)
{
+ if (ndk)
+ return media_format_ndk.create();
return media_format_jni.create();
}
-FFAMediaCodec* ff_AMediaCodec_createCodecByName(const char *name)
+FFAMediaCodec* ff_AMediaCodec_createCodecByName(const char *name, int ndk)
{
+ if (ndk)
+ return media_codec_ndk.createCodecByName(name);
return media_codec_jni.createCodecByName(name);
}
-FFAMediaCodec* ff_AMediaCodec_createDecoderByType(const char *mime_type)
+FFAMediaCodec* ff_AMediaCodec_createDecoderByType(const char *mime_type, int ndk)
{
+ if (ndk)
+ return media_codec_ndk.createDecoderByType(mime_type);
return media_codec_jni.createDecoderByType(mime_type);
}
-FFAMediaCodec* ff_AMediaCodec_createEncoderByType(const char *mime_type)
+FFAMediaCodec* ff_AMediaCodec_createEncoderByType(const char *mime_type, int ndk)
{
+ if (ndk)
+ return media_codec_ndk.createEncoderByType(mime_type);
return media_codec_jni.createEncoderByType(mime_type);
}
diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h
index 606fdbede5..b9b882f243 100644
--- a/libavcodec/mediacodec_wrapper.h
+++ b/libavcodec/mediacodec_wrapper.h
@@ -80,7 +80,7 @@ struct FFAMediaFormat {
void (*setBuffer)(FFAMediaFormat* format, const char* name, void* data, size_t size);
};
-FFAMediaFormat *ff_AMediaFormat_new(void);
+FFAMediaFormat *ff_AMediaFormat_new(int ndk);
static inline int ff_AMediaFormat_delete(FFAMediaFormat* format)
{
@@ -198,9 +198,9 @@ static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec)
return codec->getName(codec);
}
-FFAMediaCodec* ff_AMediaCodec_createCodecByName(const char *name);
-FFAMediaCodec* ff_AMediaCodec_createDecoderByType(const char *mime_type);
-FFAMediaCodec* ff_AMediaCodec_createEncoderByType(const char *mime_type);
+FFAMediaCodec* ff_AMediaCodec_createCodecByName(const char *name, int ndk);
+FFAMediaCodec* ff_AMediaCodec_createDecoderByType(const char *mime_type, int ndk);
+FFAMediaCodec* ff_AMediaCodec_createEncoderByType(const char *mime_type, int ndk);
static inline int ff_AMediaCodec_configure(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags)
{
diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c
index 2c66f38541..2e07548b77 100644
--- a/libavcodec/mediacodecdec.c
+++ b/libavcodec/mediacodecdec.c
@@ -310,7 +310,7 @@ static av_cold int mediacodec_decode_init(AVCodecContext *avctx)
FFAMediaFormat *format = NULL;
MediaCodecH264DecContext *s = avctx->priv_data;
- format = ff_AMediaFormat_new();
+ format = ff_AMediaFormat_new(0);
if (!format) {
av_log(avctx, AV_LOG_ERROR, "Failed to create media format\n");
ret = AVERROR_EXTERNAL;
diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c
index d6ce709dd8..1905e686bc 100644
--- a/libavcodec/mediacodecdec_common.c
+++ b/libavcodec/mediacodecdec_common.c
@@ -606,7 +606,7 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
}
av_log(avctx, AV_LOG_DEBUG, "Found decoder %s\n", s->codec_name);
- s->codec = ff_AMediaCodec_createCodecByName(s->codec_name);
+ s->codec = ff_AMediaCodec_createCodecByName(s->codec_name, 0);
if (!s->codec) {
av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for type %s and name %s\n", mime, s->codec_name);
ret = AVERROR_EXTERNAL;
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 07/12] avcodec/mediacodecdec: enable NDK mediacodec
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
` (5 preceding siblings ...)
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 06/12] avcodec/mediacodec: add NDK media codec wrapper Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 08/12] avutil/hwcontext_mediacodec: add ANativeWindow support Zhao Zhili
` (4 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
Changelog | 1 +
libavcodec/mediacodecdec.c | 10 +++++++++-
libavcodec/mediacodecdec_common.c | 23 +++++++++++++++++++----
libavcodec/mediacodecdec_common.h | 1 +
libavcodec/version.h | 2 +-
5 files changed, 31 insertions(+), 6 deletions(-)
diff --git a/Changelog b/Changelog
index ec9de1bd85..9e203833aa 100644
--- a/Changelog
+++ b/Changelog
@@ -18,6 +18,7 @@ version <next>:
- Media 100i decoders
- DTS to PTS reorder bsf
- ViewQuest VQC decoder
+- MediaCodec decoder via NDKMediaCodec
version 5.1:
diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c
index 2e07548b77..2c231d1a34 100644
--- a/libavcodec/mediacodecdec.c
+++ b/libavcodec/mediacodecdec.c
@@ -40,6 +40,7 @@
#include "hevc_parse.h"
#include "hwconfig.h"
#include "internal.h"
+#include "jni.h"
#include "mediacodec_wrapper.h"
#include "mediacodecdec_common.h"
@@ -54,6 +55,7 @@ typedef struct MediaCodecH264DecContext {
int delay_flush;
int amlogic_mpeg2_api23_workaround;
+ int use_ndk_codec;
} MediaCodecH264DecContext;
static av_cold int mediacodec_decode_close(AVCodecContext *avctx)
@@ -310,7 +312,10 @@ static av_cold int mediacodec_decode_init(AVCodecContext *avctx)
FFAMediaFormat *format = NULL;
MediaCodecH264DecContext *s = avctx->priv_data;
- format = ff_AMediaFormat_new(0);
+ if (s->use_ndk_codec < 0)
+ s->use_ndk_codec = !av_jni_get_java_vm(avctx);
+
+ format = ff_AMediaFormat_new(s->use_ndk_codec);
if (!format) {
av_log(avctx, AV_LOG_ERROR, "Failed to create media format\n");
ret = AVERROR_EXTERNAL;
@@ -388,6 +393,7 @@ static av_cold int mediacodec_decode_init(AVCodecContext *avctx)
}
s->ctx->delay_flush = s->delay_flush;
+ s->ctx->use_ndk_codec = s->use_ndk_codec;
if ((ret = ff_mediacodec_dec_init(avctx, s->ctx, codec_mime, format)) < 0) {
s->ctx = NULL;
@@ -528,6 +534,8 @@ static const AVCodecHWConfigInternal *const mediacodec_hw_configs[] = {
static const AVOption ff_mediacodec_vdec_options[] = {
{ "delay_flush", "Delay flush until hw output buffers are returned to the decoder",
OFFSET(delay_flush), AV_OPT_TYPE_BOOL, {.i64 = 0}, 0, 1, VD },
+ { "ndk_codec", "Use MediaCodec from NDK",
+ OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 = -1}, -1, 1, VD },
{ NULL }
};
diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c
index 1905e686bc..c3d5988063 100644
--- a/libavcodec/mediacodecdec_common.c
+++ b/libavcodec/mediacodecdec_common.c
@@ -601,12 +601,27 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
s->codec_name = ff_AMediaCodecList_getCodecNameByType(mime, profile, 0, avctx);
if (!s->codec_name) {
- ret = AVERROR_EXTERNAL;
- goto fail;
+ // getCodecNameByType() can fail due to missing JVM, while NDK
+ // mediacodec can be used without JVM.
+ if (!s->use_ndk_codec) {
+ ret = AVERROR_EXTERNAL;
+ goto fail;
+ }
+ av_log(avctx, AV_LOG_INFO, "Failed to getCodecNameByType\n");
+ } else {
+ av_log(avctx, AV_LOG_DEBUG, "Found decoder %s\n", s->codec_name);
}
- av_log(avctx, AV_LOG_DEBUG, "Found decoder %s\n", s->codec_name);
- s->codec = ff_AMediaCodec_createCodecByName(s->codec_name, 0);
+ if (s->codec_name)
+ s->codec = ff_AMediaCodec_createCodecByName(s->codec_name, s->use_ndk_codec);
+ else {
+ s->codec = ff_AMediaCodec_createDecoderByType(mime, s->use_ndk_codec);
+ if (s->codec) {
+ s->codec_name = ff_AMediaCodec_getName(s->codec);
+ if (!s->codec_name)
+ s->codec_name = av_strdup(mime);
+ }
+ }
if (!s->codec) {
av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for type %s and name %s\n", mime, s->codec_name);
ret = AVERROR_EXTERNAL;
diff --git a/libavcodec/mediacodecdec_common.h b/libavcodec/mediacodecdec_common.h
index 0b21129fee..0ab29036bd 100644
--- a/libavcodec/mediacodecdec_common.h
+++ b/libavcodec/mediacodecdec_common.h
@@ -70,6 +70,7 @@ typedef struct MediaCodecDecContext {
bool delay_flush;
atomic_int serial;
+ bool use_ndk_codec;
} MediaCodecDecContext;
int ff_mediacodec_dec_init(AVCodecContext *avctx,
diff --git a/libavcodec/version.h b/libavcodec/version.h
index f8abc803b6..43d0d9a9fc 100644
--- a/libavcodec/version.h
+++ b/libavcodec/version.h
@@ -30,7 +30,7 @@
#include "version_major.h"
#define LIBAVCODEC_VERSION_MINOR 51
-#define LIBAVCODEC_VERSION_MICRO 100
+#define LIBAVCODEC_VERSION_MICRO 101
#define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \
LIBAVCODEC_VERSION_MINOR, \
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 08/12] avutil/hwcontext_mediacodec: add ANativeWindow support
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
` (6 preceding siblings ...)
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 07/12] avcodec/mediacodecdec: enable NDK mediacodec Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 09/12] avcodec/mediacodec: " Zhao Zhili
` (3 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
libavutil/hwcontext_mediacodec.c | 56 +++++++++++++++++++++++++++++++-
libavutil/hwcontext_mediacodec.h | 11 +++++++
libavutil/version.h | 4 +--
3 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/libavutil/hwcontext_mediacodec.c b/libavutil/hwcontext_mediacodec.c
index b0d8993e15..bb1779d34d 100644
--- a/libavutil/hwcontext_mediacodec.c
+++ b/libavutil/hwcontext_mediacodec.c
@@ -18,12 +18,24 @@
#include "config.h"
+#include <android/native_window.h>
+#include <dlfcn.h>
+#include <media/NdkMediaCodec.h>
+
#include "buffer.h"
#include "common.h"
#include "hwcontext.h"
#include "hwcontext_internal.h"
#include "hwcontext_mediacodec.h"
+typedef struct MediaCodecDeviceContext {
+ AVMediaCodecDeviceContext ctx;
+
+ void *libmedia;
+ media_status_t (*create_surface)(ANativeWindow **surface);
+} MediaCodecDeviceContext;
+
+
static int mc_device_create(AVHWDeviceContext *ctx, const char *device,
AVDictionary *opts, int flags)
{
@@ -35,13 +47,55 @@ static int mc_device_create(AVHWDeviceContext *ctx, const char *device,
return 0;
}
+static int mc_device_init(AVHWDeviceContext *ctx)
+{
+ MediaCodecDeviceContext *s = ctx->hwctx;
+ AVMediaCodecDeviceContext *dev = (AVMediaCodecDeviceContext *)s;
+ ANativeWindow *native_window = NULL;
+
+ if (dev->surface)
+ return 0;
+
+ if (dev->native_window)
+ return 0;
+
+ s->libmedia = dlopen("libmediandk.so", RTLD_NOW);
+ if (!s->libmedia)
+ return AVERROR_UNKNOWN;
+
+ s->create_surface = dlsym(s->libmedia, "AMediaCodec_createPersistentInputSurface");
+ if (!s->create_surface)
+ return AVERROR_UNKNOWN;
+
+ s->create_surface(&native_window);
+ dev->native_window = native_window;
+ return 0;
+}
+
+static void mc_device_uninit(AVHWDeviceContext *ctx)
+{
+ MediaCodecDeviceContext *s = ctx->hwctx;
+ AVMediaCodecDeviceContext *dev = ctx->hwctx;
+ if (!s->libmedia)
+ return;
+
+ if (dev->native_window) {
+ ANativeWindow_release(dev->native_window);
+ dev->native_window = NULL;
+ }
+ dlclose(s->libmedia);
+ s->libmedia = NULL;
+}
+
const HWContextType ff_hwcontext_type_mediacodec = {
.type = AV_HWDEVICE_TYPE_MEDIACODEC,
.name = "mediacodec",
- .device_hwctx_size = sizeof(AVMediaCodecDeviceContext),
+ .device_hwctx_size = sizeof(MediaCodecDeviceContext),
.device_create = mc_device_create,
+ .device_init = mc_device_init,
+ .device_uninit = mc_device_uninit,
.pix_fmts = (const enum AVPixelFormat[]){
AV_PIX_FMT_MEDIACODEC,
diff --git a/libavutil/hwcontext_mediacodec.h b/libavutil/hwcontext_mediacodec.h
index 101a9806d5..920e17764f 100644
--- a/libavutil/hwcontext_mediacodec.h
+++ b/libavutil/hwcontext_mediacodec.h
@@ -31,6 +31,17 @@ typedef struct AVMediaCodecDeviceContext {
* This is the default surface used by decoders on this device.
*/
void *surface;
+
+ /**
+ * Pointer to ANativeWindow.
+ *
+ * It both surface and native_window is NULL, try to create it
+ * automatically if OS support.
+ *
+ * It can be used as output surface for decoder and input surface for
+ * encoder.
+ */
+ void *native_window;
} AVMediaCodecDeviceContext;
#endif /* AVUTIL_HWCONTEXT_MEDIACODEC_H */
diff --git a/libavutil/version.h b/libavutil/version.h
index cb0c928bd0..2df788e529 100644
--- a/libavutil/version.h
+++ b/libavutil/version.h
@@ -79,8 +79,8 @@
*/
#define LIBAVUTIL_VERSION_MAJOR 57
-#define LIBAVUTIL_VERSION_MINOR 39
-#define LIBAVUTIL_VERSION_MICRO 101
+#define LIBAVUTIL_VERSION_MINOR 40
+#define LIBAVUTIL_VERSION_MICRO 100
#define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \
LIBAVUTIL_VERSION_MINOR, \
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 09/12] avcodec/mediacodec: add ANativeWindow support
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
` (7 preceding siblings ...)
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 08/12] avutil/hwcontext_mediacodec: add ANativeWindow support Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder Zhao Zhili
` (2 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
ANativeWindow can be used without JVM.
---
libavcodec/mediacodec_surface.c | 46 ++++++++++++++++++++++++-------
libavcodec/mediacodec_surface.h | 8 ++++--
libavcodec/mediacodec_wrapper.c | 30 +++++++++++++++-----
libavcodec/mediacodec_wrapper.h | 8 ++++--
libavcodec/mediacodecdec_common.c | 4 +--
5 files changed, 72 insertions(+), 24 deletions(-)
diff --git a/libavcodec/mediacodec_surface.c b/libavcodec/mediacodec_surface.c
index 09a42295d2..ef41cdafa7 100644
--- a/libavcodec/mediacodec_surface.c
+++ b/libavcodec/mediacodec_surface.c
@@ -20,33 +20,59 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
+#include <android/native_window.h>
#include <jni.h>
+#include "libavutil/mem.h"
#include "ffjni.h"
#include "mediacodec_surface.h"
-FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *log_ctx)
+FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *native_window, void *log_ctx)
{
- JNIEnv *env = NULL;
+ FFANativeWindow *ret;
- env = ff_jni_get_env(log_ctx);
- if (!env) {
+ ret = av_mallocz(sizeof(*ret));
+ if (!ret)
return NULL;
+
+ if (surface) {
+ JNIEnv *env = NULL;
+
+ env = ff_jni_get_env(log_ctx);
+ if (env)
+ ret->surface = (*env)->NewGlobalRef(env, surface);
+ }
+
+ if (native_window) {
+ ANativeWindow_acquire(native_window);
+ ret->native_window = native_window;
}
- return (*env)->NewGlobalRef(env, surface);
+ if (!ret->surface && !ret->native_window) {
+ av_log(log_ctx, AV_LOG_ERROR, "Both surface and native_window are NULL\n");
+ av_freep(&ret);
+ }
+
+ return ret;
}
int ff_mediacodec_surface_unref(FFANativeWindow *window, void *log_ctx)
{
- JNIEnv *env = NULL;
+ if (!window)
+ return 0;
- env = ff_jni_get_env(log_ctx);
- if (!env) {
- return AVERROR_EXTERNAL;
+ if (window->surface) {
+ JNIEnv *env = NULL;
+
+ env = ff_jni_get_env(log_ctx);
+ if (env)
+ (*env)->DeleteGlobalRef(env, window->surface);
}
- (*env)->DeleteGlobalRef(env, window);
+ if (window->native_window)
+ ANativeWindow_release(window->native_window);
+
+ av_free(window);
return 0;
}
diff --git a/libavcodec/mediacodec_surface.h b/libavcodec/mediacodec_surface.h
index 933dc2bf51..e2ac1c9057 100644
--- a/libavcodec/mediacodec_surface.h
+++ b/libavcodec/mediacodec_surface.h
@@ -25,10 +25,12 @@
#include "libavcodec/avcodec.h"
-struct FFANativeWindow;
-typedef struct FFANativeWindow FFANativeWindow;
+typedef struct FFANativeWindow {
+ void *surface;
+ void *native_window;
+} FFANativeWindow;
-FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *log_ctx);
+FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *native_window, void *log_ctx);
int ff_mediacodec_surface_unref(FFANativeWindow *window, void *log_ctx);
#endif /* AVCODEC_MEDIACODEC_SURFACE_H */
diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c
index b12aced711..284d615980 100644
--- a/libavcodec/mediacodec_wrapper.c
+++ b/libavcodec/mediacodec_wrapper.c
@@ -1371,12 +1371,17 @@ fail:
return ret;
}
-static int mediacodec_jni_configure(FFAMediaCodec* ctx, const FFAMediaFormat* format_ctx, void* surface, void *crypto, uint32_t flags)
+static int mediacodec_jni_configure(FFAMediaCodec *ctx,
+ const FFAMediaFormat* format_ctx,
+ FFANativeWindow* window,
+ void *crypto,
+ uint32_t flags)
{
int ret = 0;
JNIEnv *env = NULL;
FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
const FFAMediaFormatJni *format = (FFAMediaFormatJni *)format_ctx;
+ jobject *surface = window ? window->surface : NULL;
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
@@ -2151,16 +2156,27 @@ static int mediacodec_ndk_delete(FFAMediaCodec* ctx)
return ret;
}
-static int mediacodec_ndk_configure(FFAMediaCodec* ctx, const FFAMediaFormat* format_ctx, void* surface, void *crypto, uint32_t flags)
+static int mediacodec_ndk_configure(FFAMediaCodec* ctx,
+ const FFAMediaFormat* format_ctx,
+ FFANativeWindow* window,
+ void *crypto,
+ uint32_t flags)
{
FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)format_ctx;
media_status_t status;
+ ANativeWindow *native_window = NULL;
- if (surface) {
- JNIEnv *env = NULL;
- JNI_GET_ENV_OR_RETURN(env, ctx, -1);
- codec->window = ANativeWindow_fromSurface(env, surface);
+ if (window) {
+ if (window->surface) {
+ JNIEnv *env = NULL;
+ JNI_GET_ENV_OR_RETURN(env, ctx, -1);
+ native_window = ANativeWindow_fromSurface(env, window->surface);
+ // Save for release
+ codec->window = native_window;
+ } else if (window->native_window) {
+ native_window = window->native_window;
+ }
}
if (format_ctx->class != &amediaformat_ndk_class) {
@@ -2168,7 +2184,7 @@ static int mediacodec_ndk_configure(FFAMediaCodec* ctx, const FFAMediaFormat* fo
return AVERROR(EINVAL);
}
- status = codec->configure(codec->impl, format->impl, codec->window, NULL, flags);
+ status = codec->configure(codec->impl, format->impl, native_window, NULL, flags);
if (status != AMEDIA_OK) {
av_log(codec, AV_LOG_ERROR, "configure failed, %d\n", status);
return AVERROR_EXTERNAL;
diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h
index b9b882f243..7cf3f4aecd 100644
--- a/libavcodec/mediacodec_wrapper.h
+++ b/libavcodec/mediacodec_wrapper.h
@@ -27,6 +27,7 @@
#include <sys/types.h>
#include "avcodec.h"
+#include "mediacodec_surface.h"
/**
* The following API around MediaCodec and MediaFormat is based on the
@@ -163,7 +164,7 @@ struct FFAMediaCodec {
FFAMediaCodec* (*createEncoderByType)(const char *mime_type);
int (*delete)(FFAMediaCodec* codec);
- int (*configure)(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags);
+ int (*configure)(FFAMediaCodec* codec, const FFAMediaFormat* format, FFANativeWindow* surface, void *crypto, uint32_t flags);
int (*start)(FFAMediaCodec* codec);
int (*stop)(FFAMediaCodec* codec);
int (*flush)(FFAMediaCodec* codec);
@@ -202,7 +203,10 @@ FFAMediaCodec* ff_AMediaCodec_createCodecByName(const char *name, int ndk);
FFAMediaCodec* ff_AMediaCodec_createDecoderByType(const char *mime_type, int ndk);
FFAMediaCodec* ff_AMediaCodec_createEncoderByType(const char *mime_type, int ndk);
-static inline int ff_AMediaCodec_configure(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags)
+static inline int ff_AMediaCodec_configure(FFAMediaCodec *codec,
+ const FFAMediaFormat *format,
+ FFANativeWindow *surface,
+ void *crypto, uint32_t flags)
{
return codec->configure(codec, format, surface, crypto, flags);
}
diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c
index c3d5988063..d56412aa6d 100644
--- a/libavcodec/mediacodecdec_common.c
+++ b/libavcodec/mediacodecdec_common.c
@@ -582,14 +582,14 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
if (device_ctx->type == AV_HWDEVICE_TYPE_MEDIACODEC) {
if (device_ctx->hwctx) {
AVMediaCodecDeviceContext *mediacodec_ctx = (AVMediaCodecDeviceContext *)device_ctx->hwctx;
- s->surface = ff_mediacodec_surface_ref(mediacodec_ctx->surface, avctx);
+ s->surface = ff_mediacodec_surface_ref(mediacodec_ctx->surface, mediacodec_ctx->native_window, avctx);
av_log(avctx, AV_LOG_INFO, "Using surface %p\n", s->surface);
}
}
}
if (!s->surface && user_ctx && user_ctx->surface) {
- s->surface = ff_mediacodec_surface_ref(user_ctx->surface, avctx);
+ s->surface = ff_mediacodec_surface_ref(user_ctx->surface, NULL, avctx);
av_log(avctx, AV_LOG_INFO, "Using surface %p\n", s->surface);
}
}
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
` (8 preceding siblings ...)
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 09/12] avcodec/mediacodec: " Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-26 21:17 ` Tomas Härdin
[not found] ` <CAK=uwuxEwO98S2js5fnY_T1gf1epz+WRBiWyk86qM6oQmAMMMQ@mail.gmail.com>
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 11/12] avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 12/12] fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec Zhao Zhili
11 siblings, 2 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
Changelog | 1 +
configure | 4 +
libavcodec/Makefile | 2 +
libavcodec/allcodecs.c | 2 +
libavcodec/mediacodec_wrapper.c | 102 ++++++-
libavcodec/mediacodec_wrapper.h | 8 +
libavcodec/mediacodecenc.c | 495 ++++++++++++++++++++++++++++++++
libavcodec/version.h | 4 +-
8 files changed, 611 insertions(+), 7 deletions(-)
create mode 100644 libavcodec/mediacodecenc.c
diff --git a/Changelog b/Changelog
index 9e203833aa..9e39a35972 100644
--- a/Changelog
+++ b/Changelog
@@ -19,6 +19,7 @@ version <next>:
- DTS to PTS reorder bsf
- ViewQuest VQC decoder
- MediaCodec decoder via NDKMediaCodec
+- MediaCodec encoder
version 5.1:
diff --git a/configure b/configure
index ee2e3ba6ac..5114cda13f 100755
--- a/configure
+++ b/configure
@@ -3193,6 +3193,8 @@ h264_cuvid_decoder_select="h264_mp4toannexb_bsf"
h264_mediacodec_decoder_deps="mediacodec"
h264_mediacodec_decoder_extralibs="-landroid"
h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser"
+h264_mediacodec_encoder_deps="mediacodec"
+h264_mediacodec_encoder_extralibs="-landroid"
h264_mf_encoder_deps="mediafoundation"
h264_mmal_decoder_deps="mmal"
h264_nvenc_encoder_deps="nvenc"
@@ -3212,6 +3214,8 @@ hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf"
hevc_mediacodec_decoder_deps="mediacodec"
hevc_mediacodec_decoder_extralibs="-landroid"
hevc_mediacodec_decoder_select="hevc_mp4toannexb_bsf hevc_parser"
+hevc_mediacodec_encoder_deps="mediacodec"
+hevc_mediacodec_encoder_extralibs="-landroid"
hevc_mf_encoder_deps="mediafoundation"
hevc_nvenc_encoder_deps="nvenc"
hevc_nvenc_encoder_select="atsc_a53"
diff --git a/libavcodec/Makefile b/libavcodec/Makefile
index 90c7f113a3..7d0b513eec 100644
--- a/libavcodec/Makefile
+++ b/libavcodec/Makefile
@@ -393,6 +393,7 @@ OBJS-$(CONFIG_H264_DECODER) += h264dec.o h264_cabac.o h264_cavlc.o \
OBJS-$(CONFIG_H264_AMF_ENCODER) += amfenc_h264.o
OBJS-$(CONFIG_H264_CUVID_DECODER) += cuviddec.o
OBJS-$(CONFIG_H264_MEDIACODEC_DECODER) += mediacodecdec.o
+OBJS-$(CONFIG_H264_MEDIACODEC_ENCODER) += mediacodecenc.o
OBJS-$(CONFIG_H264_MF_ENCODER) += mfenc.o mf_utils.o
OBJS-$(CONFIG_H264_MMAL_DECODER) += mmaldec.o
OBJS-$(CONFIG_H264_NVENC_ENCODER) += nvenc_h264.o nvenc.o
@@ -417,6 +418,7 @@ OBJS-$(CONFIG_HEVC_DECODER) += hevcdec.o hevc_mvs.o \
OBJS-$(CONFIG_HEVC_AMF_ENCODER) += amfenc_hevc.o
OBJS-$(CONFIG_HEVC_CUVID_DECODER) += cuviddec.o
OBJS-$(CONFIG_HEVC_MEDIACODEC_DECODER) += mediacodecdec.o
+OBJS-$(CONFIG_HEVC_MEDIACODEC_ENCODER) += mediacodecenc.o
OBJS-$(CONFIG_HEVC_MF_ENCODER) += mfenc.o mf_utils.o
OBJS-$(CONFIG_HEVC_NVENC_ENCODER) += nvenc_hevc.o nvenc.o
OBJS-$(CONFIG_HEVC_QSV_DECODER) += qsvdec.o
diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
index 46ad3b5a25..4c33a9ec3c 100644
--- a/libavcodec/allcodecs.c
+++ b/libavcodec/allcodecs.c
@@ -154,6 +154,7 @@ extern const FFCodec ff_h264_decoder;
extern const FFCodec ff_h264_crystalhd_decoder;
extern const FFCodec ff_h264_v4l2m2m_decoder;
extern const FFCodec ff_h264_mediacodec_decoder;
+extern const FFCodec ff_h264_mediacodec_encoder;
extern const FFCodec ff_h264_mmal_decoder;
extern const FFCodec ff_h264_qsv_decoder;
extern const FFCodec ff_h264_rkmpp_decoder;
@@ -842,6 +843,7 @@ extern const FFCodec ff_h264_videotoolbox_encoder;
extern const FFCodec ff_hevc_amf_encoder;
extern const FFCodec ff_hevc_cuvid_decoder;
extern const FFCodec ff_hevc_mediacodec_decoder;
+extern const FFCodec ff_hevc_mediacodec_encoder;
extern const FFCodec ff_hevc_mf_encoder;
extern const FFCodec ff_hevc_nvenc_encoder;
extern const FFCodec ff_hevc_qsv_encoder;
diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c
index 284d615980..5d1a32031d 100644
--- a/libavcodec/mediacodec_wrapper.c
+++ b/libavcodec/mediacodec_wrapper.c
@@ -212,6 +212,9 @@ struct JNIAMediaCodecFields {
jmethodID release_output_buffer_id;
jmethodID release_output_buffer_at_time_id;
+ jmethodID set_input_surface_id;
+ jmethodID signal_end_of_input_stream_id;
+
jclass mediainfo_class;
jmethodID init_id;
@@ -261,6 +264,9 @@ static const struct FFJniField jni_amediacodec_mapping[] = {
{ "android/media/MediaCodec", "releaseOutputBuffer", "(IZ)V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, release_output_buffer_id), 1 },
{ "android/media/MediaCodec", "releaseOutputBuffer", "(IJ)V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, release_output_buffer_at_time_id), 0 },
+ { "android/media/MediaCodec", "setInputSurface", "(Landroid/view/Surface;)V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, set_input_surface_id), 0 },
+ { "android/media/MediaCodec", "signalEndOfInputStream", "()V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, signal_end_of_input_stream_id), 0 },
+
{ "android/media/MediaCodec$BufferInfo", NULL, NULL, FF_JNI_CLASS, offsetof(struct JNIAMediaCodecFields, mediainfo_class), 1 },
{ "android/media/MediaCodec.BufferInfo", "<init>", "()V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, init_id), 1 },
@@ -1385,7 +1391,26 @@ static int mediacodec_jni_configure(FFAMediaCodec *ctx,
JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
- (*env)->CallVoidMethod(env, codec->object, codec->jfields.configure_id, format->object, surface, NULL, flags);
+ if (flags & codec->CONFIGURE_FLAG_ENCODE) {
+ if (surface && !codec->jfields.set_input_surface_id) {
+ av_log(ctx, AV_LOG_ERROR, "System doesn't support setInputSurface\n");
+ return AVERROR_EXTERNAL;
+ }
+
+ (*env)->CallVoidMethod(env, codec->object, codec->jfields.configure_id, format->object, NULL, NULL, flags);
+ if (ff_jni_exception_check(env, 1, codec) < 0)
+ return AVERROR_EXTERNAL;
+
+ if (!surface)
+ return 0;
+
+ (*env)->CallVoidMethod(env, codec->object, codec->jfields.set_input_surface_id, surface);
+ if (ff_jni_exception_check(env, 1, codec) < 0)
+ return AVERROR_EXTERNAL;
+ return 0;
+ } else {
+ (*env)->CallVoidMethod(env, codec->object, codec->jfields.configure_id, format->object, surface, NULL, flags);
+ }
if (ff_jni_exception_check(env, 1, codec) < 0) {
ret = AVERROR_EXTERNAL;
goto fail;
@@ -1743,6 +1768,22 @@ fail:
return ret;
}
+static int mediacodec_jni_signalEndOfInputStream(FFAMediaCodec *ctx)
+{
+ int ret = 0;
+ JNIEnv *env = NULL;
+ FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
+
+ JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
+
+ (*env)->CallVoidMethod(env, codec->object, codec->jfields.signal_end_of_input_stream_id);
+ if (ff_jni_exception_check(env, 1, codec) < 0) {
+ return AVERROR_EXTERNAL;
+ }
+
+ return 0;
+}
+
static const FFAMediaFormat media_format_jni = {
.class = &amediaformat_class,
@@ -1801,6 +1842,7 @@ static const FFAMediaCodec media_codec_jni = {
.getConfigureFlagEncode = mediacodec_jni_getConfigureFlagEncode,
.cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers,
+ .signalEndOfInputStream = mediacodec_jni_signalEndOfInputStream,
};
typedef struct FFAMediaFormatNdk {
@@ -1866,6 +1908,10 @@ typedef struct FFAMediaCodecNdk {
// Available since API level 28.
media_status_t (*getName)(AMediaCodec*, char** out_name);
void (*releaseName)(AMediaCodec*, char* name);
+
+ // Available since API level 26.
+ media_status_t (*setInputSurface)(AMediaCodec*, ANativeWindow *);
+ media_status_t (*signalEndOfInputStream)(AMediaCodec *);
} FFAMediaCodecNdk;
static const FFAMediaFormat media_format_ndk;
@@ -2098,6 +2144,9 @@ static inline FFAMediaCodec *ndk_codec_create(int method, const char *arg) {
GET_SYMBOL(getName, 0)
GET_SYMBOL(releaseName, 0)
+ GET_SYMBOL(setInputSurface, 0)
+ GET_SYMBOL(signalEndOfInputStream, 0)
+
#undef GET_SYMBOL
switch (method) {
@@ -2184,10 +2233,32 @@ static int mediacodec_ndk_configure(FFAMediaCodec* ctx,
return AVERROR(EINVAL);
}
- status = codec->configure(codec->impl, format->impl, native_window, NULL, flags);
- if (status != AMEDIA_OK) {
- av_log(codec, AV_LOG_ERROR, "configure failed, %d\n", status);
- return AVERROR_EXTERNAL;
+ if (flags & AMEDIACODEC_CONFIGURE_FLAG_ENCODE) {
+ if (native_window && !codec->setInputSurface) {
+ av_log(ctx, AV_LOG_ERROR, "System doesn't support setInputSurface\n");
+ return AVERROR_EXTERNAL;
+ }
+
+ status = codec->configure(codec->impl, format->impl, NULL, NULL, flags);
+ if (status != AMEDIA_OK) {
+ av_log(codec, AV_LOG_ERROR, "Encoder configure failed, %d\n", status);
+ return AVERROR_EXTERNAL;
+ }
+
+ if (!native_window)
+ return 0;
+
+ status = codec->setInputSurface(codec->impl, native_window);
+ if (status != AMEDIA_OK) {
+ av_log(codec, AV_LOG_ERROR, "Encoder set input surface failed, %d\n", status);
+ return AVERROR_EXTERNAL;
+ }
+ } else {
+ status = codec->configure(codec->impl, format->impl, native_window, NULL, flags);
+ if (status != AMEDIA_OK) {
+ av_log(codec, AV_LOG_ERROR, "Decoder configure failed, %d\n", status);
+ return AVERROR_EXTERNAL;
+ }
}
return 0;
@@ -2330,6 +2401,26 @@ static int mediacodec_ndk_cleanOutputBuffers(FFAMediaCodec *ctx)
return 0;
}
+static int mediacodec_ndk_signalEndOfInputStream(FFAMediaCodec *ctx)
+{
+ FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
+ media_status_t status;
+
+ if (!codec->signalEndOfInputStream) {
+ av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream unavailable\n");
+ return AVERROR_EXTERNAL;
+ }
+
+ status = codec->signalEndOfInputStream(codec->impl);
+ if (status != AMEDIA_OK) {
+ av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream failed, %d\n", status);
+ return AVERROR_EXTERNAL;
+ }
+ av_log(codec, AV_LOG_DEBUG, "signalEndOfInputStream success\n");
+
+ return 0;
+}
+
static const FFAMediaFormat media_format_ndk = {
.class = &amediaformat_ndk_class,
@@ -2388,6 +2479,7 @@ static const FFAMediaCodec media_codec_ndk = {
.getConfigureFlagEncode = mediacodec_ndk_getConfigureFlagEncode,
.cleanOutputBuffers = mediacodec_ndk_cleanOutputBuffers,
+ .signalEndOfInputStream = mediacodec_ndk_signalEndOfInputStream,
};
FFAMediaFormat *ff_AMediaFormat_new(int ndk)
diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h
index 7cf3f4aecd..f15ad66d83 100644
--- a/libavcodec/mediacodec_wrapper.h
+++ b/libavcodec/mediacodec_wrapper.h
@@ -192,6 +192,9 @@ struct FFAMediaCodec {
int (*getConfigureFlagEncode)(FFAMediaCodec *codec);
int (*cleanOutputBuffers)(FFAMediaCodec *codec);
+
+ // For encoder with FFANativeWindow as input.
+ int (*signalEndOfInputStream)(FFAMediaCodec *);
};
static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec)
@@ -311,6 +314,11 @@ static inline int ff_AMediaCodec_cleanOutputBuffers(FFAMediaCodec *codec)
return codec->cleanOutputBuffers(codec);
}
+static inline int ff_AMediaCodec_signalEndOfInputStream(FFAMediaCodec *codec)
+{
+ return codec->signalEndOfInputStream(codec);
+}
+
int ff_Build_SDK_INT(AVCodecContext *avctx);
#endif /* AVCODEC_MEDIACODEC_WRAPPER_H */
diff --git a/libavcodec/mediacodecenc.c b/libavcodec/mediacodecenc.c
new file mode 100644
index 0000000000..c81050ec80
--- /dev/null
+++ b/libavcodec/mediacodecenc.c
@@ -0,0 +1,495 @@
+/*
+ * Android MediaCodec encoders
+ *
+ * Copyright (c) 2022 Zhao Zhili <zhilizhao@tencent.com>
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include "config_components.h"
+
+#include "libavutil/avassert.h"
+#include "libavutil/hwcontext_mediacodec.h"
+#include "libavutil/imgutils.h"
+#include "libavutil/opt.h"
+
+#include "avcodec.h"
+#include "codec_internal.h"
+#include "encode.h"
+#include "hwconfig.h"
+#include "jni.h"
+#include "mediacodec.h"
+#include "mediacodec_wrapper.h"
+#include "mediacodecdec_common.h"
+
+#define INPUT_DEQUEUE_TIMEOUT_US 8000
+#define OUTPUT_DEQUEUE_TIMEOUT_US 8000
+
+typedef struct MediaCodecEncContext {
+ AVClass *avclass;
+ FFAMediaCodec *codec;
+ int use_ndk_codec;
+ FFANativeWindow *window;
+
+ int fps;
+ int width;
+ int height;
+
+ uint8_t *extradata;
+ int extradata_size;
+
+ // Since MediaCodec doesn't output DTS, use a timestamp queue to save pts
+ // of AVFrame and generate DTS for AVPacket.
+ //
+ // This doesn't work when use Surface as input, in that case frames can be
+ // sent to encoder without our notice. One exception is frames come from
+ // our MediaCodec decoder wrapper, since we can control it's render by
+ // av_mediacodec_release_buffer.
+ int64_t timestamps[32];
+ int ts_head;
+ int ts_tail;
+
+ int eof_sent;
+
+ AVFrame *frame;
+} MediaCodecEncContext;
+
+enum {
+ COLOR_FormatYUV420Planar = 0x13,
+ COLOR_FormatYUV420SemiPlanar = 0x15,
+ COLOR_FormatSurface = 0x7F000789,
+};
+
+static const struct {
+ int color_format;
+ enum AVPixelFormat pix_fmt;
+} color_formats[] = {
+ { COLOR_FormatYUV420Planar, AV_PIX_FMT_YUV420P },
+ { COLOR_FormatYUV420SemiPlanar, AV_PIX_FMT_NV12 },
+ { COLOR_FormatSurface, AV_PIX_FMT_MEDIACODEC },
+};
+
+static const enum AVPixelFormat avc_pix_fmts[] = {
+ AV_PIX_FMT_MEDIACODEC,
+ AV_PIX_FMT_YUV420P,
+ AV_PIX_FMT_NV12,
+ AV_PIX_FMT_NONE
+};
+
+static void mediacodec_output_format(AVCodecContext *avctx)
+{
+ MediaCodecEncContext *s = avctx->priv_data;
+ char *name = ff_AMediaCodec_getName(s->codec);
+ FFAMediaFormat *out_format = ff_AMediaCodec_getOutputFormat(s->codec);
+ char *str = ff_AMediaFormat_toString(out_format);
+
+ av_log(avctx, AV_LOG_DEBUG, "MediaCodec encoder %s output format %s\n",
+ name ? name : "unknown", str);
+ av_free(name);
+ av_free(str);
+ ff_AMediaFormat_delete(out_format);
+}
+
+static av_cold int mediacodec_init(AVCodecContext *avctx)
+{
+ const char *codec_mime = NULL;
+ MediaCodecEncContext *s = avctx->priv_data;
+ FFAMediaFormat *format = NULL;
+ int ret;
+ int gop;
+
+ if (s->use_ndk_codec < 0)
+ s->use_ndk_codec = !av_jni_get_java_vm(avctx);
+
+ switch (avctx->codec_id) {
+ case AV_CODEC_ID_H264:
+ codec_mime = "video/avc";
+ break;
+ case AV_CODEC_ID_HEVC:
+ codec_mime = "video/hevc";
+ break;
+ default:
+ av_assert0(0);
+ }
+
+ s->codec = ff_AMediaCodec_createEncoderByType(codec_mime, s->use_ndk_codec);
+ if (!s->codec) {
+ av_log(avctx, AV_LOG_ERROR, "Failed to create encoder for type %s\n",
+ codec_mime);
+ return AVERROR_EXTERNAL;
+ }
+
+ format = ff_AMediaFormat_new(s->use_ndk_codec);
+ if (!format) {
+ av_log(avctx, AV_LOG_ERROR, "Failed to create media format\n");
+ return AVERROR_EXTERNAL;
+ }
+
+ ff_AMediaFormat_setString(format, "mime", codec_mime);
+ s->width = FFALIGN(avctx->width, 16);
+ s->height = avctx->height;
+ ff_AMediaFormat_setInt32(format, "width", s->width);
+ ff_AMediaFormat_setInt32(format, "height", s->height);
+
+ if (avctx->pix_fmt == AV_PIX_FMT_MEDIACODEC) {
+ AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
+ if (avctx->hw_device_ctx) {
+ AVHWDeviceContext *device_ctx = (AVHWDeviceContext*)(avctx->hw_device_ctx->data);
+ AVMediaCodecDeviceContext *dev_ctx;
+
+ if (device_ctx->type != AV_HWDEVICE_TYPE_MEDIACODEC || !device_ctx->hwctx) {
+ ret = AVERROR(EINVAL);
+ goto bailout;
+ }
+ dev_ctx = device_ctx->hwctx;
+ s->window = ff_mediacodec_surface_ref(dev_ctx->surface, dev_ctx->native_window, avctx);
+ }
+
+ if (!s->window && user_ctx && user_ctx->surface)
+ s->window = ff_mediacodec_surface_ref(user_ctx->surface, NULL, avctx);
+
+ if (!s->window) {
+ ret = AVERROR(EINVAL);
+ av_log(avctx, AV_LOG_ERROR, "Missing hw_device_ctx or hwaccel_context for AV_PIX_FMT_MEDIACODEC\n");
+ goto bailout;
+ }
+ }
+
+ for (int i = 0; i < FF_ARRAY_ELEMS(color_formats); i++) {
+ if (avctx->pix_fmt == color_formats[i].pix_fmt) {
+ ff_AMediaFormat_setInt32(format, "color-format",
+ color_formats[i].color_format);
+ break;
+ }
+ }
+
+ if (avctx->bit_rate)
+ ff_AMediaFormat_setInt32(format, "bitrate", avctx->bit_rate);
+ // frame-rate and i-frame-interval are required to configure codec
+ if (avctx->framerate.num >= avctx->framerate.den && avctx->framerate.den > 0)
+ s->fps = avctx->framerate.num / avctx->framerate.den;
+ else
+ s->fps = 30;
+ gop = round(avctx->gop_size / s->fps);
+ if (gop == 0)
+ gop = 2;
+ ff_AMediaFormat_setInt32(format, "frame-rate", s->fps);
+ ff_AMediaFormat_setInt32(format, "i-frame-interval", gop);
+
+
+ ret = ff_AMediaCodec_getConfigureFlagEncode(s->codec);
+ ret = ff_AMediaCodec_configure(s->codec, format, s->window, NULL, ret);
+ if (ret) {
+ av_log(avctx, AV_LOG_ERROR, "MediaCodec configure failed, %s\n", av_err2str(ret));
+ goto bailout;
+ }
+
+ ret = ff_AMediaCodec_start(s->codec);
+ if (ret) {
+ av_log(avctx, AV_LOG_ERROR, "MediaCodec failed to start, %s\n", av_err2str(ret));
+ goto bailout;
+ }
+
+ mediacodec_output_format(avctx);
+
+ s->frame = av_frame_alloc();
+ if (!s->frame)
+ ret = AVERROR(ENOMEM);
+
+bailout:
+ if (format)
+ ff_AMediaFormat_delete(format);
+ return ret;
+}
+
+static int mediacodec_receive(AVCodecContext *avctx,
+ AVPacket *pkt,
+ int *got_packet)
+{
+ MediaCodecEncContext *s = avctx->priv_data;
+ FFAMediaCodec *codec = s->codec;
+ FFAMediaCodecBufferInfo out_info = {0};
+ uint8_t *out_buf;
+ size_t out_size = 0;
+ int ret;
+ int extradata_size = 0;
+ int64_t timeout_us = s->eof_sent ? OUTPUT_DEQUEUE_TIMEOUT_US : 0;
+ ssize_t index = ff_AMediaCodec_dequeueOutputBuffer(codec, &out_info, timeout_us);
+
+ if (ff_AMediaCodec_infoTryAgainLater(codec, index))
+ return AVERROR(EAGAIN);
+
+ if (ff_AMediaCodec_infoOutputFormatChanged(codec, index)) {
+ mediacodec_output_format(avctx);
+ return AVERROR(EAGAIN);
+ }
+
+ if (ff_AMediaCodec_infoOutputBuffersChanged(codec, index)) {
+ ff_AMediaCodec_cleanOutputBuffers(codec);
+ return AVERROR(EAGAIN);
+ }
+
+ if (index < 0)
+ return AVERROR_EXTERNAL;
+
+ if (out_info.flags & ff_AMediaCodec_getBufferFlagEndOfStream(codec))
+ return AVERROR_EOF;
+
+ out_buf = ff_AMediaCodec_getOutputBuffer(codec, index, &out_size);
+ if (!out_buf) {
+ ret = AVERROR_EXTERNAL;
+ goto bailout;
+ }
+
+ if (out_info.flags & ff_AMediaCodec_getBufferFlagCodecConfig(codec)) {
+ ret = av_reallocp(&s->extradata, out_info.size);
+ if (ret)
+ goto bailout;
+
+ s->extradata_size = out_info.size;
+ memcpy(s->extradata, out_buf + out_info.offset, out_info.size);
+ ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
+ // try immediately
+ return mediacodec_receive(avctx, pkt, got_packet);
+ }
+
+ ret = ff_get_encode_buffer(avctx, pkt, out_info.size + s->extradata_size, 0);
+ if (ret < 0)
+ goto bailout;
+
+ if (s->extradata_size) {
+ extradata_size = s->extradata_size;
+ s->extradata_size = 0;
+ memcpy(pkt->data, s->extradata, extradata_size);
+ }
+ memcpy(pkt->data + extradata_size, out_buf + out_info.offset, out_info.size);
+ pkt->pts = av_rescale_q(out_info.presentationTimeUs, AV_TIME_BASE_Q, avctx->time_base);
+ if (s->ts_tail != s->ts_head) {
+ pkt->dts = s->timestamps[s->ts_tail];
+ s->ts_tail = (s->ts_tail + 1) % FF_ARRAY_ELEMS(s->timestamps);
+ }
+
+ if (out_info.flags & ff_AMediaCodec_getBufferFlagKeyFrame(codec))
+ pkt->flags |= AV_PKT_FLAG_KEY;
+ ret = 0;
+ *got_packet = 1;
+
+ av_log(avctx, AV_LOG_TRACE, "receive packet pts %" PRId64 " dts %" PRId64
+ " flags %d extradata %d\n",
+ pkt->pts, pkt->dts, pkt->flags, extradata_size);
+
+bailout:
+ ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
+ return ret;
+}
+
+static void copy_frame_to_buffer(AVCodecContext *avctx, const AVFrame *frame, uint8_t *dst, size_t size)
+{
+ MediaCodecEncContext *s = avctx->priv_data;
+ uint8_t *dst_data[4] = {};
+ int dst_linesize[4] = {};
+ const uint8_t *src_data[4] = {
+ frame->data[0], frame->data[1], frame->data[2], frame->data[3]
+ };
+
+ if (avctx->pix_fmt == AV_PIX_FMT_YUV420P) {
+ dst_data[0] = dst;
+ dst_data[1] = dst + s->width * s->height;
+ dst_data[2] = dst_data[1] + s->width * s->height / 4;
+
+ dst_linesize[0] = s->width;
+ dst_linesize[1] = dst_linesize[2] = s->width / 2;
+ } else if (avctx->pix_fmt == AV_PIX_FMT_NV12) {
+ dst_data[0] = dst;
+ dst_data[1] = dst + s->width * s->height;
+
+ dst_linesize[0] = s->width;
+ dst_linesize[1] = s->width;
+ } else {
+ av_assert0(0);
+ }
+
+ av_image_copy(dst_data, dst_linesize, src_data, frame->linesize,
+ avctx->pix_fmt, avctx->width, avctx->height);
+}
+
+static int mediacodec_send(AVCodecContext *avctx,
+ const AVFrame *frame) {
+ MediaCodecEncContext *s = avctx->priv_data;
+ FFAMediaCodec *codec = s->codec;
+ ssize_t index;
+ uint8_t *input_buf = NULL;
+ size_t input_size = 0;
+ int64_t pts = 0;
+ uint32_t flags = 0;
+ int64_t timeout_us;
+
+ if (s->eof_sent)
+ return 0;
+
+ if (s->window) {
+ if (!frame) {
+ s->eof_sent = 1;
+ return ff_AMediaCodec_signalEndOfInputStream(codec);
+ }
+
+
+ if (frame->data[3]) {
+ pts = av_rescale_q(frame->pts, avctx->time_base, AV_TIME_BASE_Q);
+ s->timestamps[s->ts_head] = frame->pts;
+ s->ts_head = (s->ts_head + 1) % FF_ARRAY_ELEMS(s->timestamps);
+
+ av_mediacodec_release_buffer((AVMediaCodecBuffer *)frame->data[3], 1);
+ }
+ return 0;
+ }
+
+ timeout_us = INPUT_DEQUEUE_TIMEOUT_US;
+ index = ff_AMediaCodec_dequeueInputBuffer(codec, timeout_us);
+ if (ff_AMediaCodec_infoTryAgainLater(codec, index))
+ return AVERROR(EAGAIN);
+
+ if (index < 0) {
+ av_log(avctx, AV_LOG_ERROR, "dequeue input buffer failed, %zd", index);
+ return AVERROR_EXTERNAL;
+ }
+
+ if (frame) {
+ input_buf = ff_AMediaCodec_getInputBuffer(codec, index, &input_size);
+ copy_frame_to_buffer(avctx, frame, input_buf, input_size);
+
+ pts = av_rescale_q(frame->pts, avctx->time_base, AV_TIME_BASE_Q);
+
+ s->timestamps[s->ts_head] = frame->pts;
+ s->ts_head = (s->ts_head + 1) % FF_ARRAY_ELEMS(s->timestamps);
+ } else {
+ flags |= ff_AMediaCodec_getBufferFlagEndOfStream(codec);
+ s->eof_sent = 1;
+ }
+
+ ff_AMediaCodec_queueInputBuffer(codec, index, 0, input_size, pts, flags);
+ return 0;
+}
+
+static int mediacodec_encode(AVCodecContext *avctx, AVPacket *pkt)
+{
+ MediaCodecEncContext *s = avctx->priv_data;
+ int ret;
+ int got_packet = 0;
+
+ // Return on three case:
+ // 1. Serious error
+ // 2. Got a packet success
+ // 3. No AVFrame is available yet (don't return if get_frame return EOF)
+ while (1) {
+ ret = mediacodec_receive(avctx, pkt, &got_packet);
+ if (!ret)
+ return 0;
+ else if (ret != AVERROR(EAGAIN))
+ return ret;
+
+ if (!s->frame->buf[0]) {
+ ret = ff_encode_get_frame(avctx, s->frame);
+ if (ret && ret != AVERROR_EOF)
+ return ret;
+ }
+
+ ret = mediacodec_send(avctx, s->frame->buf[0] ? s->frame : NULL);
+ if (!ret)
+ av_frame_unref(s->frame);
+ else if (ret != AVERROR(EAGAIN))
+ return ret;
+ }
+
+ return 0;
+}
+
+static av_cold int mediacodec_close(AVCodecContext *avctx)
+{
+ MediaCodecEncContext *s = avctx->priv_data;
+ if (s->codec) {
+ ff_AMediaCodec_stop(s->codec);
+ ff_AMediaCodec_delete(s->codec);
+ s->codec = NULL;
+ }
+
+ if (s->window) {
+ ff_mediacodec_surface_unref(s->window, avctx);
+ s->window = NULL;
+ }
+
+ av_frame_free(&s->frame);
+
+ return 0;
+}
+
+static const AVCodecHWConfigInternal *const mediacodec_hw_configs[] = {
+ &(const AVCodecHWConfigInternal) {
+ .public = {
+ .pix_fmt = AV_PIX_FMT_MEDIACODEC,
+ .methods = AV_CODEC_HW_CONFIG_METHOD_AD_HOC |
+ AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX,
+ .device_type = AV_HWDEVICE_TYPE_MEDIACODEC,
+ },
+ .hwaccel = NULL,
+ },
+ NULL
+};
+
+#define OFFSET(x) offsetof(MediaCodecEncContext, x)
+#define VE AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM
+static const AVOption common_options[] = {
+ { "ndk_codec", "Use MediaCodec from NDK",
+ OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 = -1}, -1, 1, VE },
+ { NULL },
+};
+
+#define MEDIACODEC_ENCODER_CLASS(name) \
+static const AVClass name ## _mediacodec_class = { \
+ .class_name = #name "_mediacodec", \
+ .item_name = av_default_item_name, \
+ .option = common_options, \
+ .version = LIBAVUTIL_VERSION_INT, \
+}; \
+
+#define DECLARE_MEDIACODEC_ENCODER(short_name, long_name, codec_id) \
+MEDIACODEC_ENCODER_CLASS(short_name) \
+const FFCodec ff_ ## short_name ## _mediacodec_encoder = { \
+ .p.name = #short_name "_mediacodec", \
+ CODEC_LONG_NAME(long_name " Android MediaCodec encoder"), \
+ .p.type = AVMEDIA_TYPE_VIDEO, \
+ .p.id = codec_id, \
+ .p.capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY \
+ | AV_CODEC_CAP_HARDWARE, \
+ .priv_data_size = sizeof(MediaCodecEncContext), \
+ .p.pix_fmts = avc_pix_fmts, \
+ .init = mediacodec_init, \
+ FF_CODEC_RECEIVE_PACKET_CB(mediacodec_encode), \
+ .close = mediacodec_close, \
+ .p.priv_class = &short_name ## _mediacodec_class, \
+ .caps_internal = FF_CODEC_CAP_INIT_CLEANUP, \
+ .p.wrapper_name = "mediacodec", \
+ .hw_configs = mediacodec_hw_configs, \
+}; \
+
+#if CONFIG_H264_MEDIACODEC_ENCODER
+DECLARE_MEDIACODEC_ENCODER(h264, "H.264", AV_CODEC_ID_H264)
+#endif
+
+#if CONFIG_HEVC_MEDIACODEC_ENCODER
+DECLARE_MEDIACODEC_ENCODER(hevc, "H.265", AV_CODEC_ID_HEVC)
+#endif
diff --git a/libavcodec/version.h b/libavcodec/version.h
index 43d0d9a9fc..86ac0f3871 100644
--- a/libavcodec/version.h
+++ b/libavcodec/version.h
@@ -29,8 +29,8 @@
#include "version_major.h"
-#define LIBAVCODEC_VERSION_MINOR 51
-#define LIBAVCODEC_VERSION_MICRO 101
+#define LIBAVCODEC_VERSION_MINOR 52
+#define LIBAVCODEC_VERSION_MICRO 100
#define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \
LIBAVCODEC_VERSION_MINOR, \
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder Zhao Zhili
@ 2022-10-26 21:17 ` Tomas Härdin
2022-10-27 2:49 ` "zhilizhao(赵志立)"
[not found] ` <CAK=uwuxEwO98S2js5fnY_T1gf1epz+WRBiWyk86qM6oQmAMMMQ@mail.gmail.com>
1 sibling, 1 reply; 22+ messages in thread
From: Tomas Härdin @ 2022-10-26 21:17 UTC (permalink / raw)
To: FFmpeg development discussions and patches
mån 2022-10-24 klockan 11:16 +0800 skrev Zhao Zhili:
>
> +typedef struct MediaCodecEncContext {
> + AVClass *avclass;
> + FFAMediaCodec *codec;
> + int use_ndk_codec;
> + FFANativeWindow *window;
> +
> + int fps;
> + int width;
> + int height;
> +
> + uint8_t *extradata;
> + int extradata_size;
Why not extradata in AVCodecContext?
Some tests would be nice
/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder
2022-10-26 21:17 ` Tomas Härdin
@ 2022-10-27 2:49 ` "zhilizhao(赵志立)"
2022-10-27 9:56 ` Tomas Härdin
0 siblings, 1 reply; 22+ messages in thread
From: "zhilizhao(赵志立)" @ 2022-10-27 2:49 UTC (permalink / raw)
To: FFmpeg development discussions and patches
> On Oct 27, 2022, at 05:17, Tomas Härdin <git@haerdin.se> wrote:
>
> mån 2022-10-24 klockan 11:16 +0800 skrev Zhao Zhili:
>>
>> +typedef struct MediaCodecEncContext {
>> + AVClass *avclass;
>> + FFAMediaCodec *codec;
>> + int use_ndk_codec;
>> + FFANativeWindow *window;
>> +
>> + int fps;
>> + int width;
>> + int height;
>> +
>> + uint8_t *extradata;
>> + int extradata_size;
>
> Why not extradata in AVCodecContext?
The extradata (BUFFER_FLAG_CODEC_CONFIG) is popped after codec
init, I’m not sure if it’s OK to touch AVCodecContext->extradata
after code init.
Secondly, it isn’t specified in Android doc, but better be safe
to handle the case of BUFFER_FLAG_CODEC_CONFIG show up multiple
times.
So BUFFER_FLAG_CODEC_CONFIG is handled as in-band data, without
touch AVCodecContext->extradata too late or multiple times. Did
I overthinking or misunderstood something?
>
> Some tests would be nice
Did you mean fate test or manual test?
It’s an external codec wrapper, so maybe not in fate.
For NDK MediaCodec wrapper, good news is we can test it with ffmpeg
cmd now. For Java wrapper, it was tested by changing fftool/ffmepg
into a lib: ugly but simple.
https://github.com/quink-black/ffmpeg-ci/tree/master/android
Any suggestion is welcome.
>
> /Tomas
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder
2022-10-27 2:49 ` "zhilizhao(赵志立)"
@ 2022-10-27 9:56 ` Tomas Härdin
2022-10-27 12:41 ` "zhilizhao(赵志立)"
0 siblings, 1 reply; 22+ messages in thread
From: Tomas Härdin @ 2022-10-27 9:56 UTC (permalink / raw)
To: FFmpeg development discussions and patches
tor 2022-10-27 klockan 10:49 +0800 skrev zhilizhao(赵志立):
>
>
> > On Oct 27, 2022, at 05:17, Tomas Härdin <git@haerdin.se> wrote:
> >
> > mån 2022-10-24 klockan 11:16 +0800 skrev Zhao Zhili:
> > >
> > > +typedef struct MediaCodecEncContext {
> > > + AVClass *avclass;
> > > + FFAMediaCodec *codec;
> > > + int use_ndk_codec;
> > > + FFANativeWindow *window;
> > > +
> > > + int fps;
> > > + int width;
> > > + int height;
> > > +
> > > + uint8_t *extradata;
> > > + int extradata_size;
> >
> > Why not extradata in AVCodecContext?
>
> The extradata (BUFFER_FLAG_CODEC_CONFIG) is popped after codec
> init, I’m not sure if it’s OK to touch AVCodecContext->extradata
> after code init.
You mean that it isn't populated until after a frame has been encoded?
There's no way to specify resolution, pixel format etc to get the
extradata without encoding?
> Secondly, it isn’t specified in Android doc, but better be safe
> to handle the case of BUFFER_FLAG_CODEC_CONFIG show up multiple
> times.
Surely there's a way to signal this? I suppose one can always add a bsf
after the encoder. I'm actually not sure.
>
> > Some tests would be nice
>
> Did you mean fate test or manual test?
>
> It’s an external codec wrapper, so maybe not in fate.
It would be nice if FATE could test this on machines where the hardware
is available. We had a brief discussion on IRC about this yesterday.
Just checking which encoders actually work, what profiles are supported
and so on. Maybe tabulate it somewhere. Sometimes manufacturers specify
what codecs and profiles are supported, sometimes not.
/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder
2022-10-27 9:56 ` Tomas Härdin
@ 2022-10-27 12:41 ` "zhilizhao(赵志立)"
0 siblings, 0 replies; 22+ messages in thread
From: "zhilizhao(赵志立)" @ 2022-10-27 12:41 UTC (permalink / raw)
To: FFmpeg development discussions and patches
> On Oct 27, 2022, at 17:56, Tomas Härdin <git@haerdin.se> wrote:
>
> tor 2022-10-27 klockan 10:49 +0800 skrev zhilizhao(赵志立):
>>
>>
>>> On Oct 27, 2022, at 05:17, Tomas Härdin <git@haerdin.se> wrote:
>>>
>>> mån 2022-10-24 klockan 11:16 +0800 skrev Zhao Zhili:
>>>>
>>>> +typedef struct MediaCodecEncContext {
>>>> + AVClass *avclass;
>>>> + FFAMediaCodec *codec;
>>>> + int use_ndk_codec;
>>>> + FFANativeWindow *window;
>>>> +
>>>> + int fps;
>>>> + int width;
>>>> + int height;
>>>> +
>>>> + uint8_t *extradata;
>>>> + int extradata_size;
>>>
>>> Why not extradata in AVCodecContext?
>>
>> The extradata (BUFFER_FLAG_CODEC_CONFIG) is popped after codec
>> init, I’m not sure if it’s OK to touch AVCodecContext->extradata
>> after code init.
>
> You mean that it isn't populated until after a frame has been encoded?
> There's no way to specify resolution, pixel format etc to get the
> extradata without encoding?
That’s the case from my test on Google Pixel 3. There is no ‘csd’ inside
MediaFormat returned by ff_AMediaCodec_getOutputFormat(), until a frame
has been sent. It may depends on the device and OS.
>
>> Secondly, it isn’t specified in Android doc, but better be safe
>> to handle the case of BUFFER_FLAG_CODEC_CONFIG show up multiple
>> times.
>
> Surely there's a way to signal this? I suppose one can always add a bsf
> after the encoder. I'm actually not sure.
We can use AV_PKT_DATA_NEW_EXTRADATA to signal that along with
AVCodecContext->extradata. We still need to save extradata to a place
other than AVCodecContext->extradata temporarily.
>>
>
>>> Some tests would be nice
>>
>> Did you mean fate test or manual test?
>>
>> It’s an external codec wrapper, so maybe not in fate.
>
> It would be nice if FATE could test this on machines where the hardware
> is available. We had a brief discussion on IRC about this yesterday.
> Just checking which encoders actually work, what profiles are supported
> and so on. Maybe tabulate it somewhere. Sometimes manufacturers specify
> what codecs and profiles are supported, sometimes not.
I’m planing to add more options for codec configure and select codec by
name. Run FATE on Android is doable.
>
> /Tomas
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread
[parent not found: <CAK=uwuxEwO98S2js5fnY_T1gf1epz+WRBiWyk86qM6oQmAMMMQ@mail.gmail.com>]
* Re: [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder
[not found] ` <CAK=uwuxEwO98S2js5fnY_T1gf1epz+WRBiWyk86qM6oQmAMMMQ@mail.gmail.com>
@ 2022-11-19 17:13 ` "zhilizhao(赵志立)"
2022-11-19 18:14 ` Olivier Ayache
2022-11-20 23:44 ` Aman Karmani
0 siblings, 2 replies; 22+ messages in thread
From: "zhilizhao(赵志立)" @ 2022-11-19 17:13 UTC (permalink / raw)
To: FFmpeg development discussions and patches; +Cc: Aman Karmani
> On Nov 19, 2022, at 17:15, Aman Karmani <aman@tmm1.net> wrote:
>
>
>
> On Sun, Oct 23, 2022 at 8:17 PM Zhao Zhili <quinkblack@foxmail.com> wrote:
> From: Zhao Zhili <zhilizhao@tencent.com>
>
> Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
> ---
> Changelog | 1 +
> configure | 4 +
> libavcodec/Makefile | 2 +
> libavcodec/allcodecs.c | 2 +
> libavcodec/mediacodec_wrapper.c | 102 ++++++-
> libavcodec/mediacodec_wrapper.h | 8 +
> libavcodec/mediacodecenc.c | 495 ++++++++++++++++++++++++++++++++
> libavcodec/version.h | 4 +-
> 8 files changed, 611 insertions(+), 7 deletions(-)
> create mode 100644 libavcodec/mediacodecenc.c
>
> diff --git a/Changelog b/Changelog
> index 9e203833aa..9e39a35972 100644
> --- a/Changelog
> +++ b/Changelog
> @@ -19,6 +19,7 @@ version <next>:
> - DTS to PTS reorder bsf
> - ViewQuest VQC decoder
> - MediaCodec decoder via NDKMediaCodec
> +- MediaCodec encoder
>
>
>
> version 5.1:
> diff --git a/configure b/configure
> index ee2e3ba6ac..5114cda13f 100755
> --- a/configure
> +++ b/configure
> @@ -3193,6 +3193,8 @@ h264_cuvid_decoder_select="h264_mp4toannexb_bsf"
> h264_mediacodec_decoder_deps="mediacodec"
> h264_mediacodec_decoder_extralibs="-landroid"
> h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser"
> +h264_mediacodec_encoder_deps="mediacodec"
> +h264_mediacodec_encoder_extralibs="-landroid"
> h264_mf_encoder_deps="mediafoundation"
> h264_mmal_decoder_deps="mmal"
> h264_nvenc_encoder_deps="nvenc"
> @@ -3212,6 +3214,8 @@ hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf"
> hevc_mediacodec_decoder_deps="mediacodec"
> hevc_mediacodec_decoder_extralibs="-landroid"
> hevc_mediacodec_decoder_select="hevc_mp4toannexb_bsf hevc_parser"
> +hevc_mediacodec_encoder_deps="mediacodec"
> +hevc_mediacodec_encoder_extralibs="-landroid"
> hevc_mf_encoder_deps="mediafoundation"
> hevc_nvenc_encoder_deps="nvenc"
> hevc_nvenc_encoder_select="atsc_a53"
> diff --git a/libavcodec/Makefile b/libavcodec/Makefile
> index 90c7f113a3..7d0b513eec 100644
> --- a/libavcodec/Makefile
> +++ b/libavcodec/Makefile
> @@ -393,6 +393,7 @@ OBJS-$(CONFIG_H264_DECODER) += h264dec.o h264_cabac.o h264_cavlc.o \
> OBJS-$(CONFIG_H264_AMF_ENCODER) += amfenc_h264.o
> OBJS-$(CONFIG_H264_CUVID_DECODER) += cuviddec.o
> OBJS-$(CONFIG_H264_MEDIACODEC_DECODER) += mediacodecdec.o
> +OBJS-$(CONFIG_H264_MEDIACODEC_ENCODER) += mediacodecenc.o
> OBJS-$(CONFIG_H264_MF_ENCODER) += mfenc.o mf_utils.o
> OBJS-$(CONFIG_H264_MMAL_DECODER) += mmaldec.o
> OBJS-$(CONFIG_H264_NVENC_ENCODER) += nvenc_h264.o nvenc.o
> @@ -417,6 +418,7 @@ OBJS-$(CONFIG_HEVC_DECODER) += hevcdec.o hevc_mvs.o \
> OBJS-$(CONFIG_HEVC_AMF_ENCODER) += amfenc_hevc.o
> OBJS-$(CONFIG_HEVC_CUVID_DECODER) += cuviddec.o
> OBJS-$(CONFIG_HEVC_MEDIACODEC_DECODER) += mediacodecdec.o
> +OBJS-$(CONFIG_HEVC_MEDIACODEC_ENCODER) += mediacodecenc.o
> OBJS-$(CONFIG_HEVC_MF_ENCODER) += mfenc.o mf_utils.o
> OBJS-$(CONFIG_HEVC_NVENC_ENCODER) += nvenc_hevc.o nvenc.o
> OBJS-$(CONFIG_HEVC_QSV_DECODER) += qsvdec.o
> diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
> index 46ad3b5a25..4c33a9ec3c 100644
> --- a/libavcodec/allcodecs.c
> +++ b/libavcodec/allcodecs.c
> @@ -154,6 +154,7 @@ extern const FFCodec ff_h264_decoder;
> extern const FFCodec ff_h264_crystalhd_decoder;
> extern const FFCodec ff_h264_v4l2m2m_decoder;
> extern const FFCodec ff_h264_mediacodec_decoder;
> +extern const FFCodec ff_h264_mediacodec_encoder;
> extern const FFCodec ff_h264_mmal_decoder;
> extern const FFCodec ff_h264_qsv_decoder;
> extern const FFCodec ff_h264_rkmpp_decoder;
> @@ -842,6 +843,7 @@ extern const FFCodec ff_h264_videotoolbox_encoder;
> extern const FFCodec ff_hevc_amf_encoder;
> extern const FFCodec ff_hevc_cuvid_decoder;
> extern const FFCodec ff_hevc_mediacodec_decoder;
> +extern const FFCodec ff_hevc_mediacodec_encoder;
> extern const FFCodec ff_hevc_mf_encoder;
> extern const FFCodec ff_hevc_nvenc_encoder;
> extern const FFCodec ff_hevc_qsv_encoder;
> diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c
> index 284d615980..5d1a32031d 100644
> --- a/libavcodec/mediacodec_wrapper.c
> +++ b/libavcodec/mediacodec_wrapper.c
> @@ -212,6 +212,9 @@ struct JNIAMediaCodecFields {
> jmethodID release_output_buffer_id;
> jmethodID release_output_buffer_at_time_id;
>
> + jmethodID set_input_surface_id;
> + jmethodID signal_end_of_input_stream_id;
> +
> jclass mediainfo_class;
>
> jmethodID init_id;
> @@ -261,6 +264,9 @@ static const struct FFJniField jni_amediacodec_mapping[] = {
> { "android/media/MediaCodec", "releaseOutputBuffer", "(IZ)V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, release_output_buffer_id), 1 },
> { "android/media/MediaCodec", "releaseOutputBuffer", "(IJ)V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, release_output_buffer_at_time_id), 0 },
>
> + { "android/media/MediaCodec", "setInputSurface", "(Landroid/view/Surface;)V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, set_input_surface_id), 0 },
> + { "android/media/MediaCodec", "signalEndOfInputStream", "()V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, signal_end_of_input_stream_id), 0 },
> +
> { "android/media/MediaCodec$BufferInfo", NULL, NULL, FF_JNI_CLASS, offsetof(struct JNIAMediaCodecFields, mediainfo_class), 1 },
>
> { "android/media/MediaCodec.BufferInfo", "<init>", "()V", FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, init_id), 1 },
> @@ -1385,7 +1391,26 @@ static int mediacodec_jni_configure(FFAMediaCodec *ctx,
>
> JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
>
> - (*env)->CallVoidMethod(env, codec->object, codec->jfields.configure_id, format->object, surface, NULL, flags);
> + if (flags & codec->CONFIGURE_FLAG_ENCODE) {
> + if (surface && !codec->jfields.set_input_surface_id) {
> + av_log(ctx, AV_LOG_ERROR, "System doesn't support setInputSurface\n");
> + return AVERROR_EXTERNAL;
> + }
> +
> + (*env)->CallVoidMethod(env, codec->object, codec->jfields.configure_id, format->object, NULL, NULL, flags);
> + if (ff_jni_exception_check(env, 1, codec) < 0)
> + return AVERROR_EXTERNAL;
> +
> + if (!surface)
> + return 0;
> +
> + (*env)->CallVoidMethod(env, codec->object, codec->jfields.set_input_surface_id, surface);
> + if (ff_jni_exception_check(env, 1, codec) < 0)
> + return AVERROR_EXTERNAL;
> + return 0;
> + } else {
> + (*env)->CallVoidMethod(env, codec->object, codec->jfields.configure_id, format->object, surface, NULL, flags);
> + }
> if (ff_jni_exception_check(env, 1, codec) < 0) {
> ret = AVERROR_EXTERNAL;
> goto fail;
> @@ -1743,6 +1768,22 @@ fail:
> return ret;
> }
>
> +static int mediacodec_jni_signalEndOfInputStream(FFAMediaCodec *ctx)
> +{
> + int ret = 0;
> + JNIEnv *env = NULL;
> + FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
> +
> + JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
> +
> + (*env)->CallVoidMethod(env, codec->object, codec->jfields.signal_end_of_input_stream_id);
> + if (ff_jni_exception_check(env, 1, codec) < 0) {
> + return AVERROR_EXTERNAL;
> + }
> +
> + return 0;
> +}
> +
> static const FFAMediaFormat media_format_jni = {
> .class = &amediaformat_class,
>
> @@ -1801,6 +1842,7 @@ static const FFAMediaCodec media_codec_jni = {
>
> .getConfigureFlagEncode = mediacodec_jni_getConfigureFlagEncode,
> .cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers,
> + .signalEndOfInputStream = mediacodec_jni_signalEndOfInputStream,
> };
>
> typedef struct FFAMediaFormatNdk {
> @@ -1866,6 +1908,10 @@ typedef struct FFAMediaCodecNdk {
> // Available since API level 28.
> media_status_t (*getName)(AMediaCodec*, char** out_name);
> void (*releaseName)(AMediaCodec*, char* name);
> +
> + // Available since API level 26.
> + media_status_t (*setInputSurface)(AMediaCodec*, ANativeWindow *);
> + media_status_t (*signalEndOfInputStream)(AMediaCodec *);
> } FFAMediaCodecNdk;
>
> static const FFAMediaFormat media_format_ndk;
> @@ -2098,6 +2144,9 @@ static inline FFAMediaCodec *ndk_codec_create(int method, const char *arg) {
> GET_SYMBOL(getName, 0)
> GET_SYMBOL(releaseName, 0)
>
> + GET_SYMBOL(setInputSurface, 0)
> + GET_SYMBOL(signalEndOfInputStream, 0)
> +
> #undef GET_SYMBOL
>
> switch (method) {
> @@ -2184,10 +2233,32 @@ static int mediacodec_ndk_configure(FFAMediaCodec* ctx,
> return AVERROR(EINVAL);
> }
>
> - status = codec->configure(codec->impl, format->impl, native_window, NULL, flags);
> - if (status != AMEDIA_OK) {
> - av_log(codec, AV_LOG_ERROR, "configure failed, %d\n", status);
> - return AVERROR_EXTERNAL;
> + if (flags & AMEDIACODEC_CONFIGURE_FLAG_ENCODE) {
> + if (native_window && !codec->setInputSurface) {
> + av_log(ctx, AV_LOG_ERROR, "System doesn't support setInputSurface\n");
> + return AVERROR_EXTERNAL;
> + }
> +
> + status = codec->configure(codec->impl, format->impl, NULL, NULL, flags);
> + if (status != AMEDIA_OK) {
> + av_log(codec, AV_LOG_ERROR, "Encoder configure failed, %d\n", status);
> + return AVERROR_EXTERNAL;
> + }
> +
> + if (!native_window)
> + return 0;
> +
> + status = codec->setInputSurface(codec->impl, native_window);
> + if (status != AMEDIA_OK) {
> + av_log(codec, AV_LOG_ERROR, "Encoder set input surface failed, %d\n", status);
> + return AVERROR_EXTERNAL;
> + }
> + } else {
> + status = codec->configure(codec->impl, format->impl, native_window, NULL, flags);
> + if (status != AMEDIA_OK) {
> + av_log(codec, AV_LOG_ERROR, "Decoder configure failed, %d\n", status);
> + return AVERROR_EXTERNAL;
> + }
> }
>
> return 0;
> @@ -2330,6 +2401,26 @@ static int mediacodec_ndk_cleanOutputBuffers(FFAMediaCodec *ctx)
> return 0;
> }
>
> +static int mediacodec_ndk_signalEndOfInputStream(FFAMediaCodec *ctx)
> +{
> + FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
> + media_status_t status;
> +
> + if (!codec->signalEndOfInputStream) {
> + av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream unavailable\n");
> + return AVERROR_EXTERNAL;
> + }
> +
> + status = codec->signalEndOfInputStream(codec->impl);
> + if (status != AMEDIA_OK) {
> + av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream failed, %d\n", status);
> + return AVERROR_EXTERNAL;
> + }
> + av_log(codec, AV_LOG_DEBUG, "signalEndOfInputStream success\n");
> +
> + return 0;
> +}
> +
> static const FFAMediaFormat media_format_ndk = {
> .class = &amediaformat_ndk_class,
>
> @@ -2388,6 +2479,7 @@ static const FFAMediaCodec media_codec_ndk = {
>
> .getConfigureFlagEncode = mediacodec_ndk_getConfigureFlagEncode,
> .cleanOutputBuffers = mediacodec_ndk_cleanOutputBuffers,
> + .signalEndOfInputStream = mediacodec_ndk_signalEndOfInputStream,
> };
>
> FFAMediaFormat *ff_AMediaFormat_new(int ndk)
> diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h
> index 7cf3f4aecd..f15ad66d83 100644
> --- a/libavcodec/mediacodec_wrapper.h
> +++ b/libavcodec/mediacodec_wrapper.h
> @@ -192,6 +192,9 @@ struct FFAMediaCodec {
> int (*getConfigureFlagEncode)(FFAMediaCodec *codec);
>
> int (*cleanOutputBuffers)(FFAMediaCodec *codec);
> +
> + // For encoder with FFANativeWindow as input.
> + int (*signalEndOfInputStream)(FFAMediaCodec *);
> };
>
> static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec)
> @@ -311,6 +314,11 @@ static inline int ff_AMediaCodec_cleanOutputBuffers(FFAMediaCodec *codec)
> return codec->cleanOutputBuffers(codec);
> }
>
> +static inline int ff_AMediaCodec_signalEndOfInputStream(FFAMediaCodec *codec)
> +{
> + return codec->signalEndOfInputStream(codec);
> +}
> +
> int ff_Build_SDK_INT(AVCodecContext *avctx);
>
> #endif /* AVCODEC_MEDIACODEC_WRAPPER_H */
> diff --git a/libavcodec/mediacodecenc.c b/libavcodec/mediacodecenc.c
> new file mode 100644
> index 0000000000..c81050ec80
> --- /dev/null
> +++ b/libavcodec/mediacodecenc.c
> @@ -0,0 +1,495 @@
> +/*
> + * Android MediaCodec encoders
> + *
> + * Copyright (c) 2022 Zhao Zhili <zhilizhao@tencent.com>
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include "config_components.h"
> +
> +#include "libavutil/avassert.h"
> +#include "libavutil/hwcontext_mediacodec.h"
> +#include "libavutil/imgutils.h"
> +#include "libavutil/opt.h"
> +
> +#include "avcodec.h"
> +#include "codec_internal.h"
> +#include "encode.h"
> +#include "hwconfig.h"
> +#include "jni.h"
> +#include "mediacodec.h"
> +#include "mediacodec_wrapper.h"
> +#include "mediacodecdec_common.h"
> +
> +#define INPUT_DEQUEUE_TIMEOUT_US 8000
> +#define OUTPUT_DEQUEUE_TIMEOUT_US 8000
> +
> +typedef struct MediaCodecEncContext {
> + AVClass *avclass;
> + FFAMediaCodec *codec;
> + int use_ndk_codec;
> + FFANativeWindow *window;
> +
> + int fps;
> + int width;
> + int height;
> +
> + uint8_t *extradata;
> + int extradata_size;
> +
> + // Since MediaCodec doesn't output DTS, use a timestamp queue to save pts
> + // of AVFrame and generate DTS for AVPacket.
> + //
> + // This doesn't work when use Surface as input, in that case frames can be
> + // sent to encoder without our notice. One exception is frames come from
> + // our MediaCodec decoder wrapper, since we can control it's render by
> + // av_mediacodec_release_buffer.
> + int64_t timestamps[32];
> + int ts_head;
> + int ts_tail;
> +
> + int eof_sent;
> +
> + AVFrame *frame;
> +} MediaCodecEncContext;
> +
> +enum {
> + COLOR_FormatYUV420Planar = 0x13,
> + COLOR_FormatYUV420SemiPlanar = 0x15,
> + COLOR_FormatSurface = 0x7F000789,
> +};
> +
> +static const struct {
> + int color_format;
> + enum AVPixelFormat pix_fmt;
> +} color_formats[] = {
> + { COLOR_FormatYUV420Planar, AV_PIX_FMT_YUV420P },
> + { COLOR_FormatYUV420SemiPlanar, AV_PIX_FMT_NV12 },
> + { COLOR_FormatSurface, AV_PIX_FMT_MEDIACODEC },
> +};
> +
> +static const enum AVPixelFormat avc_pix_fmts[] = {
> + AV_PIX_FMT_MEDIACODEC,
> + AV_PIX_FMT_YUV420P,
> + AV_PIX_FMT_NV12,
> + AV_PIX_FMT_NONE
> +};
> +
> +static void mediacodec_output_format(AVCodecContext *avctx)
> +{
> + MediaCodecEncContext *s = avctx->priv_data;
> + char *name = ff_AMediaCodec_getName(s->codec);
> + FFAMediaFormat *out_format = ff_AMediaCodec_getOutputFormat(s->codec);
> + char *str = ff_AMediaFormat_toString(out_format);
> +
> + av_log(avctx, AV_LOG_DEBUG, "MediaCodec encoder %s output format %s\n",
> + name ? name : "unknown", str);
> + av_free(name);
> + av_free(str);
> + ff_AMediaFormat_delete(out_format);
> +}
> +
> +static av_cold int mediacodec_init(AVCodecContext *avctx)
> +{
> + const char *codec_mime = NULL;
> + MediaCodecEncContext *s = avctx->priv_data;
> + FFAMediaFormat *format = NULL;
> + int ret;
> + int gop;
> +
> + if (s->use_ndk_codec < 0)
> + s->use_ndk_codec = !av_jni_get_java_vm(avctx);
> +
> + switch (avctx->codec_id) {
> + case AV_CODEC_ID_H264:
> + codec_mime = "video/avc";
> + break;
> + case AV_CODEC_ID_HEVC:
> + codec_mime = "video/hevc";
> + break;
> + default:
> + av_assert0(0);
> + }
> +
> + s->codec = ff_AMediaCodec_createEncoderByType(codec_mime, s->use_ndk_codec);
> + if (!s->codec) {
> + av_log(avctx, AV_LOG_ERROR, "Failed to create encoder for type %s\n",
> + codec_mime);
> + return AVERROR_EXTERNAL;
> + }
> +
> + format = ff_AMediaFormat_new(s->use_ndk_codec);
> + if (!format) {
> + av_log(avctx, AV_LOG_ERROR, "Failed to create media format\n");
> + return AVERROR_EXTERNAL;
> + }
> +
> + ff_AMediaFormat_setString(format, "mime", codec_mime);
> + s->width = FFALIGN(avctx->width, 16);
> + s->height = avctx->height;
> + ff_AMediaFormat_setInt32(format, "width", s->width);
> + ff_AMediaFormat_setInt32(format, "height", s->height);
>
> Is it preferable to use constants like AMEDIAFORMAT_KEY_HEIGHT here?
>
> I don't have a preference either way.
These KEYs aren’t string literal, they are global variables.
extern const char* AMEDIAFORMAT_KEY_HEIGHT __INTRODUCED_IN(21);
Some basic ones are introduced very late, e.g.,
extern const char* AMEDIAFORMAT_KEY_CSD __INTRODUCED_IN(28);
extern const char* AMEDIAFORMAT_KEY_CSD_0 __INTRODUCED_IN(28);
extern const char* AMEDIAFORMAT_KEY_CSD_1 __INTRODUCED_IN(28);
extern const char* AMEDIAFORMAT_KEY_CSD_2 __INTRODUCED_IN(28);
extern const char* AMEDIAFORMAT_KEY_DISPLAY_CROP __INTRODUCED_IN(28);
So we can’t use these AMEDIAFORMAT_KEY_ directly. dlsym() these global
variables and with a fallback value is possible, just over-engineering.
Google’s API design forced us to use string literal directly.
>
> It may be worth also passing in the same values to AMEDIAFORMAT_KEY_MAX_WIDTH and AMEDIAFORMAT_KEY_MAX_HEIGHT
The documentation hints they are for decoder only:
> A key describing the maximum expected width of the content in a video decoder format, in case there are resolution changes in the video content.
https://developer.android.com/reference/android/media/MediaFormat
>
> And I think the unaligned width should be set into AMEDIAFORMAT_KEY_STRIDE
Technically, KEY_WIDTH should be unaligned width, and KEY_STRIDE should
be aligned width. However,
1. It’s well known that a lot of devices can’t handle width doesn’t aligned
to 16, they will crash, report error or produce broken files.
2. CTS tests only verify that 16 aligned resolutions are supported.
3. There are alignment info in OMX and exported via MediaCodecInfo, like
<MediaCodec name="c2.qti.avc.encoder" type="video/avc">
<Alias name="OMX.qcom.video.encoder.avc" />
<Limit name="size" min="96x96" max="4096x2304" />
<Limit name="alignment" value="2x2" />
<Limit name="block-size" value="16x16" />
It can be missing on old devices, or worse, the info doesn’t reflect the
real requirement (Maybe it was just a copy-paste from some places).
I have an idea to fix the issue: always encoding in aligned resolution,
then fix the crop setting by apply BSF (e.g., h264_metadata) on the
bitstream.
>
> Finally, avctx->sample_aspect_ratio should be propagated into aspect-width and aspect-height. Something like this:
>
> AVRational sar = avctx->sample_aspect_ratio;
> if (!sar.num || !sar.den)
> sar.num = sar.den = 1;
> av_reduce(&sar.num, &sar.den, sar.num, sar.den, 4096);
> AMediaFormat_setInt32(format, "aspect-width", sar.num);
> AMediaFormat_setInt32(format, "aspect-height", sar.den);
>
You mean sar-width/sar-height?
EXPORT const char* AMEDIAFORMAT_KEY_SAR_HEIGHT = "sar-height";
EXPORT const char* AMEDIAFORMAT_KEY_SAR_WIDTH = "sar-width";
They are available in NDK since API level 29 (Android 10)
extern const char* AMEDIAFORMAT_KEY_SAR_HEIGHT __INTRODUCED_IN(29);
extern const char* AMEDIAFORMAT_KEY_SAR_WIDTH __INTRODUCED_IN(29);
And they were added to Java MediaFormat since API level 30.
https://developer.android.com/reference/android/media/MediaFormat#KEY_PIXEL_ASPECT_RATIO_WIDTH
It’s uncommon for Java API got a feature late than NDK. I will keep
a note and do more test before setting them.
>
> +
> + if (avctx->pix_fmt == AV_PIX_FMT_MEDIACODEC) {
> + AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
> + if (avctx->hw_device_ctx) {
> + AVHWDeviceContext *device_ctx = (AVHWDeviceContext*)(avctx->hw_device_ctx->data);
> + AVMediaCodecDeviceContext *dev_ctx;
> +
> + if (device_ctx->type != AV_HWDEVICE_TYPE_MEDIACODEC || !device_ctx->hwctx) {
> + ret = AVERROR(EINVAL);
> + goto bailout;
> + }
> + dev_ctx = device_ctx->hwctx;
> + s->window = ff_mediacodec_surface_ref(dev_ctx->surface, dev_ctx->native_window, avctx);
> + }
> +
> + if (!s->window && user_ctx && user_ctx->surface)
> + s->window = ff_mediacodec_surface_ref(user_ctx->surface, NULL, avctx);
> +
> + if (!s->window) {
> + ret = AVERROR(EINVAL);
> + av_log(avctx, AV_LOG_ERROR, "Missing hw_device_ctx or hwaccel_context for AV_PIX_FMT_MEDIACODEC\n");
> + goto bailout;
> + }
> + }
> +
> + for (int i = 0; i < FF_ARRAY_ELEMS(color_formats); i++) {
> + if (avctx->pix_fmt == color_formats[i].pix_fmt) {
> + ff_AMediaFormat_setInt32(format, "color-format",
> + color_formats[i].color_format);
> + break;
>
> do we need error/fallback if no match is found?
The supported pix_fmts is specified
.priv_data_size = sizeof(MediaCodecEncContext), \
.p.pix_fmts = avc_pix_fmts, \
.init = mediacodec_init, \
It has been checked in encoder.c encode_preinit_video(), so check it again is
optional but not necessary in my opinion.
>
>
> + }
> + }
> +
> + if (avctx->bit_rate)
> + ff_AMediaFormat_setInt32(format, "bitrate", avctx->bit_rate);
> + // frame-rate and i-frame-interval are required to configure codec
> + if (avctx->framerate.num >= avctx->framerate.den && avctx->framerate.den > 0)
> + s->fps = avctx->framerate.num / avctx->framerate.den;
> + else
> + s->fps = 30;
> + gop = round(avctx->gop_size / s->fps);
> + if (gop == 0)
> + gop = 2;
>
> can we read gop from avctx? in other implementations i have seen gop hardcoded to 1
I think we should respect gop_size setting.
I will add some log message and change default gop to 1 in patch v2.
>
> for fps value, can we use av_q2d(avctx->framerate)
q2d() is for double, fps is integer here.
>
>
> + ff_AMediaFormat_setInt32(format, "frame-rate", s->fps);
> + ff_AMediaFormat_setInt32(format, "i-frame-interval", gop);
> +
>
> for H264 encoding, you can pass "profile" and "level". for example profile=0x08 for High and level=0x4000 for 5.0
>
> https://stackoverflow.com/a/38999412/332798
I’m planning to add profile/level support after the basic patch is applied.
There are some unresolved issue here:
1. I have a device which failed at configure() when profile has been set,
even for baseline.
2. DTS is missing in MediaCodec API. For the default baseline profile, there
is no problem. Trouble comes when we set profile to mainline or high. If
Surface/ANativeWindow is used, and the frames come from our MediaCodec decoder
wrapper, we can control it’s 'render' (send to encoder's surface) via
av_mediacodec_release_buffer(). A DTS generation strategy works in this case.
However, if frames comes from other sources, like a camera, there is no way
to control the 'render' yet, so DTS is missing in this case.
Configure profile/level and B frames should come together.
>
>
> +
> + ret = ff_AMediaCodec_getConfigureFlagEncode(s->codec);
> + ret = ff_AMediaCodec_configure(s->codec, format, s->window, NULL, ret);
> + if (ret) {
> + av_log(avctx, AV_LOG_ERROR, "MediaCodec configure failed, %s\n", av_err2str(ret));
> + goto bailout;
> + }
> +
> + ret = ff_AMediaCodec_start(s->codec);
> + if (ret) {
> + av_log(avctx, AV_LOG_ERROR, "MediaCodec failed to start, %s\n", av_err2str(ret));
> + goto bailout;
> + }
> +
> + mediacodec_output_format(avctx);
> +
> + s->frame = av_frame_alloc();
> + if (!s->frame)
> + ret = AVERROR(ENOMEM);
> +
> +bailout:
> + if (format)
> + ff_AMediaFormat_delete(format);
> + return ret;
> +}
> +
> +static int mediacodec_receive(AVCodecContext *avctx,
> + AVPacket *pkt,
> + int *got_packet)
> +{
> + MediaCodecEncContext *s = avctx->priv_data;
> + FFAMediaCodec *codec = s->codec;
> + FFAMediaCodecBufferInfo out_info = {0};
> + uint8_t *out_buf;
> + size_t out_size = 0;
> + int ret;
> + int extradata_size = 0;
> + int64_t timeout_us = s->eof_sent ? OUTPUT_DEQUEUE_TIMEOUT_US : 0;
> + ssize_t index = ff_AMediaCodec_dequeueOutputBuffer(codec, &out_info, timeout_us);
> +
> + if (ff_AMediaCodec_infoTryAgainLater(codec, index))
> + return AVERROR(EAGAIN);
> +
> + if (ff_AMediaCodec_infoOutputFormatChanged(codec, index)) {
> + mediacodec_output_format(avctx);
> + return AVERROR(EAGAIN);
> + }
> +
> + if (ff_AMediaCodec_infoOutputBuffersChanged(codec, index)) {
> + ff_AMediaCodec_cleanOutputBuffers(codec);
> + return AVERROR(EAGAIN);
> + }
> +
> + if (index < 0)
> + return AVERROR_EXTERNAL;
> +
> + if (out_info.flags & ff_AMediaCodec_getBufferFlagEndOfStream(codec))
> + return AVERROR_EOF;
> +
> + out_buf = ff_AMediaCodec_getOutputBuffer(codec, index, &out_size);
> + if (!out_buf) {
> + ret = AVERROR_EXTERNAL;
> + goto bailout;
> + }
> +
> + if (out_info.flags & ff_AMediaCodec_getBufferFlagCodecConfig(codec)) {
> + ret = av_reallocp(&s->extradata, out_info.size);
> + if (ret)
> + goto bailout;
> +
> + s->extradata_size = out_info.size;
> + memcpy(s->extradata, out_buf + out_info.offset, out_info.size);
> + ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
> + // try immediately
> + return mediacodec_receive(avctx, pkt, got_packet);
> + }
> +
> + ret = ff_get_encode_buffer(avctx, pkt, out_info.size + s->extradata_size, 0);
> + if (ret < 0)
> + goto bailout;
> +
> + if (s->extradata_size) {
> + extradata_size = s->extradata_size;
> + s->extradata_size = 0;
> + memcpy(pkt->data, s->extradata, extradata_size);
> + }
> + memcpy(pkt->data + extradata_size, out_buf + out_info.offset, out_info.size);
> + pkt->pts = av_rescale_q(out_info.presentationTimeUs, AV_TIME_BASE_Q, avctx->time_base);
> + if (s->ts_tail != s->ts_head) {
> + pkt->dts = s->timestamps[s->ts_tail];
> + s->ts_tail = (s->ts_tail + 1) % FF_ARRAY_ELEMS(s->timestamps);
> + }
> +
> + if (out_info.flags & ff_AMediaCodec_getBufferFlagKeyFrame(codec))
> + pkt->flags |= AV_PKT_FLAG_KEY;
> + ret = 0;
> + *got_packet = 1;
> +
> + av_log(avctx, AV_LOG_TRACE, "receive packet pts %" PRId64 " dts %" PRId64
> + " flags %d extradata %d\n",
> + pkt->pts, pkt->dts, pkt->flags, extradata_size);
> +
> +bailout:
> + ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
> + return ret;
> +}
> +
> +static void copy_frame_to_buffer(AVCodecContext *avctx, const AVFrame *frame, uint8_t *dst, size_t size)
> +{
> + MediaCodecEncContext *s = avctx->priv_data;
> + uint8_t *dst_data[4] = {};
> + int dst_linesize[4] = {};
> + const uint8_t *src_data[4] = {
> + frame->data[0], frame->data[1], frame->data[2], frame->data[3]
> + };
> +
> + if (avctx->pix_fmt == AV_PIX_FMT_YUV420P) {
> + dst_data[0] = dst;
> + dst_data[1] = dst + s->width * s->height;
> + dst_data[2] = dst_data[1] + s->width * s->height / 4;
> +
> + dst_linesize[0] = s->width;
> + dst_linesize[1] = dst_linesize[2] = s->width / 2;
> + } else if (avctx->pix_fmt == AV_PIX_FMT_NV12) {
> + dst_data[0] = dst;
> + dst_data[1] = dst + s->width * s->height;
> +
> + dst_linesize[0] = s->width;
> + dst_linesize[1] = s->width;
> + } else {
> + av_assert0(0);
> + }
> +
> + av_image_copy(dst_data, dst_linesize, src_data, frame->linesize,
> + avctx->pix_fmt, avctx->width, avctx->height);
> +}
> +
> +static int mediacodec_send(AVCodecContext *avctx,
> + const AVFrame *frame) {
> + MediaCodecEncContext *s = avctx->priv_data;
> + FFAMediaCodec *codec = s->codec;
> + ssize_t index;
> + uint8_t *input_buf = NULL;
> + size_t input_size = 0;
> + int64_t pts = 0;
> + uint32_t flags = 0;
> + int64_t timeout_us;
> +
> + if (s->eof_sent)
> + return 0;
> +
> + if (s->window) {
> + if (!frame) {
> + s->eof_sent = 1;
> + return ff_AMediaCodec_signalEndOfInputStream(codec);
> + }
> +
> +
> + if (frame->data[3]) {
> + pts = av_rescale_q(frame->pts, avctx->time_base, AV_TIME_BASE_Q);
> + s->timestamps[s->ts_head] = frame->pts;
> + s->ts_head = (s->ts_head + 1) % FF_ARRAY_ELEMS(s->timestamps);
> +
> + av_mediacodec_release_buffer((AVMediaCodecBuffer *)frame->data[3], 1);
> + }
> + return 0;
> + }
> +
> + timeout_us = INPUT_DEQUEUE_TIMEOUT_US;
> + index = ff_AMediaCodec_dequeueInputBuffer(codec, timeout_us);
> + if (ff_AMediaCodec_infoTryAgainLater(codec, index))
> + return AVERROR(EAGAIN);
> +
> + if (index < 0) {
> + av_log(avctx, AV_LOG_ERROR, "dequeue input buffer failed, %zd", index);
> + return AVERROR_EXTERNAL;
> + }
> +
> + if (frame) {
> + input_buf = ff_AMediaCodec_getInputBuffer(codec, index, &input_size);
> + copy_frame_to_buffer(avctx, frame, input_buf, input_size);
> +
> + pts = av_rescale_q(frame->pts, avctx->time_base, AV_TIME_BASE_Q);
> +
> + s->timestamps[s->ts_head] = frame->pts;
> + s->ts_head = (s->ts_head + 1) % FF_ARRAY_ELEMS(s->timestamps);
> + } else {
> + flags |= ff_AMediaCodec_getBufferFlagEndOfStream(codec);
> + s->eof_sent = 1;
> + }
> +
>
> it would be nice to propagate keyframes here (frame->pict_type == AV_PICTURE_TYPE_I). it is only possible on API26 with AMediaCodec_setParameters and PARAMETER_KEY_REQUEST_SYNC_FRAME
It’s a nice feature. I’m planning to add support after the basic function is working.
>
>
> + ff_AMediaCodec_queueInputBuffer(codec, index, 0, input_size, pts, flags);
> + return 0;
> +}
> +
> +static int mediacodec_encode(AVCodecContext *avctx, AVPacket *pkt)
> +{
> + MediaCodecEncContext *s = avctx->priv_data;
> + int ret;
> + int got_packet = 0;
> +
> + // Return on three case:
> + // 1. Serious error
> + // 2. Got a packet success
> + // 3. No AVFrame is available yet (don't return if get_frame return EOF)
> + while (1) {
> + ret = mediacodec_receive(avctx, pkt, &got_packet);
> + if (!ret)
> + return 0;
> + else if (ret != AVERROR(EAGAIN))
> + return ret;
> +
> + if (!s->frame->buf[0]) {
> + ret = ff_encode_get_frame(avctx, s->frame);
> + if (ret && ret != AVERROR_EOF)
> + return ret;
> + }
> +
> + ret = mediacodec_send(avctx, s->frame->buf[0] ? s->frame : NULL);
> + if (!ret)
> + av_frame_unref(s->frame);
> + else if (ret != AVERROR(EAGAIN))
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static av_cold int mediacodec_close(AVCodecContext *avctx)
> +{
> + MediaCodecEncContext *s = avctx->priv_data;
> + if (s->codec) {
> + ff_AMediaCodec_stop(s->codec);
> + ff_AMediaCodec_delete(s->codec);
> + s->codec = NULL;
> + }
> +
> + if (s->window) {
> + ff_mediacodec_surface_unref(s->window, avctx);
> + s->window = NULL;
> + }
> +
> + av_frame_free(&s->frame);
> +
> + return 0;
> +}
> +
> +static const AVCodecHWConfigInternal *const mediacodec_hw_configs[] = {
> + &(const AVCodecHWConfigInternal) {
> + .public = {
> + .pix_fmt = AV_PIX_FMT_MEDIACODEC,
> + .methods = AV_CODEC_HW_CONFIG_METHOD_AD_HOC |
> + AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX,
> + .device_type = AV_HWDEVICE_TYPE_MEDIACODEC,
> + },
> + .hwaccel = NULL,
> + },
> + NULL
> +};
> +
> +#define OFFSET(x) offsetof(MediaCodecEncContext, x)
> +#define VE AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM
> +static const AVOption common_options[] = {
> + { "ndk_codec", "Use MediaCodec from NDK",
> + OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 = -1}, -1, 1, VE },
> + { NULL },
> +};
> +
> +#define MEDIACODEC_ENCODER_CLASS(name) \
> +static const AVClass name ## _mediacodec_class = { \
> + .class_name = #name "_mediacodec", \
> + .item_name = av_default_item_name, \
> + .option = common_options, \
> + .version = LIBAVUTIL_VERSION_INT, \
> +}; \
> +
> +#define DECLARE_MEDIACODEC_ENCODER(short_name, long_name, codec_id) \
> +MEDIACODEC_ENCODER_CLASS(short_name) \
> +const FFCodec ff_ ## short_name ## _mediacodec_encoder = { \
> + .p.name = #short_name "_mediacodec", \
> + CODEC_LONG_NAME(long_name " Android MediaCodec encoder"), \
> + .p.type = AVMEDIA_TYPE_VIDEO, \
> + .p.id = codec_id, \
> + .p.capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY \
> + | AV_CODEC_CAP_HARDWARE, \
> + .priv_data_size = sizeof(MediaCodecEncContext), \
> + .p.pix_fmts = avc_pix_fmts, \
> + .init = mediacodec_init, \
> + FF_CODEC_RECEIVE_PACKET_CB(mediacodec_encode), \
> + .close = mediacodec_close, \
> + .p.priv_class = &short_name ## _mediacodec_class, \
> + .caps_internal = FF_CODEC_CAP_INIT_CLEANUP, \
> + .p.wrapper_name = "mediacodec", \
> + .hw_configs = mediacodec_hw_configs, \
> +}; \
> +
> +#if CONFIG_H264_MEDIACODEC_ENCODER
> +DECLARE_MEDIACODEC_ENCODER(h264, "H.264", AV_CODEC_ID_H264)
> +#endif
> +
> +#if CONFIG_HEVC_MEDIACODEC_ENCODER
> +DECLARE_MEDIACODEC_ENCODER(hevc, "H.265", AV_CODEC_ID_HEVC)
> +#endif
> diff --git a/libavcodec/version.h b/libavcodec/version.h
> index 43d0d9a9fc..86ac0f3871 100644
> --- a/libavcodec/version.h
> +++ b/libavcodec/version.h
> @@ -29,8 +29,8 @@
>
> #include "version_major.h"
>
> -#define LIBAVCODEC_VERSION_MINOR 51
> -#define LIBAVCODEC_VERSION_MICRO 101
> +#define LIBAVCODEC_VERSION_MINOR 52
> +#define LIBAVCODEC_VERSION_MICRO 100
>
> #define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \
> LIBAVCODEC_VERSION_MINOR, \
> --
> 2.25.1
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder
2022-11-19 17:13 ` "zhilizhao(赵志立)"
@ 2022-11-19 18:14 ` Olivier Ayache
2022-11-20 23:44 ` Aman Karmani
1 sibling, 0 replies; 22+ messages in thread
From: Olivier Ayache @ 2022-11-19 18:14 UTC (permalink / raw)
To: FFmpeg development discussions and patches
>That’s the case from my test on Google Pixel 3. There is no ‘csd’ inside
>MediaFormat returned by ff_AMediaCodec_getOutputFormat(), until a frame
>has been sent. It may depends on the device and OS.
Hello there, if it can help concerning the csd your asumption is right and
not only on Pixel 3. When I wrote my encoder (only using the NDK) I have to
manually add csd in extradata after encoding the first frame. I study a lot
the Android AOSP source code especially media framework and libstagefright.
Here is what I did in one of my project that use the encoder
https://github.com/olivierayache/xuggle-xuggler-android/blob/dbc8eaca734684b1763399da12ecdffa2508cc89/xuggler-android-extensions/src/main/java/com/xuggle/xuggler/record/MediaRecorder.kt#L420
At the time I worked on that I didn't do it directly in FFmpeg but it seems
interesting to store the CSD when the first frame is encoded.
Olivier Ayache
On Sat, Nov 19, 2022 at 6:13 PM "zhilizhao(赵志立)" <quinkblack@foxmail.com>
wrote:
>
>
> > On Nov 19, 2022, at 17:15, Aman Karmani <aman@tmm1.net> wrote:
> >
> >
> >
> > On Sun, Oct 23, 2022 at 8:17 PM Zhao Zhili <quinkblack@foxmail.com>
> wrote:
> > From: Zhao Zhili <zhilizhao@tencent.com>
> >
> > Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
> > ---
> > Changelog | 1 +
> > configure | 4 +
> > libavcodec/Makefile | 2 +
> > libavcodec/allcodecs.c | 2 +
> > libavcodec/mediacodec_wrapper.c | 102 ++++++-
> > libavcodec/mediacodec_wrapper.h | 8 +
> > libavcodec/mediacodecenc.c | 495 ++++++++++++++++++++++++++++++++
> > libavcodec/version.h | 4 +-
> > 8 files changed, 611 insertions(+), 7 deletions(-)
> > create mode 100644 libavcodec/mediacodecenc.c
> >
> > diff --git a/Changelog b/Changelog
> > index 9e203833aa..9e39a35972 100644
> > --- a/Changelog
> > +++ b/Changelog
> > @@ -19,6 +19,7 @@ version <next>:
> > - DTS to PTS reorder bsf
> > - ViewQuest VQC decoder
> > - MediaCodec decoder via NDKMediaCodec
> > +- MediaCodec encoder
> >
> >
> >
> > version 5.1:
> > diff --git a/configure b/configure
> > index ee2e3ba6ac..5114cda13f 100755
> > --- a/configure
> > +++ b/configure
> > @@ -3193,6 +3193,8 @@ h264_cuvid_decoder_select="h264_mp4toannexb_bsf"
> > h264_mediacodec_decoder_deps="mediacodec"
> > h264_mediacodec_decoder_extralibs="-landroid"
> > h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser"
> > +h264_mediacodec_encoder_deps="mediacodec"
> > +h264_mediacodec_encoder_extralibs="-landroid"
> > h264_mf_encoder_deps="mediafoundation"
> > h264_mmal_decoder_deps="mmal"
> > h264_nvenc_encoder_deps="nvenc"
> > @@ -3212,6 +3214,8 @@ hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf"
> > hevc_mediacodec_decoder_deps="mediacodec"
> > hevc_mediacodec_decoder_extralibs="-landroid"
> > hevc_mediacodec_decoder_select="hevc_mp4toannexb_bsf hevc_parser"
> > +hevc_mediacodec_encoder_deps="mediacodec"
> > +hevc_mediacodec_encoder_extralibs="-landroid"
> > hevc_mf_encoder_deps="mediafoundation"
> > hevc_nvenc_encoder_deps="nvenc"
> > hevc_nvenc_encoder_select="atsc_a53"
> > diff --git a/libavcodec/Makefile b/libavcodec/Makefile
> > index 90c7f113a3..7d0b513eec 100644
> > --- a/libavcodec/Makefile
> > +++ b/libavcodec/Makefile
> > @@ -393,6 +393,7 @@ OBJS-$(CONFIG_H264_DECODER) += h264dec.o
> h264_cabac.o h264_cavlc.o \
> > OBJS-$(CONFIG_H264_AMF_ENCODER) += amfenc_h264.o
> > OBJS-$(CONFIG_H264_CUVID_DECODER) += cuviddec.o
> > OBJS-$(CONFIG_H264_MEDIACODEC_DECODER) += mediacodecdec.o
> > +OBJS-$(CONFIG_H264_MEDIACODEC_ENCODER) += mediacodecenc.o
> > OBJS-$(CONFIG_H264_MF_ENCODER) += mfenc.o mf_utils.o
> > OBJS-$(CONFIG_H264_MMAL_DECODER) += mmaldec.o
> > OBJS-$(CONFIG_H264_NVENC_ENCODER) += nvenc_h264.o nvenc.o
> > @@ -417,6 +418,7 @@ OBJS-$(CONFIG_HEVC_DECODER) += hevcdec.o
> hevc_mvs.o \
> > OBJS-$(CONFIG_HEVC_AMF_ENCODER) += amfenc_hevc.o
> > OBJS-$(CONFIG_HEVC_CUVID_DECODER) += cuviddec.o
> > OBJS-$(CONFIG_HEVC_MEDIACODEC_DECODER) += mediacodecdec.o
> > +OBJS-$(CONFIG_HEVC_MEDIACODEC_ENCODER) += mediacodecenc.o
> > OBJS-$(CONFIG_HEVC_MF_ENCODER) += mfenc.o mf_utils.o
> > OBJS-$(CONFIG_HEVC_NVENC_ENCODER) += nvenc_hevc.o nvenc.o
> > OBJS-$(CONFIG_HEVC_QSV_DECODER) += qsvdec.o
> > diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
> > index 46ad3b5a25..4c33a9ec3c 100644
> > --- a/libavcodec/allcodecs.c
> > +++ b/libavcodec/allcodecs.c
> > @@ -154,6 +154,7 @@ extern const FFCodec ff_h264_decoder;
> > extern const FFCodec ff_h264_crystalhd_decoder;
> > extern const FFCodec ff_h264_v4l2m2m_decoder;
> > extern const FFCodec ff_h264_mediacodec_decoder;
> > +extern const FFCodec ff_h264_mediacodec_encoder;
> > extern const FFCodec ff_h264_mmal_decoder;
> > extern const FFCodec ff_h264_qsv_decoder;
> > extern const FFCodec ff_h264_rkmpp_decoder;
> > @@ -842,6 +843,7 @@ extern const FFCodec ff_h264_videotoolbox_encoder;
> > extern const FFCodec ff_hevc_amf_encoder;
> > extern const FFCodec ff_hevc_cuvid_decoder;
> > extern const FFCodec ff_hevc_mediacodec_decoder;
> > +extern const FFCodec ff_hevc_mediacodec_encoder;
> > extern const FFCodec ff_hevc_mf_encoder;
> > extern const FFCodec ff_hevc_nvenc_encoder;
> > extern const FFCodec ff_hevc_qsv_encoder;
> > diff --git a/libavcodec/mediacodec_wrapper.c
> b/libavcodec/mediacodec_wrapper.c
> > index 284d615980..5d1a32031d 100644
> > --- a/libavcodec/mediacodec_wrapper.c
> > +++ b/libavcodec/mediacodec_wrapper.c
> > @@ -212,6 +212,9 @@ struct JNIAMediaCodecFields {
> > jmethodID release_output_buffer_id;
> > jmethodID release_output_buffer_at_time_id;
> >
> > + jmethodID set_input_surface_id;
> > + jmethodID signal_end_of_input_stream_id;
> > +
> > jclass mediainfo_class;
> >
> > jmethodID init_id;
> > @@ -261,6 +264,9 @@ static const struct FFJniField
> jni_amediacodec_mapping[] = {
> > { "android/media/MediaCodec", "releaseOutputBuffer", "(IZ)V",
> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields,
> release_output_buffer_id), 1 },
> > { "android/media/MediaCodec", "releaseOutputBuffer", "(IJ)V",
> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields,
> release_output_buffer_at_time_id), 0 },
> >
> > + { "android/media/MediaCodec", "setInputSurface",
> "(Landroid/view/Surface;)V", FF_JNI_METHOD, offsetof(struct
> JNIAMediaCodecFields, set_input_surface_id), 0 },
> > + { "android/media/MediaCodec", "signalEndOfInputStream", "()V",
> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields,
> signal_end_of_input_stream_id), 0 },
> > +
> > { "android/media/MediaCodec$BufferInfo", NULL, NULL, FF_JNI_CLASS,
> offsetof(struct JNIAMediaCodecFields, mediainfo_class), 1 },
> >
> > { "android/media/MediaCodec.BufferInfo", "<init>", "()V",
> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, init_id), 1 },
> > @@ -1385,7 +1391,26 @@ static int mediacodec_jni_configure(FFAMediaCodec
> *ctx,
> >
> > JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
> >
> > - (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.configure_id, format->object, surface, NULL, flags);
> > + if (flags & codec->CONFIGURE_FLAG_ENCODE) {
> > + if (surface && !codec->jfields.set_input_surface_id) {
> > + av_log(ctx, AV_LOG_ERROR, "System doesn't support
> setInputSurface\n");
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.configure_id, format->object, NULL, NULL, flags);
> > + if (ff_jni_exception_check(env, 1, codec) < 0)
> > + return AVERROR_EXTERNAL;
> > +
> > + if (!surface)
> > + return 0;
> > +
> > + (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.set_input_surface_id, surface);
> > + if (ff_jni_exception_check(env, 1, codec) < 0)
> > + return AVERROR_EXTERNAL;
> > + return 0;
> > + } else {
> > + (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.configure_id, format->object, surface, NULL, flags);
> > + }
> > if (ff_jni_exception_check(env, 1, codec) < 0) {
> > ret = AVERROR_EXTERNAL;
> > goto fail;
> > @@ -1743,6 +1768,22 @@ fail:
> > return ret;
> > }
> >
> > +static int mediacodec_jni_signalEndOfInputStream(FFAMediaCodec *ctx)
> > +{
> > + int ret = 0;
> > + JNIEnv *env = NULL;
> > + FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
> > +
> > + JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
> > +
> > + (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.signal_end_of_input_stream_id);
> > + if (ff_jni_exception_check(env, 1, codec) < 0) {
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > static const FFAMediaFormat media_format_jni = {
> > .class = &amediaformat_class,
> >
> > @@ -1801,6 +1842,7 @@ static const FFAMediaCodec media_codec_jni = {
> >
> > .getConfigureFlagEncode = mediacodec_jni_getConfigureFlagEncode,
> > .cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers,
> > + .signalEndOfInputStream = mediacodec_jni_signalEndOfInputStream,
> > };
> >
> > typedef struct FFAMediaFormatNdk {
> > @@ -1866,6 +1908,10 @@ typedef struct FFAMediaCodecNdk {
> > // Available since API level 28.
> > media_status_t (*getName)(AMediaCodec*, char** out_name);
> > void (*releaseName)(AMediaCodec*, char* name);
> > +
> > + // Available since API level 26.
> > + media_status_t (*setInputSurface)(AMediaCodec*, ANativeWindow *);
> > + media_status_t (*signalEndOfInputStream)(AMediaCodec *);
> > } FFAMediaCodecNdk;
> >
> > static const FFAMediaFormat media_format_ndk;
> > @@ -2098,6 +2144,9 @@ static inline FFAMediaCodec *ndk_codec_create(int
> method, const char *arg) {
> > GET_SYMBOL(getName, 0)
> > GET_SYMBOL(releaseName, 0)
> >
> > + GET_SYMBOL(setInputSurface, 0)
> > + GET_SYMBOL(signalEndOfInputStream, 0)
> > +
> > #undef GET_SYMBOL
> >
> > switch (method) {
> > @@ -2184,10 +2233,32 @@ static int
> mediacodec_ndk_configure(FFAMediaCodec* ctx,
> > return AVERROR(EINVAL);
> > }
> >
> > - status = codec->configure(codec->impl, format->impl, native_window,
> NULL, flags);
> > - if (status != AMEDIA_OK) {
> > - av_log(codec, AV_LOG_ERROR, "configure failed, %d\n", status);
> > - return AVERROR_EXTERNAL;
> > + if (flags & AMEDIACODEC_CONFIGURE_FLAG_ENCODE) {
> > + if (native_window && !codec->setInputSurface) {
> > + av_log(ctx, AV_LOG_ERROR, "System doesn't support
> setInputSurface\n");
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + status = codec->configure(codec->impl, format->impl, NULL,
> NULL, flags);
> > + if (status != AMEDIA_OK) {
> > + av_log(codec, AV_LOG_ERROR, "Encoder configure failed,
> %d\n", status);
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + if (!native_window)
> > + return 0;
> > +
> > + status = codec->setInputSurface(codec->impl, native_window);
> > + if (status != AMEDIA_OK) {
> > + av_log(codec, AV_LOG_ERROR, "Encoder set input surface
> failed, %d\n", status);
> > + return AVERROR_EXTERNAL;
> > + }
> > + } else {
> > + status = codec->configure(codec->impl, format->impl,
> native_window, NULL, flags);
> > + if (status != AMEDIA_OK) {
> > + av_log(codec, AV_LOG_ERROR, "Decoder configure failed,
> %d\n", status);
> > + return AVERROR_EXTERNAL;
> > + }
> > }
> >
> > return 0;
> > @@ -2330,6 +2401,26 @@ static int
> mediacodec_ndk_cleanOutputBuffers(FFAMediaCodec *ctx)
> > return 0;
> > }
> >
> > +static int mediacodec_ndk_signalEndOfInputStream(FFAMediaCodec *ctx)
> > +{
> > + FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
> > + media_status_t status;
> > +
> > + if (!codec->signalEndOfInputStream) {
> > + av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream
> unavailable\n");
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + status = codec->signalEndOfInputStream(codec->impl);
> > + if (status != AMEDIA_OK) {
> > + av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream failed,
> %d\n", status);
> > + return AVERROR_EXTERNAL;
> > + }
> > + av_log(codec, AV_LOG_DEBUG, "signalEndOfInputStream success\n");
> > +
> > + return 0;
> > +}
> > +
> > static const FFAMediaFormat media_format_ndk = {
> > .class = &amediaformat_ndk_class,
> >
> > @@ -2388,6 +2479,7 @@ static const FFAMediaCodec media_codec_ndk = {
> >
> > .getConfigureFlagEncode = mediacodec_ndk_getConfigureFlagEncode,
> > .cleanOutputBuffers = mediacodec_ndk_cleanOutputBuffers,
> > + .signalEndOfInputStream = mediacodec_ndk_signalEndOfInputStream,
> > };
> >
> > FFAMediaFormat *ff_AMediaFormat_new(int ndk)
> > diff --git a/libavcodec/mediacodec_wrapper.h
> b/libavcodec/mediacodec_wrapper.h
> > index 7cf3f4aecd..f15ad66d83 100644
> > --- a/libavcodec/mediacodec_wrapper.h
> > +++ b/libavcodec/mediacodec_wrapper.h
> > @@ -192,6 +192,9 @@ struct FFAMediaCodec {
> > int (*getConfigureFlagEncode)(FFAMediaCodec *codec);
> >
> > int (*cleanOutputBuffers)(FFAMediaCodec *codec);
> > +
> > + // For encoder with FFANativeWindow as input.
> > + int (*signalEndOfInputStream)(FFAMediaCodec *);
> > };
> >
> > static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec)
> > @@ -311,6 +314,11 @@ static inline int
> ff_AMediaCodec_cleanOutputBuffers(FFAMediaCodec *codec)
> > return codec->cleanOutputBuffers(codec);
> > }
> >
> > +static inline int ff_AMediaCodec_signalEndOfInputStream(FFAMediaCodec
> *codec)
> > +{
> > + return codec->signalEndOfInputStream(codec);
> > +}
> > +
> > int ff_Build_SDK_INT(AVCodecContext *avctx);
> >
> > #endif /* AVCODEC_MEDIACODEC_WRAPPER_H */
> > diff --git a/libavcodec/mediacodecenc.c b/libavcodec/mediacodecenc.c
> > new file mode 100644
> > index 0000000000..c81050ec80
> > --- /dev/null
> > +++ b/libavcodec/mediacodecenc.c
> > @@ -0,0 +1,495 @@
> > +/*
> > + * Android MediaCodec encoders
> > + *
> > + * Copyright (c) 2022 Zhao Zhili <zhilizhao@tencent.com>
> > + *
> > + * This file is part of FFmpeg.
> > + *
> > + * FFmpeg is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public
> > + * License as published by the Free Software Foundation; either
> > + * version 2.1 of the License, or (at your option) any later version.
> > + *
> > + * FFmpeg is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with FFmpeg; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA
> > + */
> > +
> > +#include "config_components.h"
> > +
> > +#include "libavutil/avassert.h"
> > +#include "libavutil/hwcontext_mediacodec.h"
> > +#include "libavutil/imgutils.h"
> > +#include "libavutil/opt.h"
> > +
> > +#include "avcodec.h"
> > +#include "codec_internal.h"
> > +#include "encode.h"
> > +#include "hwconfig.h"
> > +#include "jni.h"
> > +#include "mediacodec.h"
> > +#include "mediacodec_wrapper.h"
> > +#include "mediacodecdec_common.h"
> > +
> > +#define INPUT_DEQUEUE_TIMEOUT_US 8000
> > +#define OUTPUT_DEQUEUE_TIMEOUT_US 8000
> > +
> > +typedef struct MediaCodecEncContext {
> > + AVClass *avclass;
> > + FFAMediaCodec *codec;
> > + int use_ndk_codec;
> > + FFANativeWindow *window;
> > +
> > + int fps;
> > + int width;
> > + int height;
> > +
> > + uint8_t *extradata;
> > + int extradata_size;
> > +
> > + // Since MediaCodec doesn't output DTS, use a timestamp queue to
> save pts
> > + // of AVFrame and generate DTS for AVPacket.
> > + //
> > + // This doesn't work when use Surface as input, in that case frames
> can be
> > + // sent to encoder without our notice. One exception is frames come
> from
> > + // our MediaCodec decoder wrapper, since we can control it's render
> by
> > + // av_mediacodec_release_buffer.
> > + int64_t timestamps[32];
> > + int ts_head;
> > + int ts_tail;
> > +
> > + int eof_sent;
> > +
> > + AVFrame *frame;
> > +} MediaCodecEncContext;
> > +
> > +enum {
> > + COLOR_FormatYUV420Planar = 0x13,
> > + COLOR_FormatYUV420SemiPlanar = 0x15,
> > + COLOR_FormatSurface = 0x7F000789,
> > +};
> > +
> > +static const struct {
> > + int color_format;
> > + enum AVPixelFormat pix_fmt;
> > +} color_formats[] = {
> > + { COLOR_FormatYUV420Planar, AV_PIX_FMT_YUV420P },
> > + { COLOR_FormatYUV420SemiPlanar, AV_PIX_FMT_NV12 },
> > + { COLOR_FormatSurface, AV_PIX_FMT_MEDIACODEC },
> > +};
> > +
> > +static const enum AVPixelFormat avc_pix_fmts[] = {
> > + AV_PIX_FMT_MEDIACODEC,
> > + AV_PIX_FMT_YUV420P,
> > + AV_PIX_FMT_NV12,
> > + AV_PIX_FMT_NONE
> > +};
> > +
> > +static void mediacodec_output_format(AVCodecContext *avctx)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + char *name = ff_AMediaCodec_getName(s->codec);
> > + FFAMediaFormat *out_format =
> ff_AMediaCodec_getOutputFormat(s->codec);
> > + char *str = ff_AMediaFormat_toString(out_format);
> > +
> > + av_log(avctx, AV_LOG_DEBUG, "MediaCodec encoder %s output format
> %s\n",
> > + name ? name : "unknown", str);
> > + av_free(name);
> > + av_free(str);
> > + ff_AMediaFormat_delete(out_format);
> > +}
> > +
> > +static av_cold int mediacodec_init(AVCodecContext *avctx)
> > +{
> > + const char *codec_mime = NULL;
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + FFAMediaFormat *format = NULL;
> > + int ret;
> > + int gop;
> > +
> > + if (s->use_ndk_codec < 0)
> > + s->use_ndk_codec = !av_jni_get_java_vm(avctx);
> > +
> > + switch (avctx->codec_id) {
> > + case AV_CODEC_ID_H264:
> > + codec_mime = "video/avc";
> > + break;
> > + case AV_CODEC_ID_HEVC:
> > + codec_mime = "video/hevc";
> > + break;
> > + default:
> > + av_assert0(0);
> > + }
> > +
> > + s->codec = ff_AMediaCodec_createEncoderByType(codec_mime,
> s->use_ndk_codec);
> > + if (!s->codec) {
> > + av_log(avctx, AV_LOG_ERROR, "Failed to create encoder for type
> %s\n",
> > + codec_mime);
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + format = ff_AMediaFormat_new(s->use_ndk_codec);
> > + if (!format) {
> > + av_log(avctx, AV_LOG_ERROR, "Failed to create media format\n");
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + ff_AMediaFormat_setString(format, "mime", codec_mime);
> > + s->width = FFALIGN(avctx->width, 16);
> > + s->height = avctx->height;
> > + ff_AMediaFormat_setInt32(format, "width", s->width);
> > + ff_AMediaFormat_setInt32(format, "height", s->height);
> >
> > Is it preferable to use constants like AMEDIAFORMAT_KEY_HEIGHT here?
> >
> > I don't have a preference either way.
>
> These KEYs aren’t string literal, they are global variables.
>
> extern const char* AMEDIAFORMAT_KEY_HEIGHT __INTRODUCED_IN(21);
>
> Some basic ones are introduced very late, e.g.,
>
> extern const char* AMEDIAFORMAT_KEY_CSD __INTRODUCED_IN(28);
> extern const char* AMEDIAFORMAT_KEY_CSD_0 __INTRODUCED_IN(28);
> extern const char* AMEDIAFORMAT_KEY_CSD_1 __INTRODUCED_IN(28);
> extern const char* AMEDIAFORMAT_KEY_CSD_2 __INTRODUCED_IN(28);
> extern const char* AMEDIAFORMAT_KEY_DISPLAY_CROP __INTRODUCED_IN(28);
>
> So we can’t use these AMEDIAFORMAT_KEY_ directly. dlsym() these global
> variables and with a fallback value is possible, just over-engineering.
>
> Google’s API design forced us to use string literal directly.
>
> >
> > It may be worth also passing in the same values to
> AMEDIAFORMAT_KEY_MAX_WIDTH and AMEDIAFORMAT_KEY_MAX_HEIGHT
>
> The documentation hints they are for decoder only:
>
> > A key describing the maximum expected width of the content in a video
> decoder format, in case there are resolution changes in the video content.
>
> https://developer.android.com/reference/android/media/MediaFormat
>
> >
> > And I think the unaligned width should be set into
> AMEDIAFORMAT_KEY_STRIDE
>
> Technically, KEY_WIDTH should be unaligned width, and KEY_STRIDE should
> be aligned width. However,
>
> 1. It’s well known that a lot of devices can’t handle width doesn’t aligned
> to 16, they will crash, report error or produce broken files.
>
> 2. CTS tests only verify that 16 aligned resolutions are supported.
>
> 3. There are alignment info in OMX and exported via MediaCodecInfo, like
>
> <MediaCodec name="c2.qti.avc.encoder" type="video/avc">
> <Alias name="OMX.qcom.video.encoder.avc" />
> <Limit name="size" min="96x96" max="4096x2304" />
> <Limit name="alignment" value="2x2" />
> <Limit name="block-size" value="16x16" />
>
> It can be missing on old devices, or worse, the info doesn’t reflect the
> real requirement (Maybe it was just a copy-paste from some places).
>
> I have an idea to fix the issue: always encoding in aligned resolution,
> then fix the crop setting by apply BSF (e.g., h264_metadata) on the
> bitstream.
>
> >
> > Finally, avctx->sample_aspect_ratio should be propagated into
> aspect-width and aspect-height. Something like this:
> >
> > AVRational sar = avctx->sample_aspect_ratio;
> > if (!sar.num || !sar.den)
> > sar.num = sar.den = 1;
> > av_reduce(&sar.num, &sar.den, sar.num, sar.den, 4096);
> > AMediaFormat_setInt32(format, "aspect-width", sar.num);
> > AMediaFormat_setInt32(format, "aspect-height", sar.den);
> >
>
> You mean sar-width/sar-height?
>
> EXPORT const char* AMEDIAFORMAT_KEY_SAR_HEIGHT = "sar-height";
> EXPORT const char* AMEDIAFORMAT_KEY_SAR_WIDTH = "sar-width";
>
> They are available in NDK since API level 29 (Android 10)
>
> extern const char* AMEDIAFORMAT_KEY_SAR_HEIGHT __INTRODUCED_IN(29);
> extern const char* AMEDIAFORMAT_KEY_SAR_WIDTH __INTRODUCED_IN(29);
>
> And they were added to Java MediaFormat since API level 30.
>
>
> https://developer.android.com/reference/android/media/MediaFormat#KEY_PIXEL_ASPECT_RATIO_WIDTH
>
> It’s uncommon for Java API got a feature late than NDK. I will keep
> a note and do more test before setting them.
>
> >
> > +
> > + if (avctx->pix_fmt == AV_PIX_FMT_MEDIACODEC) {
> > + AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
> > + if (avctx->hw_device_ctx) {
> > + AVHWDeviceContext *device_ctx =
> (AVHWDeviceContext*)(avctx->hw_device_ctx->data);
> > + AVMediaCodecDeviceContext *dev_ctx;
> > +
> > + if (device_ctx->type != AV_HWDEVICE_TYPE_MEDIACODEC ||
> !device_ctx->hwctx) {
> > + ret = AVERROR(EINVAL);
> > + goto bailout;
> > + }
> > + dev_ctx = device_ctx->hwctx;
> > + s->window = ff_mediacodec_surface_ref(dev_ctx->surface,
> dev_ctx->native_window, avctx);
> > + }
> > +
> > + if (!s->window && user_ctx && user_ctx->surface)
> > + s->window = ff_mediacodec_surface_ref(user_ctx->surface,
> NULL, avctx);
> > +
> > + if (!s->window) {
> > + ret = AVERROR(EINVAL);
> > + av_log(avctx, AV_LOG_ERROR, "Missing hw_device_ctx or
> hwaccel_context for AV_PIX_FMT_MEDIACODEC\n");
> > + goto bailout;
> > + }
> > + }
> > +
> > + for (int i = 0; i < FF_ARRAY_ELEMS(color_formats); i++) {
> > + if (avctx->pix_fmt == color_formats[i].pix_fmt) {
> > + ff_AMediaFormat_setInt32(format, "color-format",
> > + color_formats[i].color_format);
> > + break;
> >
> > do we need error/fallback if no match is found?
>
> The supported pix_fmts is specified
>
> .priv_data_size = sizeof(MediaCodecEncContext), \
> .p.pix_fmts = avc_pix_fmts, \
> .init = mediacodec_init, \
>
> It has been checked in encoder.c encode_preinit_video(), so check it again
> is
> optional but not necessary in my opinion.
>
> >
> >
> > + }
> > + }
> > +
> > + if (avctx->bit_rate)
> > + ff_AMediaFormat_setInt32(format, "bitrate", avctx->bit_rate);
> > + // frame-rate and i-frame-interval are required to configure codec
> > + if (avctx->framerate.num >= avctx->framerate.den &&
> avctx->framerate.den > 0)
> > + s->fps = avctx->framerate.num / avctx->framerate.den;
> > + else
> > + s->fps = 30;
> > + gop = round(avctx->gop_size / s->fps);
> > + if (gop == 0)
> > + gop = 2;
> >
> > can we read gop from avctx? in other implementations i have seen gop
> hardcoded to 1
>
> I think we should respect gop_size setting.
>
> I will add some log message and change default gop to 1 in patch v2.
>
> >
> > for fps value, can we use av_q2d(avctx->framerate)
>
> q2d() is for double, fps is integer here.
>
> >
> >
> > + ff_AMediaFormat_setInt32(format, "frame-rate", s->fps);
> > + ff_AMediaFormat_setInt32(format, "i-frame-interval", gop);
> > +
> >
> > for H264 encoding, you can pass "profile" and "level". for example
> profile=0x08 for High and level=0x4000 for 5.0
> >
> > https://stackoverflow.com/a/38999412/332798
>
> I’m planning to add profile/level support after the basic patch is applied.
> There are some unresolved issue here:
>
> 1. I have a device which failed at configure() when profile has been set,
> even for baseline.
>
> 2. DTS is missing in MediaCodec API. For the default baseline profile,
> there
> is no problem. Trouble comes when we set profile to mainline or high. If
> Surface/ANativeWindow is used, and the frames come from our MediaCodec
> decoder
> wrapper, we can control it’s 'render' (send to encoder's surface) via
> av_mediacodec_release_buffer(). A DTS generation strategy works in this
> case.
> However, if frames comes from other sources, like a camera, there is no way
> to control the 'render' yet, so DTS is missing in this case.
>
> Configure profile/level and B frames should come together.
>
> >
> >
> > +
> > + ret = ff_AMediaCodec_getConfigureFlagEncode(s->codec);
> > + ret = ff_AMediaCodec_configure(s->codec, format, s->window, NULL,
> ret);
> > + if (ret) {
> > + av_log(avctx, AV_LOG_ERROR, "MediaCodec configure failed,
> %s\n", av_err2str(ret));
> > + goto bailout;
> > + }
> > +
> > + ret = ff_AMediaCodec_start(s->codec);
> > + if (ret) {
> > + av_log(avctx, AV_LOG_ERROR, "MediaCodec failed to start, %s\n",
> av_err2str(ret));
> > + goto bailout;
> > + }
> > +
> > + mediacodec_output_format(avctx);
> > +
> > + s->frame = av_frame_alloc();
> > + if (!s->frame)
> > + ret = AVERROR(ENOMEM);
> > +
> > +bailout:
> > + if (format)
> > + ff_AMediaFormat_delete(format);
> > + return ret;
> > +}
> > +
> > +static int mediacodec_receive(AVCodecContext *avctx,
> > + AVPacket *pkt,
> > + int *got_packet)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + FFAMediaCodec *codec = s->codec;
> > + FFAMediaCodecBufferInfo out_info = {0};
> > + uint8_t *out_buf;
> > + size_t out_size = 0;
> > + int ret;
> > + int extradata_size = 0;
> > + int64_t timeout_us = s->eof_sent ? OUTPUT_DEQUEUE_TIMEOUT_US : 0;
> > + ssize_t index = ff_AMediaCodec_dequeueOutputBuffer(codec,
> &out_info, timeout_us);
> > +
> > + if (ff_AMediaCodec_infoTryAgainLater(codec, index))
> > + return AVERROR(EAGAIN);
> > +
> > + if (ff_AMediaCodec_infoOutputFormatChanged(codec, index)) {
> > + mediacodec_output_format(avctx);
> > + return AVERROR(EAGAIN);
> > + }
> > +
> > + if (ff_AMediaCodec_infoOutputBuffersChanged(codec, index)) {
> > + ff_AMediaCodec_cleanOutputBuffers(codec);
> > + return AVERROR(EAGAIN);
> > + }
> > +
> > + if (index < 0)
> > + return AVERROR_EXTERNAL;
> > +
> > + if (out_info.flags & ff_AMediaCodec_getBufferFlagEndOfStream(codec))
> > + return AVERROR_EOF;
> > +
> > + out_buf = ff_AMediaCodec_getOutputBuffer(codec, index, &out_size);
> > + if (!out_buf) {
> > + ret = AVERROR_EXTERNAL;
> > + goto bailout;
> > + }
> > +
> > + if (out_info.flags &
> ff_AMediaCodec_getBufferFlagCodecConfig(codec)) {
> > + ret = av_reallocp(&s->extradata, out_info.size);
> > + if (ret)
> > + goto bailout;
> > +
> > + s->extradata_size = out_info.size;
> > + memcpy(s->extradata, out_buf + out_info.offset, out_info.size);
> > + ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
> > + // try immediately
> > + return mediacodec_receive(avctx, pkt, got_packet);
> > + }
> > +
> > + ret = ff_get_encode_buffer(avctx, pkt, out_info.size +
> s->extradata_size, 0);
> > + if (ret < 0)
> > + goto bailout;
> > +
> > + if (s->extradata_size) {
> > + extradata_size = s->extradata_size;
> > + s->extradata_size = 0;
> > + memcpy(pkt->data, s->extradata, extradata_size);
> > + }
> > + memcpy(pkt->data + extradata_size, out_buf + out_info.offset,
> out_info.size);
> > + pkt->pts = av_rescale_q(out_info.presentationTimeUs,
> AV_TIME_BASE_Q, avctx->time_base);
> > + if (s->ts_tail != s->ts_head) {
> > + pkt->dts = s->timestamps[s->ts_tail];
> > + s->ts_tail = (s->ts_tail + 1) % FF_ARRAY_ELEMS(s->timestamps);
> > + }
> > +
> > + if (out_info.flags & ff_AMediaCodec_getBufferFlagKeyFrame(codec))
> > + pkt->flags |= AV_PKT_FLAG_KEY;
> > + ret = 0;
> > + *got_packet = 1;
> > +
> > + av_log(avctx, AV_LOG_TRACE, "receive packet pts %" PRId64 " dts %"
> PRId64
> > + " flags %d extradata %d\n",
> > + pkt->pts, pkt->dts, pkt->flags, extradata_size);
> > +
> > +bailout:
> > + ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
> > + return ret;
> > +}
> > +
> > +static void copy_frame_to_buffer(AVCodecContext *avctx, const AVFrame
> *frame, uint8_t *dst, size_t size)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + uint8_t *dst_data[4] = {};
> > + int dst_linesize[4] = {};
> > + const uint8_t *src_data[4] = {
> > + frame->data[0], frame->data[1], frame->data[2],
> frame->data[3]
> > + };
> > +
> > + if (avctx->pix_fmt == AV_PIX_FMT_YUV420P) {
> > + dst_data[0] = dst;
> > + dst_data[1] = dst + s->width * s->height;
> > + dst_data[2] = dst_data[1] + s->width * s->height / 4;
> > +
> > + dst_linesize[0] = s->width;
> > + dst_linesize[1] = dst_linesize[2] = s->width / 2;
> > + } else if (avctx->pix_fmt == AV_PIX_FMT_NV12) {
> > + dst_data[0] = dst;
> > + dst_data[1] = dst + s->width * s->height;
> > +
> > + dst_linesize[0] = s->width;
> > + dst_linesize[1] = s->width;
> > + } else {
> > + av_assert0(0);
> > + }
> > +
> > + av_image_copy(dst_data, dst_linesize, src_data, frame->linesize,
> > + avctx->pix_fmt, avctx->width, avctx->height);
> > +}
> > +
> > +static int mediacodec_send(AVCodecContext *avctx,
> > + const AVFrame *frame) {
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + FFAMediaCodec *codec = s->codec;
> > + ssize_t index;
> > + uint8_t *input_buf = NULL;
> > + size_t input_size = 0;
> > + int64_t pts = 0;
> > + uint32_t flags = 0;
> > + int64_t timeout_us;
> > +
> > + if (s->eof_sent)
> > + return 0;
> > +
> > + if (s->window) {
> > + if (!frame) {
> > + s->eof_sent = 1;
> > + return ff_AMediaCodec_signalEndOfInputStream(codec);
> > + }
> > +
> > +
> > + if (frame->data[3]) {
> > + pts = av_rescale_q(frame->pts, avctx->time_base,
> AV_TIME_BASE_Q);
> > + s->timestamps[s->ts_head] = frame->pts;
> > + s->ts_head = (s->ts_head + 1) %
> FF_ARRAY_ELEMS(s->timestamps);
> > +
> > + av_mediacodec_release_buffer((AVMediaCodecBuffer
> *)frame->data[3], 1);
> > + }
> > + return 0;
> > + }
> > +
> > + timeout_us = INPUT_DEQUEUE_TIMEOUT_US;
> > + index = ff_AMediaCodec_dequeueInputBuffer(codec, timeout_us);
> > + if (ff_AMediaCodec_infoTryAgainLater(codec, index))
> > + return AVERROR(EAGAIN);
> > +
> > + if (index < 0) {
> > + av_log(avctx, AV_LOG_ERROR, "dequeue input buffer failed, %zd",
> index);
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + if (frame) {
> > + input_buf = ff_AMediaCodec_getInputBuffer(codec, index,
> &input_size);
> > + copy_frame_to_buffer(avctx, frame, input_buf, input_size);
> > +
> > + pts = av_rescale_q(frame->pts, avctx->time_base,
> AV_TIME_BASE_Q);
> > +
> > + s->timestamps[s->ts_head] = frame->pts;
> > + s->ts_head = (s->ts_head + 1) % FF_ARRAY_ELEMS(s->timestamps);
> > + } else {
> > + flags |= ff_AMediaCodec_getBufferFlagEndOfStream(codec);
> > + s->eof_sent = 1;
> > + }
> > +
> >
> > it would be nice to propagate keyframes here (frame->pict_type ==
> AV_PICTURE_TYPE_I). it is only possible on API26 with
> AMediaCodec_setParameters and PARAMETER_KEY_REQUEST_SYNC_FRAME
>
> It’s a nice feature. I’m planning to add support after the basic function
> is working.
>
> >
> >
> > + ff_AMediaCodec_queueInputBuffer(codec, index, 0, input_size, pts,
> flags);
> > + return 0;
> > +}
> > +
> > +static int mediacodec_encode(AVCodecContext *avctx, AVPacket *pkt)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + int ret;
> > + int got_packet = 0;
> > +
> > + // Return on three case:
> > + // 1. Serious error
> > + // 2. Got a packet success
> > + // 3. No AVFrame is available yet (don't return if get_frame return
> EOF)
> > + while (1) {
> > + ret = mediacodec_receive(avctx, pkt, &got_packet);
> > + if (!ret)
> > + return 0;
> > + else if (ret != AVERROR(EAGAIN))
> > + return ret;
> > +
> > + if (!s->frame->buf[0]) {
> > + ret = ff_encode_get_frame(avctx, s->frame);
> > + if (ret && ret != AVERROR_EOF)
> > + return ret;
> > + }
> > +
> > + ret = mediacodec_send(avctx, s->frame->buf[0] ? s->frame :
> NULL);
> > + if (!ret)
> > + av_frame_unref(s->frame);
> > + else if (ret != AVERROR(EAGAIN))
> > + return ret;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static av_cold int mediacodec_close(AVCodecContext *avctx)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + if (s->codec) {
> > + ff_AMediaCodec_stop(s->codec);
> > + ff_AMediaCodec_delete(s->codec);
> > + s->codec = NULL;
> > + }
> > +
> > + if (s->window) {
> > + ff_mediacodec_surface_unref(s->window, avctx);
> > + s->window = NULL;
> > + }
> > +
> > + av_frame_free(&s->frame);
> > +
> > + return 0;
> > +}
> > +
> > +static const AVCodecHWConfigInternal *const mediacodec_hw_configs[] = {
> > + &(const AVCodecHWConfigInternal) {
> > + .public = {
> > + .pix_fmt = AV_PIX_FMT_MEDIACODEC,
> > + .methods = AV_CODEC_HW_CONFIG_METHOD_AD_HOC |
> > + AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX,
> > + .device_type = AV_HWDEVICE_TYPE_MEDIACODEC,
> > + },
> > + .hwaccel = NULL,
> > + },
> > + NULL
> > +};
> > +
> > +#define OFFSET(x) offsetof(MediaCodecEncContext, x)
> > +#define VE AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM
> > +static const AVOption common_options[] = {
> > + { "ndk_codec", "Use MediaCodec from NDK",
> > + OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 =
> -1}, -1, 1, VE },
> > + { NULL },
> > +};
> > +
> > +#define MEDIACODEC_ENCODER_CLASS(name) \
> > +static const AVClass name ## _mediacodec_class = { \
> > + .class_name = #name "_mediacodec", \
> > + .item_name = av_default_item_name, \
> > + .option = common_options, \
> > + .version = LIBAVUTIL_VERSION_INT, \
> > +}; \
> > +
> > +#define DECLARE_MEDIACODEC_ENCODER(short_name, long_name, codec_id)
> \
> > +MEDIACODEC_ENCODER_CLASS(short_name)
> \
> > +const FFCodec ff_ ## short_name ## _mediacodec_encoder = {
> \
> > + .p.name = #short_name "_mediacodec",
> \
> > + CODEC_LONG_NAME(long_name " Android MediaCodec encoder"),
> \
> > + .p.type = AVMEDIA_TYPE_VIDEO,
> \
> > + .p.id = codec_id,
> \
> > + .p.capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY
> \
> > + | AV_CODEC_CAP_HARDWARE,
> \
> > + .priv_data_size = sizeof(MediaCodecEncContext),
> \
> > + .p.pix_fmts = avc_pix_fmts,
> \
> > + .init = mediacodec_init,
> \
> > + FF_CODEC_RECEIVE_PACKET_CB(mediacodec_encode),
> \
> > + .close = mediacodec_close,
> \
> > + .p.priv_class = &short_name ## _mediacodec_class,
> \
> > + .caps_internal = FF_CODEC_CAP_INIT_CLEANUP,
> \
> > + .p.wrapper_name = "mediacodec",
> \
> > + .hw_configs = mediacodec_hw_configs,
> \
> > +};
> \
> > +
> > +#if CONFIG_H264_MEDIACODEC_ENCODER
> > +DECLARE_MEDIACODEC_ENCODER(h264, "H.264", AV_CODEC_ID_H264)
> > +#endif
> > +
> > +#if CONFIG_HEVC_MEDIACODEC_ENCODER
> > +DECLARE_MEDIACODEC_ENCODER(hevc, "H.265", AV_CODEC_ID_HEVC)
> > +#endif
> > diff --git a/libavcodec/version.h b/libavcodec/version.h
> > index 43d0d9a9fc..86ac0f3871 100644
> > --- a/libavcodec/version.h
> > +++ b/libavcodec/version.h
> > @@ -29,8 +29,8 @@
> >
> > #include "version_major.h"
> >
> > -#define LIBAVCODEC_VERSION_MINOR 51
> > -#define LIBAVCODEC_VERSION_MICRO 101
> > +#define LIBAVCODEC_VERSION_MINOR 52
> > +#define LIBAVCODEC_VERSION_MICRO 100
> >
> > #define LIBAVCODEC_VERSION_INT
> AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \
> >
> LIBAVCODEC_VERSION_MINOR, \
> > --
> > 2.25.1
> >
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder
2022-11-19 17:13 ` "zhilizhao(赵志立)"
2022-11-19 18:14 ` Olivier Ayache
@ 2022-11-20 23:44 ` Aman Karmani
2022-11-21 4:40 ` "zhilizhao(赵志立)"
1 sibling, 1 reply; 22+ messages in thread
From: Aman Karmani @ 2022-11-20 23:44 UTC (permalink / raw)
To: zhilizhao(赵志立)
Cc: FFmpeg development discussions and patches
On Sat, Nov 19, 2022 at 9:13 AM "zhilizhao(赵志立)" <quinkblack@foxmail.com>
wrote:
>
>
> > On Nov 19, 2022, at 17:15, Aman Karmani <aman@tmm1.net> wrote:
> >
> >
> >
> > On Sun, Oct 23, 2022 at 8:17 PM Zhao Zhili <quinkblack@foxmail.com>
> wrote:
> > From: Zhao Zhili <zhilizhao@tencent.com>
> >
> > Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
> > ---
> > Changelog | 1 +
> > configure | 4 +
> > libavcodec/Makefile | 2 +
> > libavcodec/allcodecs.c | 2 +
> > libavcodec/mediacodec_wrapper.c | 102 ++++++-
> > libavcodec/mediacodec_wrapper.h | 8 +
> > libavcodec/mediacodecenc.c | 495 ++++++++++++++++++++++++++++++++
> > libavcodec/version.h | 4 +-
> > 8 files changed, 611 insertions(+), 7 deletions(-)
> > create mode 100644 libavcodec/mediacodecenc.c
> >
> > diff --git a/Changelog b/Changelog
> > index 9e203833aa..9e39a35972 100644
> > --- a/Changelog
> > +++ b/Changelog
> > @@ -19,6 +19,7 @@ version <next>:
> > - DTS to PTS reorder bsf
> > - ViewQuest VQC decoder
> > - MediaCodec decoder via NDKMediaCodec
> > +- MediaCodec encoder
> >
> >
> >
> > version 5.1:
> > diff --git a/configure b/configure
> > index ee2e3ba6ac..5114cda13f 100755
> > --- a/configure
> > +++ b/configure
> > @@ -3193,6 +3193,8 @@ h264_cuvid_decoder_select="h264_mp4toannexb_bsf"
> > h264_mediacodec_decoder_deps="mediacodec"
> > h264_mediacodec_decoder_extralibs="-landroid"
> > h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser"
> > +h264_mediacodec_encoder_deps="mediacodec"
> > +h264_mediacodec_encoder_extralibs="-landroid"
> > h264_mf_encoder_deps="mediafoundation"
> > h264_mmal_decoder_deps="mmal"
> > h264_nvenc_encoder_deps="nvenc"
> > @@ -3212,6 +3214,8 @@ hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf"
> > hevc_mediacodec_decoder_deps="mediacodec"
> > hevc_mediacodec_decoder_extralibs="-landroid"
> > hevc_mediacodec_decoder_select="hevc_mp4toannexb_bsf hevc_parser"
> > +hevc_mediacodec_encoder_deps="mediacodec"
> > +hevc_mediacodec_encoder_extralibs="-landroid"
> > hevc_mf_encoder_deps="mediafoundation"
> > hevc_nvenc_encoder_deps="nvenc"
> > hevc_nvenc_encoder_select="atsc_a53"
> > diff --git a/libavcodec/Makefile b/libavcodec/Makefile
> > index 90c7f113a3..7d0b513eec 100644
> > --- a/libavcodec/Makefile
> > +++ b/libavcodec/Makefile
> > @@ -393,6 +393,7 @@ OBJS-$(CONFIG_H264_DECODER) += h264dec.o
> h264_cabac.o h264_cavlc.o \
> > OBJS-$(CONFIG_H264_AMF_ENCODER) += amfenc_h264.o
> > OBJS-$(CONFIG_H264_CUVID_DECODER) += cuviddec.o
> > OBJS-$(CONFIG_H264_MEDIACODEC_DECODER) += mediacodecdec.o
> > +OBJS-$(CONFIG_H264_MEDIACODEC_ENCODER) += mediacodecenc.o
> > OBJS-$(CONFIG_H264_MF_ENCODER) += mfenc.o mf_utils.o
> > OBJS-$(CONFIG_H264_MMAL_DECODER) += mmaldec.o
> > OBJS-$(CONFIG_H264_NVENC_ENCODER) += nvenc_h264.o nvenc.o
> > @@ -417,6 +418,7 @@ OBJS-$(CONFIG_HEVC_DECODER) += hevcdec.o
> hevc_mvs.o \
> > OBJS-$(CONFIG_HEVC_AMF_ENCODER) += amfenc_hevc.o
> > OBJS-$(CONFIG_HEVC_CUVID_DECODER) += cuviddec.o
> > OBJS-$(CONFIG_HEVC_MEDIACODEC_DECODER) += mediacodecdec.o
> > +OBJS-$(CONFIG_HEVC_MEDIACODEC_ENCODER) += mediacodecenc.o
> > OBJS-$(CONFIG_HEVC_MF_ENCODER) += mfenc.o mf_utils.o
> > OBJS-$(CONFIG_HEVC_NVENC_ENCODER) += nvenc_hevc.o nvenc.o
> > OBJS-$(CONFIG_HEVC_QSV_DECODER) += qsvdec.o
> > diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
> > index 46ad3b5a25..4c33a9ec3c 100644
> > --- a/libavcodec/allcodecs.c
> > +++ b/libavcodec/allcodecs.c
> > @@ -154,6 +154,7 @@ extern const FFCodec ff_h264_decoder;
> > extern const FFCodec ff_h264_crystalhd_decoder;
> > extern const FFCodec ff_h264_v4l2m2m_decoder;
> > extern const FFCodec ff_h264_mediacodec_decoder;
> > +extern const FFCodec ff_h264_mediacodec_encoder;
> > extern const FFCodec ff_h264_mmal_decoder;
> > extern const FFCodec ff_h264_qsv_decoder;
> > extern const FFCodec ff_h264_rkmpp_decoder;
> > @@ -842,6 +843,7 @@ extern const FFCodec ff_h264_videotoolbox_encoder;
> > extern const FFCodec ff_hevc_amf_encoder;
> > extern const FFCodec ff_hevc_cuvid_decoder;
> > extern const FFCodec ff_hevc_mediacodec_decoder;
> > +extern const FFCodec ff_hevc_mediacodec_encoder;
> > extern const FFCodec ff_hevc_mf_encoder;
> > extern const FFCodec ff_hevc_nvenc_encoder;
> > extern const FFCodec ff_hevc_qsv_encoder;
> > diff --git a/libavcodec/mediacodec_wrapper.c
> b/libavcodec/mediacodec_wrapper.c
> > index 284d615980..5d1a32031d 100644
> > --- a/libavcodec/mediacodec_wrapper.c
> > +++ b/libavcodec/mediacodec_wrapper.c
> > @@ -212,6 +212,9 @@ struct JNIAMediaCodecFields {
> > jmethodID release_output_buffer_id;
> > jmethodID release_output_buffer_at_time_id;
> >
> > + jmethodID set_input_surface_id;
> > + jmethodID signal_end_of_input_stream_id;
> > +
> > jclass mediainfo_class;
> >
> > jmethodID init_id;
> > @@ -261,6 +264,9 @@ static const struct FFJniField
> jni_amediacodec_mapping[] = {
> > { "android/media/MediaCodec", "releaseOutputBuffer", "(IZ)V",
> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields,
> release_output_buffer_id), 1 },
> > { "android/media/MediaCodec", "releaseOutputBuffer", "(IJ)V",
> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields,
> release_output_buffer_at_time_id), 0 },
> >
> > + { "android/media/MediaCodec", "setInputSurface",
> "(Landroid/view/Surface;)V", FF_JNI_METHOD, offsetof(struct
> JNIAMediaCodecFields, set_input_surface_id), 0 },
> > + { "android/media/MediaCodec", "signalEndOfInputStream", "()V",
> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields,
> signal_end_of_input_stream_id), 0 },
> > +
> > { "android/media/MediaCodec$BufferInfo", NULL, NULL, FF_JNI_CLASS,
> offsetof(struct JNIAMediaCodecFields, mediainfo_class), 1 },
> >
> > { "android/media/MediaCodec.BufferInfo", "<init>", "()V",
> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, init_id), 1 },
> > @@ -1385,7 +1391,26 @@ static int mediacodec_jni_configure(FFAMediaCodec
> *ctx,
> >
> > JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
> >
> > - (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.configure_id, format->object, surface, NULL, flags);
> > + if (flags & codec->CONFIGURE_FLAG_ENCODE) {
> > + if (surface && !codec->jfields.set_input_surface_id) {
> > + av_log(ctx, AV_LOG_ERROR, "System doesn't support
> setInputSurface\n");
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.configure_id, format->object, NULL, NULL, flags);
> > + if (ff_jni_exception_check(env, 1, codec) < 0)
> > + return AVERROR_EXTERNAL;
> > +
> > + if (!surface)
> > + return 0;
> > +
> > + (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.set_input_surface_id, surface);
> > + if (ff_jni_exception_check(env, 1, codec) < 0)
> > + return AVERROR_EXTERNAL;
> > + return 0;
> > + } else {
> > + (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.configure_id, format->object, surface, NULL, flags);
> > + }
> > if (ff_jni_exception_check(env, 1, codec) < 0) {
> > ret = AVERROR_EXTERNAL;
> > goto fail;
> > @@ -1743,6 +1768,22 @@ fail:
> > return ret;
> > }
> >
> > +static int mediacodec_jni_signalEndOfInputStream(FFAMediaCodec *ctx)
> > +{
> > + int ret = 0;
> > + JNIEnv *env = NULL;
> > + FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
> > +
> > + JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
> > +
> > + (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.signal_end_of_input_stream_id);
> > + if (ff_jni_exception_check(env, 1, codec) < 0) {
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > static const FFAMediaFormat media_format_jni = {
> > .class = &amediaformat_class,
> >
> > @@ -1801,6 +1842,7 @@ static const FFAMediaCodec media_codec_jni = {
> >
> > .getConfigureFlagEncode = mediacodec_jni_getConfigureFlagEncode,
> > .cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers,
> > + .signalEndOfInputStream = mediacodec_jni_signalEndOfInputStream,
> > };
> >
> > typedef struct FFAMediaFormatNdk {
> > @@ -1866,6 +1908,10 @@ typedef struct FFAMediaCodecNdk {
> > // Available since API level 28.
> > media_status_t (*getName)(AMediaCodec*, char** out_name);
> > void (*releaseName)(AMediaCodec*, char* name);
> > +
> > + // Available since API level 26.
> > + media_status_t (*setInputSurface)(AMediaCodec*, ANativeWindow *);
> > + media_status_t (*signalEndOfInputStream)(AMediaCodec *);
> > } FFAMediaCodecNdk;
> >
> > static const FFAMediaFormat media_format_ndk;
> > @@ -2098,6 +2144,9 @@ static inline FFAMediaCodec *ndk_codec_create(int
> method, const char *arg) {
> > GET_SYMBOL(getName, 0)
> > GET_SYMBOL(releaseName, 0)
> >
> > + GET_SYMBOL(setInputSurface, 0)
> > + GET_SYMBOL(signalEndOfInputStream, 0)
> > +
> > #undef GET_SYMBOL
> >
> > switch (method) {
> > @@ -2184,10 +2233,32 @@ static int
> mediacodec_ndk_configure(FFAMediaCodec* ctx,
> > return AVERROR(EINVAL);
> > }
> >
> > - status = codec->configure(codec->impl, format->impl, native_window,
> NULL, flags);
> > - if (status != AMEDIA_OK) {
> > - av_log(codec, AV_LOG_ERROR, "configure failed, %d\n", status);
> > - return AVERROR_EXTERNAL;
> > + if (flags & AMEDIACODEC_CONFIGURE_FLAG_ENCODE) {
> > + if (native_window && !codec->setInputSurface) {
> > + av_log(ctx, AV_LOG_ERROR, "System doesn't support
> setInputSurface\n");
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + status = codec->configure(codec->impl, format->impl, NULL,
> NULL, flags);
> > + if (status != AMEDIA_OK) {
> > + av_log(codec, AV_LOG_ERROR, "Encoder configure failed,
> %d\n", status);
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + if (!native_window)
> > + return 0;
> > +
> > + status = codec->setInputSurface(codec->impl, native_window);
> > + if (status != AMEDIA_OK) {
> > + av_log(codec, AV_LOG_ERROR, "Encoder set input surface
> failed, %d\n", status);
> > + return AVERROR_EXTERNAL;
> > + }
> > + } else {
> > + status = codec->configure(codec->impl, format->impl,
> native_window, NULL, flags);
> > + if (status != AMEDIA_OK) {
> > + av_log(codec, AV_LOG_ERROR, "Decoder configure failed,
> %d\n", status);
> > + return AVERROR_EXTERNAL;
> > + }
> > }
> >
> > return 0;
> > @@ -2330,6 +2401,26 @@ static int
> mediacodec_ndk_cleanOutputBuffers(FFAMediaCodec *ctx)
> > return 0;
> > }
> >
> > +static int mediacodec_ndk_signalEndOfInputStream(FFAMediaCodec *ctx)
> > +{
> > + FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
> > + media_status_t status;
> > +
> > + if (!codec->signalEndOfInputStream) {
> > + av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream
> unavailable\n");
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + status = codec->signalEndOfInputStream(codec->impl);
> > + if (status != AMEDIA_OK) {
> > + av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream failed,
> %d\n", status);
> > + return AVERROR_EXTERNAL;
> > + }
> > + av_log(codec, AV_LOG_DEBUG, "signalEndOfInputStream success\n");
> > +
> > + return 0;
> > +}
> > +
> > static const FFAMediaFormat media_format_ndk = {
> > .class = &amediaformat_ndk_class,
> >
> > @@ -2388,6 +2479,7 @@ static const FFAMediaCodec media_codec_ndk = {
> >
> > .getConfigureFlagEncode = mediacodec_ndk_getConfigureFlagEncode,
> > .cleanOutputBuffers = mediacodec_ndk_cleanOutputBuffers,
> > + .signalEndOfInputStream = mediacodec_ndk_signalEndOfInputStream,
> > };
> >
> > FFAMediaFormat *ff_AMediaFormat_new(int ndk)
> > diff --git a/libavcodec/mediacodec_wrapper.h
> b/libavcodec/mediacodec_wrapper.h
> > index 7cf3f4aecd..f15ad66d83 100644
> > --- a/libavcodec/mediacodec_wrapper.h
> > +++ b/libavcodec/mediacodec_wrapper.h
> > @@ -192,6 +192,9 @@ struct FFAMediaCodec {
> > int (*getConfigureFlagEncode)(FFAMediaCodec *codec);
> >
> > int (*cleanOutputBuffers)(FFAMediaCodec *codec);
> > +
> > + // For encoder with FFANativeWindow as input.
> > + int (*signalEndOfInputStream)(FFAMediaCodec *);
> > };
> >
> > static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec)
> > @@ -311,6 +314,11 @@ static inline int
> ff_AMediaCodec_cleanOutputBuffers(FFAMediaCodec *codec)
> > return codec->cleanOutputBuffers(codec);
> > }
> >
> > +static inline int ff_AMediaCodec_signalEndOfInputStream(FFAMediaCodec
> *codec)
> > +{
> > + return codec->signalEndOfInputStream(codec);
> > +}
> > +
> > int ff_Build_SDK_INT(AVCodecContext *avctx);
> >
> > #endif /* AVCODEC_MEDIACODEC_WRAPPER_H */
> > diff --git a/libavcodec/mediacodecenc.c b/libavcodec/mediacodecenc.c
> > new file mode 100644
> > index 0000000000..c81050ec80
> > --- /dev/null
> > +++ b/libavcodec/mediacodecenc.c
> > @@ -0,0 +1,495 @@
> > +/*
> > + * Android MediaCodec encoders
> > + *
> > + * Copyright (c) 2022 Zhao Zhili <zhilizhao@tencent.com>
> > + *
> > + * This file is part of FFmpeg.
> > + *
> > + * FFmpeg is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public
> > + * License as published by the Free Software Foundation; either
> > + * version 2.1 of the License, or (at your option) any later version.
> > + *
> > + * FFmpeg is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with FFmpeg; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA
> > + */
> > +
> > +#include "config_components.h"
> > +
> > +#include "libavutil/avassert.h"
> > +#include "libavutil/hwcontext_mediacodec.h"
> > +#include "libavutil/imgutils.h"
> > +#include "libavutil/opt.h"
> > +
> > +#include "avcodec.h"
> > +#include "codec_internal.h"
> > +#include "encode.h"
> > +#include "hwconfig.h"
> > +#include "jni.h"
> > +#include "mediacodec.h"
> > +#include "mediacodec_wrapper.h"
> > +#include "mediacodecdec_common.h"
> > +
> > +#define INPUT_DEQUEUE_TIMEOUT_US 8000
> > +#define OUTPUT_DEQUEUE_TIMEOUT_US 8000
> > +
> > +typedef struct MediaCodecEncContext {
> > + AVClass *avclass;
> > + FFAMediaCodec *codec;
> > + int use_ndk_codec;
> > + FFANativeWindow *window;
> > +
> > + int fps;
> > + int width;
> > + int height;
> > +
> > + uint8_t *extradata;
> > + int extradata_size;
> > +
> > + // Since MediaCodec doesn't output DTS, use a timestamp queue to
> save pts
> > + // of AVFrame and generate DTS for AVPacket.
> > + //
> > + // This doesn't work when use Surface as input, in that case frames
> can be
> > + // sent to encoder without our notice. One exception is frames come
> from
> > + // our MediaCodec decoder wrapper, since we can control it's render
> by
> > + // av_mediacodec_release_buffer.
> > + int64_t timestamps[32];
> > + int ts_head;
> > + int ts_tail;
> > +
> > + int eof_sent;
> > +
> > + AVFrame *frame;
> > +} MediaCodecEncContext;
> > +
> > +enum {
> > + COLOR_FormatYUV420Planar = 0x13,
> > + COLOR_FormatYUV420SemiPlanar = 0x15,
> > + COLOR_FormatSurface = 0x7F000789,
> > +};
> > +
> > +static const struct {
> > + int color_format;
> > + enum AVPixelFormat pix_fmt;
> > +} color_formats[] = {
> > + { COLOR_FormatYUV420Planar, AV_PIX_FMT_YUV420P },
> > + { COLOR_FormatYUV420SemiPlanar, AV_PIX_FMT_NV12 },
> > + { COLOR_FormatSurface, AV_PIX_FMT_MEDIACODEC },
> > +};
> > +
> > +static const enum AVPixelFormat avc_pix_fmts[] = {
> > + AV_PIX_FMT_MEDIACODEC,
> > + AV_PIX_FMT_YUV420P,
> > + AV_PIX_FMT_NV12,
> > + AV_PIX_FMT_NONE
> > +};
> > +
> > +static void mediacodec_output_format(AVCodecContext *avctx)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + char *name = ff_AMediaCodec_getName(s->codec);
> > + FFAMediaFormat *out_format =
> ff_AMediaCodec_getOutputFormat(s->codec);
> > + char *str = ff_AMediaFormat_toString(out_format);
> > +
> > + av_log(avctx, AV_LOG_DEBUG, "MediaCodec encoder %s output format
> %s\n",
> > + name ? name : "unknown", str);
> > + av_free(name);
> > + av_free(str);
> > + ff_AMediaFormat_delete(out_format);
> > +}
> > +
> > +static av_cold int mediacodec_init(AVCodecContext *avctx)
> > +{
> > + const char *codec_mime = NULL;
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + FFAMediaFormat *format = NULL;
> > + int ret;
> > + int gop;
> > +
> > + if (s->use_ndk_codec < 0)
> > + s->use_ndk_codec = !av_jni_get_java_vm(avctx);
> > +
> > + switch (avctx->codec_id) {
> > + case AV_CODEC_ID_H264:
> > + codec_mime = "video/avc";
> > + break;
> > + case AV_CODEC_ID_HEVC:
> > + codec_mime = "video/hevc";
> > + break;
> > + default:
> > + av_assert0(0);
> > + }
> > +
> > + s->codec = ff_AMediaCodec_createEncoderByType(codec_mime,
> s->use_ndk_codec);
> > + if (!s->codec) {
> > + av_log(avctx, AV_LOG_ERROR, "Failed to create encoder for type
> %s\n",
> > + codec_mime);
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + format = ff_AMediaFormat_new(s->use_ndk_codec);
> > + if (!format) {
> > + av_log(avctx, AV_LOG_ERROR, "Failed to create media format\n");
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + ff_AMediaFormat_setString(format, "mime", codec_mime);
> > + s->width = FFALIGN(avctx->width, 16);
> > + s->height = avctx->height;
> > + ff_AMediaFormat_setInt32(format, "width", s->width);
> > + ff_AMediaFormat_setInt32(format, "height", s->height);
> >
> > Is it preferable to use constants like AMEDIAFORMAT_KEY_HEIGHT here?
> >
> > I don't have a preference either way.
>
> These KEYs aren’t string literal, they are global variables.
>
> extern const char* AMEDIAFORMAT_KEY_HEIGHT __INTRODUCED_IN(21);
>
> Some basic ones are introduced very late, e.g.,
>
> extern const char* AMEDIAFORMAT_KEY_CSD __INTRODUCED_IN(28);
> extern const char* AMEDIAFORMAT_KEY_CSD_0 __INTRODUCED_IN(28);
> extern const char* AMEDIAFORMAT_KEY_CSD_1 __INTRODUCED_IN(28);
> extern const char* AMEDIAFORMAT_KEY_CSD_2 __INTRODUCED_IN(28);
> extern const char* AMEDIAFORMAT_KEY_DISPLAY_CROP __INTRODUCED_IN(28);
>
> So we can’t use these AMEDIAFORMAT_KEY_ directly. dlsym() these global
> variables and with a fallback value is possible, just over-engineering.
>
> Google’s API design forced us to use string literal directly.
>
> >
> > It may be worth also passing in the same values to
> AMEDIAFORMAT_KEY_MAX_WIDTH and AMEDIAFORMAT_KEY_MAX_HEIGHT
>
> The documentation hints they are for decoder only:
>
> > A key describing the maximum expected width of the content in a video
> decoder format, in case there are resolution changes in the video content.
>
> https://developer.android.com/reference/android/media/MediaFormat
>
> >
> > And I think the unaligned width should be set into
> AMEDIAFORMAT_KEY_STRIDE
>
> Technically, KEY_WIDTH should be unaligned width, and KEY_STRIDE should
> be aligned width. However,
>
> 1. It’s well known that a lot of devices can’t handle width doesn’t aligned
> to 16, they will crash, report error or produce broken files.
>
> 2. CTS tests only verify that 16 aligned resolutions are supported.
>
> 3. There are alignment info in OMX and exported via MediaCodecInfo, like
>
> <MediaCodec name="c2.qti.avc.encoder" type="video/avc">
> <Alias name="OMX.qcom.video.encoder.avc" />
> <Limit name="size" min="96x96" max="4096x2304" />
> <Limit name="alignment" value="2x2" />
> <Limit name="block-size" value="16x16" />
>
> It can be missing on old devices, or worse, the info doesn’t reflect the
> real requirement (Maybe it was just a copy-paste from some places).
>
> I have an idea to fix the issue: always encoding in aligned resolution,
> then fix the crop setting by apply BSF (e.g., h264_metadata) on the
> bitstream.
>
> >
> > Finally, avctx->sample_aspect_ratio should be propagated into
> aspect-width and aspect-height. Something like this:
> >
> > AVRational sar = avctx->sample_aspect_ratio;
> > if (!sar.num || !sar.den)
> > sar.num = sar.den = 1;
> > av_reduce(&sar.num, &sar.den, sar.num, sar.den, 4096);
> > AMediaFormat_setInt32(format, "aspect-width", sar.num);
> > AMediaFormat_setInt32(format, "aspect-height", sar.den);
> >
>
> You mean sar-width/sar-height?
>
> EXPORT const char* AMEDIAFORMAT_KEY_SAR_HEIGHT = "sar-height";
> EXPORT const char* AMEDIAFORMAT_KEY_SAR_WIDTH = "sar-width";
>
> They are available in NDK since API level 29 (Android 10)
>
> extern const char* AMEDIAFORMAT_KEY_SAR_HEIGHT __INTRODUCED_IN(29);
> extern const char* AMEDIAFORMAT_KEY_SAR_WIDTH __INTRODUCED_IN(29);
>
> And they were added to Java MediaFormat since API level 30.
>
>
> https://developer.android.com/reference/android/media/MediaFormat#KEY_PIXEL_ASPECT_RATIO_WIDTH
>
> It’s uncommon for Java API got a feature late than NDK. I will keep
> a note and do more test before setting them.
>
> >
> > +
> > + if (avctx->pix_fmt == AV_PIX_FMT_MEDIACODEC) {
> > + AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
> > + if (avctx->hw_device_ctx) {
> > + AVHWDeviceContext *device_ctx =
> (AVHWDeviceContext*)(avctx->hw_device_ctx->data);
> > + AVMediaCodecDeviceContext *dev_ctx;
> > +
> > + if (device_ctx->type != AV_HWDEVICE_TYPE_MEDIACODEC ||
> !device_ctx->hwctx) {
> > + ret = AVERROR(EINVAL);
> > + goto bailout;
> > + }
> > + dev_ctx = device_ctx->hwctx;
> > + s->window = ff_mediacodec_surface_ref(dev_ctx->surface,
> dev_ctx->native_window, avctx);
> > + }
> > +
> > + if (!s->window && user_ctx && user_ctx->surface)
> > + s->window = ff_mediacodec_surface_ref(user_ctx->surface,
> NULL, avctx);
> > +
> > + if (!s->window) {
> > + ret = AVERROR(EINVAL);
> > + av_log(avctx, AV_LOG_ERROR, "Missing hw_device_ctx or
> hwaccel_context for AV_PIX_FMT_MEDIACODEC\n");
> > + goto bailout;
> > + }
> > + }
> > +
> > + for (int i = 0; i < FF_ARRAY_ELEMS(color_formats); i++) {
> > + if (avctx->pix_fmt == color_formats[i].pix_fmt) {
> > + ff_AMediaFormat_setInt32(format, "color-format",
> > + color_formats[i].color_format);
> > + break;
> >
> > do we need error/fallback if no match is found?
>
> The supported pix_fmts is specified
>
> .priv_data_size = sizeof(MediaCodecEncContext), \
> .p.pix_fmts = avc_pix_fmts, \
> .init = mediacodec_init, \
>
> It has been checked in encoder.c encode_preinit_video(), so check it again
> is
> optional but not necessary in my opinion.
>
> >
> >
> > + }
> > + }
> > +
> > + if (avctx->bit_rate)
> > + ff_AMediaFormat_setInt32(format, "bitrate", avctx->bit_rate);
> > + // frame-rate and i-frame-interval are required to configure codec
> > + if (avctx->framerate.num >= avctx->framerate.den &&
> avctx->framerate.den > 0)
> > + s->fps = avctx->framerate.num / avctx->framerate.den;
> > + else
> > + s->fps = 30;
> > + gop = round(avctx->gop_size / s->fps);
> > + if (gop == 0)
> > + gop = 2;
> >
> > can we read gop from avctx? in other implementations i have seen gop
> hardcoded to 1
>
> I think we should respect gop_size setting.
>
> I will add some log message and change default gop to 1 in patch v2.
>
> >
> > for fps value, can we use av_q2d(avctx->framerate)
>
> q2d() is for double, fps is integer here.
>
> >
> >
> > + ff_AMediaFormat_setInt32(format, "frame-rate", s->fps);
> > + ff_AMediaFormat_setInt32(format, "i-frame-interval", gop);
> > +
> >
> > for H264 encoding, you can pass "profile" and "level". for example
> profile=0x08 for High and level=0x4000 for 5.0
> >
> > https://stackoverflow.com/a/38999412/332798
>
> I’m planning to add profile/level support after the basic patch is applied.
> There are some unresolved issue here:
>
> 1. I have a device which failed at configure() when profile has been set,
> even for baseline.
>
Thanks for sharing this. Today I had the same experience,
with OMX.rk.video_encoder.avc trying to pass profile/level.
I see the following message in logcat. Did you see the same behavior?
11-20 21:55:16.132 5086 5098 W ACodec : [OMX.rk.video_encoder.avc]
stopping checking profiles after 32: 8/1
11-20 21:55:16.132 5086 5098 E ACodec : [OMX.rk.video_encoder.avc]
configureCodec returning error -1010
>
> 2. DTS is missing in MediaCodec API. For the default baseline profile,
> there
> is no problem. Trouble comes when we set profile to mainline or high. If
> Surface/ANativeWindow is used, and the frames come from our MediaCodec
> decoder
> wrapper, we can control it’s 'render' (send to encoder's surface) via
> av_mediacodec_release_buffer(). A DTS generation strategy works in this
> case.
> However, if frames comes from other sources, like a camera, there is no way
> to control the 'render' yet, so DTS is missing in this case.
>
> Configure profile/level and B frames should come together.
>
> >
> >
> > +
> > + ret = ff_AMediaCodec_getConfigureFlagEncode(s->codec);
> > + ret = ff_AMediaCodec_configure(s->codec, format, s->window, NULL,
> ret);
> > + if (ret) {
> > + av_log(avctx, AV_LOG_ERROR, "MediaCodec configure failed,
> %s\n", av_err2str(ret));
> > + goto bailout;
> > + }
> > +
> > + ret = ff_AMediaCodec_start(s->codec);
> > + if (ret) {
> > + av_log(avctx, AV_LOG_ERROR, "MediaCodec failed to start, %s\n",
> av_err2str(ret));
> > + goto bailout;
> > + }
> > +
> > + mediacodec_output_format(avctx);
> > +
> > + s->frame = av_frame_alloc();
> > + if (!s->frame)
> > + ret = AVERROR(ENOMEM);
> > +
> > +bailout:
> > + if (format)
> > + ff_AMediaFormat_delete(format);
> > + return ret;
> > +}
> > +
> > +static int mediacodec_receive(AVCodecContext *avctx,
> > + AVPacket *pkt,
> > + int *got_packet)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + FFAMediaCodec *codec = s->codec;
> > + FFAMediaCodecBufferInfo out_info = {0};
> > + uint8_t *out_buf;
> > + size_t out_size = 0;
> > + int ret;
> > + int extradata_size = 0;
> > + int64_t timeout_us = s->eof_sent ? OUTPUT_DEQUEUE_TIMEOUT_US : 0;
> > + ssize_t index = ff_AMediaCodec_dequeueOutputBuffer(codec,
> &out_info, timeout_us);
> > +
> > + if (ff_AMediaCodec_infoTryAgainLater(codec, index))
> > + return AVERROR(EAGAIN);
> > +
> > + if (ff_AMediaCodec_infoOutputFormatChanged(codec, index)) {
> > + mediacodec_output_format(avctx);
> > + return AVERROR(EAGAIN);
> > + }
> > +
> > + if (ff_AMediaCodec_infoOutputBuffersChanged(codec, index)) {
> > + ff_AMediaCodec_cleanOutputBuffers(codec);
> > + return AVERROR(EAGAIN);
> > + }
> > +
> > + if (index < 0)
> > + return AVERROR_EXTERNAL;
> > +
> > + if (out_info.flags & ff_AMediaCodec_getBufferFlagEndOfStream(codec))
> > + return AVERROR_EOF;
> > +
> > + out_buf = ff_AMediaCodec_getOutputBuffer(codec, index, &out_size);
> > + if (!out_buf) {
> > + ret = AVERROR_EXTERNAL;
> > + goto bailout;
> > + }
> > +
> > + if (out_info.flags &
> ff_AMediaCodec_getBufferFlagCodecConfig(codec)) {
> > + ret = av_reallocp(&s->extradata, out_info.size);
> > + if (ret)
> > + goto bailout;
> > +
> > + s->extradata_size = out_info.size;
> > + memcpy(s->extradata, out_buf + out_info.offset, out_info.size);
> > + ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
> > + // try immediately
> > + return mediacodec_receive(avctx, pkt, got_packet);
> > + }
> > +
> > + ret = ff_get_encode_buffer(avctx, pkt, out_info.size +
> s->extradata_size, 0);
> > + if (ret < 0)
> > + goto bailout;
> > +
> > + if (s->extradata_size) {
> > + extradata_size = s->extradata_size;
> > + s->extradata_size = 0;
> > + memcpy(pkt->data, s->extradata, extradata_size);
> > + }
> > + memcpy(pkt->data + extradata_size, out_buf + out_info.offset,
> out_info.size);
> > + pkt->pts = av_rescale_q(out_info.presentationTimeUs,
> AV_TIME_BASE_Q, avctx->time_base);
> > + if (s->ts_tail != s->ts_head) {
> > + pkt->dts = s->timestamps[s->ts_tail];
> > + s->ts_tail = (s->ts_tail + 1) % FF_ARRAY_ELEMS(s->timestamps);
> > + }
> > +
> > + if (out_info.flags & ff_AMediaCodec_getBufferFlagKeyFrame(codec))
> > + pkt->flags |= AV_PKT_FLAG_KEY;
> > + ret = 0;
> > + *got_packet = 1;
> > +
> > + av_log(avctx, AV_LOG_TRACE, "receive packet pts %" PRId64 " dts %"
> PRId64
> > + " flags %d extradata %d\n",
> > + pkt->pts, pkt->dts, pkt->flags, extradata_size);
> > +
> > +bailout:
> > + ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
> > + return ret;
> > +}
> > +
> > +static void copy_frame_to_buffer(AVCodecContext *avctx, const AVFrame
> *frame, uint8_t *dst, size_t size)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + uint8_t *dst_data[4] = {};
> > + int dst_linesize[4] = {};
> > + const uint8_t *src_data[4] = {
> > + frame->data[0], frame->data[1], frame->data[2],
> frame->data[3]
> > + };
> > +
> > + if (avctx->pix_fmt == AV_PIX_FMT_YUV420P) {
> > + dst_data[0] = dst;
> > + dst_data[1] = dst + s->width * s->height;
> > + dst_data[2] = dst_data[1] + s->width * s->height / 4;
> > +
> > + dst_linesize[0] = s->width;
> > + dst_linesize[1] = dst_linesize[2] = s->width / 2;
> > + } else if (avctx->pix_fmt == AV_PIX_FMT_NV12) {
> > + dst_data[0] = dst;
> > + dst_data[1] = dst + s->width * s->height;
> > +
> > + dst_linesize[0] = s->width;
> > + dst_linesize[1] = s->width;
> > + } else {
> > + av_assert0(0);
> > + }
> > +
> > + av_image_copy(dst_data, dst_linesize, src_data, frame->linesize,
> > + avctx->pix_fmt, avctx->width, avctx->height);
> > +}
> > +
> > +static int mediacodec_send(AVCodecContext *avctx,
> > + const AVFrame *frame) {
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + FFAMediaCodec *codec = s->codec;
> > + ssize_t index;
> > + uint8_t *input_buf = NULL;
> > + size_t input_size = 0;
> > + int64_t pts = 0;
> > + uint32_t flags = 0;
> > + int64_t timeout_us;
> > +
> > + if (s->eof_sent)
> > + return 0;
> > +
> > + if (s->window) {
> > + if (!frame) {
> > + s->eof_sent = 1;
> > + return ff_AMediaCodec_signalEndOfInputStream(codec);
> > + }
> > +
> > +
> > + if (frame->data[3]) {
> > + pts = av_rescale_q(frame->pts, avctx->time_base,
> AV_TIME_BASE_Q);
> > + s->timestamps[s->ts_head] = frame->pts;
> > + s->ts_head = (s->ts_head + 1) %
> FF_ARRAY_ELEMS(s->timestamps);
> > +
> > + av_mediacodec_release_buffer((AVMediaCodecBuffer
> *)frame->data[3], 1);
> > + }
> > + return 0;
> > + }
> > +
> > + timeout_us = INPUT_DEQUEUE_TIMEOUT_US;
> > + index = ff_AMediaCodec_dequeueInputBuffer(codec, timeout_us);
> > + if (ff_AMediaCodec_infoTryAgainLater(codec, index))
> > + return AVERROR(EAGAIN);
> > +
> > + if (index < 0) {
> > + av_log(avctx, AV_LOG_ERROR, "dequeue input buffer failed, %zd",
> index);
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + if (frame) {
> > + input_buf = ff_AMediaCodec_getInputBuffer(codec, index,
> &input_size);
> > + copy_frame_to_buffer(avctx, frame, input_buf, input_size);
> > +
> > + pts = av_rescale_q(frame->pts, avctx->time_base,
> AV_TIME_BASE_Q);
> > +
> > + s->timestamps[s->ts_head] = frame->pts;
> > + s->ts_head = (s->ts_head + 1) % FF_ARRAY_ELEMS(s->timestamps);
> > + } else {
> > + flags |= ff_AMediaCodec_getBufferFlagEndOfStream(codec);
> > + s->eof_sent = 1;
> > + }
> > +
> >
> > it would be nice to propagate keyframes here (frame->pict_type ==
> AV_PICTURE_TYPE_I). it is only possible on API26 with
> AMediaCodec_setParameters and PARAMETER_KEY_REQUEST_SYNC_FRAME
>
> It’s a nice feature. I’m planning to add support after the basic function
> is working.
>
> >
> >
> > + ff_AMediaCodec_queueInputBuffer(codec, index, 0, input_size, pts,
> flags);
> > + return 0;
> > +}
> > +
> > +static int mediacodec_encode(AVCodecContext *avctx, AVPacket *pkt)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + int ret;
> > + int got_packet = 0;
> > +
> > + // Return on three case:
> > + // 1. Serious error
> > + // 2. Got a packet success
> > + // 3. No AVFrame is available yet (don't return if get_frame return
> EOF)
> > + while (1) {
> > + ret = mediacodec_receive(avctx, pkt, &got_packet);
> > + if (!ret)
> > + return 0;
> > + else if (ret != AVERROR(EAGAIN))
> > + return ret;
> > +
> > + if (!s->frame->buf[0]) {
> > + ret = ff_encode_get_frame(avctx, s->frame);
> > + if (ret && ret != AVERROR_EOF)
> > + return ret;
> > + }
> > +
> > + ret = mediacodec_send(avctx, s->frame->buf[0] ? s->frame :
> NULL);
> > + if (!ret)
> > + av_frame_unref(s->frame);
> > + else if (ret != AVERROR(EAGAIN))
> > + return ret;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static av_cold int mediacodec_close(AVCodecContext *avctx)
> > +{
> > + MediaCodecEncContext *s = avctx->priv_data;
> > + if (s->codec) {
> > + ff_AMediaCodec_stop(s->codec);
> > + ff_AMediaCodec_delete(s->codec);
> > + s->codec = NULL;
> > + }
> > +
> > + if (s->window) {
> > + ff_mediacodec_surface_unref(s->window, avctx);
> > + s->window = NULL;
> > + }
> > +
> > + av_frame_free(&s->frame);
> > +
> > + return 0;
> > +}
> > +
> > +static const AVCodecHWConfigInternal *const mediacodec_hw_configs[] = {
> > + &(const AVCodecHWConfigInternal) {
> > + .public = {
> > + .pix_fmt = AV_PIX_FMT_MEDIACODEC,
> > + .methods = AV_CODEC_HW_CONFIG_METHOD_AD_HOC |
> > + AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX,
> > + .device_type = AV_HWDEVICE_TYPE_MEDIACODEC,
> > + },
> > + .hwaccel = NULL,
> > + },
> > + NULL
> > +};
> > +
> > +#define OFFSET(x) offsetof(MediaCodecEncContext, x)
> > +#define VE AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM
> > +static const AVOption common_options[] = {
> > + { "ndk_codec", "Use MediaCodec from NDK",
> > + OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 =
> -1}, -1, 1, VE },
> > + { NULL },
> > +};
> > +
> > +#define MEDIACODEC_ENCODER_CLASS(name) \
> > +static const AVClass name ## _mediacodec_class = { \
> > + .class_name = #name "_mediacodec", \
> > + .item_name = av_default_item_name, \
> > + .option = common_options, \
> > + .version = LIBAVUTIL_VERSION_INT, \
> > +}; \
> > +
> > +#define DECLARE_MEDIACODEC_ENCODER(short_name, long_name, codec_id)
> \
> > +MEDIACODEC_ENCODER_CLASS(short_name)
> \
> > +const FFCodec ff_ ## short_name ## _mediacodec_encoder = {
> \
> > + .p.name = #short_name "_mediacodec",
> \
> > + CODEC_LONG_NAME(long_name " Android MediaCodec encoder"),
> \
> > + .p.type = AVMEDIA_TYPE_VIDEO,
> \
> > + .p.id = codec_id,
> \
> > + .p.capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY
> \
> > + | AV_CODEC_CAP_HARDWARE,
> \
> > + .priv_data_size = sizeof(MediaCodecEncContext),
> \
> > + .p.pix_fmts = avc_pix_fmts,
> \
> > + .init = mediacodec_init,
> \
> > + FF_CODEC_RECEIVE_PACKET_CB(mediacodec_encode),
> \
> > + .close = mediacodec_close,
> \
> > + .p.priv_class = &short_name ## _mediacodec_class,
> \
> > + .caps_internal = FF_CODEC_CAP_INIT_CLEANUP,
> \
> > + .p.wrapper_name = "mediacodec",
> \
> > + .hw_configs = mediacodec_hw_configs,
> \
> > +};
> \
> > +
> > +#if CONFIG_H264_MEDIACODEC_ENCODER
> > +DECLARE_MEDIACODEC_ENCODER(h264, "H.264", AV_CODEC_ID_H264)
> > +#endif
> > +
> > +#if CONFIG_HEVC_MEDIACODEC_ENCODER
> > +DECLARE_MEDIACODEC_ENCODER(hevc, "H.265", AV_CODEC_ID_HEVC)
> > +#endif
> > diff --git a/libavcodec/version.h b/libavcodec/version.h
> > index 43d0d9a9fc..86ac0f3871 100644
> > --- a/libavcodec/version.h
> > +++ b/libavcodec/version.h
> > @@ -29,8 +29,8 @@
> >
> > #include "version_major.h"
> >
> > -#define LIBAVCODEC_VERSION_MINOR 51
> > -#define LIBAVCODEC_VERSION_MICRO 101
> > +#define LIBAVCODEC_VERSION_MINOR 52
> > +#define LIBAVCODEC_VERSION_MICRO 100
> >
> > #define LIBAVCODEC_VERSION_INT
> AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \
> >
> LIBAVCODEC_VERSION_MINOR, \
> > --
> > 2.25.1
> >
>
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder
2022-11-20 23:44 ` Aman Karmani
@ 2022-11-21 4:40 ` "zhilizhao(赵志立)"
0 siblings, 0 replies; 22+ messages in thread
From: "zhilizhao(赵志立)" @ 2022-11-21 4:40 UTC (permalink / raw)
To: FFmpeg development discussions and patches
> On Nov 21, 2022, at 07:44, Aman Karmani <ffmpeg@tmm1.net> wrote:
>
> On Sat, Nov 19, 2022 at 9:13 AM "zhilizhao(赵志立)" <quinkblack@foxmail.com>
> wrote:
>
>>
>>
>>> On Nov 19, 2022, at 17:15, Aman Karmani <aman@tmm1.net> wrote:
>>>
>>>
>>>
>>> On Sun, Oct 23, 2022 at 8:17 PM Zhao Zhili <quinkblack@foxmail.com>
>> wrote:
>>> From: Zhao Zhili <zhilizhao@tencent.com>
>>>
>>> Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
>>> ---
>>> Changelog | 1 +
>>> configure | 4 +
>>> libavcodec/Makefile | 2 +
>>> libavcodec/allcodecs.c | 2 +
>>> libavcodec/mediacodec_wrapper.c | 102 ++++++-
>>> libavcodec/mediacodec_wrapper.h | 8 +
>>> libavcodec/mediacodecenc.c | 495 ++++++++++++++++++++++++++++++++
>>> libavcodec/version.h | 4 +-
>>> 8 files changed, 611 insertions(+), 7 deletions(-)
>>> create mode 100644 libavcodec/mediacodecenc.c
>>>
>>> diff --git a/Changelog b/Changelog
>>> index 9e203833aa..9e39a35972 100644
>>> --- a/Changelog
>>> +++ b/Changelog
>>> @@ -19,6 +19,7 @@ version <next>:
>>> - DTS to PTS reorder bsf
>>> - ViewQuest VQC decoder
>>> - MediaCodec decoder via NDKMediaCodec
>>> +- MediaCodec encoder
>>>
>>>
>>>
>>> version 5.1:
>>> diff --git a/configure b/configure
>>> index ee2e3ba6ac..5114cda13f 100755
>>> --- a/configure
>>> +++ b/configure
>>> @@ -3193,6 +3193,8 @@ h264_cuvid_decoder_select="h264_mp4toannexb_bsf"
>>> h264_mediacodec_decoder_deps="mediacodec"
>>> h264_mediacodec_decoder_extralibs="-landroid"
>>> h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser"
>>> +h264_mediacodec_encoder_deps="mediacodec"
>>> +h264_mediacodec_encoder_extralibs="-landroid"
>>> h264_mf_encoder_deps="mediafoundation"
>>> h264_mmal_decoder_deps="mmal"
>>> h264_nvenc_encoder_deps="nvenc"
>>> @@ -3212,6 +3214,8 @@ hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf"
>>> hevc_mediacodec_decoder_deps="mediacodec"
>>> hevc_mediacodec_decoder_extralibs="-landroid"
>>> hevc_mediacodec_decoder_select="hevc_mp4toannexb_bsf hevc_parser"
>>> +hevc_mediacodec_encoder_deps="mediacodec"
>>> +hevc_mediacodec_encoder_extralibs="-landroid"
>>> hevc_mf_encoder_deps="mediafoundation"
>>> hevc_nvenc_encoder_deps="nvenc"
>>> hevc_nvenc_encoder_select="atsc_a53"
>>> diff --git a/libavcodec/Makefile b/libavcodec/Makefile
>>> index 90c7f113a3..7d0b513eec 100644
>>> --- a/libavcodec/Makefile
>>> +++ b/libavcodec/Makefile
>>> @@ -393,6 +393,7 @@ OBJS-$(CONFIG_H264_DECODER) += h264dec.o
>> h264_cabac.o h264_cavlc.o \
>>> OBJS-$(CONFIG_H264_AMF_ENCODER) += amfenc_h264.o
>>> OBJS-$(CONFIG_H264_CUVID_DECODER) += cuviddec.o
>>> OBJS-$(CONFIG_H264_MEDIACODEC_DECODER) += mediacodecdec.o
>>> +OBJS-$(CONFIG_H264_MEDIACODEC_ENCODER) += mediacodecenc.o
>>> OBJS-$(CONFIG_H264_MF_ENCODER) += mfenc.o mf_utils.o
>>> OBJS-$(CONFIG_H264_MMAL_DECODER) += mmaldec.o
>>> OBJS-$(CONFIG_H264_NVENC_ENCODER) += nvenc_h264.o nvenc.o
>>> @@ -417,6 +418,7 @@ OBJS-$(CONFIG_HEVC_DECODER) += hevcdec.o
>> hevc_mvs.o \
>>> OBJS-$(CONFIG_HEVC_AMF_ENCODER) += amfenc_hevc.o
>>> OBJS-$(CONFIG_HEVC_CUVID_DECODER) += cuviddec.o
>>> OBJS-$(CONFIG_HEVC_MEDIACODEC_DECODER) += mediacodecdec.o
>>> +OBJS-$(CONFIG_HEVC_MEDIACODEC_ENCODER) += mediacodecenc.o
>>> OBJS-$(CONFIG_HEVC_MF_ENCODER) += mfenc.o mf_utils.o
>>> OBJS-$(CONFIG_HEVC_NVENC_ENCODER) += nvenc_hevc.o nvenc.o
>>> OBJS-$(CONFIG_HEVC_QSV_DECODER) += qsvdec.o
>>> diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
>>> index 46ad3b5a25..4c33a9ec3c 100644
>>> --- a/libavcodec/allcodecs.c
>>> +++ b/libavcodec/allcodecs.c
>>> @@ -154,6 +154,7 @@ extern const FFCodec ff_h264_decoder;
>>> extern const FFCodec ff_h264_crystalhd_decoder;
>>> extern const FFCodec ff_h264_v4l2m2m_decoder;
>>> extern const FFCodec ff_h264_mediacodec_decoder;
>>> +extern const FFCodec ff_h264_mediacodec_encoder;
>>> extern const FFCodec ff_h264_mmal_decoder;
>>> extern const FFCodec ff_h264_qsv_decoder;
>>> extern const FFCodec ff_h264_rkmpp_decoder;
>>> @@ -842,6 +843,7 @@ extern const FFCodec ff_h264_videotoolbox_encoder;
>>> extern const FFCodec ff_hevc_amf_encoder;
>>> extern const FFCodec ff_hevc_cuvid_decoder;
>>> extern const FFCodec ff_hevc_mediacodec_decoder;
>>> +extern const FFCodec ff_hevc_mediacodec_encoder;
>>> extern const FFCodec ff_hevc_mf_encoder;
>>> extern const FFCodec ff_hevc_nvenc_encoder;
>>> extern const FFCodec ff_hevc_qsv_encoder;
>>> diff --git a/libavcodec/mediacodec_wrapper.c
>> b/libavcodec/mediacodec_wrapper.c
>>> index 284d615980..5d1a32031d 100644
>>> --- a/libavcodec/mediacodec_wrapper.c
>>> +++ b/libavcodec/mediacodec_wrapper.c
>>> @@ -212,6 +212,9 @@ struct JNIAMediaCodecFields {
>>> jmethodID release_output_buffer_id;
>>> jmethodID release_output_buffer_at_time_id;
>>>
>>> + jmethodID set_input_surface_id;
>>> + jmethodID signal_end_of_input_stream_id;
>>> +
>>> jclass mediainfo_class;
>>>
>>> jmethodID init_id;
>>> @@ -261,6 +264,9 @@ static const struct FFJniField
>> jni_amediacodec_mapping[] = {
>>> { "android/media/MediaCodec", "releaseOutputBuffer", "(IZ)V",
>> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields,
>> release_output_buffer_id), 1 },
>>> { "android/media/MediaCodec", "releaseOutputBuffer", "(IJ)V",
>> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields,
>> release_output_buffer_at_time_id), 0 },
>>>
>>> + { "android/media/MediaCodec", "setInputSurface",
>> "(Landroid/view/Surface;)V", FF_JNI_METHOD, offsetof(struct
>> JNIAMediaCodecFields, set_input_surface_id), 0 },
>>> + { "android/media/MediaCodec", "signalEndOfInputStream", "()V",
>> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields,
>> signal_end_of_input_stream_id), 0 },
>>> +
>>> { "android/media/MediaCodec$BufferInfo", NULL, NULL, FF_JNI_CLASS,
>> offsetof(struct JNIAMediaCodecFields, mediainfo_class), 1 },
>>>
>>> { "android/media/MediaCodec.BufferInfo", "<init>", "()V",
>> FF_JNI_METHOD, offsetof(struct JNIAMediaCodecFields, init_id), 1 },
>>> @@ -1385,7 +1391,26 @@ static int mediacodec_jni_configure(FFAMediaCodec
>> *ctx,
>>>
>>> JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
>>>
>>> - (*env)->CallVoidMethod(env, codec->object,
>> codec->jfields.configure_id, format->object, surface, NULL, flags);
>>> + if (flags & codec->CONFIGURE_FLAG_ENCODE) {
>>> + if (surface && !codec->jfields.set_input_surface_id) {
>>> + av_log(ctx, AV_LOG_ERROR, "System doesn't support
>> setInputSurface\n");
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> +
>>> + (*env)->CallVoidMethod(env, codec->object,
>> codec->jfields.configure_id, format->object, NULL, NULL, flags);
>>> + if (ff_jni_exception_check(env, 1, codec) < 0)
>>> + return AVERROR_EXTERNAL;
>>> +
>>> + if (!surface)
>>> + return 0;
>>> +
>>> + (*env)->CallVoidMethod(env, codec->object,
>> codec->jfields.set_input_surface_id, surface);
>>> + if (ff_jni_exception_check(env, 1, codec) < 0)
>>> + return AVERROR_EXTERNAL;
>>> + return 0;
>>> + } else {
>>> + (*env)->CallVoidMethod(env, codec->object,
>> codec->jfields.configure_id, format->object, surface, NULL, flags);
>>> + }
>>> if (ff_jni_exception_check(env, 1, codec) < 0) {
>>> ret = AVERROR_EXTERNAL;
>>> goto fail;
>>> @@ -1743,6 +1768,22 @@ fail:
>>> return ret;
>>> }
>>>
>>> +static int mediacodec_jni_signalEndOfInputStream(FFAMediaCodec *ctx)
>>> +{
>>> + int ret = 0;
>>> + JNIEnv *env = NULL;
>>> + FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx;
>>> +
>>> + JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL);
>>> +
>>> + (*env)->CallVoidMethod(env, codec->object,
>> codec->jfields.signal_end_of_input_stream_id);
>>> + if (ff_jni_exception_check(env, 1, codec) < 0) {
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> static const FFAMediaFormat media_format_jni = {
>>> .class = &amediaformat_class,
>>>
>>> @@ -1801,6 +1842,7 @@ static const FFAMediaCodec media_codec_jni = {
>>>
>>> .getConfigureFlagEncode = mediacodec_jni_getConfigureFlagEncode,
>>> .cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers,
>>> + .signalEndOfInputStream = mediacodec_jni_signalEndOfInputStream,
>>> };
>>>
>>> typedef struct FFAMediaFormatNdk {
>>> @@ -1866,6 +1908,10 @@ typedef struct FFAMediaCodecNdk {
>>> // Available since API level 28.
>>> media_status_t (*getName)(AMediaCodec*, char** out_name);
>>> void (*releaseName)(AMediaCodec*, char* name);
>>> +
>>> + // Available since API level 26.
>>> + media_status_t (*setInputSurface)(AMediaCodec*, ANativeWindow *);
>>> + media_status_t (*signalEndOfInputStream)(AMediaCodec *);
>>> } FFAMediaCodecNdk;
>>>
>>> static const FFAMediaFormat media_format_ndk;
>>> @@ -2098,6 +2144,9 @@ static inline FFAMediaCodec *ndk_codec_create(int
>> method, const char *arg) {
>>> GET_SYMBOL(getName, 0)
>>> GET_SYMBOL(releaseName, 0)
>>>
>>> + GET_SYMBOL(setInputSurface, 0)
>>> + GET_SYMBOL(signalEndOfInputStream, 0)
>>> +
>>> #undef GET_SYMBOL
>>>
>>> switch (method) {
>>> @@ -2184,10 +2233,32 @@ static int
>> mediacodec_ndk_configure(FFAMediaCodec* ctx,
>>> return AVERROR(EINVAL);
>>> }
>>>
>>> - status = codec->configure(codec->impl, format->impl, native_window,
>> NULL, flags);
>>> - if (status != AMEDIA_OK) {
>>> - av_log(codec, AV_LOG_ERROR, "configure failed, %d\n", status);
>>> - return AVERROR_EXTERNAL;
>>> + if (flags & AMEDIACODEC_CONFIGURE_FLAG_ENCODE) {
>>> + if (native_window && !codec->setInputSurface) {
>>> + av_log(ctx, AV_LOG_ERROR, "System doesn't support
>> setInputSurface\n");
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> +
>>> + status = codec->configure(codec->impl, format->impl, NULL,
>> NULL, flags);
>>> + if (status != AMEDIA_OK) {
>>> + av_log(codec, AV_LOG_ERROR, "Encoder configure failed,
>> %d\n", status);
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> +
>>> + if (!native_window)
>>> + return 0;
>>> +
>>> + status = codec->setInputSurface(codec->impl, native_window);
>>> + if (status != AMEDIA_OK) {
>>> + av_log(codec, AV_LOG_ERROR, "Encoder set input surface
>> failed, %d\n", status);
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> + } else {
>>> + status = codec->configure(codec->impl, format->impl,
>> native_window, NULL, flags);
>>> + if (status != AMEDIA_OK) {
>>> + av_log(codec, AV_LOG_ERROR, "Decoder configure failed,
>> %d\n", status);
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> }
>>>
>>> return 0;
>>> @@ -2330,6 +2401,26 @@ static int
>> mediacodec_ndk_cleanOutputBuffers(FFAMediaCodec *ctx)
>>> return 0;
>>> }
>>>
>>> +static int mediacodec_ndk_signalEndOfInputStream(FFAMediaCodec *ctx)
>>> +{
>>> + FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx;
>>> + media_status_t status;
>>> +
>>> + if (!codec->signalEndOfInputStream) {
>>> + av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream
>> unavailable\n");
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> +
>>> + status = codec->signalEndOfInputStream(codec->impl);
>>> + if (status != AMEDIA_OK) {
>>> + av_log(codec, AV_LOG_ERROR, "signalEndOfInputStream failed,
>> %d\n", status);
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> + av_log(codec, AV_LOG_DEBUG, "signalEndOfInputStream success\n");
>>> +
>>> + return 0;
>>> +}
>>> +
>>> static const FFAMediaFormat media_format_ndk = {
>>> .class = &amediaformat_ndk_class,
>>>
>>> @@ -2388,6 +2479,7 @@ static const FFAMediaCodec media_codec_ndk = {
>>>
>>> .getConfigureFlagEncode = mediacodec_ndk_getConfigureFlagEncode,
>>> .cleanOutputBuffers = mediacodec_ndk_cleanOutputBuffers,
>>> + .signalEndOfInputStream = mediacodec_ndk_signalEndOfInputStream,
>>> };
>>>
>>> FFAMediaFormat *ff_AMediaFormat_new(int ndk)
>>> diff --git a/libavcodec/mediacodec_wrapper.h
>> b/libavcodec/mediacodec_wrapper.h
>>> index 7cf3f4aecd..f15ad66d83 100644
>>> --- a/libavcodec/mediacodec_wrapper.h
>>> +++ b/libavcodec/mediacodec_wrapper.h
>>> @@ -192,6 +192,9 @@ struct FFAMediaCodec {
>>> int (*getConfigureFlagEncode)(FFAMediaCodec *codec);
>>>
>>> int (*cleanOutputBuffers)(FFAMediaCodec *codec);
>>> +
>>> + // For encoder with FFANativeWindow as input.
>>> + int (*signalEndOfInputStream)(FFAMediaCodec *);
>>> };
>>>
>>> static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec)
>>> @@ -311,6 +314,11 @@ static inline int
>> ff_AMediaCodec_cleanOutputBuffers(FFAMediaCodec *codec)
>>> return codec->cleanOutputBuffers(codec);
>>> }
>>>
>>> +static inline int ff_AMediaCodec_signalEndOfInputStream(FFAMediaCodec
>> *codec)
>>> +{
>>> + return codec->signalEndOfInputStream(codec);
>>> +}
>>> +
>>> int ff_Build_SDK_INT(AVCodecContext *avctx);
>>>
>>> #endif /* AVCODEC_MEDIACODEC_WRAPPER_H */
>>> diff --git a/libavcodec/mediacodecenc.c b/libavcodec/mediacodecenc.c
>>> new file mode 100644
>>> index 0000000000..c81050ec80
>>> --- /dev/null
>>> +++ b/libavcodec/mediacodecenc.c
>>> @@ -0,0 +1,495 @@
>>> +/*
>>> + * Android MediaCodec encoders
>>> + *
>>> + * Copyright (c) 2022 Zhao Zhili <zhilizhao@tencent.com>
>>> + *
>>> + * This file is part of FFmpeg.
>>> + *
>>> + * FFmpeg is free software; you can redistribute it and/or
>>> + * modify it under the terms of the GNU Lesser General Public
>>> + * License as published by the Free Software Foundation; either
>>> + * version 2.1 of the License, or (at your option) any later version.
>>> + *
>>> + * FFmpeg is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
>>> + * Lesser General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU Lesser General Public
>>> + * License along with FFmpeg; if not, write to the Free Software
>>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA
>>> + */
>>> +
>>> +#include "config_components.h"
>>> +
>>> +#include "libavutil/avassert.h"
>>> +#include "libavutil/hwcontext_mediacodec.h"
>>> +#include "libavutil/imgutils.h"
>>> +#include "libavutil/opt.h"
>>> +
>>> +#include "avcodec.h"
>>> +#include "codec_internal.h"
>>> +#include "encode.h"
>>> +#include "hwconfig.h"
>>> +#include "jni.h"
>>> +#include "mediacodec.h"
>>> +#include "mediacodec_wrapper.h"
>>> +#include "mediacodecdec_common.h"
>>> +
>>> +#define INPUT_DEQUEUE_TIMEOUT_US 8000
>>> +#define OUTPUT_DEQUEUE_TIMEOUT_US 8000
>>> +
>>> +typedef struct MediaCodecEncContext {
>>> + AVClass *avclass;
>>> + FFAMediaCodec *codec;
>>> + int use_ndk_codec;
>>> + FFANativeWindow *window;
>>> +
>>> + int fps;
>>> + int width;
>>> + int height;
>>> +
>>> + uint8_t *extradata;
>>> + int extradata_size;
>>> +
>>> + // Since MediaCodec doesn't output DTS, use a timestamp queue to
>> save pts
>>> + // of AVFrame and generate DTS for AVPacket.
>>> + //
>>> + // This doesn't work when use Surface as input, in that case frames
>> can be
>>> + // sent to encoder without our notice. One exception is frames come
>> from
>>> + // our MediaCodec decoder wrapper, since we can control it's render
>> by
>>> + // av_mediacodec_release_buffer.
>>> + int64_t timestamps[32];
>>> + int ts_head;
>>> + int ts_tail;
>>> +
>>> + int eof_sent;
>>> +
>>> + AVFrame *frame;
>>> +} MediaCodecEncContext;
>>> +
>>> +enum {
>>> + COLOR_FormatYUV420Planar = 0x13,
>>> + COLOR_FormatYUV420SemiPlanar = 0x15,
>>> + COLOR_FormatSurface = 0x7F000789,
>>> +};
>>> +
>>> +static const struct {
>>> + int color_format;
>>> + enum AVPixelFormat pix_fmt;
>>> +} color_formats[] = {
>>> + { COLOR_FormatYUV420Planar, AV_PIX_FMT_YUV420P },
>>> + { COLOR_FormatYUV420SemiPlanar, AV_PIX_FMT_NV12 },
>>> + { COLOR_FormatSurface, AV_PIX_FMT_MEDIACODEC },
>>> +};
>>> +
>>> +static const enum AVPixelFormat avc_pix_fmts[] = {
>>> + AV_PIX_FMT_MEDIACODEC,
>>> + AV_PIX_FMT_YUV420P,
>>> + AV_PIX_FMT_NV12,
>>> + AV_PIX_FMT_NONE
>>> +};
>>> +
>>> +static void mediacodec_output_format(AVCodecContext *avctx)
>>> +{
>>> + MediaCodecEncContext *s = avctx->priv_data;
>>> + char *name = ff_AMediaCodec_getName(s->codec);
>>> + FFAMediaFormat *out_format =
>> ff_AMediaCodec_getOutputFormat(s->codec);
>>> + char *str = ff_AMediaFormat_toString(out_format);
>>> +
>>> + av_log(avctx, AV_LOG_DEBUG, "MediaCodec encoder %s output format
>> %s\n",
>>> + name ? name : "unknown", str);
>>> + av_free(name);
>>> + av_free(str);
>>> + ff_AMediaFormat_delete(out_format);
>>> +}
>>> +
>>> +static av_cold int mediacodec_init(AVCodecContext *avctx)
>>> +{
>>> + const char *codec_mime = NULL;
>>> + MediaCodecEncContext *s = avctx->priv_data;
>>> + FFAMediaFormat *format = NULL;
>>> + int ret;
>>> + int gop;
>>> +
>>> + if (s->use_ndk_codec < 0)
>>> + s->use_ndk_codec = !av_jni_get_java_vm(avctx);
>>> +
>>> + switch (avctx->codec_id) {
>>> + case AV_CODEC_ID_H264:
>>> + codec_mime = "video/avc";
>>> + break;
>>> + case AV_CODEC_ID_HEVC:
>>> + codec_mime = "video/hevc";
>>> + break;
>>> + default:
>>> + av_assert0(0);
>>> + }
>>> +
>>> + s->codec = ff_AMediaCodec_createEncoderByType(codec_mime,
>> s->use_ndk_codec);
>>> + if (!s->codec) {
>>> + av_log(avctx, AV_LOG_ERROR, "Failed to create encoder for type
>> %s\n",
>>> + codec_mime);
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> +
>>> + format = ff_AMediaFormat_new(s->use_ndk_codec);
>>> + if (!format) {
>>> + av_log(avctx, AV_LOG_ERROR, "Failed to create media format\n");
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> +
>>> + ff_AMediaFormat_setString(format, "mime", codec_mime);
>>> + s->width = FFALIGN(avctx->width, 16);
>>> + s->height = avctx->height;
>>> + ff_AMediaFormat_setInt32(format, "width", s->width);
>>> + ff_AMediaFormat_setInt32(format, "height", s->height);
>>>
>>> Is it preferable to use constants like AMEDIAFORMAT_KEY_HEIGHT here?
>>>
>>> I don't have a preference either way.
>>
>> These KEYs aren’t string literal, they are global variables.
>>
>> extern const char* AMEDIAFORMAT_KEY_HEIGHT __INTRODUCED_IN(21);
>>
>> Some basic ones are introduced very late, e.g.,
>>
>> extern const char* AMEDIAFORMAT_KEY_CSD __INTRODUCED_IN(28);
>> extern const char* AMEDIAFORMAT_KEY_CSD_0 __INTRODUCED_IN(28);
>> extern const char* AMEDIAFORMAT_KEY_CSD_1 __INTRODUCED_IN(28);
>> extern const char* AMEDIAFORMAT_KEY_CSD_2 __INTRODUCED_IN(28);
>> extern const char* AMEDIAFORMAT_KEY_DISPLAY_CROP __INTRODUCED_IN(28);
>>
>> So we can’t use these AMEDIAFORMAT_KEY_ directly. dlsym() these global
>> variables and with a fallback value is possible, just over-engineering.
>>
>> Google’s API design forced us to use string literal directly.
>>
>>>
>>> It may be worth also passing in the same values to
>> AMEDIAFORMAT_KEY_MAX_WIDTH and AMEDIAFORMAT_KEY_MAX_HEIGHT
>>
>> The documentation hints they are for decoder only:
>>
>>> A key describing the maximum expected width of the content in a video
>> decoder format, in case there are resolution changes in the video content.
>>
>> https://developer.android.com/reference/android/media/MediaFormat
>>
>>>
>>> And I think the unaligned width should be set into
>> AMEDIAFORMAT_KEY_STRIDE
>>
>> Technically, KEY_WIDTH should be unaligned width, and KEY_STRIDE should
>> be aligned width. However,
>>
>> 1. It’s well known that a lot of devices can’t handle width doesn’t aligned
>> to 16, they will crash, report error or produce broken files.
>>
>> 2. CTS tests only verify that 16 aligned resolutions are supported.
>>
>> 3. There are alignment info in OMX and exported via MediaCodecInfo, like
>>
>> <MediaCodec name="c2.qti.avc.encoder" type="video/avc">
>> <Alias name="OMX.qcom.video.encoder.avc" />
>> <Limit name="size" min="96x96" max="4096x2304" />
>> <Limit name="alignment" value="2x2" />
>> <Limit name="block-size" value="16x16" />
>>
>> It can be missing on old devices, or worse, the info doesn’t reflect the
>> real requirement (Maybe it was just a copy-paste from some places).
>>
>> I have an idea to fix the issue: always encoding in aligned resolution,
>> then fix the crop setting by apply BSF (e.g., h264_metadata) on the
>> bitstream.
>>
>>>
>>> Finally, avctx->sample_aspect_ratio should be propagated into
>> aspect-width and aspect-height. Something like this:
>>>
>>> AVRational sar = avctx->sample_aspect_ratio;
>>> if (!sar.num || !sar.den)
>>> sar.num = sar.den = 1;
>>> av_reduce(&sar.num, &sar.den, sar.num, sar.den, 4096);
>>> AMediaFormat_setInt32(format, "aspect-width", sar.num);
>>> AMediaFormat_setInt32(format, "aspect-height", sar.den);
>>>
>>
>> You mean sar-width/sar-height?
>>
>> EXPORT const char* AMEDIAFORMAT_KEY_SAR_HEIGHT = "sar-height";
>> EXPORT const char* AMEDIAFORMAT_KEY_SAR_WIDTH = "sar-width";
>>
>> They are available in NDK since API level 29 (Android 10)
>>
>> extern const char* AMEDIAFORMAT_KEY_SAR_HEIGHT __INTRODUCED_IN(29);
>> extern const char* AMEDIAFORMAT_KEY_SAR_WIDTH __INTRODUCED_IN(29);
>>
>> And they were added to Java MediaFormat since API level 30.
>>
>>
>> https://developer.android.com/reference/android/media/MediaFormat#KEY_PIXEL_ASPECT_RATIO_WIDTH
>>
>> It’s uncommon for Java API got a feature late than NDK. I will keep
>> a note and do more test before setting them.
>>
>>>
>>> +
>>> + if (avctx->pix_fmt == AV_PIX_FMT_MEDIACODEC) {
>>> + AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
>>> + if (avctx->hw_device_ctx) {
>>> + AVHWDeviceContext *device_ctx =
>> (AVHWDeviceContext*)(avctx->hw_device_ctx->data);
>>> + AVMediaCodecDeviceContext *dev_ctx;
>>> +
>>> + if (device_ctx->type != AV_HWDEVICE_TYPE_MEDIACODEC ||
>> !device_ctx->hwctx) {
>>> + ret = AVERROR(EINVAL);
>>> + goto bailout;
>>> + }
>>> + dev_ctx = device_ctx->hwctx;
>>> + s->window = ff_mediacodec_surface_ref(dev_ctx->surface,
>> dev_ctx->native_window, avctx);
>>> + }
>>> +
>>> + if (!s->window && user_ctx && user_ctx->surface)
>>> + s->window = ff_mediacodec_surface_ref(user_ctx->surface,
>> NULL, avctx);
>>> +
>>> + if (!s->window) {
>>> + ret = AVERROR(EINVAL);
>>> + av_log(avctx, AV_LOG_ERROR, "Missing hw_device_ctx or
>> hwaccel_context for AV_PIX_FMT_MEDIACODEC\n");
>>> + goto bailout;
>>> + }
>>> + }
>>> +
>>> + for (int i = 0; i < FF_ARRAY_ELEMS(color_formats); i++) {
>>> + if (avctx->pix_fmt == color_formats[i].pix_fmt) {
>>> + ff_AMediaFormat_setInt32(format, "color-format",
>>> + color_formats[i].color_format);
>>> + break;
>>>
>>> do we need error/fallback if no match is found?
>>
>> The supported pix_fmts is specified
>>
>> .priv_data_size = sizeof(MediaCodecEncContext), \
>> .p.pix_fmts = avc_pix_fmts, \
>> .init = mediacodec_init, \
>>
>> It has been checked in encoder.c encode_preinit_video(), so check it again
>> is
>> optional but not necessary in my opinion.
>>
>>>
>>>
>>> + }
>>> + }
>>> +
>>> + if (avctx->bit_rate)
>>> + ff_AMediaFormat_setInt32(format, "bitrate", avctx->bit_rate);
>>> + // frame-rate and i-frame-interval are required to configure codec
>>> + if (avctx->framerate.num >= avctx->framerate.den &&
>> avctx->framerate.den > 0)
>>> + s->fps = avctx->framerate.num / avctx->framerate.den;
>>> + else
>>> + s->fps = 30;
>>> + gop = round(avctx->gop_size / s->fps);
>>> + if (gop == 0)
>>> + gop = 2;
>>>
>>> can we read gop from avctx? in other implementations i have seen gop
>> hardcoded to 1
>>
>> I think we should respect gop_size setting.
>>
>> I will add some log message and change default gop to 1 in patch v2.
>>
>>>
>>> for fps value, can we use av_q2d(avctx->framerate)
>>
>> q2d() is for double, fps is integer here.
>>
>>>
>>>
>>> + ff_AMediaFormat_setInt32(format, "frame-rate", s->fps);
>>> + ff_AMediaFormat_setInt32(format, "i-frame-interval", gop);
>>> +
>>>
>>> for H264 encoding, you can pass "profile" and "level". for example
>> profile=0x08 for High and level=0x4000 for 5.0
>>>
>>> https://stackoverflow.com/a/38999412/332798
>>
>> I’m planning to add profile/level support after the basic patch is applied.
>> There are some unresolved issue here:
>>
>> 1. I have a device which failed at configure() when profile has been set,
>> even for baseline.
>>
>
> Thanks for sharing this. Today I had the same experience,
> with OMX.rk.video_encoder.avc trying to pass profile/level.
>
> I see the following message in logcat. Did you see the same behavior?
>
> 11-20 21:55:16.132 5086 5098 W ACodec : [OMX.rk.video_encoder.avc]
> stopping checking profiles after 32: 8/1
> 11-20 21:55:16.132 5086 5098 E ACodec : [OMX.rk.video_encoder.avc]
> configureCodec returning error -1010
It’s a device with SnapDragon 845.
Success without setting profile:
I/ExtendedACodec: setupVideoEncoder()
W/OMXUtils: do not know color format 0x7fa30c06 = 2141391878
W/OMXUtils: do not know color format 0x7fa30c04 = 2141391876
W/OMXUtils: do not know color format 0x7fa30c00 = 2141391872
W/OMXUtils: do not know color format 0x7fa30c09 = 2141391881
W/OMXUtils: do not know color format 0x7fa30c0a = 2141391882
W/OMXUtils: do not know color format 0x7fa30c08 = 2141391880
W/OMXUtils: do not know color format 0x7fa30c07 = 2141391879
W/OMXUtils: do not know color format 0x7f000789 = 2130708361
I/ACodec: setupAVCEncoderParameters with [profile: High] [level: Level1]
I/ACodec: [OMX.qcom.video.encoder.avc] cannot encode HDR static metadata. Ignoring.
I/ACodec: setupVideoEncoder succeeded
Failed when pass profile (no matter which profile)
I/ExtendedACodec: setupVideoEncoder()
W/OMXUtils: do not know color format 0x7fa30c06 = 2141391878
W/OMXUtils: do not know color format 0x7fa30c04 = 2141391876
W/OMXUtils: do not know color format 0x7fa30c00 = 2141391872
W/OMXUtils: do not know color format 0x7fa30c09 = 2141391881
W/OMXUtils: do not know color format 0x7fa30c0a = 2141391882
W/OMXUtils: do not know color format 0x7fa30c08 = 2141391880
W/OMXUtils: do not know color format 0x7fa30c07 = 2141391879
W/OMXUtils: do not know color format 0x7f000789 = 2130708361
E/ExtendedACodec: [OMX.qcom.video.encoder.avc] configureCodec returning error -38
E/ACodec: signalError(omxError 0x80001001, internalError -2147483648)
E/ACodec: [OMX.qcom.video.encoder.avc] configureCodec returning error -38
E/ACodec: signalError(omxError 0x80001001, internalError -2147483648)
E/MediaCodec: Codec reported err 0x80001001, actionCode 0, while in state 3
E/MediaCodec: Codec reported err 0x80001001, actionCode 0, while in state 0
E/MediaCodec: configure failed with err 0x80001001, resetting…
The device isn’t Pixel 3. Pixel 3 has the almost the same SOC but got
no problem.
>
>
>>
>> 2. DTS is missing in MediaCodec API. For the default baseline profile,
>> there
>> is no problem. Trouble comes when we set profile to mainline or high. If
>> Surface/ANativeWindow is used, and the frames come from our MediaCodec
>> decoder
>> wrapper, we can control it’s 'render' (send to encoder's surface) via
>> av_mediacodec_release_buffer(). A DTS generation strategy works in this
>> case.
>> However, if frames comes from other sources, like a camera, there is no way
>> to control the 'render' yet, so DTS is missing in this case.
>>
>> Configure profile/level and B frames should come together.
>>
>>>
>>>
>>> +
>>> + ret = ff_AMediaCodec_getConfigureFlagEncode(s->codec);
>>> + ret = ff_AMediaCodec_configure(s->codec, format, s->window, NULL,
>> ret);
>>> + if (ret) {
>>> + av_log(avctx, AV_LOG_ERROR, "MediaCodec configure failed,
>> %s\n", av_err2str(ret));
>>> + goto bailout;
>>> + }
>>> +
>>> + ret = ff_AMediaCodec_start(s->codec);
>>> + if (ret) {
>>> + av_log(avctx, AV_LOG_ERROR, "MediaCodec failed to start, %s\n",
>> av_err2str(ret));
>>> + goto bailout;
>>> + }
>>> +
>>> + mediacodec_output_format(avctx);
>>> +
>>> + s->frame = av_frame_alloc();
>>> + if (!s->frame)
>>> + ret = AVERROR(ENOMEM);
>>> +
>>> +bailout:
>>> + if (format)
>>> + ff_AMediaFormat_delete(format);
>>> + return ret;
>>> +}
>>> +
>>> +static int mediacodec_receive(AVCodecContext *avctx,
>>> + AVPacket *pkt,
>>> + int *got_packet)
>>> +{
>>> + MediaCodecEncContext *s = avctx->priv_data;
>>> + FFAMediaCodec *codec = s->codec;
>>> + FFAMediaCodecBufferInfo out_info = {0};
>>> + uint8_t *out_buf;
>>> + size_t out_size = 0;
>>> + int ret;
>>> + int extradata_size = 0;
>>> + int64_t timeout_us = s->eof_sent ? OUTPUT_DEQUEUE_TIMEOUT_US : 0;
>>> + ssize_t index = ff_AMediaCodec_dequeueOutputBuffer(codec,
>> &out_info, timeout_us);
>>> +
>>> + if (ff_AMediaCodec_infoTryAgainLater(codec, index))
>>> + return AVERROR(EAGAIN);
>>> +
>>> + if (ff_AMediaCodec_infoOutputFormatChanged(codec, index)) {
>>> + mediacodec_output_format(avctx);
>>> + return AVERROR(EAGAIN);
>>> + }
>>> +
>>> + if (ff_AMediaCodec_infoOutputBuffersChanged(codec, index)) {
>>> + ff_AMediaCodec_cleanOutputBuffers(codec);
>>> + return AVERROR(EAGAIN);
>>> + }
>>> +
>>> + if (index < 0)
>>> + return AVERROR_EXTERNAL;
>>> +
>>> + if (out_info.flags & ff_AMediaCodec_getBufferFlagEndOfStream(codec))
>>> + return AVERROR_EOF;
>>> +
>>> + out_buf = ff_AMediaCodec_getOutputBuffer(codec, index, &out_size);
>>> + if (!out_buf) {
>>> + ret = AVERROR_EXTERNAL;
>>> + goto bailout;
>>> + }
>>> +
>>> + if (out_info.flags &
>> ff_AMediaCodec_getBufferFlagCodecConfig(codec)) {
>>> + ret = av_reallocp(&s->extradata, out_info.size);
>>> + if (ret)
>>> + goto bailout;
>>> +
>>> + s->extradata_size = out_info.size;
>>> + memcpy(s->extradata, out_buf + out_info.offset, out_info.size);
>>> + ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
>>> + // try immediately
>>> + return mediacodec_receive(avctx, pkt, got_packet);
>>> + }
>>> +
>>> + ret = ff_get_encode_buffer(avctx, pkt, out_info.size +
>> s->extradata_size, 0);
>>> + if (ret < 0)
>>> + goto bailout;
>>> +
>>> + if (s->extradata_size) {
>>> + extradata_size = s->extradata_size;
>>> + s->extradata_size = 0;
>>> + memcpy(pkt->data, s->extradata, extradata_size);
>>> + }
>>> + memcpy(pkt->data + extradata_size, out_buf + out_info.offset,
>> out_info.size);
>>> + pkt->pts = av_rescale_q(out_info.presentationTimeUs,
>> AV_TIME_BASE_Q, avctx->time_base);
>>> + if (s->ts_tail != s->ts_head) {
>>> + pkt->dts = s->timestamps[s->ts_tail];
>>> + s->ts_tail = (s->ts_tail + 1) % FF_ARRAY_ELEMS(s->timestamps);
>>> + }
>>> +
>>> + if (out_info.flags & ff_AMediaCodec_getBufferFlagKeyFrame(codec))
>>> + pkt->flags |= AV_PKT_FLAG_KEY;
>>> + ret = 0;
>>> + *got_packet = 1;
>>> +
>>> + av_log(avctx, AV_LOG_TRACE, "receive packet pts %" PRId64 " dts %"
>> PRId64
>>> + " flags %d extradata %d\n",
>>> + pkt->pts, pkt->dts, pkt->flags, extradata_size);
>>> +
>>> +bailout:
>>> + ff_AMediaCodec_releaseOutputBuffer(codec, index, false);
>>> + return ret;
>>> +}
>>> +
>>> +static void copy_frame_to_buffer(AVCodecContext *avctx, const AVFrame
>> *frame, uint8_t *dst, size_t size)
>>> +{
>>> + MediaCodecEncContext *s = avctx->priv_data;
>>> + uint8_t *dst_data[4] = {};
>>> + int dst_linesize[4] = {};
>>> + const uint8_t *src_data[4] = {
>>> + frame->data[0], frame->data[1], frame->data[2],
>> frame->data[3]
>>> + };
>>> +
>>> + if (avctx->pix_fmt == AV_PIX_FMT_YUV420P) {
>>> + dst_data[0] = dst;
>>> + dst_data[1] = dst + s->width * s->height;
>>> + dst_data[2] = dst_data[1] + s->width * s->height / 4;
>>> +
>>> + dst_linesize[0] = s->width;
>>> + dst_linesize[1] = dst_linesize[2] = s->width / 2;
>>> + } else if (avctx->pix_fmt == AV_PIX_FMT_NV12) {
>>> + dst_data[0] = dst;
>>> + dst_data[1] = dst + s->width * s->height;
>>> +
>>> + dst_linesize[0] = s->width;
>>> + dst_linesize[1] = s->width;
>>> + } else {
>>> + av_assert0(0);
>>> + }
>>> +
>>> + av_image_copy(dst_data, dst_linesize, src_data, frame->linesize,
>>> + avctx->pix_fmt, avctx->width, avctx->height);
>>> +}
>>> +
>>> +static int mediacodec_send(AVCodecContext *avctx,
>>> + const AVFrame *frame) {
>>> + MediaCodecEncContext *s = avctx->priv_data;
>>> + FFAMediaCodec *codec = s->codec;
>>> + ssize_t index;
>>> + uint8_t *input_buf = NULL;
>>> + size_t input_size = 0;
>>> + int64_t pts = 0;
>>> + uint32_t flags = 0;
>>> + int64_t timeout_us;
>>> +
>>> + if (s->eof_sent)
>>> + return 0;
>>> +
>>> + if (s->window) {
>>> + if (!frame) {
>>> + s->eof_sent = 1;
>>> + return ff_AMediaCodec_signalEndOfInputStream(codec);
>>> + }
>>> +
>>> +
>>> + if (frame->data[3]) {
>>> + pts = av_rescale_q(frame->pts, avctx->time_base,
>> AV_TIME_BASE_Q);
>>> + s->timestamps[s->ts_head] = frame->pts;
>>> + s->ts_head = (s->ts_head + 1) %
>> FF_ARRAY_ELEMS(s->timestamps);
>>> +
>>> + av_mediacodec_release_buffer((AVMediaCodecBuffer
>> *)frame->data[3], 1);
>>> + }
>>> + return 0;
>>> + }
>>> +
>>> + timeout_us = INPUT_DEQUEUE_TIMEOUT_US;
>>> + index = ff_AMediaCodec_dequeueInputBuffer(codec, timeout_us);
>>> + if (ff_AMediaCodec_infoTryAgainLater(codec, index))
>>> + return AVERROR(EAGAIN);
>>> +
>>> + if (index < 0) {
>>> + av_log(avctx, AV_LOG_ERROR, "dequeue input buffer failed, %zd",
>> index);
>>> + return AVERROR_EXTERNAL;
>>> + }
>>> +
>>> + if (frame) {
>>> + input_buf = ff_AMediaCodec_getInputBuffer(codec, index,
>> &input_size);
>>> + copy_frame_to_buffer(avctx, frame, input_buf, input_size);
>>> +
>>> + pts = av_rescale_q(frame->pts, avctx->time_base,
>> AV_TIME_BASE_Q);
>>> +
>>> + s->timestamps[s->ts_head] = frame->pts;
>>> + s->ts_head = (s->ts_head + 1) % FF_ARRAY_ELEMS(s->timestamps);
>>> + } else {
>>> + flags |= ff_AMediaCodec_getBufferFlagEndOfStream(codec);
>>> + s->eof_sent = 1;
>>> + }
>>> +
>>>
>>> it would be nice to propagate keyframes here (frame->pict_type ==
>> AV_PICTURE_TYPE_I). it is only possible on API26 with
>> AMediaCodec_setParameters and PARAMETER_KEY_REQUEST_SYNC_FRAME
>>
>> It’s a nice feature. I’m planning to add support after the basic function
>> is working.
>>
>>>
>>>
>>> + ff_AMediaCodec_queueInputBuffer(codec, index, 0, input_size, pts,
>> flags);
>>> + return 0;
>>> +}
>>> +
>>> +static int mediacodec_encode(AVCodecContext *avctx, AVPacket *pkt)
>>> +{
>>> + MediaCodecEncContext *s = avctx->priv_data;
>>> + int ret;
>>> + int got_packet = 0;
>>> +
>>> + // Return on three case:
>>> + // 1. Serious error
>>> + // 2. Got a packet success
>>> + // 3. No AVFrame is available yet (don't return if get_frame return
>> EOF)
>>> + while (1) {
>>> + ret = mediacodec_receive(avctx, pkt, &got_packet);
>>> + if (!ret)
>>> + return 0;
>>> + else if (ret != AVERROR(EAGAIN))
>>> + return ret;
>>> +
>>> + if (!s->frame->buf[0]) {
>>> + ret = ff_encode_get_frame(avctx, s->frame);
>>> + if (ret && ret != AVERROR_EOF)
>>> + return ret;
>>> + }
>>> +
>>> + ret = mediacodec_send(avctx, s->frame->buf[0] ? s->frame :
>> NULL);
>>> + if (!ret)
>>> + av_frame_unref(s->frame);
>>> + else if (ret != AVERROR(EAGAIN))
>>> + return ret;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static av_cold int mediacodec_close(AVCodecContext *avctx)
>>> +{
>>> + MediaCodecEncContext *s = avctx->priv_data;
>>> + if (s->codec) {
>>> + ff_AMediaCodec_stop(s->codec);
>>> + ff_AMediaCodec_delete(s->codec);
>>> + s->codec = NULL;
>>> + }
>>> +
>>> + if (s->window) {
>>> + ff_mediacodec_surface_unref(s->window, avctx);
>>> + s->window = NULL;
>>> + }
>>> +
>>> + av_frame_free(&s->frame);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static const AVCodecHWConfigInternal *const mediacodec_hw_configs[] = {
>>> + &(const AVCodecHWConfigInternal) {
>>> + .public = {
>>> + .pix_fmt = AV_PIX_FMT_MEDIACODEC,
>>> + .methods = AV_CODEC_HW_CONFIG_METHOD_AD_HOC |
>>> + AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX,
>>> + .device_type = AV_HWDEVICE_TYPE_MEDIACODEC,
>>> + },
>>> + .hwaccel = NULL,
>>> + },
>>> + NULL
>>> +};
>>> +
>>> +#define OFFSET(x) offsetof(MediaCodecEncContext, x)
>>> +#define VE AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM
>>> +static const AVOption common_options[] = {
>>> + { "ndk_codec", "Use MediaCodec from NDK",
>>> + OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 =
>> -1}, -1, 1, VE },
>>> + { NULL },
>>> +};
>>> +
>>> +#define MEDIACODEC_ENCODER_CLASS(name) \
>>> +static const AVClass name ## _mediacodec_class = { \
>>> + .class_name = #name "_mediacodec", \
>>> + .item_name = av_default_item_name, \
>>> + .option = common_options, \
>>> + .version = LIBAVUTIL_VERSION_INT, \
>>> +}; \
>>> +
>>> +#define DECLARE_MEDIACODEC_ENCODER(short_name, long_name, codec_id)
>> \
>>> +MEDIACODEC_ENCODER_CLASS(short_name)
>> \
>>> +const FFCodec ff_ ## short_name ## _mediacodec_encoder = {
>> \
>>> + .p.name = #short_name "_mediacodec",
>> \
>>> + CODEC_LONG_NAME(long_name " Android MediaCodec encoder"),
>> \
>>> + .p.type = AVMEDIA_TYPE_VIDEO,
>> \
>>> + .p.id = codec_id,
>> \
>>> + .p.capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY
>> \
>>> + | AV_CODEC_CAP_HARDWARE,
>> \
>>> + .priv_data_size = sizeof(MediaCodecEncContext),
>> \
>>> + .p.pix_fmts = avc_pix_fmts,
>> \
>>> + .init = mediacodec_init,
>> \
>>> + FF_CODEC_RECEIVE_PACKET_CB(mediacodec_encode),
>> \
>>> + .close = mediacodec_close,
>> \
>>> + .p.priv_class = &short_name ## _mediacodec_class,
>> \
>>> + .caps_internal = FF_CODEC_CAP_INIT_CLEANUP,
>> \
>>> + .p.wrapper_name = "mediacodec",
>> \
>>> + .hw_configs = mediacodec_hw_configs,
>> \
>>> +};
>> \
>>> +
>>> +#if CONFIG_H264_MEDIACODEC_ENCODER
>>> +DECLARE_MEDIACODEC_ENCODER(h264, "H.264", AV_CODEC_ID_H264)
>>> +#endif
>>> +
>>> +#if CONFIG_HEVC_MEDIACODEC_ENCODER
>>> +DECLARE_MEDIACODEC_ENCODER(hevc, "H.265", AV_CODEC_ID_HEVC)
>>> +#endif
>>> diff --git a/libavcodec/version.h b/libavcodec/version.h
>>> index 43d0d9a9fc..86ac0f3871 100644
>>> --- a/libavcodec/version.h
>>> +++ b/libavcodec/version.h
>>> @@ -29,8 +29,8 @@
>>>
>>> #include "version_major.h"
>>>
>>> -#define LIBAVCODEC_VERSION_MINOR 51
>>> -#define LIBAVCODEC_VERSION_MICRO 101
>>> +#define LIBAVCODEC_VERSION_MINOR 52
>>> +#define LIBAVCODEC_VERSION_MICRO 100
>>>
>>> #define LIBAVCODEC_VERSION_INT
>> AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \
>>>
>> LIBAVCODEC_VERSION_MINOR, \
>>> --
>>> 2.25.1
>>>
>>
>>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread
* [FFmpeg-devel] [PATCH 11/12] avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
` (9 preceding siblings ...)
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 10/12] avcodec: add MediaCodec encoder Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 12/12] fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec Zhao Zhili
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
---
libavutil/hwcontext.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/libavutil/hwcontext.c b/libavutil/hwcontext.c
index ab9ad3703e..3396598269 100644
--- a/libavutil/hwcontext.c
+++ b/libavutil/hwcontext.c
@@ -397,10 +397,14 @@ int av_hwframe_transfer_get_formats(AVBufferRef *hwframe_ref,
static int transfer_data_alloc(AVFrame *dst, const AVFrame *src, int flags)
{
- AVHWFramesContext *ctx = (AVHWFramesContext*)src->hw_frames_ctx->data;
+ AVHWFramesContext *ctx;
AVFrame *frame_tmp;
int ret = 0;
+ if (!src->hw_frames_ctx)
+ return AVERROR(EINVAL);
+ ctx = (AVHWFramesContext*)src->hw_frames_ctx->data;
+
frame_tmp = av_frame_alloc();
if (!frame_tmp)
return AVERROR(ENOMEM);
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread* [FFmpeg-devel] [PATCH 12/12] fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec
[not found] <20221024031658.1026118-1-quinkblack@foxmail.com>
` (10 preceding siblings ...)
2022-10-24 3:16 ` [FFmpeg-devel] [PATCH 11/12] avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc Zhao Zhili
@ 2022-10-24 3:16 ` Zhao Zhili
11 siblings, 0 replies; 22+ messages in thread
From: Zhao Zhili @ 2022-10-24 3:16 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: matthieu.bouron, Zhao Zhili, aman
From: Zhao Zhili <zhilizhao@tencent.com>
---
fftools/ffmpeg_opt.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c
index 9245e02813..9c15858c05 100644
--- a/fftools/ffmpeg_opt.c
+++ b/fftools/ffmpeg_opt.c
@@ -785,6 +785,10 @@ static void add_input_streams(OptionsContext *o, AVFormatContext *ic)
"with old commandlines. This behaviour is DEPRECATED and will be removed "
"in the future. Please explicitly set \"-hwaccel_output_format qsv\".\n");
ist->hwaccel_output_format = AV_PIX_FMT_QSV;
+ } else if (!hwaccel_output_format && hwaccel && !strcmp(hwaccel, "mediacodec")) {
+ // There is no real AVHWFrameContext implementation. Set
+ // hwaccel_output_format to avoid av_hwframe_transfer_data error.
+ ist->hwaccel_output_format = AV_PIX_FMT_MEDIACODEC;
} else if (hwaccel_output_format) {
ist->hwaccel_output_format = av_get_pix_fmt(hwaccel_output_format);
if (ist->hwaccel_output_format == AV_PIX_FMT_NONE) {
--
2.25.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 22+ messages in thread