Re: [FFmpeg-devel] [PATCH v8 3/6] avformat/flvenc: support mux av1 in enhanced flv

[Neal Gompa <ngompa13@gmail.com>]

On Wed, May 10, 2023 at 7:13 AM Steven Liu <lingjiujianke@gmail.com> wrote:
>
> Neal Gompa <ngompa13@gmail.com> 于2023年5月10日周三 19:05写道：
> >
> > On Tue, May 9, 2023 at 6:48 AM Gyan Doshi <ffmpeg@gyani.pro> wrote:
> > >
> > >
> > >
> > > On 2023-05-09 04:05 pm, Neal Gompa wrote:
> > > > On Tue, May 9, 2023 at 12:14 AM Gyan Doshi <ffmpeg@gyani.pro> wrote:
> > > >>
> > > >>
> > > >> On 2023-05-09 08:11 am, Neal Gompa wrote:
> > > >>> On Mon, May 8, 2023 at 9:55 PM Steven Liu <lingjiujianke@gmail.com> wrote:
> > > >>>> Neal Gompa <ngompa13@gmail.com> 于2023年5月9日周二 07:08写道：
> > > >>>>> On Thu, Apr 13, 2023 at 5:45 AM Steven Liu <lq@chinaffmpeg.org> wrote:
> > > >>>>>> Signed-off-by: Steven Liu <lq@chinaffmpeg.org>
> > > >>>>>> ---
> > > >>>>>>    libavformat/flvenc.c | 25 ++++++++++++++++++++-----
> > > >>>>>>    1 file changed, 20 insertions(+), 5 deletions(-)
> > > >>>>>>
> > > >>>>>> diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c
> > > >>>>>> index 57a26245ff..7b43ecaefa 100644
> > > >>>>>> --- a/libavformat/flvenc.c
> > > >>>>>> +++ b/libavformat/flvenc.c
> > > >>>>>> @@ -28,6 +28,7 @@
> > > >>>>>>    #include "libavcodec/mpeg4audio.h"
> > > >>>>>>    #include "avio.h"
> > > >>>>>>    #include "avc.h"
> > > >>>>>> +#include "av1.h"
> > > >>>>>>    #include "hevc.h"
> > > >>>>>>    #include "avformat.h"
> > > >>>>>>    #include "flv.h"
> > > >>>>>> @@ -48,6 +49,7 @@ static const AVCodecTag flv_video_codec_ids[] = {
> > > >>>>>>        { AV_CODEC_ID_VP6A,     FLV_CODECID_VP6A },
> > > >>>>>>        { AV_CODEC_ID_H264,     FLV_CODECID_H264 },
> > > >>>>>>        { AV_CODEC_ID_HEVC,     MKBETAG('h', 'v', 'c', '1') },
> > > >>>>>> +    { AV_CODEC_ID_AV1,      MKBETAG('a', 'v', '0', '1') },
> > > >>>>>>        { AV_CODEC_ID_NONE,     0 }
> > > >>>>>>    };
> > > >>>>>>
> > > >>>>>> @@ -494,7 +496,8 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i
> > > >>>>>>        FLVContext *flv = s->priv_data;
> > > >>>>>>
> > > >>>>>>        if (par->codec_id == AV_CODEC_ID_AAC || par->codec_id == AV_CODEC_ID_H264
> > > >>>>>> -            || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC) {
> > > >>>>>> +            || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC
> > > >>>>>> +            || par->codec_id == AV_CODEC_ID_AV1) {
> > > >>>>>>            int64_t pos;
> > > >>>>>>            avio_w8(pb,
> > > >>>>>>                    par->codec_type == AVMEDIA_TYPE_VIDEO ?
> > > >>>>>> @@ -540,6 +543,9 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i
> > > >>>>>>                if (par->codec_id == AV_CODEC_ID_HEVC) {
> > > >>>>>>                    avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeSequenceStart); // ExVideoTagHeader mode with PacketTypeSequenceStart
> > > >>>>>>                    avio_write(pb, "hvc1", 4);
> > > >>>>>> +            } else if (par->codec_id == AV_CODEC_ID_AV1) {
> > > >>>>>> +                avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeSequenceStart);
> > > >>>>>> +                avio_write(pb, "av01", 4);
> > > >>>>>>                } else {
> > > >>>>>>                    avio_w8(pb, par->codec_tag | FLV_FRAME_KEY); // flags
> > > >>>>>>                    avio_w8(pb, 0); // AVC sequence header
> > > >>>>>> @@ -548,6 +554,8 @@ static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par, i
> > > >>>>>>
> > > >>>>>>                if (par->codec_id == AV_CODEC_ID_HEVC)
> > > >>>>>>                    ff_isom_write_hvcc(pb, par->extradata, par->extradata_size, 0);
> > > >>>>>> +            else if (par->codec_id == AV_CODEC_ID_AV1)
> > > >>>>>> +                ff_isom_write_av1c(pb, par->extradata, par->extradata_size, 1);
> > > >>>>>>                else
> > > >>>>>>                    ff_isom_write_avcc(pb, par->extradata, par->extradata_size);
> > > >>>>>>            }
> > > >>>>>> @@ -640,7 +648,8 @@ static int flv_init(struct AVFormatContext *s)
> > > >>>>>>
> > > >>>>>>                if (par->codec_id == AV_CODEC_ID_MPEG4 ||
> > > >>>>>>                    par->codec_id == AV_CODEC_ID_H263 ||
> > > >>>>>> -                par->codec_id == AV_CODEC_ID_HEVC) {
> > > >>>>>> +                par->codec_id == AV_CODEC_ID_HEVC ||
> > > >>>>>> +                par->codec_id == AV_CODEC_ID_AV1) {
> > > >>>>>>                    int error = s->strict_std_compliance > FF_COMPLIANCE_UNOFFICIAL;
> > > >>>>>>                    av_log(s, error ? AV_LOG_ERROR : AV_LOG_WARNING,
> > > >>>>>>                           "Codec %s is not supported in the official FLV specification,\n", avcodec_get_name(par->codec_id));
> > > >>>>>> @@ -848,13 +857,15 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt)
> > > >>>>>>        if (par->codec_id == AV_CODEC_ID_VP6F || par->codec_id == AV_CODEC_ID_VP6A ||
> > > >>>>>>            par->codec_id == AV_CODEC_ID_VP6  || par->codec_id == AV_CODEC_ID_AAC)
> > > >>>>>>            flags_size = 2;
> > > >>>>>> -    else if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC)
> > > >>>>>> +    else if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4 ||
> > > >>>>>> +             par->codec_id == AV_CODEC_ID_HEVC || par->codec_id == AV_CODEC_ID_AV1)
> > > >>>>>>            flags_size = 5;
> > > >>>>>>        else
> > > >>>>>>            flags_size = 1;
> > > >>>>>>
> > > >>>>>>        if (par->codec_id == AV_CODEC_ID_AAC || par->codec_id == AV_CODEC_ID_H264
> > > >>>>>> -            || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC) {
> > > >>>>>> +            || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC
> > > >>>>>> +            || par->codec_id == AV_CODEC_ID_AV1) {
> > > >>>>>>            size_t side_size;
> > > >>>>>>            uint8_t *side = av_packet_get_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, &side_size);
> > > >>>>>>            if (side && side_size > 0 && (side_size != par->extradata_size || memcmp(side, par->extradata, side_size))) {
> > > >>>>>> @@ -874,7 +885,8 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt)
> > > >>>>>>                   "Packets are not in the proper order with respect to DTS\n");
> > > >>>>>>            return AVERROR(EINVAL);
> > > >>>>>>        }
> > > >>>>>> -    if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4 || par->codec_id == AV_CODEC_ID_HEVC) {
> > > >>>>>> +    if (par->codec_id == AV_CODEC_ID_H264 || par->codec_id == AV_CODEC_ID_MPEG4 ||
> > > >>>>>> +        par->codec_id == AV_CODEC_ID_HEVC ||  par->codec_id == AV_CODEC_ID_AV1) {
> > > >>>>>>            if (pkt->pts == AV_NOPTS_VALUE) {
> > > >>>>>>                av_log(s, AV_LOG_ERROR, "Packet is missing PTS\n");
> > > >>>>>>                return AVERROR(EINVAL);
> > > >>>>>> @@ -987,6 +999,9 @@ static int flv_write_packet(AVFormatContext *s, AVPacket *pkt)
> > > >>>>>>            if (par->codec_id == AV_CODEC_ID_HEVC) {
> > > >>>>>>                avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeCodedFramesX); // ExVideoTagHeader mode with PacketTypeCodedFramesX
> > > >>>>>>                avio_write(pb, "hvc1", 4);
> > > >>>>>> +        } else if (par->codec_id == AV_CODEC_ID_AV1) {
> > > >>>>>> +            avio_w8(pb, FLV_IS_EX_HEADER | PacketTypeCodedFrames);
> > > >>>>>> +            avio_write(pb, "av01", 4);
> > > >>>>>>            } else {
> > > >>>>>>                avio_w8(pb, flags);
> > > >>>>>>            }
> > > >>>>>> --
> > > >>>>>> 2.40.0
> > > >>>>>>
> > > >>>>> I tested this by applying the patches on top of ffmpeg 6.0 and it
> > > >>>>> doesn't seem to work:
> > > >>>>>
> > > >>>>>     libavutil      58.  2.100 / 58.  2.100
> > > >>>>>     libavcodec     60.  3.100 / 60.  3.100
> > > >>>>>     libavformat    60.  3.100 / 60.  3.100
> > > >>>>>     libavdevice    60.  1.100 / 60.  1.100
> > > >>>>>     libavfilter     9.  3.100 /  9.  3.100
> > > >>>>>     libswscale      7.  1.100 /  7.  1.100
> > > >>>>>     libswresample   4. 10.100 /  4. 10.100
> > > >>>>>     libpostproc    57.  1.100 / 57.  1.100
> > > >>>>> [libdav1d @ 0x55c9ea8cae80] libdav1d 1.1.0
> > > >>>>> Input #0, matroska,webm, from '2023-05-08 18-52-42.mkv':
> > > >>>>>     Metadata:
> > > >>>>>       ENCODER         : Lavf60.3.100
> > > >>>>>     Duration: 00:02:00.53, start: 0.000000, bitrate: 1155 kb/s
> > > >>>>>     Stream #0:0: Video: av1 (Main), yuv420p(tv, bt709), 1280x720, 30
> > > >>>>> fps, 30 tbr, 1k tbn
> > > >>>>>       Metadata:
> > > >>>>>         DURATION        : 00:02:00.533000000
> > > >>>>>     Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp
> > > >>>>>       Metadata:
> > > >>>>>         title           : Track1
> > > >>>>>         DURATION        : 00:02:00.533000000
> > > >>>>> [flv @ 0x55c9ea936140] Codec av1 is not supported in the official FLV
> > > >>>>> specification,
> > > >>>>> [flv @ 0x55c9ea936140] use vstrict=-1 / -strict -1 to use it anyway.
> > > >>>> Pay attention this warning message.
> > > >>>>
> > > >>>>> [out#0/flv @ 0x55c9ea8cda80] Could not write header (incorrect codec
> > > >>>>> parameters ?): Invalid argument
> > > >>>>> [aost#0:1/copy @ 0x55c9ea97ae00] Error initializing output stream:
> > > >>>>> Stream mapping:
> > > >>>>>     Stream #0:0 -> #0:0 (copy)
> > > >>>>>     Stream #0:1 -> #0:1 (copy)
> > > >>>>>       Last message repeated 1 times
> > > >>> Well, this is my command:
> > > >>>
> > > >>> $ ffmpeg -i "input.mkv" -vcodec copy -acodec copy "output.flv"
> > > >>>
> > > >>> What should I be doing instead?
> > > >>     ffmpeg -i "input.mkv" -vcodec copy -acodec copy -strict -1 "output.flv"
> > > >>
> > > > That defeats the point of this patch. This patch adds proper support
> > > > for AV1 to the FLV code, so it should not need that.
> > >
> > > This patch does not add HEVC or AV1 support to the FLV *specification*,
> > > hence the guard.
> > >
> >
> > The thing is, this patch series is about implementing support for the
> > enhanced FLV specification.
> No, That documentation is not flv official specification, it just a document.

Frankly, so was the original one. The new specification is being
stewarded by a group that has the current owner of the RTMP spec as a
member (Veriskope).

RTMP was never IETF or ISO standardized, and neither was FLV. F4V was
derived from MP4/MOV, but that's not what RTMP uses.

This argument of it being "just a document" applies to the original
spec from Adobe/Veriskope too.

I still think it's reasonable that the enhanced FLV spec would be
supported without requiring disabling "strict" mode. That said, I
think it's fair to warn that we're using *new* features of the FLV
spec and there may be incompatibilities with older software.




--
真実はいつも一つ！/ Always, there's only one truth!
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".