Re: [FFmpeg-devel] [PATCH 2/2] avformat/lc3: Add file format for LC3/LC3plus transport

[Paul B Mahol <onemda@gmail.com>]

On Sat, Mar 30, 2024 at 10:51 PM Antoine Soulier <asoulier@google.com>
wrote:

> I am not sure about the block:
>
>> +    lc3->timestamp += lc3->dt;
>> +    if (lc3->timestamp > lc3->length) {
>> +        pkt->duration -= lc3->timestamp - lc3->length;
>> +        lc3->timestamp = lc3->length;
>> +    }
>
>
> The purpose is to reduce the duration of the last packet.
> When converting a file, the last frame can contain "zeros" (or irrelevant)
> samples that are not present in the source file (alignment to frame
> boundary). These samples should be discarded.
> > If I remove the "read_seek()" implementation, I lose the PTS / timestamp
> information, and I don't know how to detect the last frame in the stream.
> Do you know how I can handle a reduced duration for the last frame without
> having an implementation of "read_seek()"?
>
> (I can play with the position and EOF information, but can lead to strange
> behaviors if the encoder adds extra frames at the end).
>

Looks like format limitation. If duration is not stored in each packet than
demuxer can not do it and no spagetti code can fix it in demuxer. Still
output frame duration is set by decoder too when decoding, so make sure
that decoder outputs valid output and timestamps/durations.
So its not really possible to have also correct and robust duration of last
packet returned by demuxer. Even more the file may be manipulated in such
way that its no longer correct to derive last packet duration from full
duration stored in header when the EOF reached.


>
>
> On Sat, Mar 30, 2024 at 4:46 AM Paul B Mahol <onemda@gmail.com> wrote:
>
>>
>>
>> On Fri, Mar 29, 2024 at 6:30 PM Antoine Soulier via ffmpeg-devel <
>> ffmpeg-devel@ffmpeg.org> wrote:
>>
>>> A file format is described in Bluetooth SIG LC3 and ETSI TS 103 634, for
>>> test purpose. This is the format implemented here.
>>>
>>> Signed-off-by: Antoine Soulier <asoulier@google.com>
>>> ---
>>>  Changelog                |   1 +
>>>  doc/muxers.texi          |   6 ++
>>>  libavformat/Makefile     |   2 +
>>>  libavformat/allformats.c |   2 +
>>>  libavformat/lc3dec.c     | 164 +++++++++++++++++++++++++++++++++++++++
>>>  libavformat/lc3enc.c     | 100 ++++++++++++++++++++++++
>>>  6 files changed, 275 insertions(+)
>>>  create mode 100644 libavformat/lc3dec.c
>>>  create mode 100644 libavformat/lc3enc.c
>>>
>>> diff --git a/Changelog b/Changelog
>>> index 83a4cf7888..08c200a41d 100644
>>> --- a/Changelog
>>> +++ b/Changelog
>>> @@ -2,6 +2,7 @@ Entries are sorted chronologically from oldest to
>>> youngest within each release,
>>>  releases are sorted from youngest to oldest.
>>>
>>>  version <next>:
>>> +- LC3/LC3plus demuxer and muxer
>>>  - LC3/LC3plus decoding/encoding using external library liblc3
>>>
>>>
>>> diff --git a/doc/muxers.texi b/doc/muxers.texi
>>> index a10a8e216f..9687746c30 100644
>>> --- a/doc/muxers.texi
>>> +++ b/doc/muxers.texi
>>> @@ -2612,6 +2612,12 @@ WebDAV server every second:
>>>  ffmpeg -f x11grab -framerate 1 -i :0.0 -q:v 6 -update 1 -protocol_opts
>>> method=PUT http://example.com/desktop.jpg
>>>  @end example
>>>
>>> +@section lc3
>>> +Bluetooth SIG Low Complexity Communication Codec audio (LC3), or
>>> +ETSI TS 103 634 Low Complexity Communication Codec plus (LC3plus).
>>> +
>>> +This muxer accepts a single @code{lc3} audio stream.
>>> +
>>>  @section matroska
>>>
>>>  Matroska container muxer.
>>> diff --git a/libavformat/Makefile b/libavformat/Makefile
>>> index 44aa485029..4961c42852 100644
>>> --- a/libavformat/Makefile
>>> +++ b/libavformat/Makefile
>>> @@ -332,6 +332,8 @@ OBJS-$(CONFIG_KVAG_DEMUXER)              += kvag.o
>>>  OBJS-$(CONFIG_KVAG_MUXER)                += kvag.o rawenc.o
>>>  OBJS-$(CONFIG_LAF_DEMUXER)               += lafdec.o
>>>  OBJS-$(CONFIG_LATM_MUXER)                += latmenc.o rawenc.o
>>> +OBJS-$(CONFIG_LC3_DEMUXER)               += lc3dec.o
>>> +OBJS-$(CONFIG_LC3_MUXER)                 += lc3enc.o
>>>  OBJS-$(CONFIG_LMLM4_DEMUXER)             += lmlm4.o
>>>  OBJS-$(CONFIG_LOAS_DEMUXER)              += loasdec.o rawdec.o
>>>  OBJS-$(CONFIG_LUODAT_DEMUXER)            += luodatdec.o
>>> diff --git a/libavformat/allformats.c b/libavformat/allformats.c
>>> index 9df42bb87a..0b36a7c3eb 100644
>>> --- a/libavformat/allformats.c
>>> +++ b/libavformat/allformats.c
>>> @@ -252,6 +252,8 @@ extern const FFInputFormat  ff_kvag_demuxer;
>>>  extern const FFOutputFormat ff_kvag_muxer;
>>>  extern const FFInputFormat  ff_laf_demuxer;
>>>  extern const FFOutputFormat ff_latm_muxer;
>>> +extern const FFInputFormat  ff_lc3_demuxer;
>>> +extern const FFOutputFormat ff_lc3_muxer;
>>>  extern const FFInputFormat  ff_lmlm4_demuxer;
>>>  extern const FFInputFormat  ff_loas_demuxer;
>>>  extern const FFInputFormat  ff_luodat_demuxer;
>>> diff --git a/libavformat/lc3dec.c b/libavformat/lc3dec.c
>>> new file mode 100644
>>> index 0000000000..74d6794d00
>>> --- /dev/null
>>> +++ b/libavformat/lc3dec.c
>>> @@ -0,0 +1,164 @@
>>> +/*
>>> + * LC3 demuxer
>>> + * Copyright (C) 2024  Antoine Soulier <asoulier@google.com>
>>> + *
>>> + * This file is part of FFmpeg.
>>> + *
>>> + * FFmpeg is free software; you can redistribute it and/or
>>> + * modify it under the terms of the GNU Lesser General Public
>>> + * License as published by the Free Software Foundation; either
>>> + * version 2.1 of the License, or (at your option) any later version.
>>> + *
>>> + * FFmpeg is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>> + * Lesser General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU Lesser General Public
>>> + * License along with FFmpeg; if not, write to the Free Software
>>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
>>> 02110-1301 USA
>>> + */
>>> +
>>> +/**
>>> + * @file
>>> + * Based on the file format specified by :
>>> + *
>>> + * - Bluetooth SIG - Low Complexity Communication Codec Test Suite
>>> + *
>>> https://www.bluetooth.org/docman/handlers/downloaddoc.ashx?doc_id=502301
>>> + *   3.2.8.2 Reference LC3 Codec Bitstream Format
>>> + *
>>> + * - ETSI TI 103 634 V1.4.1 - Low Complexity Communication Codec plus
>>> + *
>>> https://www.etsi.org/deliver/etsi_ts/103600_103699/103634/01.04.01_60/ts_103634v010401p.pdf
>>> + *   LC3plus conformance script package
>>> + */
>>> +
>>> +#include "libavcodec/packet.h"
>>> +#include "libavutil/intreadwrite.h"
>>> +
>>> +#include "avformat.h"
>>> +#include "avio.h"
>>> +#include "demux.h"
>>> +#include "internal.h"
>>> +
>>> +typedef struct LC3DemuxContext {
>>> +    int dt, delay;
>>> +    int64_t timestamp;
>>> +    int64_t length;
>>> +} LC3DemuxContext;
>>> +
>>> +static int lc3_read_header(AVFormatContext *s)
>>> +{
>>> +    LC3DemuxContext *lc3 = s->priv_data;
>>> +    AVStream *st = NULL;
>>> +    uint16_t tag, hdr_size;
>>> +    uint16_t frame_10us;
>>> +    uint32_t length;
>>> +    int ep_mode, hr_mode;
>>> +    int srate_hz, channels, bit_rate;
>>> +    int num_extra_params, ret;
>>> +
>>> +    tag = avio_rb16(s->pb);
>>> +    hdr_size = avio_rl16(s->pb);
>>> +
>>> +    if (tag != 0x1ccc || hdr_size < 9 * sizeof(uint16_t))
>>> +        return AVERROR_INVALIDDATA;
>>> +
>>> +    num_extra_params = hdr_size / sizeof(uint16_t) - 9;
>>> +
>>> +    srate_hz   = avio_rl16(s->pb) * 100;
>>> +    bit_rate   = avio_rl16(s->pb) * 100;
>>> +    channels   = avio_rl16(s->pb);
>>> +    frame_10us = avio_rl16(s->pb);
>>> +    ep_mode    = avio_rl16(s->pb) != 0;
>>> +    length     = avio_rl32(s->pb);
>>> +    hr_mode    = num_extra_params >= 1 && avio_rl16(s->pb);
>>> +
>>> +    st = avformat_new_stream(s, NULL);
>>> +    if (!st)
>>> +        return AVERROR(ENOMEM);
>>> +
>>> +    avpriv_set_pts_info(st, 64, 1, srate_hz);
>>> +    st->start_time = 0;
>>> +    st->duration = length;
>>> +
>>> +    st->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
>>> +    st->codecpar->codec_id = AV_CODEC_ID_LC3;
>>> +    st->codecpar->sample_rate = srate_hz;
>>> +    st->codecpar->bit_rate = bit_rate;
>>> +    st->codecpar->ch_layout.nb_channels = channels;
>>> +
>>> +    if ((ret = ff_alloc_extradata(st->codecpar, 6)) < 0) {
>>> +        av_log(s, AV_LOG_ERROR, "Could not allocate extradata.\n");
>>> +        return ret;
>>> +    }
>>> +
>>> +    AV_WL16(st->codecpar->extradata + 0, frame_10us);
>>> +    AV_WL16(st->codecpar->extradata + 2, ep_mode);
>>> +    AV_WL16(st->codecpar->extradata + 4, hr_mode);
>>> +
>>> +    lc3->dt = av_rescale(frame_10us, srate_hz, 100*1000);
>>> +    lc3->delay = av_rescale(frame_10us == 750 ? 400 : 250, srate_hz,
>>> 100*1000);
>>> +
>>> +    lc3->timestamp = -lc3->delay;
>>> +    lc3->length = st->duration;
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int lc3_read_packet(AVFormatContext *s, AVPacket *pkt)
>>> +{
>>> +    LC3DemuxContext *lc3 = s->priv_data;
>>> +    AVIOContext *pb = s->pb;
>>> +    int ret;
>>> +
>>>
>>
>> int64_t pos = avio_tell(pb);
>>
>>
>>> +    ret = av_get_packet(s->pb, pkt, avio_rl16(pb));
>>> +    if (ret < 0)
>>> +        return ret;
>>> +
>>> +    pkt->pos -= 2;
>>>
>>
>> pkt->pos = pos;
>>
>> +    pkt->pts = FFMAX(lc3->timestamp, 0);
>>>
>>
>> Remove that line above.
>>
>>
>>> +    pkt->duration = lc3->dt;
>>> +
>>> +    lc3->timestamp += lc3->dt;
>>> +    if (lc3->timestamp > lc3->length) {
>>> +        pkt->duration -= lc3->timestamp - lc3->length;
>>> +        lc3->timestamp = lc3->length;
>>> +    }
>>> +
>>>
>>
>> Remove above block.
>>
>>
>>> +    return 0;
>>> +}
>>> +
>>> +static int lc3_read_seek(AVFormatContext *s, int stream_index, int64_t
>>> timestamp, int flags)
>>> +{
>>> +    AVStream *st = s->streams[stream_index];
>>> +    FFStream *const sti = ffstream(st);
>>> +    LC3DemuxContext *lc3 = s->priv_data;
>>> +    const AVIndexEntry *ie;
>>> +    int index;
>>> +
>>> +    timestamp = FFMAX(timestamp - lc3->delay, 0);
>>> +    if (timestamp >= lc3->length)
>>> +        return AVERROR_EOF;
>>> +
>>> +    index = av_index_search_timestamp(st, timestamp, flags);
>>> +    ie = index < 0 ? NULL : &sti->index_entries[index];
>>> +    if (!ie || avio_seek(s->pb, ie->pos, SEEK_SET) < 0)
>>> +        return -1;
>>> +
>>> +    lc3->timestamp = ie->timestamp;
>>> +    if (lc3->timestamp == 0)
>>> +      lc3->timestamp -= lc3->delay;
>>> +
>>> +    return 0;
>>> +}
>>>
>>
>> This logic is unnecessary.
>> Format does not store timestamp internally in bitstream.
>> Just set packet duration to number of samples stored in that packet.
>> No need for pts overdesign, and remove this read_seek_function.
>> And just leave GENERIC_INDEX flags bellow.
>>
>>
>>> +
>>> +const FFInputFormat ff_lc3_demuxer = {
>>> +    .p.name         = "lc3",
>>> +    .p.long_name    = NULL_IF_CONFIG_SMALL("LC3 (Low Complexity
>>> Communication Codec)"),
>>> +    .p.extensions   = "lc3",
>>> +    .p.flags        = AVFMT_GENERIC_INDEX,
>>> +    .priv_data_size = sizeof(LC3DemuxContext),
>>> +    .read_header    = lc3_read_header,
>>> +    .read_packet    = lc3_read_packet,
>>> +    .read_seek      = lc3_read_seek,
>>> +};
>>> diff --git a/libavformat/lc3enc.c b/libavformat/lc3enc.c
>>> new file mode 100644
>>> index 0000000000..e768212f63
>>> --- /dev/null
>>> +++ b/libavformat/lc3enc.c
>>> @@ -0,0 +1,100 @@
>>> +/*
>>> + * LC3 muxer
>>> + * Copyright (C) 2024  Antoine Soulier <asoulier@google.com>
>>> + *
>>> + * This file is part of FFmpeg.
>>> + *
>>> + * FFmpeg is free software; you can redistribute it and/or
>>> + * modify it under the terms of the GNU Lesser General Public
>>> + * License as published by the Free Software Foundation; either
>>> + * version 2.1 of the License, or (at your option) any later version.
>>> + *
>>> + * FFmpeg is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>> + * Lesser General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU Lesser General Public
>>> + * License along with FFmpeg; if not, write to the Free Software
>>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
>>> 02110-1301 USA
>>> + */
>>> +
>>> +/**
>>> + * @file
>>> + * Based on the file format specified by :
>>> + *
>>> + * - Bluetooth SIG - Low Complexity Communication Codec Test Suite
>>> + *
>>> https://www.bluetooth.org/docman/handlers/downloaddoc.ashx?doc_id=502301
>>> + *   3.2.8.2 Reference LC3 Codec Bitstream Format
>>> + *
>>> + * - ETSI TI 103 634 V1.4.1 - Low Complexity Communication Codec plus
>>> + *
>>> https://www.etsi.org/deliver/etsi_ts/103600_103699/103634/01.04.01_60/ts_103634v010401p.pdf
>>> + *   LC3plus conformance script package
>>> + */
>>> +
>>> +#include "libavutil/intreadwrite.h"
>>> +
>>> +#include "avformat.h"
>>> +#include "avio.h"
>>> +#include "mux.h"
>>> +#include "internal.h"
>>> +
>>> +static av_cold int lc3_init(AVFormatContext *s)
>>> +{
>>> +    if (s->nb_streams != 1) {
>>> +        av_log(s, AV_LOG_ERROR, "This muxer only supports a single
>>> stream.\n");
>>> +        return AVERROR(EINVAL);
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int lc3_write_header(AVFormatContext *s)
>>> +{
>>> +    AVStream *st = s->streams[0];
>>> +    int channels = st->codecpar->ch_layout.nb_channels;
>>> +    int srate_hz = st->codecpar->sample_rate;
>>> +    int bit_rate = st->codecpar->bit_rate;
>>> +    int frame_us, ep_mode, hr_mode;
>>> +    uint32_t nb_samples = av_rescale_q(
>>> +        st->duration, st->time_base, (AVRational){ 1, srate_hz });
>>> +
>>> +    if (st->codecpar->extradata_size < 6)
>>> +        return AVERROR_INVALIDDATA;
>>> +
>>> +    frame_us = AV_RL16(st->codecpar->extradata + 0) * 10;
>>> +    ep_mode = AV_RL16(st->codecpar->extradata + 2) != 0;
>>> +    hr_mode = AV_RL16(st->codecpar->extradata + 4) != 0;
>>> +
>>> +    avio_wb16(s->pb, 0x1ccc);
>>> +    avio_wl16(s->pb, (9 + hr_mode) * sizeof(uint16_t));
>>> +    avio_wl16(s->pb, srate_hz / 100);
>>> +    avio_wl16(s->pb, bit_rate / 100);
>>> +    avio_wl16(s->pb, channels);
>>> +    avio_wl16(s->pb, frame_us / 10);
>>> +    avio_wl16(s->pb, ep_mode);
>>> +    avio_wl32(s->pb, nb_samples);
>>> +    if (hr_mode)
>>> +        avio_wl16(s->pb, hr_mode);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int lc3_write_packet(AVFormatContext *s, AVPacket *pkt)
>>> +{
>>> +    avio_wl16(s->pb, pkt->size);
>>> +    avio_write(s->pb, pkt->data, pkt->size);
>>> +    return 0;
>>> +}
>>> +
>>> +const FFOutputFormat ff_lc3_muxer = {
>>> +    .p.name        = "lc3",
>>> +    .p.long_name   = NULL_IF_CONFIG_SMALL("LC3 (Low Complexity
>>> Communication Codec)"),
>>> +    .p.extensions  = "lc3",
>>> +    .p.audio_codec = AV_CODEC_ID_LC3,
>>> +    .p.video_codec = AV_CODEC_ID_NONE,
>>> +    .p.flags       = AVFMT_NOTIMESTAMPS,
>>> +    .init          = lc3_init,
>>> +    .write_header  = lc3_write_header,
>>> +    .write_packet  = lc3_write_packet,
>>> +};
>>> --
>>> 2.44.0.478.gd926399ef9-goog
>>>
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel@ffmpeg.org
>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>>>
>>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".