[FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Tomas Härdin]

Hi

In my fiddling with fragmented indexes in mov.c I ran cross
mov_fix_index(), which has been in the codebase since September 2016.
The intent of this function is to implement limited support for edit
lists (elst). More rudely one could say it implements half-assed
support for edit lists. Besides the way it's implemented being
incredibly cursed, I feel strongly that this is not something that a
demuxer should do, partly because it's not something a demuxer *can*
do. There are a number of reasons why:

1) demuxers are not part of the presentation layer
2) the demuxer must lie about what the file actually contains
3) the ISOBMFF spec mandates being able to cut on samples, not just
frames. for audio, there is no way to do that from a demuxer
4) similarly, there is no way (I think) to implement "dwell" segments

The second point becomes painfully clear when one looks at
mov_fix_index(). FFStream.index_entries is completely replaced. The
function is not idempotent. For the kind of work I'm doing right now
this is a huge problem.

As I see it, the way forward is:

1) introduce API for edit lists, so that users can decide themselves
what to do
2) implement basic support for edit lists in the ffmpeg CLI
3) remove edit list hacks from all demuxers, especially mov.c

The main issue I see with this is that it risks turning ffmpeg into an
NLE tool, which would be a waste of developer effort better spent on
melt and related tools (kdenlive, shotcut etc). The present level of
edit list support could be reimplemented by just fiddling with the -ss
and -t options. That is, the ffmpeg CLI could automatically derive -ss
and -t from the edit list extracted by lavf.

If this stuff is done correctly then it should be possible to remux MOV
to MXF while keeping the edit lists and the original uncut essence. And
of course back again to MOV if so desired.

I haven't sketched out any API yet, but it would probably closely
mirror how elst works in MOV. Care should be taken that MXF operational
pattern 3a can also fit into this API. Possibly even OP3c.

Thoughts?

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Kieran Kunhya via ffmpeg-devel]

[-- Attachment #1: Type: message/rfc822, Size: 5657 bytes --]

From: Kieran Kunhya <kieran618@googlemail.com>
To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Subject: Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers
Date: Fri, 13 Jun 2025 14:51:45 +0200
Message-ID: <CABGuwEkEw7NZHMST3-_a6Gx_+A3ERHtbdA2mxnSDE-Q2fh43hQ@mail.gmail.com>

On Fri, 13 Jun 2025, 12:56 Tomas Härdin, <git@haerdin.se> wrote:

> Hi
>
> In my fiddling with fragmented indexes in mov.c I ran cross
> mov_fix_index(), which has been in the codebase since September 2016.
> The intent of this function is to implement limited support for edit
> lists (elst). More rudely one could say it implements half-assed
> support for edit lists. Besides the way it's implemented being
> incredibly cursed, I feel strongly that this is not something that a
> demuxer should do, partly because it's not something a demuxer *can*
> do. There are a number of reasons why:
>
> 1) demuxers are not part of the presentation layer
> 2) the demuxer must lie about what the file actually contains
> 3) the ISOBMFF spec mandates being able to cut on samples, not just
> frames. for audio, there is no way to do that from a demuxer
> 4) similarly, there is no way (I think) to implement "dwell" segments
>
> The second point becomes painfully clear when one looks at
> mov_fix_index(). FFStream.index_entries is completely replaced. The
> function is not idempotent. For the kind of work I'm doing right now
> this is a huge problem.
>
> As I see it, the way forward is:
>
> 1) introduce API for edit lists, so that users can decide themselves
> what to do
> 2) implement basic support for edit lists in the ffmpeg CLI
> 3) remove edit list hacks from all demuxers, especially mov.c
>
> The main issue I see with this is that it risks turning ffmpeg into an
> NLE tool, which would be a waste of developer effort better spent on
> melt and related tools (kdenlive, shotcut etc). The present level of
> edit list support could be reimplemented by just fiddling with the -ss
> and -t options. That is, the ffmpeg CLI could automatically derive -ss
> and -t from the edit list extracted by lavf.
>
> If this stuff is done correctly then it should be possible to remux MOV
> to MXF while keeping the edit lists and the original uncut essence. And
> of course back again to MOV if so desired.
>
> I haven't sketched out any API yet, but it would probably closely
> mirror how elst works in MOV. Care should be taken that MXF operational
> pattern 3a can also fit into this API. Possibly even OP3c.
>
> Thoughts?


Derek did this in 2018:

https://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/227437.html

Kieran

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Derek Buitenhuis]

On 6/13/2025 1:51 PM, Kieran Kunhya via ffmpeg-devel wrote:
> Derek did this in 2018:
> 
> https://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/227437.html

elenril had some good comments, others were so bad I never sent a v2
because every time I looked at the replies I was demotivated by their
severe asshole-ish-ness.

I think it is a good starting point; this was an issue I have been pointing
out since before the edit list support patch set from youtube even went in,
in the reviews for it, but was overruled as "some support is better than none".

The key point of the set was generic support (e.g. MKV too) and to make
sure it was, as noted by Thomas too, it is applied at presentation level,
after decode.



- Derek
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2487 bytes --]

Hi Tomas

On Fri, Jun 13, 2025 at 12:55:56PM +0200, Tomas Härdin wrote:
> Hi
> 
> In my fiddling with fragmented indexes in mov.c I ran cross
> mov_fix_index(), which has been in the codebase since September 2016.
> The intent of this function is to implement limited support for edit
> lists (elst). More rudely one could say it implements half-assed
> support for edit lists. Besides the way it's implemented being
> incredibly cursed, I feel strongly that this is not something that a
> demuxer should do, partly because it's not something a demuxer *can*
> do. There are a number of reasons why:
> 
> 1) demuxers are not part of the presentation layer
> 2) the demuxer must lie about what the file actually contains
> 3) the ISOBMFF spec mandates being able to cut on samples, not just
> frames. for audio, there is no way to do that from a demuxer
> 4) similarly, there is no way (I think) to implement "dwell" segments
> 
> The second point becomes painfully clear when one looks at
> mov_fix_index(). FFStream.index_entries is completely replaced. The
> function is not idempotent. For the kind of work I'm doing right now
> this is a huge problem.
> 
> As I see it, the way forward is:
> 

> 1) introduce API for edit lists, so that users can decide themselves
> what to do

+1


> 2) implement basic support for edit lists in the ffmpeg CLI

+1


> 3) remove edit list hacks from all demuxers, especially mov.c

+1 (with ABI +2 bump)


> 
> The main issue I see with this is that it risks turning ffmpeg into an
> NLE tool,

If ffmpeg could do NLE, that would not be a bad thing
And a full implementation of edit lists may be equivalent to this.
I think thats a win not a loss


[...]
> If this stuff is done correctly then it should be possible to remux MOV
> to MXF while keeping the edit lists and the original uncut essence. And
> of course back again to MOV if so desired.

yeah


> 
> I haven't sketched out any API yet, but it would probably closely
> mirror how elst works in MOV. Care should be taken that MXF operational
> pattern 3a can also fit into this API. Possibly even OP3c.
> 
> Thoughts?

it would be cool if someone implemented this, its not trivial as all
sides need to support it. demuxer, muxer, player, ffmpeg

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

What is money laundering? Its paying someone and not telling the government.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Tomas Härdin]

> Derek did this in 2018:
> 
> https://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/227437.html

Interesting. I didn't think to use side data for it. Putting the edit
lists in AVStream seems wrong. They belong to AVFormatContext. Else we
can't support ganged or alternate packages in MXF (OP1b, OP1c, OP2b,
OP2c, OP3b, OP3c), nor could we support IMF. Yes, I know imfdec.c
exists. It too is cursed.

The point about ABI stability is a good one, but can be dealt with by
an extra level of indirection. A bit annoying but probably a good idea
in the long run.

Also this point by Rostislav is a good one:
https://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/227446.html

> It wouldn't be breaking anything - technically making this part of
> lavf in the first place and enabling it by default broke behaviour.

In other words, by default ffmpeg currently *throws away essence*. This
is very bad.

---

Here's what I came up with so far for reference. The idea is to just
stick a pointer to AVEditLists in AVFormatContext. This was written
before I saw the ABI point and the indirection solution to it.

Some of the fields can probably be removed since they're intended
mostly for MXF, and adding fields later is easier than removing them.

Anyway, here are some structs:

// the terminology here is mostly lifted from ISOBMFF and the MXF spec
#define AV_EDIT_ITEM_FILLER 1       ///< Filler. Only duration is of relevance.

/**
 * Edit item.
 * Corresponds to elst entries in MOV, SourceClips in MXF.
 */
typedef struct AVEditItem {
    int stream;                     ///< Source stream for this item. TODO: other files? Could maybe be handled by assigning some kind of global stream ID to streams
    int64_t start;                  ///< Offset in terms of the source's presentation time base, for example sample rate for audio
    int64_t duration;               ///< Duration in terms of edit rate for this track. Must be non-negative. Zero means duration of the rest of the stream (see section 8.6.6.1 of ISO/IEC 14496-12:2015(E))
    AVRational rate;                ///< Relative playback rate. Zero = dwell, repeat pointed-to frame for the duration. 1 = normal. 2 = play twice as fast. TODO: let rate = -1 mean filler like in mov?
    int flags;                      ///< Flags. See AV_EDIT_ITEM_*
    AVDictionary *metadata;         ///< Metadata such as SourcePackageID, SourceTrackID, StartTimecode and so on.
} AVEditItem;

/**
 * Edit track.
 * A collection of edit items in temporal order.
 */
typedef struct AVEditTrack {
    enum AVMediaType type;          ///< Media type of this track.
    AVRational edit_rate;           ///< Edit rate. May be different from the frame rates of the streams pointed to by the edit items.
    int64_t origin;                 ///< Start of this track. Necessary for compatibility with MXF.
    AVEditItem *items;              ///< Edit items.
    int nb_items;                   ///< Number of edit items.
    unsigned items_allocated_size;  ///< Number of bytes allocated for edit items.
    AVDictionary *metadata;         ///< Metadata such as TrackNumber, TrackName and so on.
};

/**
 * Edit list.
 * Multiple edit tracks of different types, all collected as a unit.
 * Constituent tracks may have different origin and duration.
 */
typedef struct AVEditList {
    AVEditTrack *tracks;            ///< Tracks.
    int nb_tracks;                  ///< Number of tracks.
    unsigned tracks_allocated_size; ///< Number of bytes allocated for tracks.
    int loop_count;                 ///< Number of loops. 0 = infinite, >0 = that number of times. Used for HEIF/AVIF. TODO: put in metadata?
    AVDictionary *metadata;         ///< Metadata such as PackageUID, Name and so on.
} AVEditList;

/**
 * Collection of edit lists.
 * This is used for formats like MXF and IMF that support multiple "variants" of the same file,
 * for example different language variants of one movie.
 * In MXF parlance these are called "material packages".
 */
typedef struct AVEditLists {
    AVEditList *edit_lists;             ///< Edit lists.
    int nb_edit_lists;                  ///< Number of edit lists.
    unsigned edit_lists_allocated_size; ///< Number of bytes allocated for edit lists.
    AVDictionary *metadata;             ///< Metadata specific to this collection of edit lists.
} AVEditLists;

---

More fine points to consider that are touched on above:

* how do we express a timeline that points to multiple files, including external files?
* when should playback start?
* should we add an AVMediaType for timecodes?

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Gyan Doshi]

On 2025-06-13 04:25 pm, Tomas Härdin wrote:
> The present level of
> edit list support could be reimplemented by just fiddling with the -ss
> and -t options. That is, the ffmpeg CLI could automatically derive -ss
> and -t from the edit list extracted by lavf.

In mov.c, edit lists are per-track and can compose their edited tracks 
differently. So if the stream index/tables are to be no longer 
reconstituted within mov,c, -ss and -t may not return synchronized 
packets (as per the intended elst) from all streams.

Regards,
Gyan


_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Derek Buitenhuis]

On 6/13/2025 3:34 PM, Tomas Härdin wrote:
> Interesting. I didn't think to use side data for it. Putting the edit
> lists in AVStream seems wrong. They belong to AVFormatContext. Else we
> can't support ganged or alternate packages in MXF (OP1b, OP1c, OP2b,
> OP2c, OP3b, OP3c), nor could we support IMF. Yes, I know imfdec.c
> exists. It too is cursed.

I don't know what any of that means, but I definitely wasn't aware of any
MXF stuff back then, so probably didn't design for it.

> Here's what I came up with so far for reference. The idea is to just
> stick a pointer to AVEditLists in AVFormatContext. This was written
> before I saw the ABI point and the indirection solution to it.

My main concern here is that it is extremely complex from API
perspective, and approximately zero users will end up using it
correctly (or at all), outside of ffmpeg.c. (From what I can
tell this complexity solely comes from IMF/MXF.)

Probably part of such an RFC needs to contain how it is intended to be
used (integration with existing APIs, functions, etc.). If it
is just these structs, it's essentially unusably complex as
far as I am concerned. (This was an issue with my old RFC too,
but this is even more complex, and hindsight is 20/20). Some
amount of "just works", or a higher level API needs to exist,
I think.

- Derek
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Tomas Härdin]

fre 2025-06-13 klockan 16:21 +0200 skrev Michael Niedermayer:
> > 3) remove edit list hacks from all demuxers, especially mov.c
> 
> +1 (with ABI +2 bump)

I'm not sure why a (major?) bump would be necessary. Are removal of
options a major bump, not a minor one? Either way, we could make the
options ffmpeg.c ones rather than mov.c ones. There's going to be pain
either way, partly because doing this in the demuxer was the wrong
thing to do in the first place. Another case of "oh let's just do a
quick and dirty thing" coming back to bite us.

Come to think of this, perhaps hiding this behind a major bump is not
such a bad idea. That allows making more sweeping changes, including
behavioral ones.

> > The main issue I see with this is that it risks turning ffmpeg into
> > an
> > NLE tool,
> 
> If ffmpeg could do NLE, that would not be a bad thing
> And a full implementation of edit lists may be equivalent to this.
> I think thats a win not a loss

Fair enough. So long as I don't have to implement it or think
particularly much about it.

> > I haven't sketched out any API yet, but it would probably closely
> > mirror how elst works in MOV. Care should be taken that MXF
> > operational
> > pattern 3a can also fit into this API. Possibly even OP3c.
> > 
> > Thoughts?
> 
> it would be cool if someone implemented this, its not trivial as all
> sides need to support it. demuxer, muxer, player, ffmpeg

We don't need it everywhere immediately. But yeah, it's a lot. We
probably need a layer above lavf at some point. libavpresentation?
Probably with a cache. That's what libmlt does. It's also why I'm
skeptical of going too far with this, because that's dev time not spent
on libmlt.

I probably need to double-check that I can spend the time to implement
this on the demuxer side, and possibly in fftools, before I commit to
it. This being a blocker for getting segmented indexes working and
upstreamed might help. I'm assuming complete removal of elst support is
out of the question.

For now I'll probably dummy out elst support and the associated tests
just to make progress on segmented indexes.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Gyan Doshi]

On 2025-06-13 08:23 pm, Tomas Härdin wrote:
> For now I'll probably dummy out elst support and the associated tests
> just to make progress on segmented indexes.

Does `-ignore_editlist true`   not work for your use-case?

Regards,
Gyan
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2398 bytes --]

Hi

On Fri, Jun 13, 2025 at 04:53:14PM +0200, Tomas Härdin wrote:
> fre 2025-06-13 klockan 16:21 +0200 skrev Michael Niedermayer:
> > > 3) remove edit list hacks from all demuxers, especially mov.c
> > 
> > +1 (with ABI +2 bump)
> 
> I'm not sure why a (major?) bump would be necessary. Are removal of
> options a major bump, not a minor one?

removial of things is major, it can break user applications


> Either way, we could make the
> options ffmpeg.c ones rather than mov.c ones. There's going to be pain

unless iam missing something, this would still break library users


> either way, partly because doing this in the demuxer was the wrong
> thing to do in the first place.

yes, you are correct!


> Another case of "oh let's just do a
> quick and dirty thing" coming back to bite us.
> 
> Come to think of this, perhaps hiding this behind a major bump is not
> such a bad idea. That allows making more sweeping changes, including
> behavioral ones.

yeah


> 
> > > The main issue I see with this is that it risks turning ffmpeg into
> > > an
> > > NLE tool,
> > 
> > If ffmpeg could do NLE, that would not be a bad thing
> > And a full implementation of edit lists may be equivalent to this.
> > I think thats a win not a loss
> 
> Fair enough. So long as I don't have to implement it or think
> particularly much about it.

it will be needed to think about "if its possible" to implement
during API design

That said if
* proper edit list implementation is bascially NLE support

Then not only is it alot of work, it also allows us to attract a new
group of developers and users

That is either way it will be alot of work, no way around this.

But if we go all the way and implement NLE support, we also get
as reward more developers to help with that work and maintaince
and likely sponsors paying it.

if OTOH, we do it halfway and do edit lists but avoid NLE, its still
going to be about the same "alot of work" but in that case we get
no extra developers and no extra sponsors thus it could be actually more
work per person

Thus i suggest we pick the choice with more sponsors and more developers

[...]

thx

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Those who are too smart to engage in politics are punished by being
governed by those who are dumber. -- Plato 

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 809 bytes --]

Hi Tomas

On Fri, Jun 13, 2025 at 04:53:14PM +0200, Tomas Härdin wrote:
[...]
> > > I haven't sketched out any API yet, but it would probably closely
> > > mirror how elst works in MOV. Care should be taken that MXF
> > > operational
> > > pattern 3a can also fit into this API. Possibly even OP3c.
> > > 
> > > Thoughts?
> > 
> > it would be cool if someone implemented this, its not trivial as all
> > sides need to support it. demuxer, muxer, player, ffmpeg
> 
> We don't need it everywhere immediately.

yes,

spliting the work in manageble parts is also certainly a good idea

thx


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If you think the mosad wants you dead since a long time then you are either
wrong or dead since a long time.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Nicolas George]

Tomas Härdin (HE12025-06-13):
> 3) remove edit list hacks from all demuxers, especially mov.c

That would mean that applications that currently show files with edit
lists correctly or almost correctly thanks to these hacks would no
longer do so.

If you manage to get something clean working, these hacks should be made
as unobtrusive as possible, but certainly not removed for a long time.

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Tomas Härdin]

fre 2025-06-13 klockan 20:27 +0530 skrev Gyan Doshi:
> 
> 
> On 2025-06-13 08:23 pm, Tomas Härdin wrote:
> > For now I'll probably dummy out elst support and the associated
> > tests
> > just to make progress on segmented indexes.
> 
> Does `-ignore_editlist true`   not work for your use-case?

There is no use case here, except making refactoring possible, so as to
enable opening fragmented files. The current elst hack is a blocker for
that.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Tomas Härdin]

fre 2025-06-13 klockan 18:19 +0200 skrev Michael Niedermayer:
> Hi
> 
> On Fri, Jun 13, 2025 at 04:53:14PM +0200, Tomas Härdin wrote:
> > fre 2025-06-13 klockan 16:21 +0200 skrev Michael Niedermayer:
> > > > 3) remove edit list hacks from all demuxers, especially mov.c
> > > 
> > > +1 (with ABI +2 bump)
> > 
> > I'm not sure why a (major?) bump would be necessary. Are removal of
> > options a major bump, not a minor one?
> 
> removial of things is major, it can break user applications
> 
> 
> > Either way, we could make the
> > options ffmpeg.c ones rather than mov.c ones. There's going to be
> > pain
> 
> unless iam missing something, this would still break library users
> 
> 
> > either way, partly because doing this in the demuxer was the wrong
> > thing to do in the first place.
> 
> yes, you are correct!
> 
> 
> > Another case of "oh let's just do a
> > quick and dirty thing" coming back to bite us.
> > 
> > Come to think of this, perhaps hiding this behind a major bump is
> > not
> > such a bad idea. That allows making more sweeping changes,
> > including
> > behavioral ones.
> 
> yeah

Sounds like you're not opposed to removing the hacks from the demuxer
then, even if this results in a change in behavior, assuming it is done
after a major version bump. Very good.

> > > > The main issue I see with this is that it risks turning ffmpeg
> > > > into
> > > > an
> > > > NLE tool,
> > > 
> > > If ffmpeg could do NLE, that would not be a bad thing
> > > And a full implementation of edit lists may be equivalent to
> > > this.
> > > I think thats a win not a loss
> > 
> > Fair enough. So long as I don't have to implement it or think
> > particularly much about it.
> 
> it will be needed to think about "if its possible" to implement
> during API design
> 
> That said if
> * proper edit list implementation is bascially NLE support
> 
> Then not only is it alot of work, it also allows us to attract a new
> group of developers and users

*We* shouldn't necessarily be attracting them. Libre multimedia is more
than just FFmpeg. It is better to focus NLE efforts on melt. We can
explicitly declare that FFmpeg's task is to carry the necessary
timeline metadata, and the essence, unmolested between formats. When
this is not possible, such as remuxing OP3c MXF to MOV, the ffmpeg CLI
should complain loudly. The present state of things is bad, because
essence is silently discarded by default. I'm sure Google's use case
can be accommodated with a small shell script combining ffmpeg and melt
(the elst hack appears to originate from Google)

Exposing this metadata on the other hand is highly useful to projects
like melt, because it allows proper handling of MOV-elst and higher
operational pattern MXF in tools like kdenlive. kdenlive can already
generate "scripts" that can be used to render timelines on the terminal
via melt. There is no need for us to reimplement such functionality.

FFmpeg's architecture is wholly unsuitable for NLE work. That way lies
madness. This becomes apparent when looking at melt. I suggest more
devs take a look at it.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Tomas Härdin]

fre 2025-06-13 klockan 20:07 +0530 skrev Gyan Doshi:
> 
> 
> On 2025-06-13 04:25 pm, Tomas Härdin wrote:
> > The present level of
> > edit list support could be reimplemented by just fiddling with the
> > -ss
> > and -t options. That is, the ffmpeg CLI could automatically derive
> > -ss
> > and -t from the edit list extracted by lavf.
> 
> In mov.c, edit lists are per-track and can compose their edited
> tracks 
> differently. So if the stream index/tables are to be no longer 
> reconstituted within mov,c, -ss and -t may not return synchronized 
> packets (as per the intended elst) from all streams.

The present hack doesn't handle this kind of stuff correctly anyway,
because it cannot cut audio except on frame boundaries.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 3797 bytes --]

Hi Tomas

On Tue, Jun 17, 2025 at 11:15:25PM +0200, Tomas Härdin wrote:
> fre 2025-06-13 klockan 18:19 +0200 skrev Michael Niedermayer:
[...]
> > > > > The main issue I see with this is that it risks turning ffmpeg
> > > > > into
> > > > > an
> > > > > NLE tool,
> > > > 
> > > > If ffmpeg could do NLE, that would not be a bad thing
> > > > And a full implementation of edit lists may be equivalent to
> > > > this.
> > > > I think thats a win not a loss
> > > 
> > > Fair enough. So long as I don't have to implement it or think
> > > particularly much about it.
> > 
> > it will be needed to think about "if its possible" to implement
> > during API design
> > 
> > That said if
> > * proper edit list implementation is bascially NLE support
> > 
> > Then not only is it alot of work, it also allows us to attract a new
> > group of developers and users
> 
> *We* shouldn't necessarily be attracting them. Libre multimedia is more
> than just FFmpeg. It is better to focus NLE efforts on melt.

Projects grow or they shrink there is no standing still.

Not attracting new developers is madness and would be the end of FFmpeg.
We all surely realize we are mostly a bunch of old man, we do need new
developers.


> We can

If we dont have developers or maintainers we can do nothing

What you suggest or hint toward to me, in plain english sounds like,
drop the ffmpeg command line tool because it would otherwise need to have
NLE support.
Do a vote about this, if you belive people support a ffmpeg that
doesnt support mov/mp4

My way is forward,
    a streaming server (IF there are people who want to work on it),
    NLE (IF there are people who want to work on it)
    anything else multimedia related (IF there are people who want to work on it)

If the community chooses not to move forward even though it would be a
lower effort to move forward. Then I clearly am in the wrong community.


[...]

> FFmpeg's architecture is wholly unsuitable for NLE work. That way lies
> madness. This becomes apparent when looking at melt. I suggest more
> devs take a look at it.

Just move toward a full implementation of edit lists, and you have done
everything i ask for.

If we dont implement full edit list support, then we should not pretend
    1. ffmpeg even supports mov or mp4 (and thats the end of ffmpeg.)
    2. that this could replace the hacks in the demuxer

Again theres no standing still. either FFmpeg grows or it shrinks
and if it shrinks it will die.

Iam NOT asking anyone to compete with or replace any NLE tool.
That may or may not happen as a sideffect.

Now technically, for edit list support you need at minimum
1. demuxers exporting metadata,
2. muxers importing it
3. a way to apply a edit list, so mov (with edit list) ->  mov (without edit list)
   works
   There are 2 cases here, one with a decoder and encoder and one without.
   the first must always work the 2nd only sometimes and this 2nd one is
   equivalent to our hacks more or less.
Whats optional:
4. export and import edit lists in a json format or something like that,
   allow the user to make changes

1-4 == basic NLE support,
yes theres UI, intermediate formats, caches, transitions and and and, but
for basic NLE support 1-4 is whats needed

now i probbaly missed something but the technical details where not the
point of my mail anyway.
also iam wrting this after 5am so probably i missed something or said something
dumb :)

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Whats the most studid thing your enemy could do ? Blow himself up
Whats the most studid thing you could do ? Give up your rights and
freedom because your enemy blew himself up.


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Gyan Doshi]

On 2025-06-18 03:03 am, Tomas Härdin wrote:
> fre 2025-06-13 klockan 20:07 +0530 skrev Gyan Doshi:
>>
>> On 2025-06-13 04:25 pm, Tomas Härdin wrote:
>>> The present level of
>>> edit list support could be reimplemented by just fiddling with the
>>> -ss
>>> and -t options. That is, the ffmpeg CLI could automatically derive
>>> -ss
>>> and -t from the edit list extracted by lavf.
>> In mov.c, edit lists are per-track and can compose their edited
>> tracks
>> differently. So if the stream index/tables are to be no longer
>> reconstituted within mov,c, -ss and -t may not return synchronized
>> packets (as per the intended elst) from all streams.
> The present hack doesn't handle this kind of stuff correctly anyway,
> because it cannot cut audio except on frame boundaries.

The present level of (limited) edit list support can't be reimplemented 
using CLI ss/t is all I'm saying.

Regards,
Gyan

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Nicolas George]

Tomas Härdin (HE12025-06-17):
> The present hack doesn't handle this kind of stuff correctly anyway,
> because it cannot cut audio except on frame boundaries.

Oh, the fallacious argument “if it is not perfect it is worthless”!

It is true that the current hacks cannot deliver perfect accuracy. It is
entirely false and hypocritical to pretend they do not provide valuable
service. In a lot of use cases, the small inaccuracy they impose is
perfectly negligible. This is in particular true for simple players,
whose A-V sync code will just smooth over the inaccuracy.

And before you reply that a proper API will do that without the
inaccuracy, please take the time to reflect that a new proper API will
not magically bring the features to existing applications. Applications
need to be updated to make use of the API.

What we should have is both a proper API to expose the information to
high-level applications and the current hacks to bring the feature
transparently if not perfectly to existing applications.

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 1030 bytes --]

On Wed, Jun 18, 2025 at 05:55:24AM +0200, Michael Niedermayer wrote:
[...]
> 3. a way to apply a edit list, so mov (with edit list) ->  mov (without edit list)
>    works
>    There are 2 cases here, one with a decoder and encoder and one without.
>    the first must always work the 2nd only sometimes and this 2nd one is
>    equivalent to our hacks more or less.

better examle:

3. a way to apply a edit list, so
   mov (with edit list) ->  mpeg-ts (no edit lists) works

   also another example for "apply" is a player, which also needs to apply
   the edit lists to be able to present a file to a human
   thats ffplay but also many other players that use libavcodec & format


[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If the United States is serious about tackling the national security threats 
related to an insecure 5G network, it needs to rethink the extent to which it
values corporate profits and government espionage over security.-Bruce Schneier

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Nicolas George]

Michael Niedermayer (HE12025-06-18):
> 3. a way to apply a edit list, so
>    mov (with edit list) ->  mpeg-ts (no edit lists) works

With transcoding, libavfilter with the concat and trim filters can do
that.

Without transcoding, it is impossible in the general case.

>    also another example for "apply" is a player, which also needs to apply
>    the edit lists to be able to present a file to a human
>    thats ffplay but also many other players that use libavcodec & format

For that, we need to add support for seeking in libavfilter and the
basic filter. Not that hard, in fact.

Regards,

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

[Tomas Härdin]

ons 2025-06-18 klockan 05:55 +0200 skrev Michael Niedermayer:
> What you suggest or hint toward to me, in plain english sounds like,
> drop the ffmpeg command line tool because it would otherwise need to
> have NLE support.

I am not suggesting that. As many others have been pointing out
recently, we don't have to support literally every use case. We can say
that we will transcode essence and expose/map relevant metadata/side
data. We can say that it is up to users to implement the relevant
business logic. melt again serves as a useful example. Another way
could be writing a Python program that parses edit list data supplied
by ffprobe, building an ffmpeg command line that it then executes to
effect the desired rendering


> My way is forward,
>     a streaming server (IF there are people who want to work on it),
>     NLE (IF there are people who want to work on it)
>     anything else multimedia related (IF there are people who want to
> work on it)

You should know by now that a significant fraction of developers
disagree with the kind of scope creep this implies


> If we dont implement full edit list support, then we should not
> pretend
>     1. ffmpeg even supports mov or mp4 (and thats the end of ffmpeg.)

FFmpeg does not support Flash or 3D meshes in mov files either. Despite
this, the world has not ended

> 3. a way to apply a edit list, so
>    mov (with edit list) ->  mpeg-ts (no edit lists) works

There is no "correct" way to do this that satisfies every user. This is
why I say this kind of stuff is business logic. It could be handled by
a small shell script. Or Python. Either way, it does not belong in lavf

>    also another example for "apply" is a player, which also needs to
> apply
>    the edit lists to be able to present a file to a human
>    thats ffplay but also many other players that use libavcodec &
> format

Yes, such as melt. Which already exists and does all the stuff
necessary for NLE work, so there's no need for us to reimplement it

In order to unblock work on fragmented indexes, which is important for
both mov and mxf, I need to be absolutely certain that the change in
behavior that this entails can be effected. I'm already working on the
assumption that it can be guarded by a major version bump. I can put
the API in place, after which someone else can implement the necessary
rendering if they really want to. It is not reasonable at all that an
already complicated task (opening .mov without having to parse the
entire damn file) is predicated on implementing a whole damn NLE
framework as well. Especially when such code already exists (libmlt)

The upside of a proper API is that we can export edit lists from other
formats as well, such as:

* IMF
* MKV
* MXF

Sharp minds should have realized back in 2016 that any hack that
"implements" edit lists in mov.c would also have to be duplicated in
demuxers for other formats with similar functionality. Which is just
silly. It was wrong then and it is wrong now

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".