Re: [FFmpeg-devel] [RFC] Moving edit list handling out of demuxers

["Tomas Härdin" <git@haerdin.se>]

> Derek did this in 2018:
> 
> https://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/227437.html

Interesting. I didn't think to use side data for it. Putting the edit
lists in AVStream seems wrong. They belong to AVFormatContext. Else we
can't support ganged or alternate packages in MXF (OP1b, OP1c, OP2b,
OP2c, OP3b, OP3c), nor could we support IMF. Yes, I know imfdec.c
exists. It too is cursed.

The point about ABI stability is a good one, but can be dealt with by
an extra level of indirection. A bit annoying but probably a good idea
in the long run.

Also this point by Rostislav is a good one:
https://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/227446.html

> It wouldn't be breaking anything - technically making this part of
> lavf in the first place and enabling it by default broke behaviour.

In other words, by default ffmpeg currently *throws away essence*. This
is very bad.

---

Here's what I came up with so far for reference. The idea is to just
stick a pointer to AVEditLists in AVFormatContext. This was written
before I saw the ABI point and the indirection solution to it.

Some of the fields can probably be removed since they're intended
mostly for MXF, and adding fields later is easier than removing them.

Anyway, here are some structs:

// the terminology here is mostly lifted from ISOBMFF and the MXF spec
#define AV_EDIT_ITEM_FILLER 1       ///< Filler. Only duration is of relevance.

/**
 * Edit item.
 * Corresponds to elst entries in MOV, SourceClips in MXF.
 */
typedef struct AVEditItem {
    int stream;                     ///< Source stream for this item. TODO: other files? Could maybe be handled by assigning some kind of global stream ID to streams
    int64_t start;                  ///< Offset in terms of the source's presentation time base, for example sample rate for audio
    int64_t duration;               ///< Duration in terms of edit rate for this track. Must be non-negative. Zero means duration of the rest of the stream (see section 8.6.6.1 of ISO/IEC 14496-12:2015(E))
    AVRational rate;                ///< Relative playback rate. Zero = dwell, repeat pointed-to frame for the duration. 1 = normal. 2 = play twice as fast. TODO: let rate = -1 mean filler like in mov?
    int flags;                      ///< Flags. See AV_EDIT_ITEM_*
    AVDictionary *metadata;         ///< Metadata such as SourcePackageID, SourceTrackID, StartTimecode and so on.
} AVEditItem;

/**
 * Edit track.
 * A collection of edit items in temporal order.
 */
typedef struct AVEditTrack {
    enum AVMediaType type;          ///< Media type of this track.
    AVRational edit_rate;           ///< Edit rate. May be different from the frame rates of the streams pointed to by the edit items.
    int64_t origin;                 ///< Start of this track. Necessary for compatibility with MXF.
    AVEditItem *items;              ///< Edit items.
    int nb_items;                   ///< Number of edit items.
    unsigned items_allocated_size;  ///< Number of bytes allocated for edit items.
    AVDictionary *metadata;         ///< Metadata such as TrackNumber, TrackName and so on.
};

/**
 * Edit list.
 * Multiple edit tracks of different types, all collected as a unit.
 * Constituent tracks may have different origin and duration.
 */
typedef struct AVEditList {
    AVEditTrack *tracks;            ///< Tracks.
    int nb_tracks;                  ///< Number of tracks.
    unsigned tracks_allocated_size; ///< Number of bytes allocated for tracks.
    int loop_count;                 ///< Number of loops. 0 = infinite, >0 = that number of times. Used for HEIF/AVIF. TODO: put in metadata?
    AVDictionary *metadata;         ///< Metadata such as PackageUID, Name and so on.
} AVEditList;

/**
 * Collection of edit lists.
 * This is used for formats like MXF and IMF that support multiple "variants" of the same file,
 * for example different language variants of one movie.
 * In MXF parlance these are called "material packages".
 */
typedef struct AVEditLists {
    AVEditList *edit_lists;             ///< Edit lists.
    int nb_edit_lists;                  ///< Number of edit lists.
    unsigned edit_lists_allocated_size; ///< Number of bytes allocated for edit lists.
    AVDictionary *metadata;             ///< Metadata specific to this collection of edit lists.
} AVEditLists;

---

More fine points to consider that are touched on above:

* how do we express a timeline that points to multiple files, including external files?
* when should playback start?
* should we add an AVMediaType for timecodes?

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".