[FFmpeg-devel] what AVCodecID to use for copying full vanc data between .mxf and .mcc

[FFmpeg-devel] what AVCodecID to use for copying full vanc data between .mxf and .mcc

[Jacob Lifshay]

I'm currently writing a .mcc muxer, it currently translates from eia-608/708 to full vanc packets before outputting a .mcc file:
https://github.com/programmerjake/FFmpeg/tree/add-mcc-mux

I want to add the ability to the mxf and mcc muxers/demuxers to keep the full vanc data when doing stream copies, what do you think is the best way to do that?

* add a new AVCodecID for mxf vbi_vanc_smpte_436M
* change the muxers/demuxers to be able to use AV_CODEC_ID_SMPTE_KLV or AV_CODEC_ID_SMPTE_2038 (they hold the full vanc packets, right?)
* use the mxf vbi_vanc_smpte_436M stream as is -- AV_CODEC_ID_NONE and with vbi_vanc_smpte_436M in the metadata

additionally, it'd be nice to be able to output a eia-608/708 stream to mxf, so that would either need a codec/filter of some sort to translate to full vanc data in whatever format the mxf muxer ends up supporting, or to have that translation built-in to the mxf muxer, like it is built-in to the mxf demuxer (with eia608_extract).

What do you think?

Jacob
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] what AVCodecID to use for copying full vanc data between .mxf and .mcc

[Devin Heitmueller]

Hi Jake,


On Tue, Jul 8, 2025 at 12:12 AM Jacob Lifshay
<programmerjake-at-gmail.com@ffmpeg.org> wrote:
>
> I'm currently writing a .mcc muxer, it currently translates from eia-608/708 to full vanc packets before outputting a .mcc file:
> https://github.com/programmerjake/FFmpeg/tree/add-mcc-mux

I took a quick look at your branch.  So this is a good start, IMHO...

Note that I took a look at the MCC demuxer last year, and it's got a
bunch of issues where the parsing makes a bunch of assumptions (e.g.
doesn't filter out other VANC types which might be present in an MCC
such as AFD).  And it only extracts 608 (i.e. doesn't handle 708).
I'm mentioning this should you try using the MCC demuxer while
developing a muxer.  The demuxer would need some love before I would
trust it for anything in production.

> I want to add the ability to the mxf and mcc muxers/demuxers to keep the full vanc data when doing stream copies, what do you think is the best way to do that?
>
> * add a new AVCodecID for mxf vbi_vanc_smpte_436M
> * change the muxers/demuxers to be able to use AV_CODEC_ID_SMPTE_KLV or AV_CODEC_ID_SMPTE_2038 (they hold the full vanc packets, right?)

I would stay away from KLV entirely.

The SMPTE 2038 standard does support all VANC data types, but support
in ffmpeg really is only oriented around passthrough (i.e. routing
2038 packets from an input TS to an output TS).  It currently doesn't
attempt to decode the 2038 payloads into something easier to work
with.  I've built stuff like bitstream filters that take in 2038
packets and use libklvanc to parse out the individual VANC sections,
but none of that is upstream (happy to send you links to where it can
be found in my public GitHub repo).  In other words, you might have
difficulty working with the structures if you have to deserialize them
and reserialize them when passing them between modules (although
libklvanc does technically support both).

> * use the mxf vbi_vanc_smpte_436M stream as is -- AV_CODEC_ID_NONE and with vbi_vanc_smpte_436M in the metadata

I can't speak to whether this would be a good idea.

> additionally, it'd be nice to be able to output a eia-608/708 stream to mxf, so that would either need a codec/filter of some sort to translate to full vanc data in whatever format the mxf muxer ends up supporting, or to have that translation built-in to the mxf muxer, like it is built-in to the mxf demuxer (with eia608_extract).

I've done this sort of thing with bitstream filters.  The BSF takes in
2038, extracts the 608, and outputs 608 packets.  You could do
something comparable for MXF.  Also, I set up the muxers to
automatically insert the BSF if needed, which makes it painless for
the user.

The 608/708 stuff is kind of a mess in terms of what is in those
payloads.  Some modules will expect 608 only, others will contain
tuples that include 708.  And there continues to be no easy way to go
back and forth between 608 packets and SEI data embedded within the
video stream (in either direction:  SEI to 608 packet streams, or 608
packet streams into SEI).

Devin

-- 
Devin Heitmueller, Senior Software Engineer
LTN Global Communications
o: +1 (301) 363-1001
w: https://ltnglobal.com  e: devin.heitmueller@ltnglobal.com
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] what AVCodecID to use for copying full vanc data between .mxf and .mcc

[Jacob Lifshay]

On Tue, Jul 8, 2025 at 5:08 PM Devin Heitmueller
<devin.heitmueller@ltnglobal.com> wrote:
> I took a quick look at your branch.  So this is a good start, IMHO...

thanks!

> Note that I took a look at the MCC demuxer last year, and it's got a
> bunch of issues where the parsing makes a bunch of assumptions (e.g.
> doesn't filter out other VANC types which might be present in an MCC
> such as AFD).  And it only extracts 608 (i.e. doesn't handle 708).

it does handle 708, since they're using the exact same triples. I've
been using it in the casparcg branch I've been working on and it seems
to work fine, giving me the 608 and 708 captions streams. I haven't
tried to use it with any .mcc files that have anything other than 708
captions vanc packets however -- do those actually exist? it seems
like the format supports them, but every file I've seen just uses it
for 708 captions. I couldn't find any by searching github for:
path:*.mcc /^[0-9:]{11}\t[^T]/

> I'm mentioning this should you try using the MCC demuxer while
> developing a muxer.  The demuxer would need some love before I would
> trust it for anything in production.
>
> > I want to add the ability to the mxf and mcc muxers/demuxers to keep the full vanc data when doing stream copies, what do you think is the best way to do that?
> >
> > * add a new AVCodecID for mxf vbi_vanc_smpte_436M

I'm thinking this is the best option, since it's a pretty simple
format so the required [de]serialization is pretty trivial. it's just
the data in Table 7 (starts on page 13) of:
https://pub.smpte.org/latest/st436/s436m-2006.pdf

(note it uses a klv array which is just u32be array len, u32be array
element size (always 1 here), and the array contents)

> > * change the muxers/demuxers to be able to use AV_CODEC_ID_SMPTE_KLV or AV_CODEC_ID_SMPTE_2038 (they hold the full vanc packets, right?)
>
> I would stay away from KLV entirely.

ok.

> In other words, you might have
> difficulty working with the structures

yeah, it looks waay overcomplicated for what I want.

> > additionally, it'd be nice to be able to output a eia-608/708 stream to mxf, so that would either need a codec/filter of some sort to translate to full vanc data in whatever format the mxf muxer ends up supporting, or to have that translation built-in to the mxf muxer, like it is built-in to the mxf demuxer (with eia608_extract).
>
> I've done this sort of thing with bitstream filters.  The BSF takes in
> 2038, extracts the 608, and outputs 608 packets.  You could do
> something comparable for MXF.

so I could write bitstream filters to interconvert between 608/708 and
mxf vbi_vanc_smpte_436M and modify the muxers to auto-insert the
bitstream filters?

> Also, I set up the muxers to
> automatically insert the BSF if needed, which makes it painless for
> the user.

so, just add the right ff_stream_add_bitstream_filter calls in the
muxers' init functions?

Thanks,
Jacob
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] what AVCodecID to use for copying full vanc data between .mxf and .mcc

[Devin Heitmueller]

On Tue, Jul 8, 2025 at 10:38 PM Jacob Lifshay
<programmerjake-at-gmail.com@ffmpeg.org> wrote:
> > Note that I took a look at the MCC demuxer last year, and it's got a
> > bunch of issues where the parsing makes a bunch of assumptions (e.g.
> > doesn't filter out other VANC types which might be present in an MCC
> > such as AFD).  And it only extracts 608 (i.e. doesn't handle 708).
>
> it does handle 708, since they're using the exact same triples. I've
> been using it in the casparcg branch I've been working on and it seems
> to work fine, giving me the 608 and 708 captions streams.

Last I looked it was throwing everything on the floor other than the
608 tuples, but perhaps that has changed.

> I haven't
> tried to use it with any .mcc files that have anything other than 708
> captions vanc packets however -- do those actually exist? it seems
> like the format supports them, but every file I've seen just uses it
> for 708 captions. I couldn't find any by searching github for:
> path:*.mcc /^[0-9:]{11}\t[^T]/

Yeah, so I've seen MCC files that contain stuff like AFD.  I'll dig
around and see if I can find you a sample.  IIRC the demuxer wasn't
even looking at the DID/SDID, so it would treat every line in the file
as a CDP payload.

<snip>

> > I've done this sort of thing with bitstream filters.  The BSF takes in
> > 2038, extracts the 608, and outputs 608 packets.  You could do
> > something comparable for MXF.
>
> so I could write bitstream filters to interconvert between 608/708 and
> mxf vbi_vanc_smpte_436M and modify the muxers to auto-insert the
> bitstream filters?

Yeah, I've done this in the past, for example I had a BSF that
converted SCTE-35 to SCTE-104, and if you tried to map the SCTE-35
stream into the decklink muxer it would automatically add the
bitstream filter.
>
> > Also, I set up the muxers to
> > automatically insert the BSF if needed, which makes it painless for
> > the user.
>
> so, just add the right ff_stream_add_bitstream_filter calls in the
> muxers' init functions?

Yes.  You can see an example of it here:

https://github.com/LTNGlobal-opensource/FFmpeg-ltn/blob/lted2b/libavdevice/decklink_enc.cpp#L860

Devin

-- 
Devin Heitmueller, Senior Software Engineer
LTN Global Communications
o: +1 (301) 363-1001
w: https://ltnglobal.com  e: devin.heitmueller@ltnglobal.com
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] what AVCodecID to use for copying full vanc data between .mxf and .mcc

[Tomas Härdin]

mån 2025-07-07 klockan 21:12 -0700 skrev Jacob Lifshay:
> I'm currently writing a .mcc muxer, it currently translates from eia-
> 608/708 to full vanc packets before outputting a .mcc file:
> https://github.com/programmerjake/FFmpeg/tree/add-mcc-mux
> 
> I want to add the ability to the mxf and mcc muxers/demuxers to keep
> the full vanc data when doing stream copies, what do you think is the
> best way to do that?

Stream copy should be easy enough. I'm not sure whether BSFs is the way
to go in the long run, since full ANC support may require dealing with
full uncropped frames and the ability to encode/decode VBI data. We
also currently lack proper 608/708 support, so I'm not sure what the
utility of converting from 436M to 608 is as things stand at the
moment. Also horizontal ANC data exists. Stream copying 436M sounds
reasonable enough.

Consider some potential use cases:

* copying 436M data from MXF to MXF
* decoding 8-bit 436M data to 1-bit data
* "encoding" (rendering) 1-bit 436M data to 8-bit data
* extracting teletext data from 436M and rendering it
* combining 436M and video essence to a full frame for playout
* embedding/de-embedding 608/708 CC:s and transcoding to/from regular
subtitles
* "encoding" subtitles as teletext, then embedding the resulting ANC
data in whatever way is appropriate (as video samples, as 436M data or
whatever)
* possibly some cursed combination of all of the above, like doing
teletext and closed captions at the same time

> * add a new AVCodecID for mxf vbi_vanc_smpte_436M
> * change the muxers/demuxers to be able to use AV_CODEC_ID_SMPTE_KLV
> or AV_CODEC_ID_SMPTE_2038 (they hold the full vanc packets, right?)

Having key and length as part of the "essence" for these types of
streams sounds wrong and cursed. Shouldn't we do the same thing we do
for audio and video, and convert either to some common format?
Otherwise, wouldn't we need to write BSFs for the outer product of
every type of ANC "codec"? Perhaps the way subtitles are handled is the
way to go, where there are two categories of subs (text and bitmap).

> additionally, it'd be nice to be able to output a eia-608/708 stream
> to mxf, so that would either need a codec/filter of some sort to
> translate to full vanc data in whatever format the mxf muxer ends up
> supporting, or to have that translation built-in to the mxf muxer,
> like it is built-in to the mxf demuxer (with eia608_extract).

There's more than one way to skin this cat with MXF. You can either put
the 608/708 data in the video essence and set the cropping information
appropriately, or as 436M packets, or inside the H.264 packets as SEI
messages. Or all three at the same time.

It's unfortunate that libav* has chosen to separate audio and video,
rather than carrying them as combined edit units as libmlt does, into
which we could also slip ANC data, subtitles or whatever.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] what AVCodecID to use for copying full vanc data between .mxf and .mcc

[Jacob Lifshay]

On July 22, 2025 5:21:24 AM PDT, "Tomas Härdin" <git@haerdin.se> wrote:
> mån 2025-07-07 klockan 21:12 -0700 skrev Jacob Lifshay:
> > I'm currently writing a .mcc muxer, it currently translates from eia-
> > 608/708 to full vanc packets before outputting a .mcc file:
> > https://github.com/programmerjake/FFmpeg/tree/add-mcc-mux

I've since changed the mcc muxer to support 436m, see below.
> > 
> > I want to add the ability to the mxf and mcc muxers/demuxers to keep
> > the full vanc data when doing stream copies, what do you think is the
> > best way to do that?
> 
> Stream copy should be easy enough. I'm not sure whether BSFs is the way
> to go in the long run, since full ANC support may require dealing with
> full uncropped frames and the ability to encode/decode VBI data.

imo that needs the ability for libavfilter to work on subtitles streams which seems unlikely to happen anytime soon based on how softworkz last attempt went. so I decided to write bitstream filters for now because they work on subtitles streams too.

> We
> also currently lack proper 608/708 support, so I'm not sure what the
> utility of converting from 436M to 608 is as things stand at the
> moment.

we actually have waay more 608/708 support than 436M, there are codecs to decode it to subtitles and subcc can extract it from video side data. 436M (until I merge the code I'm working on) only supports mxf muxing/demuxing (which is also just as much support as any of the other fully general vanc formats afaik).

The code I'm working on adds 436m output from the mcc decoder, adds two bitstream filters for conversion between 608/708 and 436m, adds a avpriv_* API for manipulating 436m data and extracting 608/708 data, and adds a mcc encoder that supports both 608/708 and 436m input by auto-inserting the 608 to 436m bitstream filter (this is changed from my earlier email). it also tries to add 608/708 output in the mxf muxer by auto-inserting that bitstream filter but I haven't finished figuring out all the code that needs changing to support that without the output being different from manually inserting that bitstream filter so the mxf muxer only ever sees a 436m data stream.

> Also horizontal ANC data exists.

mxf 436m supports HANC. i'd have to double check the code I wrote, since iirc it checks for only vanc data in some places, i assumed the mxf 436m is only vanc since vanc was in the metadata name back when it was a AV_CODEC_ID_NONE (as it is currently upstream, my code changes that to a new codec id).

> Stream copying 436M sounds
> reasonable enough.
> 
> Consider some potential use cases:
> 
> * copying 436M data from MXF to MXF
supported upstream now, my code retains support for that.

> * decoding 8-bit 436M data to 1-bit data
1-bit data is not allowed with the essence key I'm using according to the 436M specification, there's a separate essence key for VBI data that I'm not changing how ffmpeg handles it, so it's still AV_CODEC_ID_NONE with metadata set to VBI something. the 436m manipulation API I wrote has enumerants and AVOptions for it but not decoding/encoding support, it only supports ANC for that.

https://code.ffmpeg.org/programmerjake/FFmpeg/src/branch/add-mcc-mux/libavcodec/vanc_smpte_436m.h (i may rename that -- vanc -> anc)

> * "encoding" (rendering) 1-bit 436M data to 8-bit data
same as previous

> * extracting teletext data from 436M and rendering it
that should be easily doable similarly to how 

> * combining 436M and video essence to a full frame for playout
iirc ffmpeg architecturally doesn't support merging a 608 stream into video side data since bitstream filters only have one input and avframe filters don't support subtitles. I've had to work around that with custom code in casparcg. 436m data is similar.

> * embedding/de-embedding 608/708 CC:s and transcoding to/from regular
> subtitles

there are codecs for decoding 608/708 (but no encoders) and libavfilter has custom code for de-embedding from a video stream (subcc). embedding in a video stream isn't supported due to the architectural issues.

> * "encoding" subtitles as teletext, then embedding the resulting ANC
> data in whatever way is appropriate (as video samples, as 436M data or
> whatever)
> * possibly some cursed combination of all of the above, like doing
> teletext and closed captions at the same time

I haven't thought much about teletext, I'm assuming it'll follow a very similar path to 608/708. I'm working on all this for a TV station in the USA so only needed to bother with 608/708 data.

> > * add a new AVCodecID for mxf vbi_vanc_smpte_436M
> > * change the muxers/demuxers to be able to use AV_CODEC_ID_SMPTE_KLV
> > or AV_CODEC_ID_SMPTE_2038 (they hold the full vanc packets, right?)
> 
> Having key and length as part of the "essence" for these types of
> streams sounds wrong and cursed. Shouldn't we do the same thing we do
> for audio and video, and convert either to some common format?

sure, for now I've picked 436m as the common format and left other vanc formats <-> 436m bitstream filters for others to implement.

> Otherwise, wouldn't we need to write BSFs for the outer product of
> every type of ANC "codec"?

technically you can chain bsfs using av_bsf_list_finalize so we just have to support every anc codec <-> 436m. so when those eventually get implemented by someone else you can do AV_CODEC_ID_SMPTE_KLV -> AV_CODEC_ID_SMPTE_2038 via using 436m as an intermediate format.

Jacob
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".