Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

Hi Marth64,


first of all, thanks for taking on solving this long-standing broken feature. My earlier submission (https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=5031) didn't find much love, so I had given up proposing, yet I can say that multiple millions of files got probed with it meanwhile. I'll get back later to the reason for mentioning this.

Following the conversation about your submission I could see that your earlier approaches were similar to mine, but unfortunately you have been talked out of it.

Comments like

"Strong no. Internal properties are internal...."

without proposing a better, more agreeable way - without even caring about what you are trying to achieve, even though it's about fixing a long-standing important bug - that's a blueprint example for what I mentioned recently, that the predominant understanding of "democratic" participation - whether member in a committee or GA - is about having the right to say "no" to something. This doesn't bring things forward and that's not the way how leadership should look like, neither can it work successfully this way.
(this is meant to address everyone giving answers like that, selection is purely accidental, it doesn't matter who actually said it)


This serves as a good example for how these things are doing no good to the project. In earlier days, the tagline was like "world's fastest audio and video encoder", but this is rather heading for "world's slowest media prober".

Try this for example:

ffprobe -loglevel verbose -analyze_frames -show_entries stream=closed_captions:disposition=:side_data=  "http://streams.videolan.org/streams/ts/CC/NewsStream-608-ac3.ts"

This correctly indicates CC availability with 66,147,986 bytes read (file size: 60.428.088)

Without analyze_frames: 5,715,518 bytes read
So, the actual probing requires 5.7 MB. To get a similar coverage I set -read_intervals "%+3"
to analyze frames from the first 3s. But now it reads 11MB, because it processes the data twice.


For comparison, with the patch I had submitted in 2021 it correctly detects CC with only 5,718,438 bytes read - i.e. without any extra processing needed.


What's happening?

When running ffprobe with show_frames (and now also with analyze_frames), it (also) does the actual probing first. This part (#1) stops kind of like "as soon as it has all required information" or the analyze_duration is exceeded.
Then, for (#2) it seeks back to the start and iterates through the frames to determine the presence of CC.

But #2 isn't needed at all, because the information about CC data is already available after #1 is done. ffprobe just cannot access it, because somebody said it must be private: well done!
Normally, everything in ffmpeg is optimized to the max with SIMD and assembly code (which is great), but here, ffprobe is taking multiple times longer (see below) to execute than it would need to - because of somebody's idealized picture of which data has to be private. 
That's one of the reasons why I'm saying that a stronger leadership is needed - like for recognizing and clearing up such nonsense in a timely manner. Calling for a "democratic" vote each time for every little bit - who wants that? As a GA member, I wouldn't want to have to vote and as a contributor I would rather give up (like I did) or choose an inferior way (like Marth64 did).


Notes

It had been argued that the CC indication in the codec properties would not be reliable as it might not have analyzed a sufficient amount of frames "to be sure":

While this is a very valid concern for some kinds of frame side data, it does not apply to CC data. It's either in every frame or none. If a provider generally broadcasts CC, then it's always present in every frame, even during programs for which no CC is available - it's always there. Like I mentioned at the top, we're using the properties field from the codec (via codec_par) and there hasn’t been a single case reported where the CC detection this way would have been incorrect.


Why not just use the first video frame, then?

There are many reasons why this is not reliable:

- A broadcast stream can have multiple programs and video streams
- The tuner might not have fully locked at the start, packets can be corrupted, audio packets are smaller, less prone to corruption, so you might get valid audio packets before getting valid video frames, same for SD vs. HD, etc. 
- Some programs might be scrambled and get decrypted after some delay, while fta programs work instantly
- so which value to choose for -read_intervals? You just can't be sure, so you need to choose a value by which you are on the safe side - maybe 5s -> just for doing something which has been done already.

Any value you would choose won't ever hit the right point (and even then it would still be doubled work), because the right point is exactly when the probing (#1) is complete - as that is determined dynamically, up to the configured limit, and then you can be sure that when a video stream has been successfully probed, the CC information is valid and when not it doesn't matter anyway.
That's why the right place for CC detection is clearly in the codec and not in a duplicate sequence of analyzing frames in ffprobe.

How that information can find its way from the codec to ffprobe is surely debatable, but comments like "Strong no" or a plain "This data doesn't belong there" is not a debate - it's only destructive. It would be fine if this would be the first part of an answer which is suggesting an alternative way (like Marton's reply on Nov 28), but this is just happening too rarely. 
Again, I'm not after blaming specific persons. This kind of toxicity is infectious and I'm sure that many don't even want to be like that and probably don't even behave like that elsewhere. It seems more like an ill adaption process, where one has learned that it's more safe here to point at something wrong than to make a suggestion which might then be instantly bashed by others.
Rather than complaining and blaming I would rather like to motivate people not to be like that, try to break those chains of behavior and go forward with a more positive attitude, ignoring and stop feeding the negativism of others. 😊


On the subject: I like the -analyze_frames method in general, had something similar in mind actually, because there's other data (like HDR10+ or DVB subtitle presence and screen size) which can occur at any position. I would like to extend this in a way that you can specify for what you are actually looking and also exit in case when all of it has been found.

I don't know anything about the recurrence pattern of the film-grain data, but at least for the CC detection there has to be a better way, using the field from the decoder. When probing libraries with thousands of files the ffprobe execution time matters a lot.

Best
sw




_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Kieran Kunhya via ffmpeg-devel]

>
> While this is a very valid concern for some kinds of frame side data, it
> does not apply to CC data. It's either in every frame or none. If a
> provider generally broadcasts CC, then it's always present in every frame,
> even during programs for which no CC is available - it's always there. Like
> I mentioned at the top, we're using the properties field from the codec
> (via codec_par) and there hasn’t been a single case reported where the CC
> detection this way would have been incorrect.
>

This isn't true, CC data is sparse.
I have no opinions about the rest of the text.

Kieran

>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Kieran Kunhya via ffmpeg-devel
> Sent: Monday, January 27, 2025 10:40 AM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Cc: Kieran Kunhya <kieran618@googlemail.com>; Marth64
> <marth64@proxyid.net>
> Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection
> and ffprobe fields (cover letter)
> 
> >
> > While this is a very valid concern for some kinds of frame side
> data, it
> > does not apply to CC data. It's either in every frame or none. If a
> > provider generally broadcasts CC, then it's always present in every
> frame,
> > even during programs for which no CC is available - it's always
> there. Like
> > I mentioned at the top, we're using the properties field from the
> codec
> > (via codec_par) and there hasn’t been a single case reported where
> the CC
> > detection this way would have been incorrect.
> >
> 
> This isn't true, CC data is sparse.
> I have no opinions about the rest of the text.
> 
> Kieran



Hi Kieran,

Thanks for your reply.

In expectation of doubt, I had yesterday slightly modified ffprobe to count frames with and without CC side data like this:

if (par->codec_type == AVMEDIA_TYPE_VIDEO) {
    if (!is_sub && do_analyze_frames) {
        int has_cc = 0;

        for (int i = 0; i < frame->nb_side_data; i++) {
            if (frame->side_data[i]->type == AV_FRAME_DATA_A53_CC) {
                streams_with_closed_captions[pkt->stream_index]++;
                has_cc = 1;
            }
            else if (frame->side_data[i]->type == AV_FRAME_DATA_FILM_GRAIN_PARAMS)
                streams_with_film_grain[pkt->stream_index] = 1;
        }

        if (!has_cc) {
            streams_without_closed_captions[pkt->stream_index]++;
        }
    }
}


Then I ran it on all kinds of mpegts recordings I had at hand, and that's the result:

All video streams only, some were from the same file, others recorded in the same way, hence the duplications.

+--------+----------------+-------------------+
|        | Frames with CC | Frames without CC |
+--------+----------------+-------------------+
| stream |            786 |                 0 |
+--------+----------------+-------------------+
| stream |            786 |                 0 |
+--------+----------------+-------------------+
| stream |            786 |                 0 |
+--------+----------------+-------------------+
| stream |              0 |               232 |
+--------+----------------+-------------------+
| stream |              0 |               238 |
+--------+----------------+-------------------+
| stream |              0 |               239 |
+--------+----------------+-------------------+
| stream |              0 |               237 |
+--------+----------------+-------------------+
| stream |              0 |               235 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            112 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |            118 |                 0 |
+--------+----------------+-------------------+
| stream |             58 |                 0 |
+--------+----------------+-------------------+
| stream |             51 |                 0 |
+--------+----------------+-------------------+
| stream |             42 |                 0 |
+--------+----------------+-------------------+
| stream |             45 |                 0 |
+--------+----------------+-------------------+
| stream |             42 |                 0 |
+--------+----------------+-------------------+
| stream |            102 |                 0 |
+--------+----------------+-------------------+
| stream |             45 |                 0 |
+--------+----------------+-------------------+
| stream |            104 |                 0 |
+--------+----------------+-------------------+
| stream |             42 |                 0 |
+--------+----------------+-------------------+
| stream |            102 |                 0 |
+--------+----------------+-------------------+
| stream |             44 |                 0 |
+--------+----------------+-------------------+
| stream |            104 |                 0 |
+--------+----------------+-------------------+
| stream |             42 |                 0 |
+--------+----------------+-------------------+
| stream |            106 |                 0 |
+--------+----------------+-------------------+
| stream |             44 |                 0 |
+--------+----------------+-------------------+
| stream |             99 |                 0 |
+--------+----------------+-------------------+
| stream |             41 |                 0 |
+--------+----------------+-------------------+
| stream |            101 |                 0 |
+--------+----------------+-------------------+
| stream |             45 |                 0 |
+--------+----------------+-------------------+
| stream |             99 |                 0 |
+--------+----------------+-------------------+
| stream |             32 |                 0 |
+--------+----------------+-------------------+
| stream |             92 |                 0 |
+--------+----------------+-------------------+
| stream |             40 |                 0 |
+--------+----------------+-------------------+
| stream |             95 |                 0 |
+--------+----------------+-------------------+
| stream |             37 |                 0 |
+--------+----------------+-------------------+
| stream |             97 |                 0 |
+--------+----------------+-------------------+
| stream |             34 |                 0 |
+--------+----------------+-------------------+
| stream |             94 |                 0 |
+--------+----------------+-------------------+
| stream |             37 |                 0 |
+--------+----------------+-------------------+
| stream |             96 |                 0 |
+--------+----------------+-------------------+
| stream |             34 |                 0 |
+--------+----------------+-------------------+
| stream |             94 |                 0 |
+--------+----------------+-------------------+
| stream |             35 |                 0 |
+--------+----------------+-------------------+
| stream |             91 |                 0 |
+--------+----------------+-------------------+
| stream |             43 |                 0 |
+--------+----------------+-------------------+
| stream |            103 |                 0 |
+--------+----------------+-------------------+
| stream |             40 |                 0 |
+--------+----------------+-------------------+
| stream |            101 |                 0 |
+--------+----------------+-------------------+
| stream |             43 |                 0 |
+--------+----------------+-------------------+
| stream |             59 |                 0 |
+--------+----------------+-------------------+
| stream |              0 |              5002 |
+--------+----------------+-------------------+
| stream |              0 |               725 |
+--------+----------------+-------------------+
| stream |              0 |               528 |
+--------+----------------+-------------------+
| stream |              0 |              7423 |
+--------+----------------+-------------------+
| stream |              0 |              3532 |
+--------+----------------+-------------------+
| stream |              0 |              9088 |
+--------+----------------+-------------------+
| stream |              0 |            105006 |
+--------+----------------+-------------------+
| stream |              0 |            104868 |
+--------+----------------+-------------------+
| stream |              0 |             50770 |
+--------+----------------+-------------------+
| stream |          74595 |                 0 |
+--------+----------------+-------------------+
| stream |              0 |                90 |
+--------+----------------+-------------------+
| stream |              0 |                90 |
+--------+----------------+-------------------+
| stream |              0 |             95652 |
+--------+----------------+-------------------+
| stream |              0 |              8745 |
+--------+----------------+-------------------+
| stream |              0 |              2832 |
+--------+----------------+-------------------+
| stream |              0 |              7204 |
+--------+----------------+-------------------+
| stream |              0 |              7204 |
+--------+----------------+-------------------+
| stream |              0 |              7204 |
+--------+----------------+-------------------+
| stream |              0 |              7197 |
+--------+----------------+-------------------+
| stream |              0 |              7197 |
+--------+----------------+-------------------+

I can provide all (most) files privately. Happy to look into any examples if you have.

Best
sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Soft Works
> Sent: Monday, January 27, 2025 11:00 AM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Cc: Marth64 <marth64@proxyid.net>; Kieran Kunhya
> <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection
> and ffprobe fields (cover letter)
> 
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Kieran Kunhya via ffmpeg-devel
> > Sent: Monday, January 27, 2025 10:40 AM
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel@ffmpeg.org>
> > Cc: Kieran Kunhya <kieran618@googlemail.com>; Marth64
> > <marth64@proxyid.net>
> > Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC
> detection
> > and ffprobe fields (cover letter)
> >

I almost forgot these:

These for example are from full-transponder recordings of terrestrial ATSC in Utah:

+---------+------------+
| With CC | Without CC |
+---------+------------+
|    1869 |          0 |
+---------+------------+
|    1876 |          0 |
+---------+------------+
|    3794 |          0 |
+---------+------------+
|    1873 |          0 |
+---------+------------+
|    1639 |          0 |
+---------+------------+
|    1640 |          0 |
+---------+------------+
|    1628 |          1 |
+---------+------------+
|    1655 |          0 |
+---------+------------+
|    1636 |          0 |
+---------+------------+
|    1634 |          0 |
+---------+------------+
|    1633 |          0 |
+---------+------------+
|       0 |       1881 |
+---------+------------+
|       0 |       1880 |
+---------+------------+
|    3763 |          0 |
+---------+------------+
|    1862 |          0 |
+---------+------------+
|    1468 |          0 |
+---------+------------+
|       0 |       1821 |
+---------+------------+
|    1878 |          0 |
+---------+------------+
|    3721 |          0 |
+---------+------------+
|    1876 |          0 |
+---------+------------+
|    1841 |          0 |
+---------+------------+
|    1901 |          0 |
+---------+------------+
|    3757 |          0 |
+---------+------------+
|    1898 |          0 |
+---------+------------+
|    1824 |          0 |
+---------+------------+
|    1864 |          0 |
+---------+------------+
|    1889 |          0 |
+---------+------------+
|    3748 |          0 |
+---------+------------+
|    1891 |          0 |
+---------+------------+
|    1884 |          0 |
+---------+------------+
|    1870 |          0 |
+---------+------------+
|     119 |       1775 |
+---------+------------+
|    1891 |          0 |
+---------+------------+
|    1884 |          2 |
+---------+------------+
|    1897 |          0 |
+---------+------------+
|    1897 |          0 |
+---------+------------+
|    1889 |          0 |
+---------+------------+
|    1893 |          0 |
+---------+------------+
|    1892 |          0 |
+---------+------------+
|    1898 |          0 |
+---------+------------+
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Kieran Kunhya via ffmpeg-devel
> Sent: Monday, January 27, 2025 10:40 AM
> > While this is a very valid concern for some kinds of frame side
> data, it
> > does not apply to CC data. It's either in every frame or none. If a
> > provider generally broadcasts CC, then it's always present in every
> frame,
> > even during programs for which no CC is available - it's always
> there. Like
> > I mentioned at the top, we're using the properties field from the
> codec
> > (via codec_par) and there hasn’t been a single case reported where
> the CC
> > detection this way would have been incorrect.
> >
> 
> This isn't true, CC data is sparse.
> I have no opinions about the rest of the text.
> 
> Kieran

Hi Kieran,

I found it 😊

It's in  EIA-708-A from 1999 as well as in the latest successor spec ANSI/CTA-708-E S-2023

from chapter 4.1 (1999): 

"for DTV and DTVCC specific (Le., ATSC A/53) caption encoding, the DTV Closed-Captioning channel is a continuous 9600 bps stream allocated from the DTV signal capacity"
(the latest version adds more variations)

and chapter 4.2 (1999): "Pre-Allocated Bandwidth"

"The DTVCC Transport Channel is a fixed, pre-allocated stream which exists in all DTV-system bit streams, even though captions may not be present. This ever-present bandwidth (which includes NTSC and DTVCC caption data) allows encoders to easily insert caption data into the DTV bit-stream at the point of origin and at multiple down-stream encoding points without having to perform complex picture data processing and bandwidth reallocation"


Maybe you just mixed it up with DVB subtitles which are actually sparse?

sw



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Kieran Kunhya via ffmpeg-devel]

On Mon, Jan 27, 2025 at 7:03 PM Soft Works <softworkz@hotmail.com> wrote:
>
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Kieran Kunhya via ffmpeg-devel
> > Sent: Monday, January 27, 2025 10:40 AM
> > > While this is a very valid concern for some kinds of frame side
> > data, it
> > > does not apply to CC data. It's either in every frame or none. If a
> > > provider generally broadcasts CC, then it's always present in every
> > frame,
> > > even during programs for which no CC is available - it's always
> > there. Like
> > > I mentioned at the top, we're using the properties field from the
> > codec
> > > (via codec_par) and there hasn’t been a single case reported where
> > the CC
> > > detection this way would have been incorrect.
> > >
> >
> > This isn't true, CC data is sparse.
> > I have no opinions about the rest of the text.
> >
> > Kieran
>
> Hi Kieran,
>
> I found it 😊
>
> It's in  EIA-708-A from 1999 as well as in the latest successor spec ANSI/CTA-708-E S-2023
>
> from chapter 4.1 (1999):
>
> "for DTV and DTVCC specific (Le., ATSC A/53) caption encoding, the DTV Closed-Captioning channel is a continuous 9600 bps stream allocated from the DTV signal capacity"
> (the latest version adds more variations)
>
> and chapter 4.2 (1999): "Pre-Allocated Bandwidth"
>
> "The DTVCC Transport Channel is a fixed, pre-allocated stream which exists in all DTV-system bit streams, even though captions may not be present. This ever-present bandwidth (which includes NTSC and DTVCC caption data) allows encoders to easily insert caption data into the DTV bit-stream at the point of origin and at multiple down-stream encoding points without having to perform complex picture data processing and bandwidth reallocation"

Not everyone follows this.

Kieran
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> -----Original Message-----
> From: Kieran Kunhya <kieran618@googlemail.com>
> Sent: Monday, January 27, 2025 8:26 PM
> To: Soft Works <softworkz@hotmail.com>
> Cc: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>; Marth64 <marth64@proxyid.net>
> Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection
> and ffprobe fields (cover letter)
> 
> On Mon, Jan 27, 2025 at 7:03 PM Soft Works <softworkz@hotmail.com>
> wrote:
> >
> > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > > Kieran Kunhya via ffmpeg-devel
> > > Sent: Monday, January 27, 2025 10:40 AM
> > > > While this is a very valid concern for some kinds of frame side
> > > data, it
> > > > does not apply to CC data. It's either in every frame or none.
> If a
> > > > provider generally broadcasts CC, then it's always present in
> every
> > > frame,
> > > > even during programs for which no CC is available - it's always
> > > there. Like
> > > > I mentioned at the top, we're using the properties field from
> the
> > > codec
> > > > (via codec_par) and there hasn’t been a single case reported
> where
> > > the CC
> > > > detection this way would have been incorrect.
> > > >
> > >
> > > This isn't true, CC data is sparse.
> > > I have no opinions about the rest of the text.
> > >
> > > Kieran
> >
> > Hi Kieran,
> >
> > I found it 😊
> >
> > It's in  EIA-708-A from 1999 as well as in the latest successor
> spec ANSI/CTA-708-E S-2023
> >
> > from chapter 4.1 (1999):
> >
> > "for DTV and DTVCC specific (Le., ATSC A/53) caption encoding, the
> DTV Closed-Captioning channel is a continuous 9600 bps stream
> allocated from the DTV signal capacity"
> > (the latest version adds more variations)
> >
> > and chapter 4.2 (1999): "Pre-Allocated Bandwidth"
> >
> > "The DTVCC Transport Channel is a fixed, pre-allocated stream which
> exists in all DTV-system bit streams, even though captions may not be
> present. This ever-present bandwidth (which includes NTSC and DTVCC
> caption data) allows encoders to easily insert caption data into the
> DTV bit-stream at the point of origin and at multiple down-stream
> encoding points without having to perform complex picture data
> processing and bandwidth reallocation"
> 
> Not everyone follows this.
> 
> Kieran

Do you have an example stream recording?
And who are those who wouldn't be following? Broadcasters? 


As mentioned previously, we have users from all over the US who are recording ATSC streams from all kinds of broadcasters and not a single case has been seen where the CC detection are part of the codec probing has led to incorrect results (in 4 years).

And since when do we care about off-spec?
Finally, it's not about breaking any existing behavior. The CC detection has been defunct for years now.

sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Devin Heitmueller]

> Do you have an example stream recording?
> And who are those who wouldn't be following? Broadcasters?

For what it's worth, I have seen this and it can definitely happen.
It's not common though since it both technically violates the spec and
also not best practice (for example, some televisions won't show you
the option to enable captions if the caption stream isn't present at
all).  Generally it occurs when the broadcaster is doing splicing of
TS streams entirely in the compressed domain (e.g. for ad insertion),
or cases where ads or programs don't contain captions and the
encoder/transcoder isn't smart enough to generate empty caption
packets and include them in the output.

That said, I think it's very defensible to say, "We don't set the flag
saying captions are present if not detected within the probing
window".

Devin

-- 
Devin Heitmueller, Senior Software Engineer
LTN Global Communications
o: +1 (301) 363-1001
w: https://ltnglobal.com  e: devin.heitmueller@ltnglobal.com
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Devin Heitmueller
> Sent: Monday, January 27, 2025 9:16 PM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Cc: Marth64 <marth64@proxyid.net>; Kieran Kunhya
> <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection
> and ffprobe fields (cover letter)
> 
> > Do you have an example stream recording?
> > And who are those who wouldn't be following? Broadcasters?
> 
> For what it's worth, I have seen this and it can definitely happen.
> It's not common though since it both technically violates the spec
> and
> also not best practice (for example, some televisions won't show you
> the option to enable captions if the caption stream isn't present at
> all).  Generally it occurs when the broadcaster is doing splicing of
> TS streams entirely in the compressed domain (e.g. for ad insertion),
> or cases where ads or programs don't contain captions and the
> encoder/transcoder isn't smart enough to generate empty caption
> packets and include them in the output.
> 
> That said, I think it's very defensible to say, "We don't set the
> flag
> saying captions are present if not detected within the probing
> window".
> 
> Devin

Hi Devin,

thanks a lot for your insights. Sure that those kinds of cases exist. Another example would be shared program slots where one broadcaster sends CC and the other doesn't.

But that's different from what Kieran said (= "sparse"), which would mean that there's no continuous stream and caption data comes around only when there's some caption content to display. I was about to say that this isn't even possible due to the way TVs are working, but I didn't feel to have sufficient evidence, so thanks for confirming this.

In case of "sparse" data, it would have meant that there could be captions which can only be detected by scanning frames for a certain time range (like 20s).
But as we agree - the result from the initial parsing is sufficient to reliably know whether there are captions at the start of the stream - same like we determine the frame size. Both can change at a later time, but both are definitive for the probed section of the stream.

Besides: one doesn't exclude the other, we should anyway keep the -analyze_frames feature because it's very useful for other side data, and that means you'd also be able to get an answer to the question whether there's CC data _anywhere_ in the stream - if you want it.

Thanks
sw 






_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Marth64]

> For what it's worth, I have seen this and it can definitely happen.
Hi, I have seen it too, especially with SCTE-20 samples. Thanks
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

From: Marth64 <marth64@proxyid.net> 
Sent: Thursday, January 30, 2025 5:44 AM
To: Soft Works <softworkz@hotmail.com>
Cc: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>; Kieran Kunhya <kieran618@googlemail.com>
Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

> > For what it's worth, I have seen this and it can definitely happen.
> Hi, I have seen it too, especially with SCTE-20 samples. Thanks

Which kind of recurrence pattern are you talking about?
I don't think this is conforming to spec. I've been contacted by somebody who was too afraid to post on this ML. He said that the one source from which "sparse" CC data is coming from is a library named libcaption which produces off-spec CC data.

As mentioned before: off-spec CC data or a CC stream appearing due to changes in broadcast can still be detected via analyze_frames. That's not much different to frame size changes, which can also happen at any later point in time. But what's relevant is the frame size at the beginning and CC data in the same way. For spec-conforming streams, this method is very reliable. Most importantly: it doesn't waste precious time by processing the initial frames twice.

Even if it was just 98% reliable, those 2% could not serve as justification for why probing of the 98% takes twice or thrice the time.

sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Marth64]

> Which kind of recurrence pattern are you talking about?
I am talking about really sparse pattern. e.g. nothing for a long
time, then ads inserted or program changes, and embedded bit stream
starts.
It's not that often but it happens with SCTE-20, or sloppy DVD edits
for example.

>  Even if it was just 98% reliable, those 2% could not serve as justification for why probing of the 98% takes twice or thrice the time.
I do agree. Having the "fast but catches most" solution as an option
would be useful. I had tried to put up 3 proposals and -analyze_frames was
the only one that seemed to stick since it did not break API or make
things awkward with dispositions.

For now, -analyze_frames helps to at least make the feature workable and has
a utilitarian purpose for flexible CC probing needs.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> -----Original Message-----
> From: Marth64 <marth64@proxyid.net>
> Sent: Thursday, January 30, 2025 6:07 AM
> To: Soft Works <softworkz@hotmail.com>
> Cc: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>; Kieran Kunhya <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection
> and ffprobe fields (cover letter)
> 
> > Which kind of recurrence pattern are you talking about?
> I am talking about really sparse pattern. e.g. nothing for a long
> time, then ads inserted or program changes, and embedded bit stream
> starts.
> It's not that often but it happens with SCTE-20, or sloppy DVD edits
> for example.
> 
> >  Even if it was just 98% reliable, those 2% could not serve as
> justification for why probing of the 98% takes twice or thrice the
> time.
> I do agree. Having the "fast but catches most" solution as an option
> would be useful. I had tried to put up 3 proposals and -
> analyze_frames was
> the only one that seemed to stick since it did not break API or make
> things awkward with dispositions.
> 
> For now, -analyze_frames helps to at least make the feature workable
> and has
> a utilitarian purpose for flexible CC probing needs.

For us, this is completely unusable. As mentioned, there are often thousands of files to probe and every few milliseconds matter. Sometimes probing is also done as part of a start playback request, and again it's not acceptable to increase the delay just why some people think that the data must not be shared between one lib and another - while the same people are implementing ways to their liking if they need it themselves.

There are many ways for transferring the that information, some good, some worse, to some extent also depending on point of views - that's all fine and I really don't care which way it should be. Just what's not acceptable IMO is to say: "No, you must not use this data, let ffprobe process the data twice".

So, this has to be worked out in some way. Do you want to submit something or shall I?

Thanks,
sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Marth64]

> So, this has to be worked out in some way. Do you want to submit something or shall I?
https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=12356
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> -----Original Message-----
> From: Marth64 <marth64@proxyid.net>
> Sent: Thursday, January 30, 2025 6:25 AM
> To: Soft Works <softworkz@hotmail.com>
> Cc: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>; Kieran Kunhya <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection
> and ffprobe fields (cover letter)
> 
> > So, this has to be worked out in some way. Do you want to submit
> something or shall I?
> https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=12356

LOL 😊 
https://ffmpeg.org/pipermail/ffmpeg-devel/2021-October/286678.html
https://ffmpeg.org/pipermail/ffmpeg-devel/2021-October/286682.html

(mine's no longer on PatchWork...)

Another use case is in combination with Subtitle Filtering: If there's auto-conversion desired in ffmpeg (not ffprobe), it's also necessary to have that information (presence of CC) available after coded probe to determine whether to set up filters accordingly. There's also no room for reading all frames once another time.

sw




_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Marth64]

> LOL 😊
lol indeed :)

> Another use case is in combination with Subtitle Filtering: If there's auto-conversion desired in ffmpeg
Yeah that's reasonable. I would request a manual switch or some other
way to establish the filter unconditionally, for those who do want to
operate on the sparse files.

Thanks
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Marth64]

That said, the concept of exposing the properties was rejected. I
understand why since they are internal.
So the conversation would have to be re-opened, or maybe an
alternative solution devised.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> -----Original Message-----
> From: Marth64 <marth64@proxyid.net>
> Sent: Thursday, January 30, 2025 6:46 AM
> To: Soft Works <softworkz@hotmail.com>
> Cc: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>; Kieran Kunhya <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection
> and ffprobe fields (cover letter)
> 
> That said, the concept of exposing the properties was rejected. I
> understand why since they are internal.
> So the conversation would have to be re-opened, or maybe an
> alternative solution devised.


I found something! It won't require an API change at all.

In AVCodecParameters there's an array member coded_side_data of type AVPacketSideData which is defined like this:

typedef struct AVPacketSideData {
    uint8_t *data;
    size_t   size;
    enum AVPacketSideDataType type;
} AVPacketSideData;

and the enum AVPacketSideDataType has this member: AV_PKT_DATA_A53_CC

So, this would allow to add a packet to the codec parameters with AV_PKT_DATA_A53_CC side data - as an indication for the existence of CC data. Without any API change...

What do you think about this?


sw


_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Marth64]

> What do you think about this?
On first glance it sounds promising. I think it's worth exploring.
I've not used coded_side_data before so I'd have to look again with
fresh eyes to make a better opinion.

Thanks
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> -----Original Message-----
> From: Marth64 <marth64@proxyid.net>
> Sent: Thursday, January 30, 2025 6:41 AM
> To: Soft Works <softworkz@hotmail.com>
> Cc: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>; Kieran Kunhya <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection
> and ffprobe fields (cover letter)
> 
> > LOL 😊
> lol indeed :)
> 
> > Another use case is in combination with Subtitle Filtering: If
> there's auto-conversion desired in ffmpeg
> Yeah that's reasonable. I would request a manual switch or some other
> way to establish the filter unconditionally, for those who do want to
> operate on the sparse files.

Sure. Either that or filters can always be set up manually as well - whatever Is desired...


The second thing I wanted to talk about with you is extending the analyze_frames feature for more data. If you hadn't done this, I would have submitted something very similar 😊 


The immediate candidates for additional data I'm interested in are HDR10+ side data and DVB subtitles, specifically the target window size, which you would actually want to know beforehand, but DVB subs can appear anywhere or not at all. With analyze_frames and read_intervals it will be possible to define something like 3 or 4 few-minnute time ranges (like start-middle-end) to scan for DVB subs.
There are also other kinds of data which might be of interest, and that's where I'm wondering whether we should keep adding more arrays like streams_with_closed_captions and streams_with_film_grain or whether a more generic would be desireable?

What's even more important for this feature is some kind of auto-exit mechanism similar to this:

Via an option you specify which data you want to scan for - e.g. filmGrain and HDR10+. That would mean that the frames and side data are being processed just like normal, but as soon as all the information has been found, ffprobe would exit instead of running through all defined intervals till the end.

Does that make sense to you?

Best,
sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Marth64]

Hi,

If extended it will have to be more generic. But for only two fields
present now I did not want to over engineer this in the initial
addition of the option.
It makes sense except for the DVB subs. I thought those were discrete
graphics or text streams. Wouldn't analyzeduration catch them?
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection and ffprobe fields (cover letter)

[Soft Works]

> -----Original Message-----
> From: Marth64 <marth64@proxyid.net>
> Sent: Thursday, January 30, 2025 7:56 AM
> To: Soft Works <softworkz@hotmail.com>
> Cc: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>; Kieran Kunhya <kieran618@googlemail.com>
> Subject: Re: [FFmpeg-devel] [PATCH v2 00/11] fix broken CC detection
> and ffprobe fields (cover letter)
> 
> Hi,
> 
> If extended it will have to be more generic. But for only two fields
> present now I did not want to over engineer this in the initial
> addition of the option.
> It makes sense except for the DVB subs. I thought those were discrete
> graphics or text streams. Wouldn't analyzeduration catch them?

Yes, these are graphical subtitles. The problem here is not the presence detection of such streams You get that instantly from corresponding DVB descriptors.
But there's a target size which is sometimes important (can differ from video size) and you need to catch and actual subtitle event to see it.

sw
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".