From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id A1580404EF for ; Sun, 24 Jul 2022 18:38:50 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 35C4D68B73F; Sun, 24 Jul 2022 21:38:47 +0300 (EEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11olkn2010.outbound.protection.outlook.com [40.92.20.10]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D25B568B451 for ; Sun, 24 Jul 2022 21:38:40 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SqYOkAcuK/FNW5GP5x/ra5ty/m7SjtQ5IZNFb+N4fOOcqmZJ+Y7MU0+nqhut8OszQpGCGXMB7WBFu465nSJa7w6EIKrsw86PY576MEf9kl/0ZFoRMxYIwBH9NYCQznn9fLEhuKwU+zcbvJA2gonP1OQh5dVWhzgV9Lm7gqaWjsExgIWHBjiqQNfOyXDfh7hFb82W/yNA4HpWmV1bVU0CR7vbgmBuLTyG20vbHUNQoU6blzC8qr6wRKgUY4KulzPPwTyEctK3vWvBOIC+rw19zWJqVQTSBI/tXMC7JVlzZ4W38tj0v1FmcM7lsUOU5heij6KX0L8+DoZc1sWmKldJtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8d7gO3kwGwaBWAQCQDJQmgf+d3GAPvmaFTv+RqjRnck=; b=bL4y8XVgxPPmdH+OfYG1oqHMiwC368+yKMaq/OY2ps5ZY1KrrUUcgmC8RzMB1s2CsOFuoEJkGcTpLeMSYhdk38s9YwxWiPFRKSxZejoVliDejXELAphMPd7FcbmHY1o4swFmcDp6tE2Bf5MWdEuYBU8iOIhj+fP4KR3r6f4NLc1UeFRmMZVBfX4dwOYvaU77BHWu7AX4EtxnL11AwxQhkH1H1qxps0cHqpivlYgZjMfT8zKE0IRmPjoLDnYFwLaNCIsy2w59lKJZVUIaevAMb44dqXdDFhmOJ0lcX5OkkoCs1El51NlrHHYkIsmfN17lMRslKi52GfzlmdJzJMHKmg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8d7gO3kwGwaBWAQCQDJQmgf+d3GAPvmaFTv+RqjRnck=; b=g0dtvStKtwSHjORKMJzzA07lU9QMiiSGabvG/IDyo2BGU0wkbBgJGL9+jGtCHddaSKn4+7luccLfyvDdRYkl3E7hlEhIUOt7Oaq0IEj0Kh42UaIgyaNC8V9sanl7U8iKdbs9hOngnfFuauwI/6hBO2d9cEbLTq7ADNsiFQTSDwmx7u8SXwE8sCSJn5ZBwI7UvE6aEMbQWR3s7qTFcTIineLr9Pqq8xPtiFvRouffbHaXa++EO51diWPiZ58FKvqWnwWo8KFJLnplBCYRTiC2cfPC0d3W0L7bJ4jA1DFinOK0S79lG32ehlUtujt0yKtwLFeZMUCbTjBVaIfO473vGQ== Received: from DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM (2603:10b6:8:b::20) by SJ0P223MB0590.NAMP223.PROD.OUTLOOK.COM (2603:10b6:a03:47e::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5458.19; Sun, 24 Jul 2022 18:38:37 +0000 Received: from DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM ([fe80::d9e4:ced6:ab31:c231]) by DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM ([fe80::d9e4:ced6:ab31:c231%2]) with mapi id 15.20.5458.024; Sun, 24 Jul 2022 18:38:37 +0000 From: Soft Works To: FFmpeg development discussions and patches Thread-Topic: [FFmpeg-devel] [PATCH v5 00/25] Subtitle Filtering 2022 Thread-Index: AQHYiHoQ/s8zpruhyEis6n+SHo1FCK1rU4+AgAAK6wCAAY9AAIAg4EiAgAAKy2A= Date: Sun, 24 Jul 2022 18:38:37 +0000 Message-ID: References: <20220703170714.GF396728@pb2> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-tmn: [pYlFLHF9NNYFig9kFUqNR/kajkpKNSK6] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7ec99caf-6e1e-4676-0ed8-08da6da3b901 x-ms-traffictypediagnostic: SJ0P223MB0590:EE_ x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: zc8WuLyqDorLbDKCH439b6Xc88SrVZPMgmaEx020RQWdKUj/xtNfITWnDQp6ykB5TOtxUkT7YTpdhrJz7U/rLvikQ4kV0SKZwn36JYRhgM7X+AmaXdotltfY051gbrSHKsQPrE/I8o52WSSx4R5yJRAtbPVVbnR28TCkS7p+eexoRYUGda/XGuuJiBaDM/D6ZYSOGbEuePDGWJ1NDHG9H/X2RO2p2rzWEOfELsxRlFdamCbxFriCnF+8uSCjPr+a2E5Bz+/CefJENzwuylKYaP7qO4Epk9z4hX9UqHBv1TO/RUQvqwZLdoCqyHUxq48eFMTHlcsdZ2k/ttqLbeesURlEZ4BRIB6NNPY7me0IyH4JNt6LGJutn9nmId0IHIUbUZyltLxFLEnEo4AXo0WMyZRpUFbB2wadv7P/dD+TKZVoOL2OdNSg3b882et6Ikw7B9Mdfr0xbeb9eXxgGN6H/HCweT/HEX25N6cSvrHiKE8FehgeDXa1JjiNaY9eiF910UM5doSuI0FOpasO/fFQxPtbnOzWQqWflxIMwu8WJ6W2IjtMw3rec2sQEWsrAdo8ah0GzV1VPC6jIOKfyS07VHxiB+I0VIZqKfe9dfiFv25SHtCIu5rN9szW3EzRdMRdyiuknsmYp0hkyjS03SeWhuij22VahETUrT24RkFu/7Sta/8zPqcVGRwOf376M8eW x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?R0tTdjFtVnZGcGR1MTZocnhUc2JUb0pmT3grOUNkZENKcWQ3TmFVUE9tWXh6?= =?utf-8?B?TnNQbHozeUhvRTQ4c0hwWURhRjYrWDJpcUZ1MWZCM2QyVktkeXFpcGF5L09q?= =?utf-8?B?OVg0T1FpRllBb0xuSmx5aVM3dzhBS1A1VExDN0EwSldMWXFvczFPRkxndkpj?= =?utf-8?B?bUR3Y2M1cDdaNXVCc20zaVNZdk83QnowdnhqOG5veXFJWTcxa2lSRndWVkhq?= =?utf-8?B?cmlvR0pmN2FqZlFaTDFLWTZGc1kwTURpY1NqWWFIWFQ3aWlMdVU4VldCYlRL?= =?utf-8?B?aXFXQXorMWRDN0E1UTRtTXNnSnRDSVhoREg0d0RLdWF6TjBRbGllM0FEdWlK?= =?utf-8?B?TVkvUURmbk5uTjhuQnRKQStHSkZiZEVib24raGNEdFVzN2tWU3phRmZ2L1lZ?= =?utf-8?B?QkpYb0lVZGpJVkNlR05SN1lnMjBiYUtsNlF6Mk5yNUNkWUVERU82QVRRZEZ5?= =?utf-8?B?RndHSC9TWlpNczNZUk8xbHRDZ3NkZWlpdTE5U1VHVXJYQ2JaZm5PVnFYRDd5?= =?utf-8?B?WjdlZTZaZ2F5N2MyekJ1aHdzNjg3a2RFQlgrenpNQ0N6QitCenZKUDNxWnJS?= =?utf-8?B?YUtBbHF5Z1RTb0prWWJxNU5vOG9hZlNBRWtOR2FVRDRSTnJqMFlUN1BmZFJn?= =?utf-8?B?R3JPRnFHZmJ1QndwU0xza01ubmx4MysvYXczVEVZVDVCUXplNWxDWEZoY1B2?= =?utf-8?B?RDdJc0VtTEJDVDl0aFE4cC9uREF5UmdScC9rR3dMT3B5cG9lQWxCSzFZQ29U?= =?utf-8?B?Y21QQkxiRWRGUElMRThjenR1TCttOE9FNk0wNG1xeWgycncwbXBibGZUMEhn?= =?utf-8?B?L3JRWXRtVkx4QkZQaU9FNW83dUY0OHRsNkF3SlQ0Y2hhN2JJa0w0UzdYbkxJ?= =?utf-8?B?b0lNSGIwbzVkbnFRRjBlT3BVYWV2TzF3V3IzbXluTTluZHBnWmVEN0dBYW1z?= =?utf-8?B?V245U3NGSmVVYTB3aDRBcUd1V1VDSy9JUCt0MzhIcE1tN0xPaXg5RVhPR0RG?= =?utf-8?B?b0kvUmx3amhYSmtRcHpCS2xoRFgyelQ1OWNOWXdMT2hyTk13aDhhRlVsWkpC?= =?utf-8?B?NkYydjQrMWlISjdCbmJuYm9kc2tqenNKdDdvMHdhLzZpRUpUWHNuNEVGcUdU?= =?utf-8?B?UUdSZCthMGdSRW9FRkx1TVhXdzJ1SS9xL2ZWcUQ3SUdaRW9Da25EelNBMGJ0?= =?utf-8?B?Zm91S0xVb3hubUJDNUFGOUVzSUVjSzVjQXBMVytVNHZ3bmFrVjNvU0JwRkVn?= =?utf-8?B?TDZ0clNPbjZkcVJsT0VYUjcxakZ6RExKZHhzeWZOeTAzY1BZRWtCbGRlRGk4?= =?utf-8?B?cmZqOXF3SDRDcElYK05sVEpVLzZtWWp4NEthQkJHNWRCUnR5aW9nb0QwWFVH?= =?utf-8?B?bzFGS2ZVbUJtWHdHKy9icGR3L2FpdUlwYS9OODNWcDR3ZFp6NWdGelF1aDZB?= =?utf-8?B?UDBSMnIyRkVJN0kwVlZoRXE3RzdsME95b01DbGtYOXhQZ2dXU1NreEhTbGNB?= =?utf-8?B?VUJsblZXaG1mVXFiVnFyNlFZV0Y1dk5SQWtjdksyMUN1QWxHZ2ZpVXZhTVMx?= =?utf-8?B?bVhKQWVTWDB6Y2R6UnQ4UFhmU1NZbXRUREU4by80d2NZNTJoNk0wcjh1eHdO?= =?utf-8?B?cVY0V2lMeSt4MWRwUjBleE96b0R3SkZHN3ZmUTY1YmIzS0ZOM0puMTc1eWlY?= =?utf-8?Q?ETNSTMMPBTpiACoCosgw?= MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-1ff67.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 7ec99caf-6e1e-4676-0ed8-08da6da3b901 X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Jul 2022 18:38:37.6828 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0P223MB0590 Subject: Re: [FFmpeg-devel] [PATCH v5 00/25] Subtitle Filtering 2022 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: > -----Original Message----- > From: ffmpeg-devel On Behalf Of > Nicolas George > Sent: Sunday, July 24, 2022 5:10 PM > To: FFmpeg development discussions and patches devel@ffmpeg.org> > Subject: Re: [FFmpeg-devel] [PATCH v5 00/25] Subtitle Filtering 2022 > > I hesitated a long time before replying, but all considering, at this > point expressing what is weighing on my heart cannot make things > worse. > > > Michael Niedermayer (12022-07-03): > > What is the timeline for the audio+video merge ? > > I cannot give a timeline because I do not work on FFmpeg on a > schedule, > I work on FFmpeg in my free time, for fun. And lately, working on > FFmpeg > has been really un-fun. Recently, my contributions have been met with > indifference at best (see when I proposed a XML parser six months > ago: > nobody cared), outright hostility at worst. Under these > circumstances, > whenever I am considering working on something FFmpeg-related, I > usually > find something else more fun to do. > > I do not recognize the project I started contributing to more than > fifteen years ago. I do not even recognize the project that boasted > the > clever optimization framework that made FFVP9 possible, it has become > increasingly hostile to trying new and more efficient ways of doing > things in favor of a corporate never-take-risks style of coding. I am > more and more often considering giving up and cutting my losses. > > > IIUC this would resolve this deadlock (with extra work adapting the > patchset > > so it would be work for SW adapting it and it would be work for you > finishing > > the merge) > > Also can others help nicolas moving his work forward > > > > What i suggest is to pick a time and then try to finish the merge > before. > > If it succeeds this patchset needs updating and can move forward > without > > this main objection > > OTOH if the time is not hit, we agree that the objection can no > longer be > > used > > My answer as maintainer of the framework of libavfilter is: no. > > Of course, maintainers are not dictators. The members of the project > can > collectively decide otherwise. But I have to warn you about the > consequences. > > First, the issue about negotiation is not he only severe flaw in this > patch series. Negotiation hasn't been implemented for audio+video yet. Neither does that patchset do it for audio+video+subtitles. It is out of scope of this patchset. It can be done later or never, not everybody is a fan of doing so, as comments have shown. Clearly, this is in no way a showstopping reason as you had conceded yourself recently. > I can immediately quote another one: for text > subtitles, > the approach of this proposal to synchronization is to feed > everything > to libass as it comes and see what comes out. It will work on easy > cases, when the subtitles are interleaved with the video or come from > a separate file. - or come from decoded closed captions - or come from graphic subtitles converted with the graphicsub2text filter - or come from the subfeed filter after fixing durations - or come from the subfeed filter ensuring a regular repetition (heartbeat) Subtitle events don't need to come in linear order. Multiple events can have identical start times, subtitle events can overlap. The overlaytextsubs filter is meant to be a direct replacement for the existing subtitles filter, which performs additional opening, parsing and decoding of the source file in parallel, and avoiding that was one of the primary objectives I had for starting development. That's why it was very important for me to preserve the exact same behavior as the overlaytextsubs filter exposes. Other approaches for implementatino are surely possible as well. Traian, who did the text2graphicsub filter had initially an implementation that handled the timing manually instead of letting libass do it, but it turned out that this can quickly become a really complex task, especially when overlapping events or animations are part of the game, so it came down to feeding everything to libass in the end, like the overlaytextsubs filter and the subtitles filters do. The nice thing about having subtitle filtering is that there is no fixed functionality involved where you can argue about right or wrong: anyone is free to contribute another filter which is pursuing a different approach. I would welcome that and there may be cases where an alternative method could be advantageous, but it surely won't be superior in general. > But as soon as a filter will, for example, adjust the > timing to make the subtitles more early, it will just not work. Of > course, it was not tested because this patch series does not even > offer the feature to adjust the time of subtitles, which is frankly > ridiculous, it is one of the most obvious thing people might want to > do. The only reason why there is no timing adjustment filter is that I didn't need one. It is really easy to implement such filter. The abilities of such kind of filter are limited by the individual circumstances, though: You can always delay subtitle presentation, but the amount of time that you can move them ahead is limited by the situation. It depends on the amount of time they are muxed ahead in the source. This can range from zero (for example when originating from closed captions) to a few seconds (e.g. with ocr-ed DVB subs). In the latter case it's easy to subtract one or two seconds to show subtitles earlier - in the former case, there is no room for this - unless you would let the video frames queue up at the overlaytextsubs filter to synchronize with the subtitle frames. You are absolutely right here: the overlaytextsubs filter doesn't do that, it doesn't use framesync. Besides the reason to produce equal results to the existing subtitles filter (which framesync would interfere with), there's another consideration which kept me from using framesync: The benefit would be small and very limited. Let's look at an example: assuming we have a 4k 30fps video onto which we want to overlay text subtitles. The text subtitles are muxed "just-in-time" in the source and we want the subtitles to be shown 3 seconds earlier. In order to make this possible, it would be required to queue up 3s * 30 fps = 90 frames at the overlay filter. And this is not the muxing queue, it's inside a filtergraph where we have uncompressed frames. Assuming 4 bytes per pixel for simplicity * 3840 * 2160 = 32 MB per frame. For 90 frames, this makes 2.8 GB memory for 3 seconds delay. It is not unusual that subtitles are off by even larger time spans (e.g. video has cut off intro but sub timings assume intro to be present). Even with sufficient RAM - as soon as hardware acceleration is being used, you really wouldn't want to use previous GPU memory for subtitle timing adjustments. Eventually this brought me to the conclusion that this isn't a suitable approach for adjusting subtitle offsets except for small corrections. It's easy to add such filter to change timings and it's very well possible to implement an overlay filter with a different behavior - I would be sincerely interested to see which alternative solution you would come up for this. > this patch series does not even > offer the feature to adjust the time of subtitles, which is frankly > ridiculous, it is one of the most obvious thing people might want to > do. How did you come to assume that it wouldn't be possible to adjust the timing of subtitles at all? It's just that a much better and universal way than using a filter is the typical approach used for audio video sync correction, which can even be combined with the overlay filter: ffmpeg -i subtitlevideo.mkv -itsoffset -3 -i subtitlevideo.mkv -filter_complex "[1:s]subfeed[sub1];[0:v][sub1]overlaytextsubs[fout]" -map [fout] -map 0:a -sn -y out.mkv > Note that I did not have to perform a full review of the patch series > to > find this flaw. I have been preparing to implement subtitles > filtering > for years now, I know which aspects are tricky and hard to implement > properly. I only had to check precisely how it was done. And it turns > out it was not done at all. Then it's very weird that it's all working: dozens of examples for new functionality and all existing functionality is preserved. > I suspect that if I were to do a full review, I would find a few > other > flaws. But the author has made painfully clear that they did not > respect > my expertise in this area, You kept talking about your expertise and the lack of mine, without ever talking about any technical matters. This time, you mentioned technical issues and you see me answering in a very detailed way. We have seen how it cannot work and now you see how it could work. If you stick to technical matters without dropping the typical discrediting subtext, it could all go well. Thanks, softworkz _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".