From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.ffmpeg.org (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id 3B95040F1E for ; Tue, 20 May 2025 19:18:40 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTP id 9455268BD36; Tue, 20 May 2025 22:18:36 +0300 (EEST) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05olkn2055.outbound.protection.outlook.com [40.92.90.55]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTPS id 583C868BD36 for ; Tue, 20 May 2025 22:18:30 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=r/YqSSYlqaWK2j87OgihoPHgcHHpVYQVc+OUSx0BTRv2UHpRERxAS9mfEIahtFunP8xamvu4wjOTpwCQlugKKujz1WRS/DaW5t8frgxa0iEi32D/dNPmRp2lf+coHQvtjVbF8+xhTtFxMtjgdRLJ1GKMvEr+AYux1FxqnYWKQ7IHO5v1Bgho3nI4PvF0KOElc7DlWxxLVU4yRorwQ60XWwebT3qMx0kpf80uoaIc6UxiTN0yNvZcDvdaTn5huaKD11YFqqdjSG6xYbosO+A0D6UsvF6Tpxbdf+g/eAkxDT2FvM4wVzMsFWv6DqBj+UydXz83XG39U+32Xxgm+0n78w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OJByGJYgBmAW2oLINdUy92M61iB0W3osBS0Jp40TF0k=; b=ByPx0/dNl0lAEVvWBPiJyZROZpyUITyN+y+LBpJkptWWrsavplekJP7WnLBtWbyFxcvLthTTs8tZivRbhIXe+UsmCS50IvMDNrlDuwP/oLjL2vScsKaMtUbdcOR9r85DsPG2hoEsRNx7KleIUtkk4PSMfMPoIdrpstpmyBwB1k9ZMSt2sTAaNCnd/QGeUqIZe/dM/x8y1unanHA4zDegaXUTKt59TtWzXPFba5iYMoI8s8djPaYRTGTGA8wd9vmHgJnQIpbVNLbV026s2ClpXM4qcCNouRJZtcQ2lOOkspPhLv+5cQjuWdHFy0qMBkmSm69zAfphSzhZRaZmcjvdgQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OJByGJYgBmAW2oLINdUy92M61iB0W3osBS0Jp40TF0k=; b=WkscZ8S8wCKiIHn0CUWskr7sdm9MAXMghcDAiVb2JowtF5Vj14XRnMp13bB6B55tM+nIYXMMQWmrLmWnLel26BJlfJsosUnbkdE9BrfcBUOVmbDgZ+PCoMIeXXksDm1VxTwQjt+3MynPcCeizCasknL4ub5y1YX/Db1DzWhhRpiwih5yAs3lYLqYiBOMGkthlbDNtIFNU48QDgjyaUpgMUyO0qoTQNjcbn4IO78qZvHCc1yXNjXoLOwWvgJsiZGA6KAMWHgjTKBbD+jHVNA0aAfw50nWX+V27pTXgzDfwrtWiZvz+g8VP9QH+Z5KF8DYEggk3fdhrFB6Od6ZemRyzA== Received: from AS8P250MB0744.EURP250.PROD.OUTLOOK.COM (2603:10a6:20b:541::14) by DU0P250MB0970.EURP250.PROD.OUTLOOK.COM (2603:10a6:10:419::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8746.30; Tue, 20 May 2025 19:18:24 +0000 Received: from AS8P250MB0744.EURP250.PROD.OUTLOOK.COM ([fe80::384d:40d4:ecb7:1c9]) by AS8P250MB0744.EURP250.PROD.OUTLOOK.COM ([fe80::384d:40d4:ecb7:1c9%4]) with mapi id 15.20.8722.027; Tue, 20 May 2025 19:18:24 +0000 Message-ID: Date: Tue, 20 May 2025 21:18:22 +0200 User-Agent: Mozilla Thunderbird To: ffmpeg-devel@ffmpeg.org References: <20250520152717.15343-1-derek.buitenhuis@gmail.com> Content-Language: en-US From: Andreas Rheinhardt In-Reply-To: <20250520152717.15343-1-derek.buitenhuis@gmail.com> X-ClientProxiedBy: FR4P281CA0053.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:cc::18) To AS8P250MB0744.EURP250.PROD.OUTLOOK.COM (2603:10a6:20b:541::14) X-Microsoft-Original-Message-ID: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8P250MB0744:EE_|DU0P250MB0970:EE_ X-MS-Office365-Filtering-Correlation-Id: 84e3d268-ed70-4753-9e75-08dd97d31706 X-MS-Exchange-SLBlob-MailProps: AZnQBsB9XmoaApi1aWVt9ILKZbGoHsd6C3u6qyicx1t9/1Nj9oz04B9IV+iAYcJ/871GBpsH8zMZMzKAGF0b8KXG9fME+P78LDKzaRFfJXztbchfnsHsSb4VSDDmagq+4nZbrtYskJrs8QJLLVI9vrjtTtTjYgu5+2b+f/e4Lx7Mv4Zb41kWQGRpEWcHDGiY4M8xlvuzkRs5clyybtPgHzKkrGr2MDNIm4I6Kun1azw9HjH8yfo+j9XmIjIz+xB+oe19PlyKZG5G9VOwrq+JxVP5D2/YhrkUUk99k0g/SpXw0pvl35xlrIKj0DBmJVR369bXC9amXowPiYN+a0brjwiGGB3WVtKA76sbRPLRT7cM0dHbsKJZJyjWkkQEYPqDoy4Rm1RUrVCkwNPFRVzj/2v21dDgBmPceTmpATPvt+ZTppxLcWJvWWAdvbSYjHGu6JMEV+sZQI2nxDsra/XccR0k3HvpF/XCgYmzFBANtLLaD95U25UYm+iBcWURjfsYr5rwNAv9i08aGppSWXQdFb/I1drH6MNFMnsVgBeBax0+pKDcTQDs7I9OKi1I/MvZTxftQdiFiTqjhuMeonrmYJmDE0MVD2W8SgIJCULcTMLpyOxpslOt5Snxqt1JqOcmoloBFpli+voziIgmu9p9lVO9TQhoDRdKCZVvjMarH/gQ/xn2WWauoRk455mHQGK3MYLu4AJONfGaBW3Wro78LaSEMlP++W5qCxOZP0su3zQAhIDEJ1q2HAOJzFM4bs4RC6KSO4ThdzM= X-Microsoft-Antispam: BCL:0; ARA:14566002|461199028|6090799003|41001999006|19110799006|15080799009|7092599006|8060799009|5072599009|440099028|3412199025; X-Microsoft-Antispam-Message-Info: =?utf-8?B?L01YUGZuRHMvblpzMXluSzF4Z1hUM2RXeGlIT2F6ODF2NE1MVEYrTnpwV1F5?= =?utf-8?B?dWF1enE0NkZiWnpmZTVDaTR2VXMxdjkvVWZQUUZJZlJwVkFBQWNpTWtudUtO?= =?utf-8?B?OXZDdVdNL0Fra0FmMW9RUmhKQUgxWVJ2SUY5eXFDRkdmQUJEVU9MRlZQOUEv?= =?utf-8?B?bGxVR01WQlEzdWVzOXYyZUdpQ3l6bzNNVTZ2aDdiOTU1R1h6YU1kOHRuWW94?= =?utf-8?B?cFY0STdwRnFZRm9FeVVTc0F3cGRnSmtWVGJYUXZWSU9kbzVQcElWSk02aTl2?= =?utf-8?B?QXRCSFgyaEFBT1pINys3Uk9NUzlJU1Uwc2Y4RjVmNTRTbmU1Smo3RXpiTXUv?= =?utf-8?B?R3BYL1hxSUJ3K1RvV1J0RTdSNU01M2ZqbHorK2lBdzliaFQ5TmRQZWU4NDYx?= =?utf-8?B?UVI4NmdwMVdGWGZjaTY1Q2FhRWd4M3lZd2Nuc1pzM0dCb2pRdW0xOHcrbU12?= =?utf-8?B?amNUVFJRT2JYeWZyUUR2NTBwZFA1em1vOTljRGZ4alNhOTd1Znd1VCsvWjN0?= =?utf-8?B?WnZPWEZpWHlyU2xUNUpsOUgvbXMvWnVvR1pva2RhN2IvZE9zVitodXh6SDl1?= =?utf-8?B?QlRYSS9DMUY4SXBnc3FxTUYwMG5yM053eE5TOUxMV3h1ZHZqck16TDVKVDhN?= =?utf-8?B?ZWNkeVlXcVNwdGtqbGFZLzZ0Q1V1RU5PdmdldEJlSzlTRWp3dGw4OUk2OXYw?= =?utf-8?B?OXJwbkZvR1RPOTg1Q3B3NWk5Um4vTGFUUlFwajFBZDhDU2czUDdCaFNmbHps?= =?utf-8?B?MDM5S3ZrbWZzUVExWjZWUm8xdW5EZ3VxVSt6UXhGT3kwcHdvTUx3RnRBNUJj?= =?utf-8?B?SHVtNUIxTWpqOGN2WGRCVDRucjdURFBQbGxCaXhxRzViSTNBT3JJWnNKZkVF?= =?utf-8?B?NitCK3g3a0ZKcGJZa1BBY1VaOEhyMHNNV0t1QTl6RkJNRmtmYXdhUDJuVFR5?= =?utf-8?B?UVFhblFhQmRERUtlVEp0WmtheC9KSWtqNkMzNnc5UzViSEFHZHlTQzgyOURp?= =?utf-8?B?cHNETHpiN2Jyd2tSTTc1cFczUTNJNzVjZm0rdW1OTlRQZ3JwenVRUCtFL0dM?= =?utf-8?B?RDVWYXRWNytlKzJtbTFaNkVJam5MeUxPbFVzek5ZZ0VoMnI3RHMrMDJuZ2tw?= =?utf-8?B?RDBwS2s5OWZoUWlRSDJzRDlpdWI1NktNazg5UnJrZFlTMHdpMHFtUnRSY2th?= =?utf-8?B?a3BDTVp3YVo1VzlQejdhOEFTVUNnRHFmM3B0VHowUDZlOEZXb0tET0t3SldU?= =?utf-8?B?VS9qVURPb0drNUcwRTdacWxDWTc5dzlhcFdaQjJnNTlxOTRBajJPZ000MnZU?= =?utf-8?B?UkxDRFh1L2dMbEZRRGJ1TlR1V2NJWG5CM1g1U3dGSVFEaXVoaTZPNk1PUFg5?= =?utf-8?B?T0NlRndxVlk1UUI2L3BqRUUwa2I2S0dtbUFSL2xVRU5kMXE5dGRLY21SYmR4?= =?utf-8?B?OE1SbmFYT0FWa1FOalprc0RuWTMwY2NFV3ZvZFpTb2REak95dXJiSWdQOHJJ?= =?utf-8?B?QXBSTFcvTXhPcFVDc3I1MTRKUWVJcXo0dTZSanZBbmtUSkN4SHNQSXdiSHpt?= =?utf-8?Q?wzPfeXjkBUYWoRa9Nc8yve62M=3D?= X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bTZiaEYzNzRxTER0RDFiUnMyNDRHR3oyZnBvS0Qrak1DNXpxVW9ZZmNxRHlB?= =?utf-8?B?N0o1WDFzSnpoZHJQQUNMWnVNOU9RcVkzVjR6d0JBd0NKdFVCbzVMa1R2bjZK?= =?utf-8?B?U2k3VXNIZlQ0MnhsblRqV1N6VHdGZHVUNzZzNnNNalFvZ2hhRkhQQnVvTito?= =?utf-8?B?UXhJdmJ4MTg1eDkvaHFsanJCdVYxeUhibFB3VTVpTWRJTGxaTnpoL0lvSVJa?= =?utf-8?B?WWJGRHBCZ21xVzk5QzNYYjA4T01PL3prWFo4OU1WUUpiR2hCTGY4R3oxR2Jl?= =?utf-8?B?TlROdEIxMVhxM3lpRFdIY1ZEUTZXMEt3ejBUYWtjSGVIckdRa3k0bDRkVVR2?= =?utf-8?B?ZkovbkJaK0FuWXk5M0lRNytEcVFHdWplK3J2NXFpOEpMd3lMSXQ0cjIwY2oy?= =?utf-8?B?UzhCS0ZMbGJaTE1JcXlSL0pwWkljTERJUThMMHBNaVVwdDJLWVhqOC9nK2lM?= =?utf-8?B?RW1PeUpRRjc4Y3BtcFFVcXdxM2NwcVNFOTJqSWlvRk1JVXo5eFlSS1FMc3Nx?= =?utf-8?B?Y3NFZ3V6REZuNHlTaXoyczBhaC93RUExU01kbCtlYkk2cUVnRkVySWdsYXZj?= =?utf-8?B?UkNuQXMzZzYwcHpqaER4Vld6WHhURnRMM0tIUlQ2RVFhZ0E0blFQV3FiSkZi?= =?utf-8?B?cWhDczhKbUUzR3BickJqR2ZKa1Z5WUhLZ3pKekZ2aGZVN2xreXdFak9uOWNC?= =?utf-8?B?V0FYdy96Q0hRcEJPVTMvSTg0ME5QcGJzME4zQy8ycDVCSGpZSGxQc2l2UW9W?= =?utf-8?B?UFdIbGw3VnNiemQvTmw5MEpjZzhtZVNMVXIxMW4wMVFWRFl3ZVJFL09DMjFl?= =?utf-8?B?UWpKVU9XTWtHWjRvc1RXZWRDS1U1R08wNUZOOHkwa0w3S1hjczEyUnhzVUpB?= =?utf-8?B?TXEzaUlZRXBBQ0toL3R0M3JFK1dQOHV5c0cwTTlLWHBEczZYamdXd0tvQjls?= =?utf-8?B?bHdEVkU5bWx6bVlBQ3NUMEtkNTBPUjM1UU9WcFF5RVVXMFF1K094OHl0RHpt?= =?utf-8?B?eE5RUWg5T2dnNk9xR01rVVZqeGt1ZTFnc2xMRWdXL3NpTDdSUU9GUlJTZ3oy?= =?utf-8?B?NWtnLzRENzIvSTBCWEtMSVUyY29QZUd3cWNPV0ppemR4VDc4N2tuQXpHTkgr?= =?utf-8?B?SGFSZnJaZU5VNVduSUQ5NlpKTk1VdXdyV1EyRzJNcXdwYVBrZUpmUXpSbVhx?= =?utf-8?B?cnVJMEdzdnZ5c0pPWDBCblZ5MUptdnNtMjBGQ0x5QzVhVCtTQndxWEFwSThZ?= =?utf-8?B?WWp3dU1MQ0VSb1RxWXNZM0l4TFZoNHZSU2syc05VNStqQVF4TVl0YzNpa1dB?= =?utf-8?B?Um9Qb2x0ZElYTGhxUDd5M2lkWmVDbVliVlVIYmpSTVROeitwTXFUaGU1SFBU?= =?utf-8?B?UUtEWU8wK2szeUlvYUd4Slc1QnNuN0pMeGlXM1JvVm5KNkpjb2YvS3RIbWRs?= =?utf-8?B?dk5FUDA5dHJCYlNNcGJHV0lqd3F6YVpzZUFJUjJneFVWdFJmMVd2d1ZnSXRM?= =?utf-8?B?S0pQajRzMzBiQzdwY2ZmTm4wL0sxNzFaSDN5dDFuYWZIK210OHF4cTluV2tt?= =?utf-8?B?UjQ3Wk52VGxJMVBEQ2JVenlhV2F5bGVEbmNpdVMyNDV1MUhiNStqL3RjRFpq?= =?utf-8?B?QzE1c2h2d29xRHRzb3Y3TGlDdTNaVi9kc0xaZlJuQTA1SzF3TGNnbTBXL1Fa?= =?utf-8?B?WllCNVRVNmRxczg5QTkyNkt4VVdLMmVqdnN2VEJJeHB4SzU0akpBd1dZZUsv?= =?utf-8?B?V0lkODdwYmJsNGtqUnBJTHBvanp6bGRleXB4K1hzQzZ3eGtMRUY3Mms3L2Zj?= =?utf-8?B?d3liSm5XUVZodjhiNk14UT09?= X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 84e3d268-ed70-4753-9e75-08dd97d31706 X-MS-Exchange-CrossTenant-AuthSource: AS8P250MB0744.EURP250.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 May 2025 19:18:23.9594 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0P250MB0970 Subject: Re: [FFmpeg-devel] [PATCH v2] avformat/dhav: fix backward scanning for get_duration and optimize seeking X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: Derek Buitenhuis: > From: Justin Ruggles > > The backwards scanning done for incomplete final packets should not > assume a specific alignment at the end of the file. Truncated files > result in hundreds of thousands of seeks if the final packet does not > fall on a specific byte boundary, which can be extremely slow. > For example, with HTTP, each backwards seek results in a separate > HTTP request. > > This changes the scanning to check for the end tag 1 byte at a time > and buffers the last 1 MiB using ffio_ensure_seekback to avoid additional > seek operations. > > Signed-off-by: Justin Ruggles > Signed-off-by: Derek Buitenhuis > --- > Addresses Andreas' comments from the previous round. > --- > libavformat/dhav.c | 48 +++++++++++++++++++++++++++++++++++----------- > 1 file changed, 37 insertions(+), 11 deletions(-) > > diff --git a/libavformat/dhav.c b/libavformat/dhav.c > index b2ead99609..27e2f4db23 100644 > --- a/libavformat/dhav.c > +++ b/libavformat/dhav.c > @@ -22,6 +22,7 @@ > > #include > > +#include "libavutil/intreadwrite.h" > #include "libavutil/mem.h" > #include "libavutil/parseutils.h" > #include "avio_internal.h" > @@ -232,34 +233,59 @@ static void get_timeinfo(unsigned date, struct tm *timeinfo) > timeinfo->tm_sec = sec; > } > > +#define MAX_DURATION_BUFFER_SIZE (1024*1024) > + > static int64_t get_duration(AVFormatContext *s) > { > DHAVContext *dhav = s->priv_data; > int64_t start_pos = avio_tell(s->pb); > + int64_t end_pos = -1; > int64_t start = 0, end = 0; > struct tm timeinfo; > - int max_interations = 100000; > + uint8_t *end_buffer; > + int64_t end_buffer_size; > + int64_t end_buffer_pos; > + int64_t offset; > + int ret; > > if (!s->pb->seekable) > return 0; > > - avio_seek(s->pb, avio_size(s->pb) - 8, SEEK_SET); > - while (avio_tell(s->pb) > 12 && max_interations--) { > - if (avio_rl32(s->pb) == MKTAG('d','h','a','v')) { > - int64_t seek_back = avio_rl32(s->pb); > + end_buffer_size = FFMIN(MAX_DURATION_BUFFER_SIZE, avio_size(s->pb)); > + end_buffer = av_malloc(end_buffer_size); > + if (!end_buffer) > + return 0; > + end_buffer_pos = avio_size(s->pb) - end_buffer_size; > + avio_seek(s->pb, end_buffer_pos, SEEK_SET); > + ret = ffio_ensure_seekback(s->pb, end_buffer_size); > + if (ret < 0) > + return 0; > + avio_read(s->pb, end_buffer, end_buffer_size); > > - avio_seek(s->pb, -seek_back, SEEK_CUR); > - read_chunk(s); > - get_timeinfo(dhav->date, &timeinfo); > - end = av_timegm(&timeinfo) * 1000LL; > + offset = end_buffer_size - 8; > + while (offset > 0) { > + if (AV_RL32(end_buffer + offset) == MKTAG('d','h','a','v')) { > + int64_t seek_back = AV_RL32(end_buffer + offset + 4); > + end_pos = end_buffer_pos + offset - seek_back + 8; > break; > } else { > - avio_seek(s->pb, -12, SEEK_CUR); > + offset -= 9; > } > } > > - avio_seek(s->pb, start_pos, SEEK_SET); > + av_freep(&end_buffer); > + > + if (end_pos < 0) { > + avio_seek(s->pb, start_pos, SEEK_SET); > + return 0; > + } > + > + avio_seek(s->pb, end_pos, SEEK_SET); > + read_chunk(s); > + get_timeinfo(dhav->date, &timeinfo); > + end = av_timegm(&timeinfo) * 1000LL; The only thing that you want from parsing the end is the end timestamp and this is given by data alone. date is a 32bit le number at offset 16 from the start of the dhav tag; it can be read directly from the buffer, so you do not need to seek again and use read_chunk to get the end. It also means that your ffio_ensure_seekback() is unnecessary (unless you wanted to ensure that the seek back to start_pos works (which is currently not guaranteed at all and not ensured by your code and would probably also be bad in general given that this would cache the whole file in memory). (The above is based on the presumption that we are not really interested in what parse_ext() may parse in the chunk at the end; I don't know whether this is true or not.) - Andreas _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".