From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 42837497CC for ; Mon, 19 Feb 2024 16:49:12 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A459E68D3DA; Mon, 19 Feb 2024 18:49:10 +0200 (EET) Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-he1eur04olkn2048.outbound.protection.outlook.com [40.92.73.48]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 18F9868D390 for ; Mon, 19 Feb 2024 18:49:03 +0200 (EET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=c+Ggcmp0O/43xrUyjahGjVd2RneegS1f/CAI/2IOUEH3c2rRIN9b8mshudQ3wMtDf9OQZKNr3T+3RGa5Gmzat4uA/uxVviNrtv0xnL07ao+DUgkD4GHBE6/aJXKhd/qv0r/BWZAJBhyHRjTCIoq4QelUdqmGPcz8xlRcdOyySEp3jel2CzfNoGpSpZc4ytK8/Ja+LyazYdfV+zj9UulshTh82M0dtAxu2nHJuSNrVqjgh2xQUf10tSZGZsHOMVuS0OYhYCJpEiMlZZ58WoZQiSupB44x30mkf88ZKR9HDLgwZvqzkfJkQrBt6zBUGxN3IiI8mQftVjEImp2k6pIveQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GM/E4Wd0psMognN4IHynCaG1fgKjtcQcDnI1a+KO6GA=; b=IfT8nOtQD87V2T0LxVU0fTKKO88YGrnS2E4fEiQmWD6NqTjBAEGcejy7kzRu7ISVx2OyGUUBUItruzvjLiD9vlo/gzhvbsdqm0B3BDQtSGCgeL70Gq4mREuVTYukum7MlIxAbT5HUBeto5h0I0C+yJ7U4PK3E4XvYfeqnPqNeBy0xhq63Xp8K8CTe3J5a4ec7W/dX+X+t12HyWU/8cVXiqk6dEBOd4hnvzqIcSogIHwlxRiE8OhCtal0UIII9sVoZYl0OiJTsp7rz4mHAHn+y8WPASdR4oCPDJL+ieBJ30wEKjSc4C+4SqULYE6cOPBMs8owV7AbkaHX8tmfFVwivA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GM/E4Wd0psMognN4IHynCaG1fgKjtcQcDnI1a+KO6GA=; b=V5GwvQ/5obDuz5FuA9+ZmVJIm3DltkYXwnKzqZYC4bAksOG82cSlNBEYgnouFaXm6bxqlQoq5b7vTDbJgQ6tgsRIMOJqCO1skmjkW/H9yQ09mQ36aA9XDjbfOAUPfd4S1Wpu0884wVhjjLkc3FJfVHzWEud36vymzvkm3EmxBTW5pqhyHj6Fqq/qM60peBpG0LAn9oXqhQHlvh5sfRNR9uDv0/ubOlfzv9dey1yWEeGujHT5r+9oCOLmfcJnRDIUAFoZTqupiaVQ4CmX7VXtVzqPPgm7jRJMqHQedgDCUJc3WkpGAw4Ba5FJhR9yLEeiCpQzeHOivky8uFFpLAU/2A== Received: from AS8P250MB0744.EURP250.PROD.OUTLOOK.COM (2603:10a6:20b:541::14) by PR3P250MB0323.EURP250.PROD.OUTLOOK.COM (2603:10a6:102:17e::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.38; Mon, 19 Feb 2024 16:49:01 +0000 Received: from AS8P250MB0744.EURP250.PROD.OUTLOOK.COM ([fe80::65aa:deb0:a18e:d48d]) by AS8P250MB0744.EURP250.PROD.OUTLOOK.COM ([fe80::65aa:deb0:a18e:d48d%5]) with mapi id 15.20.7292.033; Mon, 19 Feb 2024 16:49:01 +0000 Message-ID: Date: Mon, 19 Feb 2024 17:50:57 +0100 User-Agent: Mozilla Thunderbird Content-Language: en-US To: ffmpeg-devel@ffmpeg.org References: <20240205194142.37049-1-thilo.borgmann@mail.de> <20240205194142.37049-3-thilo.borgmann@mail.de> From: Andreas Rheinhardt In-Reply-To: <20240205194142.37049-3-thilo.borgmann@mail.de> X-TMN: [yU48faen5rCDGe0CnOqSvon1umgee1S1bwkQ5hpcRc0=] X-ClientProxiedBy: FR0P281CA0186.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:ab::16) To AS8P250MB0744.EURP250.PROD.OUTLOOK.COM (2603:10a6:20b:541::14) X-Microsoft-Original-Message-ID: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8P250MB0744:EE_|PR3P250MB0323:EE_ X-MS-Office365-Filtering-Correlation-Id: 9e3474b0-fe0a-4e2a-4a53-08dc316aac58 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fZ3wRSrTMnjhGJJU7aEj5Rt5T3VoJGUGsVjfMJh3Ogso6qVFDzaNPsy1IDbbHliMrkGf4g1D4ZYar9w3Lgr36G99opHkpWkyYxaWZDjeOG4qAQVEG6hvju4qAgz9TAnEdp2L7aNlqx63tKQ1YrYnALM01nNRnAOw7Xnh6XXd2Wzw+n6TuJ2xCXx/Ra6f/6X7sDrqtG5SVmmeVsAqlQbiPzPKROz2NtD62tUds1i1rcN/XvqOcdpGUt8GV6GK2XBrqqrVsTvIEihrdOSxh/nm9cvD/n7gZMCrG3jOWggYC3Sds1Q6Z/1+56ZF91H+vRGhgl2HA22MwF0lQuGXUOvy3LGXttQXWILLC3jKLX2xRt6i0TUGd/kqmKjTYssf+I89cphc+JvQfE6rhUq+BEcsAEJbrSbZqCcja/s84GxYhXHxzQLL6Rbufr8zy/e33I28ifjmZ3zahOGcvhNqdPfZ7bFK+Kam2g2P3WhtZyC1hfjF0/Ces3OqB6+P+u4kuD5dlg/MsmMHp5Qtt7EM7Ura2QnOZ8EF7ygS43NNkgWcsZ91iSjkjWOCVD9lYU12R7ZB7fGjh8wkDfL/D2qZtVRSjNbVubwxT6oKII6ThXfReePWofWEOmEXBN607ylN3LlE5NnEYznEAoqnGbbg27sE0Q== X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eFpQcHhuN3JOQ1dKZERWaFBqeWlPMnU3ZU54cUJlaDViL1Q4aUZoV0J5R1Zp?= =?utf-8?B?YzZQYUdMZFJKbVJnbFVnYmZ3eERHcHYxRE5GQmROMHJkMWFyd1psNEFZN3d4?= =?utf-8?B?OEk3S3hrcHN5NlpMNUNRTi96WHVRVkhNeU9Xcm9CcGdZYk1PSlUwZGp2S2k3?= =?utf-8?B?VGlET2w0QldhVHVCR3JDR2VET29zcmRiMVhuQ0w5OHlLSTBkMlJCK2Ezcmtz?= =?utf-8?B?Rmo4bjNYUG8xR1ZtTVFPZ2VBNkdQVlluVURMR3J2MWZIUmJucnNUNXNpNkJv?= =?utf-8?B?VFZjSGxaK1UvdGtmVGQwQVlZMmsvay9tUjg0Z2w1cUpxMzlPcDd2dzZ6dFlx?= =?utf-8?B?V3ZxVVlsbjVaTkdSS25PQmt6QlNXQTE1YWtBVzcvK25vd2luNk5qQXRPRnBK?= =?utf-8?B?WjVuSjRIZnlNYU41U3ZHaVM2K2pvMThLakl4emUwOGtkMDJjaXV5Q1g0TXQz?= =?utf-8?B?a3JtNENZMktiVUZiNTRibmVCSnR6RUtpSTNHTmhrQU9taWE1cis1cC9GZysz?= =?utf-8?B?N082aElDbHVwTUhNcnJIdEd4Z3FsbktPd2JMa0duY3IwWGlPVk9WVE5TVk5L?= =?utf-8?B?K2NLN3dENzNPYnZhWXRTZ3dPMUNKWmxCd3I4WVZjMWQ2OE14RXNRUlkyOWpL?= =?utf-8?B?cUtmOW9KbFRqdlNOcmI4akZjZ1Qxajl6QkhxZDFoMFZPNVcrS2ZkMlFoaHpB?= =?utf-8?B?cWV6Zk9lWDNBT2l4aUhIY1k3N0NTK0dNNm1XK3pQNmxaam9MMCtRY09ZMG5T?= =?utf-8?B?bjdTVW8rNmF4U2hIajFHVGxyQkJxYSsxemtQc2ZlZWd5TE1DbC91b21hVnd0?= =?utf-8?B?V3lYY2J5dmMrTHZ5Zm9mcVhiWlUraXhOamJDYTlnSTNpbUlUQ3VRc0h2TE8r?= =?utf-8?B?YTJmK2FFQmZBVEUvTklIcVlGOE5nZGVTY0diTEVIaGNJVkg4cHpGamxXdkRB?= =?utf-8?B?Q2JLb21LOHVpTUJXUVFVY0RobE5sQ1NUaUJSTkZtRlZsVVAwUXMzRDlxYVM0?= =?utf-8?B?ajNvaEFUN0grd2lGcXdYeGlYMlRsRkZKMGs3N0lPb2tKVXpWL2dvUmNCTnc2?= =?utf-8?B?TXlONTh2RlluVU1CZlFNTkY3TStmajNDZjZOblhTUlYxWkptNUFkbkh1YTVi?= =?utf-8?B?djNqNHBVZnVIUVNFbFpJbDdjUkpTZGUxKzhYcW0xYXBXekxCZ3FCVkZjb2hX?= =?utf-8?B?U0pubmd3VXZiUmZsZi9ybnB5SVdDNE9ON3NkRXlhakN4YTF3eCt5RGdOS1lT?= =?utf-8?B?b1NHdHhINjNoWG5zWXBBWGV5d0tVOUdxRlFTaTEyeVdsOW11TlhDa29FTFgw?= =?utf-8?B?d3AwdHlub3NlTWRxVURtd2ZoejhXOCtIejJreGFlUnNGUkd4eW5teElZU0NE?= =?utf-8?B?ZTZqNFF5ekx2ZEVlQkJINlFveHQxdmJwTUZBWXJMV1pSeTlWWTU3blpFSlps?= =?utf-8?B?RkFNNGtvREg1M2lzVzlqR2tpSUladXF0WkhQWEppMlJ1NEZCU3R2TXg1aWdF?= =?utf-8?B?ZjhZamV5aCt2YW1TQjg0UytUNFBUdnl3bk5KYTJRczIwRUlRcWRFaDQwdUZQ?= =?utf-8?B?anNKTHdqZWxsUldmZ3dMMmRNdE9FOUNVVi9BdnU1R241UWNNc2FXLzV0ZmNP?= =?utf-8?B?THFqRDFXSTFFNWxLbnFUMXZKWlBDV20zSXpBbVNFV2h4eDB3MDZzSlBtd2d4?= =?utf-8?B?d0swaHpKdVdJbzZzak04VVdtOWxIZi8vNDhJZ3I4MytRQnJCcTd1dTZBPT0=?= X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9e3474b0-fe0a-4e2a-4a53-08dc316aac58 X-MS-Exchange-CrossTenant-AuthSource: AS8P250MB0744.EURP250.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2024 16:49:00.9928 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR3P250MB0323 Subject: Re: [FFmpeg-devel] [PATCH v10 2/5] libavcodec/webp: add support for animated WebP X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: Thilo Borgmann via ffmpeg-devel: > From: Josef Zlomek > > Fixes: 4907 > > Adds support for decoding of animated WebP. > > The WebP decoder adds the animation related features according to the specs: > https://developers.google.com/speed/webp/docs/riff_container#animation > The frames of the animation may be smaller than the image canvas. > Therefore, the frame is decoded to a temporary frame, > then it is blended into the canvas, the canvas is copied to the output frame, > and finally the frame is disposed from the canvas. > > The output to AV_PIX_FMT_YUVA420P/AV_PIX_FMT_YUV420P is still supported. > The background color is specified only as BGRA in the WebP file > so it is converted to YUVA if YUV formats are output. > > Signed-off-by: Josef Zlomek > --- > Changelog | 1 + > libavcodec/codec_desc.c | 3 +- > libavcodec/version.h | 2 +- > libavcodec/webp.c | 704 +++++++++++++++++++++++++++++++++++++--- > 4 files changed, 654 insertions(+), 56 deletions(-) > > +static int webp_decode_frame(AVCodecContext *avctx, AVFrame *p, > + int *got_frame, AVPacket *avpkt) > +{ > + WebPContext *s = avctx->priv_data; > + AVFrame *canvas = s->canvas_frame.f; > + int ret; > + int key_frame = avpkt->flags & AV_PKT_FLAG_KEY; > + > + *got_frame = 0; > + > + if (key_frame) { > + // The canvas is passed from one thread to another in a sequence > + // starting with a key frame followed by non-key frames. > + // The key frame reports progress 1, > + // the N-th non-key frame awaits progress N = s->await_progress > + // and reports progress N + 1. > + s->await_progress = 0; > + } > + > + // reset the frame params > + s->anmf_flags = 0; > + s->width = 0; > + s->height = 0; > + s->pos_x = 0; > + s->pos_y = 0; > + s->has_alpha = 0; > + > + ret = webp_decode_frame_common(avctx, avpkt->data, avpkt->size, got_frame, key_frame); > + if (ret < 0) > + goto end; > + > + if (s->vp8x_flags & VP8X_FLAG_ANIMATION) { > + // VP8 decoder might have changed the width and height of the frame > + AVFrame *frame = s->frame; > + ret = av_frame_copy_props(canvas, frame); > + if (ret < 0) > + return ret; > + > + ret = ff_set_dimensions(s->avctx, canvas->width, canvas->height); > + if (ret < 0) > + return ret; > + > + s->avctx->pix_fmt = canvas->format; > + } > + > + ff_thread_finish_setup(s->avctx); 1. Up until now, when decoding a series of stand-alone WebP pictures, the multiple decoder instances don't wait for each other (because the WebP decoder had no update_thread_context callback). You added such a callback and now you are calling it after the main picture has already been decoded, effectively serializing everything. You can test this for yourself: Create lots of files via ffmpeg -i -c:v libwebp -f webp%d.webp and decode them (don't use -stream_loop on a single input picture, as this will flush the decoder after every single picture, so that everything is always serialized). 2. To fix this, ff_thread_finish_setup() needs to be called as soon as possible. This means that you have to abandon the approach of letting the inner VP8 decoder set the frame dimensions and then overwriting them again in the WebP decoder. 3. Your WebP demuxer (from 4/5) splits an animation into a series of packets. The problem is the design of the extended file format (https://developers.google.com/speed/webp/docs/riff_container#extended_file_format): Certain metadata chunks (in particular exif) that pertain to the whole animation are stored only after the image data. If you split the input into packets, the decoder won't be able to attach it to every frame except the last one. 4. Due to this, splitting the input packet should be avoided. But this causes great complications in the decoder: It now needs to output multiple frames per packet (if said packet contained an animation). I see three ways to do this: a) Use the receive frame callback for this decoder. This will necessitate changes to pthread_frame.c (which currently can't handle receive_frame decoders) and even then it will have the further downside that decoding a single animation is single-threaded when using frame-threading (given that animations provide opportunity to decode parts of a packet in parallel, they work naturally with slice threading, not frame-threading). But non-animations should work fine as now. b) Somehow implement this via AV_CODEC_CAP_OTHER_THREADS, potentially via the AVExecutor API. This would have the upside that it could dynamically switch between frame- and slice threading (depending upon what the input is). c) Accept packets containing whole animations, but use a BSF to split the data so that the metadata arrives at the decoder before/together with the real frame data. I am not sure whether this would necessitate changes to the decode API, too (basically: if there are threads that are currently not busy and the BSF has not signalled EAGAIN yet, then try to extract another packet from the BSF). Notice that the BSF I have in mind would not be a public BSF, but a private one (given that the output of the BSF would be spec-incompliant due to the wrong ordering it should not be public), i.e. one not accessible via av_bsf_get_by_name() or av_bsf_iterate(). > + > + if (*got_frame) { > + if (!(s->vp8x_flags & VP8X_FLAG_ANIMATION)) { > + // no animation, output the decoded frame > + av_frame_move_ref(p, s->frame); > + } else { > + if (!key_frame) { > + ff_thread_await_progress(&s->canvas_frame, s->await_progress, 0); > + > + ret = dispose_prev_frame_in_canvas(s); > + if (ret < 0) > + goto end; > + } > + > + ret = blend_frame_into_canvas(s); > + if (ret < 0) > + goto end; > + > + ret = copy_canvas_to_frame(s, p, key_frame); > + if (ret < 0) > + goto end; > + > + ff_thread_report_progress(&s->canvas_frame, s->await_progress + 1, 0); > + } > + > + p->pts = avpkt->pts; > + } > + > + ret = avpkt->size; > + > +end: > + av_frame_unref(s->frame); > + return ret; > } > > const FFCodec ff_webp_decoder = { > .p.name = "webp", > CODEC_LONG_NAME("WebP image"), > .p.type = AVMEDIA_TYPE_VIDEO, > .p.id = AV_CODEC_ID_WEBP, > .priv_data_size = sizeof(WebPContext), > + UPDATE_THREAD_CONTEXT(webp_update_thread_context), > .init = webp_decode_init, > FF_CODEC_DECODE_CB(webp_decode_frame), > .close = webp_decode_close, > + .flush = webp_decode_flush, > .p.capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_FRAME_THREADS, > - .caps_internal = FF_CODEC_CAP_ICC_PROFILES, > + .caps_internal = FF_CODEC_CAP_ICC_PROFILES | FF_CODEC_CAP_ALLOCATE_PROGRESS, > }; _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".