From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 4038240097 for ; Sun, 19 Dec 2021 12:51:56 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1010568AE9D; Sun, 19 Dec 2021 14:51:54 +0200 (EET) Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 029BA68AE6F for ; Sun, 19 Dec 2021 14:51:47 +0200 (EET) Received: from localhost (213-47-68-29.cable.dynamic.surfer.at [213.47.68.29]) (Authenticated sender: michael@niedermayer.cc) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id CDFA01C0006 for ; Sun, 19 Dec 2021 12:51:46 +0000 (UTC) Date: Sun, 19 Dec 2021 13:51:45 +0100 From: Michael Niedermayer To: FFmpeg development discussions and patches Message-ID: <20211219125145.GO2829255@pb2> References: <20211216132151.8216-1-jamrial@gmail.com> <92861361-f316-78a5-3c72-1e279d2a9f8c@passwd.hu> <20211217112417.GG2829255@pb2> <5d14188-a4b0-4978-104c-59595fed8137@passwd.hu> <20211218133612.GL2829255@pb2> <20211218141516.GM2829255@pb2> <31397d1-438-8bfd-5a79-a07a67982559@passwd.hu> MIME-Version: 1.0 In-Reply-To: <31397d1-438-8bfd-5a79-a07a67982559@passwd.hu> Subject: Re: [FFmpeg-devel] [PATCH 000/279 v2] New channel layout API X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: multipart/mixed; boundary="===============1212074912463231927==" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: --===============1212074912463231927== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="VGCfsDJTqEoXtsu8" Content-Disposition: inline --VGCfsDJTqEoXtsu8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Dec 19, 2021 at 12:35:11PM +0100, Marton Balint wrote: >=20 >=20 > On Sat, 18 Dec 2021, Michael Niedermayer wrote: >=20 > > On Sat, Dec 18, 2021 at 02:36:12PM +0100, Michael Niedermayer wrote: > > > On Fri, Dec 17, 2021 at 07:04:08PM +0100, Marton Balint wrote: > > > >=20 > > > >=20 > > > > On Fri, 17 Dec 2021, Michael Niedermayer wrote: > > > >=20 > > > > > On Fri, Dec 17, 2021 at 01:04:19AM +0100, Marton Balint wrote: > > > > > >=20 > > > > > >=20 > > > > > > On Thu, 16 Dec 2021, James Almer wrote: > > > > > >=20 > > > > > > > Resending the first two patches only, since this is meant to > > > > > > > show the implementation of one of the several suggestions made > > > > > > > in the previous set that need to be discussed and hopefully > > > > > > > resolved in a call. > > > > > >=20 > > > > > > Can you push the full branch somewhere? > > > > > >=20 > > > > > > >=20 > > > > > > > The proposals so far to extend the API to support either cust= om > > > > > > > labels for channels are, or some form of extra user informati= on. > > > > > > >=20 > > > > > > > - Fixed array of bytes to hold a label. Simple solution, but > > > > > > > the labels will have a hard limit that can only be extended > > > > > > > with a major bump. This is what i implemented in this versio= n. > > > > > > > - "char *name" per channel that the user may allocate and the > > > > > > > API will manage, duplicate and free. Simple solution, and the > > > > > > > name can be arbitrarily long, but inefficient (av_strdup() p= er > > > > > > > channel with a custom label on layout copy). > > > > > > > - "const char *name" per channel for compile time constants, = or > > > > > > > that the user may allocate and free. Very efficient, but for > > > > > > > non compile time strings ensuring they outlive the layout can > > > > > > > be tricky. > > > > > > > - Refcounted AVChannelCustom with a dictionary. This can't be > > > > > > > done with AVBufferRef, so it would require some other form > > > > > > > of reference counting. And a dictionary may add quite a bit = of > > > > > > > complexity to the API, as you can set anything on them. > > > > > >=20 > > > > > > Until we have proper refcounting API we can make the AVBufferRe= f in > > > > > > AVChannelLayout a void *, and only allow channel_layout functio= ns to > > > > > > dereference it as an AVBufferRef. This would mean adding some e= xtra helper > > > > > > functions to channel layout, but overall it is not unsolvable. > > > > > >=20 > > > > > > The real question is that if you want to use refcounting and ad= d helpers to > > > > > > query / replace per-channel metadata, or you find the idea too = heavy weight > > > > > > and would like to stick to flat structs. > > > > >=20 > > > > > what is the advantage of refcounting for channel metadata ? > > > > > is it about the used memory, about the reduced need to copy ? > > > >=20 > > > > Basicly it is the ability to store per-channel metadata in avdictio= nary, > > > > because otherwise it would have to be copyed, and avdictionary is v= ery > > > > ineffective at copying because of many mallocs. > > > >=20 > > > > >=20 > > > > > what kind of metadata and what size do you expect ? > > > > > bytes, kilobytes, megabytes, gigabytes per channel ? > > > >=20 > > > > Usually, nothing, because most format don't have support for per-ch= annel > > > > metadata. In some cases it is going to be a couple of textual metad= ata > > > > key-value pairs, such as language, label, group, speaker, positon, = so 4-5 > > > > dynamically allocated string pairs, plus the AVDictionary itself, m= ultiplied > > > > by the number of channels in a layout. > > > >=20 > > > > >=20 > > > > > what is the overhead for dynamic allocation and ref counting? > > > > > that is at which point does it even make sense ? > > > >=20 > > > > I don't have exact measurements. It is generally felt that copying > > > > AVDictionary per-channel is a huge overhead for something as lightw= eight as > > > > an audio frame which is a 2-4 kB per channel at most and only a cou= ple of > > > > allocs usually not dependant on the number of channels. That's why > > > > refcounting was proposed. > > >=20 > > > I was thinking more at a AVStream / AVCodecParameters level. > >=20 > > > How will a demuxer transport such metadata over a AVPacket into a dec= oder > > > outputting metadata-filled AVFrames? > >=20 > > or is this never needed ? >=20 > I am not sure I understand. Usually metadata is passed from demuxer to > decoder by avcodec_parameters_to_context(), this is used for all metadata > which is in AVCodecParameters. >=20 > For per-packet metadata ff_decode_frame_props() has some automatic packet > side data -> frame side data transfer. >=20 > AVStream side data may be transferred to AVPacket side data if > av_format_inject_global_side_data() is used, but it is not enabled by > default. The sidedata in AVPacket is not channel specific, the data in AVFrames new channels is. The later is as you wrote expensive to copy/alloc so it needs ref counting what i was trying to point to was that if we need a way to transfer this data on a per packet base from demuxer forward then we need a flat format. At least with the current AVPackets And a flat format is much lighter to copy around so then one is pushed toward the question, "should that be used for AVFrames too?" a small <10kb audio frame with 8 audio channels and a ref counted dictionary or whatever per each of the 8 channels vs. a flat description. an extra is that a flat format is very simple to memcmp() to check if it changed. I have no position in this above, iam just wanting to make sure nothing is missed in this somewhat rushed design. also theres yet another orthogonal aspect which has been missed i think This per channel description whatever its form will be is usefull for anoth= er case. And that are the video/image channels. generally this is=20 red/green/blue/(alpha) but there is material with other channels,=20 IR/NIR/UV/... and i imagine probably depth from some lidar/radar system Ideally such a future image channel API can share parts with the audio one thx [...] --=20 Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Opposition brings concord. Out of discord comes the fairest harmony. -- Heraclitus --VGCfsDJTqEoXtsu8 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iF0EABEIAB0WIQSf8hKLFH72cwut8TNhHseHBAsPqwUCYb8q3QAKCRBhHseHBAsP q3QKAJ9T0ItHpta4fPMCxp3Lm1Js3uOO3wCeJYuqCDNPqlns8ihincGWoXzEn8I= =Zuu3 -----END PGP SIGNATURE----- --VGCfsDJTqEoXtsu8-- --===============1212074912463231927== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". --===============1212074912463231927==--