From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <ffmpeg-devel-bounces@ffmpeg.org>
Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100])
by master.gitmailbox.com (Postfix) with ESMTPS id 06E544ABDF
for <ffmpegdev@gitmailbox.com>; Tue, 25 Mar 2025 02:26:33 +0000 (UTC)
Received: from [127.0.1.1] (localhost [127.0.0.1])
by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 46E5A687B95;
Tue, 25 Mar 2025 04:26:30 +0200 (EET)
Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net
[217.70.183.196])
by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0DCE76879F8
for <ffmpeg-devel@ffmpeg.org>; Tue, 25 Mar 2025 04:26:23 +0200 (EET)
Received: by mail.gandi.net (Postfix) with ESMTPSA id 5194344364
for <ffmpeg-devel@ffmpeg.org>; Tue, 25 Mar 2025 02:26:23 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=niedermayer.cc;
s=gm1; t=1742869583;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:mime-version:mime-version:content-type:content-type:
in-reply-to:in-reply-to:references:references;
bh=T6IIN8a82jdfxLmNB+UEU+PW9Q+MD43U1S1UWbnHEXc=;
b=D6mT/WC02fCS6zA/1Ypb6XY9RDPdsUO500GqRH9Rl4pp1uN9StcKYfpLlM+qNSArXetBgw
9LLrVuJ8VQqttpcbvvEDweKC/HwR9yrAOgEmuE60VkPGRNJUzhdiYz/25r2upJpJA13455
IIAQpsZMG+AS15/crpQbc4+v1Lb7JdYYH+oY4CRG9TVP7Og+/VHz6u4muHg2Q6TL8XGkfZ
YLU+IVJZQl74If0jgocVdgdY/Tk4oUSDvAv/fA0rfzioKQBBlw+Kms0JailxU79PfL9ouK
3kcNnVmd/uY0tS34C9dOud+XO/OUSgIinDAi2To5r6m80TApQU3AyEp2LU0A0A==
Date: Tue, 25 Mar 2025 03:26:22 +0100
From: Michael Niedermayer <michael@niedermayer.cc>
To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Message-ID: <20250325022622.GV4991@pb2>
References: <20250324222051.19096-1-jamrial@gmail.com>
<20250325015053.GU4991@pb2>
<b667e162-df8c-4954-b240-1803bab810d6@gmail.com>
MIME-Version: 1.0
In-Reply-To: <b667e162-df8c-4954-b240-1803bab810d6@gmail.com>
X-GND-State: clean
X-GND-Score: -85
X-GND-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdduiedugeehucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuifetpfffkfdpucggtfgfnhhsuhgsshgtrhhisggvnecuuegrihhlohhuthemuceftddunecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenfghrlhcuvffnffculdduhedmnecujfgurhepfffhvffukfhfgggtuggjsehgtderredttddunecuhfhrohhmpefoihgthhgrvghlucfpihgvuggvrhhmrgihvghruceomhhitghhrggvlhesnhhivgguvghrmhgrhigvrhdrtggtqeenucggtffrrghtthgvrhhnpedutedvhfduuedugedufefghefhvedvgffgffekhfdvgfdvtefftdejkeehteefheenucfkphepgedurdeiiedrieejrdduudefnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepgedurdeiiedrieejrdduudefpdhhvghloheplhhotggrlhhhohhsthdpmhgrihhlfhhrohhmpehmihgthhgrvghlsehnihgvuggvrhhmrgihvghrrdgttgdpnhgspghrtghpthhtohepuddprhgtphhtthhopehffhhmphgvghdquggvvhgvlhesfhhfmhhpvghgrdhorhhg
X-GND-Sasl: michael@niedermayer.cc
Subject: Re: [FFmpeg-devel] [PATCH] avcodec/ffv1enc: further reduce stack
usage
X-BeenThere: ffmpeg-devel@ffmpeg.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: FFmpeg development discussions and patches <ffmpeg-devel.ffmpeg.org>
List-Unsubscribe: <https://ffmpeg.org/mailman/options/ffmpeg-devel>,
<mailto:ffmpeg-devel-request@ffmpeg.org?subject=unsubscribe>
List-Archive: <https://ffmpeg.org/pipermail/ffmpeg-devel>
List-Post: <mailto:ffmpeg-devel@ffmpeg.org>
List-Help: <mailto:ffmpeg-devel-request@ffmpeg.org?subject=help>
List-Subscribe: <https://ffmpeg.org/mailman/listinfo/ffmpeg-devel>,
<mailto:ffmpeg-devel-request@ffmpeg.org?subject=subscribe>
Reply-To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Content-Type: multipart/mixed; boundary="===============7653569320134358671=="
Errors-To: ffmpeg-devel-bounces@ffmpeg.org
Sender: "ffmpeg-devel" <ffmpeg-devel-bounces@ffmpeg.org>
Archived-At: <https://master.gitmailbox.com/ffmpegdev/20250325022622.GV4991@pb2/>
List-Archive: <https://master.gitmailbox.com/ffmpegdev/>
List-Post: <mailto:ffmpegdev@gitmailbox.com>
--===============7653569320134358671==
Content-Type: multipart/signed; micalg=pgp-sha512;
protocol="application/pgp-signature"; boundary="JPFEczjN1Zb2Zqlb"
Content-Disposition: inline
--JPFEczjN1Zb2Zqlb
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Mon, Mar 24, 2025 at 10:57:28PM -0300, James Almer wrote:
> On 3/24/2025 10:50 PM, Michael Niedermayer wrote:
> > Hi
> >=20
> > On Mon, Mar 24, 2025 at 07:20:50PM -0300, James Almer wrote:
> > > Continues from commit 702239bc500b, fixing FATE failures on MacOS.
> > >=20
> > > Signed-off-by: James Almer <jamrial@gmail.com>
> > > ---
> > > Confirmed by Martin Storsj=F6. Float encoding untested.
> > >=20
> > > libavcodec/ffv1.h | 16 ++++
> > > libavcodec/ffv1enc.c | 177 +++++++++++++++++-----------------------=
---
> > > 2 files changed, 84 insertions(+), 109 deletions(-)
> > >=20
> > > diff --git a/libavcodec/ffv1.h b/libavcodec/ffv1.h
> > > index 09118e0b7d..d1c239f138 100644
> > > --- a/libavcodec/ffv1.h
> > > +++ b/libavcodec/ffv1.h
> > > @@ -115,6 +115,22 @@ typedef struct FFV1SliceContext {
> > > uint32_t val; //this is unneeded if you accept a dereferenc=
e on each access
> > > uint16_t ndx;
> > > } unit[4][65536];
> > > + struct RemapEncoderState {
> > > + int delta_stack[65536]; //We need to encode the run valu=
e before the adjustments, this stores the adjustments until we know the len=
gth of the run
> > > + int16_t index_stack[65537]; //only needed with multiple segm=
ents
> > > + uint8_t state[2][3][32];
> > > + int mul[4096+1];
> > > + RangeCoder rc;
> > > + int lu;
> > > + int run;
> > > + int64_t last_val;
> > > + int compact_index;
> > > + int mul_count;
> > > + int i;
> > > + int pixel_num;
> > > + int p;
> > > + int current_mul_index;
> > > + } remap_state;
> > > } FFV1SliceContext;
> >=20
> > please provide a link to the failure
>=20
> Martin will have to do that. I can't seem to find any FATE instance faili=
ng,
> but he said it affected his OSX machines.
yeah, i also looked and couldnt find it
>=20
> >=20
> > This makes the code increasingly ugly.
> >=20
> > i dont understand why this breaks fate, fate should not use
> > any of the float code as none should be run in fate ATM.
> > its also all under -strict -2 checks
>=20
> It also surprised me, since these are functions that need to be called,
> unlike the fix in 702239bc500b which was in a function actually called by
> existing tests.
for the record macosx should habe 512k stack per thread
>=20
> >=20
> > this is temporary data not needed outside float32
> > and not needed outside the remap table writing.
> >=20
> > we may need more than one such state.
> > (if we dont use a heuristic but actually
> > encode bruteforce / trial and error)
> >=20
> > t conflicts with all work i did today
> >=20
> > theres tons of unused memory.
> >=20
> > We ATM do 2 things in encode_float32_remap_segment()
> > one is encoding the table
> > the other is writing the remaped pixels into sc->bitmap
> > by using unit[s.p][s.i].ndx
> >=20
> > sc->bitmap is unused before, unit[s.p][s.i].ndx unused afterwards
> > the input image itself is also not used again
> > half of fltmap32 is unused (thats 512kb alone here)
>=20
> Yeah, ideally all this is allocated only when needed rather than
> unconditionally in the slice context. But i didn't go that far since i ca=
n't
> even reproduce this issue.
>=20
> >=20
> > the code can be writen so it doesnt need the stack
> > but just runs twice over the stuff (not sure how clean this
> > would be but if you try _please_ do it on top of the patches
> > i posted today, the code is simpler and less buggy after
> > these patches
> >=20
> > But i really dont understand why fate fails in relation to code
> > it never executes.
>=20
> For the issue fixed in 702239bc500b, i guess it did attempt to reserve st=
ack
> space even if it never used it. For this one? Beats me.
maybe some smart compiler inlining this
ATM before sleeping over it i would suggest as solution
1. apply my patches from today (code will be cleaner)
2. eliminate the RemapEncoderState struct put things on the stack
3. use the space after fltmap (in fltmap32) for the 2 stack arrays
with some struct and union to keep it clear
moving RemapEncoderState to the context makes no sense, theres tons
of small bits in it that dont exist outside their local use in the
code using them.
I guess the struct causes more problems than it solves
ill look into this tomorrow, need to sleep now as theres some
noisy workers across the street building a house so i need to
sleep at night, but dont hesitate to work on it if you want
or just disable the code
thx
[...]
--=20
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
Let us carefully observe those good qualities wherein our enemies excel us
and endeavor to excel them, by avoiding what is faulty, and imitating what
is excellent in them. -- Plutarch
--JPFEczjN1Zb2Zqlb
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iF0EABEKAB0WIQSf8hKLFH72cwut8TNhHseHBAsPqwUCZ+IUSgAKCRBhHseHBAsP
q83bAJsGbRCh6oHebw1kQRhDaoulIHnLkQCeKiYVRLCgQxASLcxmPop3QY7oe/A=
=AHVI
-----END PGP SIGNATURE-----
--JPFEczjN1Zb2Zqlb--
--===============7653569320134358671==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
--===============7653569320134358671==--