[FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[Victor Luchitz]

[-- Attachment #1: Type: text/plain, Size: 103 bytes --]

Re-posting the patch as an attachment. Sorry for the inconvenience!

-- 
Best regards,
 Victor Luchitz

[-- Attachment #2: 0001-libavcodec-add-bit-rate-support-to-RoQ-video-encoder.patch --]
[-- Type: application/octet-stream, Size: 11943 bytes --]

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[Tomas Härdin]

[-- Attachment #1: Type: text/plain, Size: 4626 bytes --]

tis 2024-01-23 klockan 11:33 +0300 skrev Victor Luchitz:
> Re-posting the patch as an attachment. Sorry for the inconvenience!

Yep, this one applies and also passes FATE.

I notice ffplay_buffer whines a lot when playing RoQ files:

> [swscaler @ 0x7fcb44043240] deprecated pixel format used, make sure you did set range correctly
> [ffplay_buffer @ 0x7fcb44032fc0] filter context - w: 256 h: 256 fmt: 14 csp: unknown range: unknown, incoming frame - w: 256 h: 256 fmt: 14 csp: unknown range: pc pts_time: 0
> [ffplay_buffer @ 0x7fcb44032fc0] Changing video frame properties on the fly is not supported by all filters.

I don't think this problem relates to this patch however.

Anyway, using -b:v 100k causes the encoder to effectively become stuck
on the first frame, being unable to go below 621 kbps and increasing
qscale very slowly. But you mentioned this already of course. Perhaps
there should be a faster "startup" phase? Subsequent frames being P-
frames may lead to the average hitting the target bitrate.

Even when using 1 Mbps the encoding is very slow:

./ffmpeg -r:v 30 -t 1 -f lavfi -i testsrc2 -s 256x256 -b:v 1000k -y
foo-1000k.roq

> [roqvideo @ 0x3112980] 
> Generated a frame too big for desired bit rate (1489 kbps), now switching to a bigger qscale value (257).
> [roqvideo @ 0x3112980] 
> Generated a frame too big for desired bit rate (1489 kbps), now switching to a bigger qscale value (259).
> [...]
> Generated a frame too big for desired bit rate (1100 kbps), now switching to a bigger qscale value (196863).
> [roqvideo @ 0x3112980] .0 size=       0kB time=N/A bitrate=N/A speed=N/A    
> Generated a frame too big for desired bit rate (1069 kbps), now switching to a bigger qscale value (262399).
> [roqvideo @ 0x3112980] 
> Generated a frame too small for desired bit rate (601 kbps), reverting lambda and using smaller inc on qscale (196863).
> [roqvideo @ 0x3112980] 0.0 size=      17kB time=00:00:00.16 bitrate= 812.7kbits/s speed=0.0476x    
> Generated a frame too small for desired bit rate (915 kbps), qscale value cannot be lowered any further (1).
> [...]
> [roqvideo @ 0x3112980] 
> Generated a frame too big for desired bit rate (1084 kbps), now switching to a bigger qscale value (327680).
> [roqvideo @ 0x3112980] 
> Generated a frame too big for desired bit rate (1068 kbps), now switching to a bigger qscale value (393216).
> [roqvideo @ 0x3112980] 0.0 size=      42kB time=00:00:00.40 bitrate= 866.5kbits/s speed=0.0667x    
> Generated a frame too small for desired bit rate (514 kbps), reverting lambda and using smaller inc on qscale (327680).
> [roqvideo @ 0x3112980] 0.0 size=      62kB time=00:00:00.56 bitrate= 896.2kbits/s speed=0.0708x    
> Generated a frame too small for desired bit rate (901 kbps), qscale value cannot be lowered any further (1).

This suggests an unstable regulation loop. It also makes the encoder
slower than the Cinepak encoder, amazingly. 59.6 vs 21.57 seconds for
encoding 90 frames of dimension 256x256.

-bt should have a default of 0, and similar default logic as -b. The
default of bitrate/20 also seems excessively small? If I change the
default to be the same as the bitrate then the encoding time above
shrinks to 11.7 seconds.

There is an excessive newline visible here:

> +                   "\nGenerated a frame too big for desired bit rate
> (%d kbps), "

And elsewhere.

Using non-integer framerates causes the encoder to go bananas. For
example 24000/1001 fps makes the ratecontrol think it's running at
24000 fps. An easy fix is to reject avctx->time_base.num != 1 on init.
roq_write_header() rejects non-integer framerates.

I'm also curious about this hunk:

> -        if (enc->lambda > 100000) {
> +        if (enc->lambda > 100000000) {
>              av_log(roq->logctx, AV_LOG_ERROR, "Cannot encode video in Quake compatible form\n");
>              return AVERROR(EINVAL);
>          }

Where in the Quake (3?) source code is this limitation? Seems rather
that there should be a retry limit. There's probably no harm in keeping
going so long as the packet doesn't end up above the limit.

A quick logarithmic regression on bitrate vs qscale suggests it scales
with somewhere between qscale^-0.2833 and qscale^-0.2842. Conversely,
to hit a specific bitrate, try scaling qscale with the bitrate ratio
raised to around 3.5. A more conservative exponent like 2 is probably
also fine. See patch attached.

A more systematic approach might be to do a binary search for qscale,
with an initial guess per the formula above.

/Tomas


[-- Attachment #2: 0001-Fast-ratecontrol-could-be-better.patch --]
[-- Type: text/x-patch, Size: 6460 bytes --]

From 5995119843f57ba68dc8d66cb6c53cd8ec386ab9 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Tomas=20H=C3=A4rdin?= <git@haerdin.se>
Date: Tue, 23 Jan 2024 18:40:55 +0100
Subject: [PATCH] Fast ratecontrol, could be better

---
 libavcodec/roqvideoenc.c | 48 +++++++++++++++++++++++++++-------------
 1 file changed, 33 insertions(+), 15 deletions(-)

diff --git a/libavcodec/roqvideoenc.c b/libavcodec/roqvideoenc.c
index 976015d918..0fd70e85c4 100644
--- a/libavcodec/roqvideoenc.c
+++ b/libavcodec/roqvideoenc.c
@@ -894,7 +894,7 @@ static int roq_encode_video(AVCodecContext *avctx)
     RoqEncContext *const enc = avctx->priv_data;
     RoqTempData *const tempData = &enc->tmp_data;
     RoqContext *const roq = &enc->common;
-    int ret;
+    int ret, num_fast_qscale_deltas = 0, num_tries = 0;
 
     memset(tempData, 0, sizeof(*tempData));
 
@@ -908,12 +908,16 @@ static int roq_encode_video(AVCodecContext *avctx)
     }
 
  retry_encode:
+    num_tries++;
     for (int i = 0; i < roq->width * roq->height / 64; i++)
         gather_data_for_cel(enc->cel_evals + i, enc);
 
     /* Quake 3 can't handle chunks bigger than 65535 bytes */
+#define MAX_LAMBDA_DELTA 3000000
+#define MAX_TRIES 30
+#define MAX_FAST_TRIES 5
     if (tempData->mainChunkSize/8 > 65535 && enc->quake3_compat) {
-        if (enc->lambda > 100000000) {
+        if (num_tries > MAX_TRIES) {
             av_log(roq->logctx, AV_LOG_ERROR, "Cannot encode video in Quake compatible form\n");
             return AVERROR(EINVAL);
         }
@@ -941,28 +945,40 @@ static int roq_encode_video(AVCodecContext *avctx)
         int64_t tol = avctx->bit_rate_tolerance;
 
         /* tolerance > bit rate, set to 5% of the bit rate */
-        if (tol > avctx->bit_rate)
+        if (!tol)
             tol = avctx->bit_rate / 20;
 
         av_log(roq->logctx, AV_LOG_VERBOSE,
-               "\nDesired bit rate (%d kbps), "
+               "Desired bit rate (%d kbps), "
+               "Current bit rate (%d kbps), "
                "Bit rate tolerance (%d), "
                "Frame rate (%d)\n",
-               (int)avctx->bit_rate, (int)tol, avctx->time_base.den);
+               (int)avctx->bit_rate / 1000, ftotal / 1000, (int)tol, avctx->time_base.den);
 
         if (ftotal > (avctx->bit_rate + tol)) {
             /* frame is too big - increase qscale */
-            if (enc->lambda > 100000000) {
-                av_log(roq->logctx, AV_LOG_ERROR, "\nCannot encode video at desired bitrate\n");
-                return AVERROR(EINVAL);
+            if (num_tries > MAX_TRIES) {
+                av_log(roq->logctx, AV_LOG_WARNING, "Cannot encode video at desired bitrate (got %d kbps)\n", ftotal / 1000);
+                // don't fail hard
+                goto keepgoing;
+            }
+            enc->lambda_delta = enc->lambda_delta <= 0 ? 1 : enc->lambda_delta < MAX_LAMBDA_DELTA/2 ? enc->lambda_delta*2 : MAX_LAMBDA_DELTA;
+            if (num_fast_qscale_deltas < MAX_FAST_TRIES) {
+                float ratio;
+                int fast_lambda_delta;
+                ratio = powf((float)ftotal / avctx->bit_rate, 2) - 1; // -1 because it's a delta
+                av_log(roq->logctx, AV_LOG_VERBOSE, "ratio = %f\n", ratio);
+                fast_lambda_delta = FFMAX(1, FFMIN(enc->lambda * ratio, MAX_LAMBDA_DELTA));
+                // sometimes ratio ends up quite small, in which case we fall back to the simple doubling logic
+                enc->lambda_delta = FFMAX(enc->lambda_delta, fast_lambda_delta);
+                num_fast_qscale_deltas++;
             }
-            enc->lambda_delta = enc->lambda_delta <= 0 ? 1 : enc->lambda_delta < 65536 ? enc->lambda_delta*2 : 65536;
             enc->last_lambda = enc->lambda;
             enc->lambda += enc->lambda_delta;
             av_log(roq->logctx, AV_LOG_INFO,
-                   "\nGenerated a frame too big for desired bit rate (%d kbps), "
-                   "now switching to a bigger qscale value (%d).\n",
-                   ftotal / 1000, (int)enc->lambda);
+                   "Generated a frame too big for desired bit rate (%d kbps), "
+                   "now switching to a bigger qscale value (%d -> %d).\n",
+                   ftotal / 1000, (int)enc->last_lambda, (int)enc->lambda);
             tempData->mainChunkSize = 0;
             memset(tempData->used_option, 0, sizeof(tempData->used_option));
             memset(tempData->codebooks.usedCB4, 0,
@@ -975,12 +991,12 @@ static int roq_encode_video(AVCodecContext *avctx)
             /* frame is too small - decrease qscale */
             if (enc->lambda <= 1) {
                 av_log(roq->logctx, AV_LOG_WARNING,
-                       "\nGenerated a frame too small for desired bit rate (%d kbps), "
+                       "Generated a frame too small for desired bit rate (%d kbps), "
                        "qscale value cannot be lowered any further (%d).\n",
                        ftotal / 1000, (int)enc->lambda);
             } else if ((enc->lambda - enc->last_lambda) == 1) {
                 av_log(roq->logctx, AV_LOG_WARNING,
-                       "\nCannot find qscale that gives desired bit rate within desired tolerance, "
+                       "Cannot find qscale that gives desired bit rate within desired tolerance, "
                        "using lower bitrate (%d kbps) with higher qscale value (%d).\n",
                        ftotal / 1000, (int)enc->lambda);
             } else {
@@ -992,7 +1008,7 @@ static int roq_encode_video(AVCodecContext *avctx)
                     enc->lambda = enc->last_lambda;
                     //enc->lambda *= (float)(tempData->mainChunkSize * avctx->time_base.den) / avctx->bit_rate;
                     av_log(roq->logctx, AV_LOG_INFO,
-                           "\nGenerated a frame too small for desired bit rate (%d kbps), "
+                           "Generated a frame too small for desired bit rate (%d kbps), "
                            "reverting lambda and using smaller inc on qscale (%d).\n",
                            ftotal / 1000, (int)enc->lambda);
                 }
@@ -1007,6 +1023,7 @@ static int roq_encode_video(AVCodecContext *avctx)
             }
         }
     }
+keepgoing:
 
     write_codebooks(enc);
 
@@ -1200,6 +1217,7 @@ static const AVClass roq_class = {
 
 static const FFCodecDefault roq_defaults[] = {
     { "b",                "0" },
+    { "bt",               "0" },
     { NULL },
 };
 
-- 
2.39.2


[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[Victor Luchitz]

On Tue, Jan 23, 2024 at 8:44 PM Tomas Härdin <git@haerdin.se> wrote:

> tis 2024-01-23 klockan 11:33 +0300 skrev Victor Luchitz:
> > Re-posting the patch as an attachment. Sorry for the inconvenience!
>
> Yep, this one applies and also passes FATE.
>
> I notice ffplay_buffer whines a lot when playing RoQ files:
>
> > [swscaler @ 0x7fcb44043240] deprecated pixel format used, make sure you
> did set range correctly
> > [ffplay_buffer @ 0x7fcb44032fc0] filter context - w: 256 h: 256 fmt: 14
> csp: unknown range: unknown, incoming frame - w: 256 h: 256 fmt: 14 csp:
> unknown range: pc pts_time: 0
> > [ffplay_buffer @ 0x7fcb44032fc0] Changing video frame properties on the
> fly is not supported by all filters.
>
> I don't think this problem relates to this patch however.
>

It definitely doesn't. If we decide this needs ironing out, it'd go in a
separate
patch in my opinion.


>
> Anyway, using -b:v 100k causes the encoder to effectively become stuck
> on the first frame, being unable to go below 621 kbps and increasing
> qscale very slowly. But you mentioned this already of course. Perhaps
> there should be a faster "startup" phase? Subsequent frames being P-
> frames may lead to the average hitting the target bitrate.
>

I'm not sure we'd go as far as special-casing the encoder for extremely low
bit-rates at high image resolutions. I mean, if you want low picture
quality,
the proper way for RoQ would be to scale down your video to a lower
resolution
instead of turning the video into a huge ugly mess of macroblocks.

The main issues with allowing a larger keyframe and then trying to make it
up on subsequent frames are: doable for FIRST frame, may cause buffer
underrun on later keyframes; we are targeting CDROM media with this patch,
be it 1X, 2X, whatever. We cannot have too much variance in the frame sizes
(bigger). Smaller is allowable, but not bigger.

It's OK if encoding takes some time :)


> Even when using 1 Mbps the encoding is very slow:
>
> ./ffmpeg -r:v 30 -t 1 -f lavfi -i testsrc2 -s 256x256 -b:v 1000k -y
> foo-1000k.roq
>
> > [roqvideo @ 0x3112980]
> > Generated a frame too big for desired bit rate (1489 kbps), now
> switching to a bigger qscale value (257).
> > [roqvideo @ 0x3112980]
> > Generated a frame too big for desired bit rate (1489 kbps), now
> switching to a bigger qscale value (259).
> > [...]
> > Generated a frame too big for desired bit rate (1100 kbps), now
> switching to a bigger qscale value (196863).
> > [roqvideo @ 0x3112980] .0 size=       0kB time=N/A bitrate=N/A
> speed=N/A
> > Generated a frame too big for desired bit rate (1069 kbps), now
> switching to a bigger qscale value (262399).
> > [roqvideo @ 0x3112980]
> > Generated a frame too small for desired bit rate (601 kbps), reverting
> lambda and using smaller inc on qscale (196863).
> > [roqvideo @ 0x3112980] 0.0 size=      17kB time=00:00:00.16 bitrate=
> 812.7kbits/s speed=0.0476x
> > Generated a frame too small for desired bit rate (915 kbps), qscale
> value cannot be lowered any further (1).
> > [...]
> > [roqvideo @ 0x3112980]
> > Generated a frame too big for desired bit rate (1084 kbps), now
> switching to a bigger qscale value (327680).
> > [roqvideo @ 0x3112980]
> > Generated a frame too big for desired bit rate (1068 kbps), now
> switching to a bigger qscale value (393216).
> > [roqvideo @ 0x3112980] 0.0 size=      42kB time=00:00:00.40 bitrate=
> 866.5kbits/s speed=0.0667x
> > Generated a frame too small for desired bit rate (514 kbps), reverting
> lambda and using smaller inc on qscale (327680).
> > [roqvideo @ 0x3112980] 0.0 size=      62kB time=00:00:00.56 bitrate=
> 896.2kbits/s speed=0.0708x
> > Generated a frame too small for desired bit rate (901 kbps), qscale
> value cannot be lowered any further (1).
>
> This suggests an unstable regulation loop. It also makes the encoder
> slower than the Cinepak encoder, amazingly. 59.6 vs 21.57 seconds for
> encoding 90 frames of dimension 256x256.
>
> -bt should have a default of 0, and similar default logic as -b. The
> default of bitrate/20 also seems excessively small? If I change the
> default to be the same as the bitrate then the encoding time above
> shrinks to 11.7 seconds.
>

The reason for the default tight bitrate tolerance is explained above:
we're targeting slow 1x and 2x speed CD-ROM drives..

However, if you feel we need to change it to something else, that's fine,
as long as we can override it via the command line :)


>
> There is an excessive newline visible here:
>
> > +                   "\nGenerated a frame too big for desired bit rate
> > (%d kbps), "
>
> And elsewhere.
>
> Using non-integer framerates causes the encoder to go bananas. For
> example 24000/1001 fps makes the ratecontrol think it's running at
> 24000 fps. An easy fix is to reject avctx->time_base.num != 1 on init.
> roq_write_header() rejects non-integer framerates.
>

Seems to me like a somewhat generic cleanup change that shouldn't
necessarily go with this patch. I'm OK with bundling it with everything
else though.


>
> I'm also curious about this hunk:
>
> > -        if (enc->lambda > 100000) {
> > +        if (enc->lambda > 100000000) {
> >              av_log(roq->logctx, AV_LOG_ERROR, "Cannot encode video in
> Quake compatible form\n");
> >              return AVERROR(EINVAL);
> >          }
>
> Where in the Quake (3?) source code is this limitation? Seems rather
> that there should be a retry limit. There's probably no harm in keeping
> going so long as the packet doesn't end up above the limit.
>

In our tests we found that lambda can go way higher than the original
100000. Stopping at 100000 is artificially restricting yourself when you
can go much further. 100000000 is high enough to allow 1X  bitrates
even on extreme frames.


>
> A quick logarithmic regression on bitrate vs qscale suggests it scales
> with somewhere between qscale^-0.2833 and qscale^-0.2842. Conversely,
> to hit a specific bitrate, try scaling qscale with the bitrate ratio
> raised to around 3.5. A more conservative exponent like 2 is probably
> also fine. See patch attached.
>

Thanks so much for the patch! I've tested it on a pretty long hires video at
835kbps and everything seems to work fine. The runtime was 2m39s for
both: the original and your version.



>
> A more systematic approach might be to do a binary search for qscale,
> with an initial guess per the formula above.
>
> /Tomas
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>


-- 
Best regards,
 Victor Luchitz
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[Tomas Härdin]

ons 2024-01-24 klockan 11:50 +0300 skrev Victor Luchitz:
> On Tue, Jan 23, 2024 at 8:44 PM Tomas Härdin <git@haerdin.se> wrote:
> > 
> > Anyway, using -b:v 100k causes the encoder to effectively become
> > stuck
> > on the first frame, being unable to go below 621 kbps and
> > increasing
> > qscale very slowly. But you mentioned this already of course.
> > Perhaps
> > there should be a faster "startup" phase? Subsequent frames being
> > P-
> > frames may lead to the average hitting the target bitrate.
> > 
> 
> I'm not sure we'd go as far as special-casing the encoder for
> extremely low
> bit-rates at high image resolutions. I mean, if you want low picture
> quality,
> the proper way for RoQ would be to scale down your video to a lower
> resolution
> instead of turning the video into a huge ugly mess of macroblocks.

I'm not really suggesting such low bitrates is a good idea, but they
shouldn't cause the encoder to choke. What happens if there's suddenly
a frame that's very complex, say white noise?

> The main issues with allowing a larger keyframe and then trying to
> make it
> up on subsequent frames are: doable for FIRST frame, may cause buffer
> underrun on later keyframes; we are targeting CDROM media with this
> patch,
> be it 1X, 2X, whatever. We cannot have too much variance in the frame
> sizes
> (bigger). Smaller is allowable, but not bigger.

How bursty is CD-ROM anyway? I suspect something similar to VBV would
help here. Surely there are buffers involved?

> It's OK if encoding takes some time :)

Yeah but it shouldn't hang

> 
> 
> > 
> > I'm also curious about this hunk:
> > 
> > > -        if (enc->lambda > 100000) {
> > > +        if (enc->lambda > 100000000) {
> > >              av_log(roq->logctx, AV_LOG_ERROR, "Cannot encode
> > > video in
> > Quake compatible form\n");
> > >              return AVERROR(EINVAL);
> > >          }
> > 
> > Where in the Quake (3?) source code is this limitation? Seems
> > rather
> > that there should be a retry limit. There's probably no harm in
> > keeping
> > going so long as the packet doesn't end up above the limit.
> > 
> 
> In our tests we found that lambda can go way higher than the original
> 100000. Stopping at 100000 is artificially restricting yourself when
> you
> can go much further. 100000000 is high enough to allow 1X  bitrates
> even on extreme frames.

I figured as much. But this could also be arrived at with some formula
I suspect. A higher limit isn't wrong per se.

> > A quick logarithmic regression on bitrate vs qscale suggests it
> > scales
> > with somewhere between qscale^-0.2833 and qscale^-0.2842.
> > Conversely,
> > to hit a specific bitrate, try scaling qscale with the bitrate
> > ratio
> > raised to around 3.5. A more conservative exponent like 2 is
> > probably
> > also fine. See patch attached.
> > 
> 
> Thanks so much for the patch! I've tested it on a pretty long hires
> video at
> 835kbps and everything seems to work fine. The runtime was 2m39s for
> both: the original and your version.

Yep. But, the fact that I could get it to hang at certain points
doesn't fully instill confidence.

A better approach might be to buffer an entire GOP, then find a lambda
that brings the entire GOP within the desired bitrate. If the encoder
happens to insert an extra I-frame, emit the I and P frames up to that
point and buffer up enough frames to make another GOP. Say with a GOP
size of 6, it would normally look like:

  [IPPPPP][IPPPPP]...

But sometimes like this, if nothing special is done:

  [IPPPIP][IPPPPP]...

This is probably wasteful however, so better to do

  [IPPP][IPPPPP][IP...

That second I-frame would probably have been quite cheap to fit within
the allotted bits. It might for example be a cut to white and fade in.

If I understand this patch correctly, it tries to make every packet
roughly the same size, which seems excessive. Does the machine you're
targetting not have enough RAM to buffer an entire CD-ROM? Or even a
couple of seconds?

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[Victor Luchitz]

In our case, the machine we're targeting (the Sega 32X) has only 256KB
of RAM. Even more modern consoles such as XBOX or even PS3 didn't
have enough RAM to hold an entire CD-ROM..

We also have to be concerned about how fast we can move data to the
main CPU. Say you make the first frame BIG to make it look its best, then
compensate with much smaller delta frames... is the first frame too much
data to move to the CPU quick enough? Well, maybe not an issue for the
first frame, but definitely an issue for keyframes in the middle of the
stream.



On Thu, Jan 25, 2024 at 12:30 AM Tomas Härdin <git@haerdin.se> wrote:

> ons 2024-01-24 klockan 11:50 +0300 skrev Victor Luchitz:
> > On Tue, Jan 23, 2024 at 8:44 PM Tomas Härdin <git@haerdin.se> wrote:
> > >
> > > Anyway, using -b:v 100k causes the encoder to effectively become
> > > stuck
> > > on the first frame, being unable to go below 621 kbps and
> > > increasing
> > > qscale very slowly. But you mentioned this already of course.
> > > Perhaps
> > > there should be a faster "startup" phase? Subsequent frames being
> > > P-
> > > frames may lead to the average hitting the target bitrate.
> > >
> >
> > I'm not sure we'd go as far as special-casing the encoder for
> > extremely low
> > bit-rates at high image resolutions. I mean, if you want low picture
> > quality,
> > the proper way for RoQ would be to scale down your video to a lower
> > resolution
> > instead of turning the video into a huge ugly mess of macroblocks.
>
> I'm not really suggesting such low bitrates is a good idea, but they
> shouldn't cause the encoder to choke. What happens if there's suddenly
> a frame that's very complex, say white noise?
>
> > The main issues with allowing a larger keyframe and then trying to
> > make it
> > up on subsequent frames are: doable for FIRST frame, may cause buffer
> > underrun on later keyframes; we are targeting CDROM media with this
> > patch,
> > be it 1X, 2X, whatever. We cannot have too much variance in the frame
> > sizes
> > (bigger). Smaller is allowable, but not bigger.
>
> How bursty is CD-ROM anyway? I suspect something similar to VBV would
> help here. Surely there are buffers involved?
>
> > It's OK if encoding takes some time :)
>
> Yeah but it shouldn't hang
>
> >
> >
> > >
> > > I'm also curious about this hunk:
> > >
> > > > -        if (enc->lambda > 100000) {
> > > > +        if (enc->lambda > 100000000) {
> > > >              av_log(roq->logctx, AV_LOG_ERROR, "Cannot encode
> > > > video in
> > > Quake compatible form\n");
> > > >              return AVERROR(EINVAL);
> > > >          }
> > >
> > > Where in the Quake (3?) source code is this limitation? Seems
> > > rather
> > > that there should be a retry limit. There's probably no harm in
> > > keeping
> > > going so long as the packet doesn't end up above the limit.
> > >
> >
> > In our tests we found that lambda can go way higher than the original
> > 100000. Stopping at 100000 is artificially restricting yourself when
> > you
> > can go much further. 100000000 is high enough to allow 1X  bitrates
> > even on extreme frames.
>
> I figured as much. But this could also be arrived at with some formula
> I suspect. A higher limit isn't wrong per se.
>
> > > A quick logarithmic regression on bitrate vs qscale suggests it
> > > scales
> > > with somewhere between qscale^-0.2833 and qscale^-0.2842.
> > > Conversely,
> > > to hit a specific bitrate, try scaling qscale with the bitrate
> > > ratio
> > > raised to around 3.5. A more conservative exponent like 2 is
> > > probably
> > > also fine. See patch attached.
> > >
> >
> > Thanks so much for the patch! I've tested it on a pretty long hires
> > video at
> > 835kbps and everything seems to work fine. The runtime was 2m39s for
> > both: the original and your version.
>
> Yep. But, the fact that I could get it to hang at certain points
> doesn't fully instill confidence.
>
> A better approach might be to buffer an entire GOP, then find a lambda
> that brings the entire GOP within the desired bitrate. If the encoder
> happens to insert an extra I-frame, emit the I and P frames up to that
> point and buffer up enough frames to make another GOP. Say with a GOP
> size of 6, it would normally look like:
>
>   [IPPPPP][IPPPPP]...
>
> But sometimes like this, if nothing special is done:
>
>   [IPPPIP][IPPPPP]...
>
> This is probably wasteful however, so better to do
>
>   [IPPP][IPPPPP][IP...
>
> That second I-frame would probably have been quite cheap to fit within
> the allotted bits. It might for example be a cut to white and fade in.
>
> If I understand this patch correctly, it tries to make every packet
> roughly the same size, which seems excessive. Does the machine you're
> targetting not have enough RAM to buffer an entire CD-ROM? Or even a
> couple of seconds?
>
> /Tomas
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>


-- 
Best regards,
 Victor Luchitz
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[Tomas Härdin]

tor 2024-01-25 klockan 01:09 +0300 skrev Victor Luchitz:
> In our case, the machine we're targeting (the Sega 32X) has only
> 256KB
> of RAM. Even more modern consoles such as XBOX or even PS3 didn't
> have enough RAM to hold an entire CD-ROM..
> 
> We also have to be concerned about how fast we can move data to the
> main CPU. Say you make the first frame BIG to make it look its best,
> then
> compensate with much smaller delta frames... is the first frame too
> much
> data to move to the CPU quick enough? Well, maybe not an issue for
> the
> first frame, but definitely an issue for keyframes in the middle of
> the
> stream.

Ah, that makes a lot more sense. Yeah I can see that being an issue.
And decoding time probably depends on packet size as well, so even if
you could move data around in advance, this still doesn't help that
keyframe decode any faster. I've seen similar behavior with JPEG2000
where decode time scales linearly with bitrate (plus a fixed overhead).

One thing that strikes me is if you're only aiming for linear playback
then you can set the GOP size large enough that the encoder is never
forced to insert a keyframe.

I think my main issue then comes down to the qscale search being way
too slow, especially the fact that it can effectively hang. I would
suggest a binary search using the heuristic I derived for the first
split, or something similar. Searching between 1 and 100000000 seems
excessive in most cases. It should be possible to box that interval in
quickly, again using the heuristic.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[Victor Luchitz]

On Thu, Jan 25, 2024 at 1:27 AM Tomas Härdin <git@haerdin.se> wrote:

> tor 2024-01-25 klockan 01:09 +0300 skrev Victor Luchitz:
> > In our case, the machine we're targeting (the Sega 32X) has only
> > 256KB
> > of RAM. Even more modern consoles such as XBOX or even PS3 didn't
> > have enough RAM to hold an entire CD-ROM..
> >
> > We also have to be concerned about how fast we can move data to the
> > main CPU. Say you make the first frame BIG to make it look its best,
> > then
> > compensate with much smaller delta frames... is the first frame too
> > much
> > data to move to the CPU quick enough? Well, maybe not an issue for
> > the
> > first frame, but definitely an issue for keyframes in the middle of
> > the
> > stream.
>
> Ah, that makes a lot more sense. Yeah I can see that being an issue.
> And decoding time probably depends on packet size as well, so even if
> you could move data around in advance, this still doesn't help that
> keyframe decode any faster. I've seen similar behavior with JPEG2000
> where decode time scales linearly with bitrate (plus a fixed overhead).
>
> One thing that strikes me is if you're only aiming for linear playback
> then you can set the GOP size large enough that the encoder is never
> forced to insert a keyframe.
>
> I think my main issue then comes down to the qscale search being way
> too slow, especially the fact that it can effectively hang. I would
> suggest a binary search using the heuristic I derived for the first
> split, or something similar. Searching between 1 and 100000000 seems
> excessive in most cases. It should be possible to box that interval in
> quickly, again using the heuristic.
>

Alright, we'll look into it, thanks!


>
> /Tomas
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>


-- 
Best regards,
 Victor Luchitz
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[Anton Khirnov]

Quoting Victor Luchitz (2024-01-23 09:33:04)
> Re-posting the patch as an attachment. Sorry for the inconvenience!
> 
> -- 
> Best regards,
>  Victor Luchitz
> 
> From b6825112446ee220d89cab975e8bcef823884141 Mon Sep 17 00:00:00 2001
> From: Victor Luchits <vluchits@gmail.com>
> Date: Tue, 23 Jan 2024 11:27:07 +0300
> Subject: [PATCH] libavcodec: add bit-rate support to RoQ video encoder
> 
> The bitrate option (-b:v) can now be used to specify the bit rate
> of the video stream of the RoQ encoder.
> 
> Original patch by Joseph Fenton aka Chilly Willy
> 
> Signed-off-by: Victor Luchits <vluchits@gmail.com>
> ---
>  Changelog                |   1 +
>  libavcodec/roqvideo.h    |   1 +
>  libavcodec/roqvideodec.c |  16 ++++++

What's with the decoder changes?

-- 
Anton Khirnov
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec: add bit-rate support to RoQ video encoder

[Victor Luchitz]

You're right, this is actually a part that's not necessarily tied to bitrate
support, I'll move it to a separate patch.

On Wed, Jan 24, 2024 at 2:06 PM Anton Khirnov <anton@khirnov.net> wrote:

> Quoting Victor Luchitz (2024-01-23 09:33:04)
> > Re-posting the patch as an attachment. Sorry for the inconvenience!
> >
> > --
> > Best regards,
> >  Victor Luchitz
> >
> > From b6825112446ee220d89cab975e8bcef823884141 Mon Sep 17 00:00:00 2001
> > From: Victor Luchits <vluchits@gmail.com>
> > Date: Tue, 23 Jan 2024 11:27:07 +0300
> > Subject: [PATCH] libavcodec: add bit-rate support to RoQ video encoder
> >
> > The bitrate option (-b:v) can now be used to specify the bit rate
> > of the video stream of the RoQ encoder.
> >
> > Original patch by Joseph Fenton aka Chilly Willy
> >
> > Signed-off-by: Victor Luchits <vluchits@gmail.com>
> > ---
> >  Changelog                |   1 +
> >  libavcodec/roqvideo.h    |   1 +
> >  libavcodec/roqvideodec.c |  16 ++++++
>
> What's with the decoder changes?
>
> --
> Anton Khirnov
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>


-- 
Best regards,
 Victor Luchitz
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".