Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
 help / color / mirror / Atom feed
From: James Almer <jamrial@gmail.com>
To: ffmpeg-devel@ffmpeg.org
Subject: Re: [FFmpeg-devel] [PATCH] ffmpeg CLI multithreading
Date: Mon, 13 Nov 2023 09:38:01 -0300
Message-ID: <4eedc7f4-1307-458f-a6bb-98a70d027f69@gmail.com> (raw)
In-Reply-To: <CAEu79SasD2tsA_OOkzU2pbtEw=nFXeDqDvbtNNs0eZsz_GiNzg@mail.gmail.com>

On 11/13/2023 9:34 AM, Jan Ekström wrote:
> On Sat, Nov 4, 2023 at 11:22 AM Anton Khirnov <anton@khirnov.net> wrote:
>>
>> Hi,
>> after ~2 years of work and ~700+ preparatory commits, here is finally
>> the first "fully functional" version of multithreaded ffmpeg CLI. In
>> quotes because due to the scale of the changes I'm sure some things got
>> broken and I didn't notice - more testing very much welcome.
>>
>> One thing which is most definitely broken is the
>> -fix_sub_duration_heartbeat option, which in its current form makes
>> assumptions about synchronization between distant components
>> (encoders/muxers -> decoders), which makes it fundamentally
>> non-deterministic when each of these components runs asynchronously
>> (note that its behaviour is unpredictable even now, it's just
>> deterministic across runs with the same options). I'm currently
>> disabling the option in 16/24, better suggestions on what to do with it
>> are welcome.
> 
> Tested a version of this branch without the fix_sub_duration_heartbeat
> removal (https://github.com/jeeb/ffmpeg/commits/ffmpeg_threading_with_sub_heartbeat),
> and was expecting the functionality to be completely
> non-deterministic. Yet to my surprise a single GEN=1 update worked on
> all of the x86 devices from Thinkpad x230 to a gen3 Ryzen laptop. The
> threading seems to allow for the output to fall behind by some amount
> (in this case about ~200ms?), but the FATE test seems to be very
> deterministic with these components on the systems I could test and
> with THREADS set to 16/32/128, putting a load on the system etc. As a
> test I also switched the encoder to libx264, preset placebo and the
> only thing that affected was the amount of delay, but the test was
> still seemingly deterministic.
> 
> I would request people to test
> "fate-ffmpeg-fix_sub_duration_heartbeat" on something a bit more
> exotic from this branch, just to see if the same result is attained.
> If it is, then this specific test case would seem to be stable stable
> and not blocking the threading work.

Someone with access to the M1 fate machine could do this.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

      reply	other threads:[~2023-11-13 12:37 UTC|newest]

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-11-04  7:56 Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 01/24] lavf/mux: do not apply max_interleave_delta to subtitles Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 02/24] lavfi/af_amix: make sure the output does not depend on input ordering Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 03/24] lavc/8bps: fix exporting palette after 63767b79a570404628b2521b83104108b7b6884c Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 04/24] fftools/ffmpeg: move a few inline function into a new header Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 05/24] fftools/thread_queue: do not return elements for receive-finished streams Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 06/24] fftools/thread_queue: count receive-finished streams as finished Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 07/24] fftools/ffmpeg: rework keeping track of file duration for -stream_loop Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 08/24] fftools/ffmpeg_filter: remove an unnecessary sub2video_push_ref() call Anton Khirnov
2023-11-04 14:19   ` Nicolas George
2023-11-09 10:42     ` Anton Khirnov
2023-11-09 10:47       ` Nicolas George
2023-11-09 21:29         ` Anton Khirnov
2023-11-17  9:44           ` Nicolas George
2023-11-17 11:52             ` Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 09/24] fftools/ffmpeg_filter: track input/output index in {Input, Output}FilterPriv Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 10/24] fftools/ffmpeg_filter: move filtering to a separate thread Anton Khirnov
2023-11-04 19:54   ` Michael Niedermayer
2023-11-09 11:45     ` [FFmpeg-devel] [PATCH v2 " Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 11/24] fftools/ffmpeg_filter: buffer sub2video heartbeat frames like other frames Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 12/24] fftools/ffmpeg_filter: reindent Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 13/24] fftools/ffmpeg_mux: add muxing thread private data Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 14/24] fftools/ffmpeg_mux: move bitstream filtering to the muxer thread Anton Khirnov
2023-11-04 13:39   ` James Almer
2023-11-09 11:41     ` Anton Khirnov
2023-11-09 11:47       ` James Almer
2023-11-09 12:00         ` Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 15/24] fftools/ffmpeg_demux: switch from AVThreadMessageQueue to ThreadQueue Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 16/24] fftools/ffmpeg: disable -fix_sub_duration_heartbeat Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 17/24] fftools/ffmpeg_enc: move encoding to a separate thread Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 18/24] fftools/ffmpeg: add thread-aware transcode scheduling infrastructure Anton Khirnov
2023-11-04 13:53   ` James Almer
2023-11-09 11:06     ` Anton Khirnov
2023-11-04 18:44   ` Michael Niedermayer
2023-11-09 11:36     ` [FFmpeg-devel] [PATCH v2 " Anton Khirnov
2023-11-11 15:21       ` [FFmpeg-devel] [PATCH v3 " Anton Khirnov
2023-11-09 11:39     ` [FFmpeg-devel] [PATCH " Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 19/24] fftools/ffmpeg_demux: convert to the scheduler Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 20/24] fftools/ffmpeg_dec: " Anton Khirnov
2023-11-04 18:30   ` Michael Niedermayer
2023-11-11 15:24     ` Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 21/24] fftools/ffmpeg_filter: " Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 22/24] fftools/ffmpeg_enc: " Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 23/24] fftools/ffmpeg_mux: " Anton Khirnov
2023-11-04  7:56 ` [FFmpeg-devel] [PATCH 24/24] ffmpeg: switch to scheduler Anton Khirnov
2023-11-09 11:46 ` [FFmpeg-devel] [PATCH] ffmpeg CLI multithreading Anton Khirnov
2023-11-09 12:46   ` Paul B Mahol
2023-11-11 20:10 ` Michael Niedermayer
2023-11-17 10:46   ` Anton Khirnov
2023-11-13 12:34 ` Jan Ekström
2023-11-13 12:38   ` James Almer [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4eedc7f4-1307-458f-a6bb-98a70d027f69@gmail.com \
    --to=jamrial@gmail.com \
    --cc=ffmpeg-devel@ffmpeg.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \
		ffmpegdev@gitmailbox.com
	public-inbox-index ffmpegdev

Example config snippet for mirrors.


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git