[FFmpeg-devel] [GSoC][RFC] GSoC 2026 Proposal (350 Hours): High-Performance LibTorch Backend Modernization

[FFmpeg-devel] [GSoC][RFC] GSoC 2026 Proposal (350 Hours): High-Performance LibTorch Backend Modernization

[Raja Rathour via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 3361 bytes --]

*GSoC 2026 Proposal: High-Performance LibTorch Backend Modernization*

*Candidate:* Raja Rathour
*Project Type:* Large (350 Hours)
*Mentor:* Guo Yejun
*1. Problem Statement: Bridging the Integration Gap*

While the LibTorch backend logic was present in the source tree, it was
functionally inaccessible to end-users due to a registration mismatch in
the AVOption system. Specifically, the dnn_backend unit in
vf_dnn_processing.c lacked the necessary constants to map the user input
string "torch" to the internal DNN_TH backend ID. This caused the following
failure:

*The Error:*
[Parsed_dnn_processing_0 @ 0x...] Option 'dnn_backend' not found

*The Fix:*
I have already diagnosed and resolved this by correctly registering the
torch constant in the dnn_processing_options array and updating DnnContext
offsets.
*2. Current Progress & Functional Verification*

I have verified the end-to-end inference pipeline using a local build
(--enable-libtorch). The following terminal output serves as proof of
concept, showing successful 25-frame inference at 14.8x speed using the new
LibTorch integration:

*# Verified Command Line Proof*

./ffmpeg -f lavfi -i testsrc=duration=1 -vf
"dnn_processing=model=model.pt:dnn_backend=torch" -f null -
...
Stream mapping: Stream #0:0 -> #0:0 (wrapped_avframe -> wrapped_avframe)
frame= 25 fps=0.0 time=00:00:01.00 speed=14.8x

This confirms that the "plumbing" between FFmpeg's filtergraph and the
LibTorch engine (using at::from_blob for memory wrapping) is fully
operational.
[image: image.png]


*3. 350-Hour Technical Roadmap (12-Week Plan)*

The project scope has been expanded to 350 hours to prioritize
architectural modernization and high-performance GPU utilization:

• *Phase 1: Unified Async Infrastructure (Weeks 1–4):*
Migration to the common DNNAsyncExecModule. This ensures non-blocking
execution, bringing LibTorch into alignment with the OpenVINO and
TensorFlow backends.

• *Phase 2: Zero-Copy GPU Pipeline (Weeks 5–9 – HIGH PRIORITY):*
Developing direct mapping for AV_PIX_FMT_CUDA frames. By eliminating
redundant CPU-to-GPU memory copies, we can significantly reduce PCIe
bandwidth bottlenecks for hardware-accelerated filters.

• *Phase 3: Batch-Mode Inference & Refinement (Weeks 10–12 – Optional):*
Implementing frame-accumulation buffers for (B x C x H x W) processing to
maximize hardware throughput via tensor concatenation.
*4. Proven Track Record: Merged Contributions*

My proposal is built on a foundation of successful upstream contributions.
I have already submitted and merged patches that established the initial
infrastructure for this backend, including:

• Async Infrastructure Refactoring: Initial migration steps for the
LibTorch worker lifecycle.
• Memory Safety & Buffering: Implementation of persistent input buffers and
dynamic shape reallocation logic to prevent runtime overflows.
• Worker Management: Refined management of the inference request queue to
improve stability.
*5. Community Impact & Stability*

To ensure long-term maintainability, all performance features will be
implemented as opt-in parameters. I am committed to maintaining a
synchronous fallback path and contributing to the FATE test suite to verify
these new execution paths across different environments.

[-- Attachment #1.2: image.png --]
[-- Type: image/png, Size: 348397 bytes --]

[-- Attachment #2: Type: text/plain, Size: 163 bytes --]

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org

[FFmpeg-devel] Re: [GSoC][RFC] GSoC 2026 Proposal (350 Hours): High-Performance LibTorch Backend Modernization

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 2483 bytes --]

Hi Raja, Guo

On Sun, Jan 25, 2026 at 10:30:25PM +0530, Raja Rathour via ffmpeg-devel wrote:
> *GSoC 2026 Proposal: High-Performance LibTorch Backend Modernization*
> 
> *Candidate:* Raja Rathour
> *Project Type:* Large (350 Hours)
> *Mentor:* Guo Yejun

if guo agrees then someone should add this to
https://trac.ffmpeg.org/wiki/SponsoringPrograms/GSoC/2026

we need more projects listed on the above idea page ASAP because
google will look at this and say "wtf that page is nearly empty"



> *1. Problem Statement: Bridging the Integration Gap*
> 
> While the LibTorch backend logic was present in the source tree, it was
> functionally inaccessible to end-users due to a registration mismatch in
> the AVOption system. Specifically, the dnn_backend unit in
> vf_dnn_processing.c lacked the necessary constants to map the user input
> string "torch" to the internal DNN_TH backend ID. This caused the following
> failure:
> 
> *The Error:*
> [Parsed_dnn_processing_0 @ 0x...] Option 'dnn_backend' not found
> 
> *The Fix:*
> I have already diagnosed and resolved this by correctly registering the
> torch constant in the dnn_processing_options array and updating DnnContext
> offsets.
> *2. Current Progress & Functional Verification*
> 
> I have verified the end-to-end inference pipeline using a local build
> (--enable-libtorch). The following terminal output serves as proof of
> concept, showing successful 25-frame inference at 14.8x speed using the new
> LibTorch integration:
> 
> *# Verified Command Line Proof*
> 
> ./ffmpeg -f lavfi -i testsrc=duration=1 -vf
> "dnn_processing=model=model.pt:dnn_backend=torch" -f null -
> ...
> Stream mapping: Stream #0:0 -> #0:0 (wrapped_avframe -> wrapped_avframe)
> frame= 25 fps=0.0 time=00:00:01.00 speed=14.8x
> 
> This confirms that the "plumbing" between FFmpeg's filtergraph and the
> LibTorch engine (using at::from_blob for memory wrapping) is fully
> operational.
> [image: image.png]
> 
> 
> *3. 350-Hour Technical Roadmap (12-Week Plan)*

I do like the idea of improving libtorch support
I have to admit i probably never used FFmpeg with libtorch
but i used libtorch directly in another project i wrote.

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Take away the freedom of one citizen and you will be jailed, take away
the freedom of all citizens and you will be congratulated by your peers
in Parliament.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 163 bytes --]

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org

[FFmpeg-devel] Re: [GSoC][RFC] GSoC 2026 Proposal (350 Hours): High-Performance LibTorch Backend Modernization

[Raja Rathour via ffmpeg-devel]

>
> Hi Michael,
>
> Thank you for the positive feedback and for supporting the effort to
> modernize the LibTorch backend. I'm glad to hear you see the value in this
> project.
>
> I would be happy to add this to the GSoC 2026 ideas page immediately to
> ensure the organization's project list is well-represented for Google's
> review. However, I am encountering a "404 Not Found" error whenever I
> attempt to register an account at
> https://trac.ffmpeg.org/wiki/SponsoringPrograms/GSoC/2026
>
> Could you please advise if there is a specific procedure for obtaining a
> Trac account, or should I provide the Wiki-formatted text here for a
> maintainer to post on my behalf?
>
> Best regards,
>
> Raja Rathour
>
_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org

[FFmpeg-devel] Re: [GSoC][RFC] GSoC 2026 Proposal (350 Hours): High-Performance LibTorch Backend Modernization

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 1197 bytes --]

Hi Raja

On Mon, Jan 26, 2026 at 04:16:43PM +0530, Raja Rathour via ffmpeg-devel wrote:
> Hi Michael,
> Thank you for the positive feedback and for supporting the effort to
> modernize the LibTorch backend. I'm glad to hear you see the value in
> this project.
> I would be happy to add this to the GSoC 2026 ideas page immediately
> to ensure the organization's project list is well-represented for
> Google's review. However, I am encountering a "404 Not Found" error
> whenever I attempt to register an account at  https://
> trac.ffmpeg.org/wiki/SponsoringPrograms/GSoC/2026

the link works fine here


> Could you please advise if there is a specific procedure for
> obtaining a Trac account, or should I provide the Wiki-formatted text
> here for a maintainer to post on my behalf?

there is no special procedure

thx

PS: html mail does not work on this mailing list

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If you fake or manipulate statistics in a paper in physics you will never
get a job again.
If you fake or manipulate statistics in a paper in medicin you will get
a job for life at the pharma industry.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 163 bytes --]

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org