[FFmpeg-devel] [PATCH 00/16] NVidia Tegra hardware decoding backend

[averne <averne381@gmail.com>]

Hi all,

This patch series implements a hardware decoding backend for nvidia
Tegra devices, notably the Nintendo Switch. It was primarily written
for HorizonOS (Nintendo Switch OS), but also supports nvidia's
Linux4Tegra distro. As for hardware, all Tegras later than the X1
(T210) should be supported, although the patch does not implement
features that were added to subsequent revisions of multimedia
engines (eg. 12-bit HEVC). However, since I only own T210 devices
(Switch and jetson nano), I was not able to verify this.

The backend is essentially a userspace NVDEC driver, as due to the
OS design of the Switch, we cannot link to nvidia's system libraries.
It notably uses (sparse) hardware documentation released by nvidia
here: https://github.com/NVIDIA/open-gpu-doc/tree/master/classes/video.
It supports all codecs available in hardware (MPEG1/2/4, VC1, H264,
HEVC, VP8, VP9 and JPEG), with dynamic frequency scaling, and
hardware-accelerated frame transfer.

At the moment I'm submitting the series with some nvidia headers
pulled from various sources, but I do think they should rather be
put in nv-codec-headers, let me know.

The code was tested for memory bugs and leaks with valgrind and
asan on L4T. Some quick performance testing (decoding with -f null -)
showed results in line with official software, tested against the 
nvv4l2 backend that was posted here a while ago:
https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2020-June/263759.html.
Note that the numbers are skewed because frame transfer cannot be 
disabled in nvidia's backend.
- HEVC Main 10 @ 4k   (~80Mbps): nvtegra  79fps, nvv4l2  66fps
- HEVC Main 10 @ 1080p (~5Mbps): nvtegra 402fps, nvv4l2 229fps
- H264         @ 1080p (~3Mbps): nvtegra 286fps, nvv4l2 260fps

Several homebrew applications have been using this backend for some
time, with no bugs reported. As far as I'm aware, this is the
complete list of them:
- NXMP, a media player based on mpv: https://github.com/proconsule/nxmp
- WiliWili, a bilibili client: https://github.com/xfangfang/wiliwili
- Switchfin, a Jellyfin client: https://github.com/dragonflylee/switchfin
- Moonlight-Switch, a Moonlight client: https://github.com/XITRIX/Moonlight-Switch
- chiaki: https://git.sr.ht/~kkwong/chiaki/
- My own media player, unreleased at this time

Nintendo Switch support assumes a working devkitA64 homebrew
environment, instructions regarding setup can be found here:
https://devkitpro.org/wiki/devkitPro_pacman. The hwaccel can then be
configured by eg.:
```
source /opt/devkitpro/switchvars.sh && ./configure
--cross-prefix=aarch64-none-elf- --enable-cross-compile --arch=aarch64
--cpu=cortex-a57 --target-os=horizon --enable-pic --enable-gpl
--enable-nvtegra
```

It should probably be noted that NVDEC usage on discrete gpus is
very similar. As far as I know, the main difference is that the 
interfacing is done through the GPFIFO block (same block that 
manages the 3D engine), instead of host1x.

Thank you for your consideration.


averne (16):
  avutil/buffer: add helper to allocate aligned memory
  configure,avutil: add support for HorizonOS
  avutil: add ioctl definitions for tegra devices
  avutil: add hardware definitions for NVDEC, NVJPG and VIC
  avutil: add common code for nvtegra
  avutil: add nvtegra hwcontext
  hwcontext_nvtegra: add dynamic frequency scaling routines
  nvtegra: add common hardware decoding code
  nvtegra: add mpeg1/2 hardware decoding
  nvtegra: add mpeg4 hardware decoding
  nvtegra: add vc1 hardware decoding
  nvtegra: add h264 hardware decoding
  nvtegra: add hevc hardware decoding
  nvtegra: add vp8 hardware decoding
  nvtegra: add vp9 hardware decoding
  nvtegra: add mjpeg hardware decoding

 configure                      |   30 +
 libavcodec/Makefile            |   11 +
 libavcodec/h263dec.c           |    6 +
 libavcodec/h264_slice.c        |    6 +-
 libavcodec/h264dec.c           |    3 +
 libavcodec/hevcdec.c           |   17 +-
 libavcodec/hevcdec.h           |    2 +
 libavcodec/hwaccels.h          |   10 +
 libavcodec/hwconfig.h          |    2 +
 libavcodec/mjpegdec.c          |    6 +
 libavcodec/mpeg12dec.c         |   12 +
 libavcodec/mpeg4videodec.c     |    3 +
 libavcodec/nvtegra_decode.c    |  517 +++++++++
 libavcodec/nvtegra_decode.h    |   94 ++
 libavcodec/nvtegra_h264.c      |  506 +++++++++
 libavcodec/nvtegra_hevc.c      |  633 +++++++++++
 libavcodec/nvtegra_mjpeg.c     |  336 ++++++
 libavcodec/nvtegra_mpeg12.c    |  319 ++++++
 libavcodec/nvtegra_mpeg4.c     |  344 ++++++
 libavcodec/nvtegra_vc1.c       |  455 ++++++++
 libavcodec/nvtegra_vp8.c       |  334 ++++++
 libavcodec/nvtegra_vp9.c       |  665 ++++++++++++
 libavcodec/vc1dec.c            |    9 +
 libavcodec/vp8.c               |    6 +
 libavcodec/vp9.c               |   10 +-
 libavutil/Makefile             |    9 +
 libavutil/buffer.c             |   31 +
 libavutil/buffer.h             |    7 +
 libavutil/clb0b6.h             |  303 ++++++
 libavutil/clc5b0.h             |  436 ++++++++
 libavutil/cle7d0.h             |  129 +++
 libavutil/cpu.c                |    7 +
 libavutil/hwcontext.c          |    4 +
 libavutil/hwcontext.h          |    1 +
 libavutil/hwcontext_internal.h |    1 +
 libavutil/hwcontext_nvtegra.c  | 1045 ++++++++++++++++++
 libavutil/hwcontext_nvtegra.h  |   92 ++
 libavutil/nvdec_drv.h          | 1858 ++++++++++++++++++++++++++++++++
 libavutil/nvhost_ioctl.h       |  511 +++++++++
 libavutil/nvjpg_drv.h          |  189 ++++
 libavutil/nvmap_ioctl.h        |  451 ++++++++
 libavutil/nvtegra.c            | 1035 ++++++++++++++++++
 libavutil/nvtegra.h            |  258 +++++
 libavutil/nvtegra_host1x.h     |   94 ++
 libavutil/pixdesc.c            |    4 +
 libavutil/pixfmt.h             |    8 +
 libavutil/vic_drv.h            |  279 +++++
 47 files changed, 11085 insertions(+), 3 deletions(-)
 create mode 100644 libavcodec/nvtegra_decode.c
 create mode 100644 libavcodec/nvtegra_decode.h
 create mode 100644 libavcodec/nvtegra_h264.c
 create mode 100644 libavcodec/nvtegra_hevc.c
 create mode 100644 libavcodec/nvtegra_mjpeg.c
 create mode 100644 libavcodec/nvtegra_mpeg12.c
 create mode 100644 libavcodec/nvtegra_mpeg4.c
 create mode 100644 libavcodec/nvtegra_vc1.c
 create mode 100644 libavcodec/nvtegra_vp8.c
 create mode 100644 libavcodec/nvtegra_vp9.c
 create mode 100644 libavutil/clb0b6.h
 create mode 100644 libavutil/clc5b0.h
 create mode 100644 libavutil/cle7d0.h
 create mode 100644 libavutil/hwcontext_nvtegra.c
 create mode 100644 libavutil/hwcontext_nvtegra.h
 create mode 100644 libavutil/nvdec_drv.h
 create mode 100644 libavutil/nvhost_ioctl.h
 create mode 100644 libavutil/nvjpg_drv.h
 create mode 100644 libavutil/nvmap_ioctl.h
 create mode 100644 libavutil/nvtegra.c
 create mode 100644 libavutil/nvtegra.h
 create mode 100644 libavutil/nvtegra_host1x.h
 create mode 100644 libavutil/vic_drv.h

--
2.45.1

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".