From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id A004E47CC5 for ; Tue, 17 Oct 2023 22:12:00 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E4E1168CA0A; Wed, 18 Oct 2023 01:11:58 +0300 (EEST) Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com [209.85.167.53]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6813068C620 for ; Wed, 18 Oct 2023 01:11:52 +0300 (EEST) Received: by mail-lf1-f53.google.com with SMTP id 2adb3069b0e04-507a98517f3so3871279e87.0 for ; Tue, 17 Oct 2023 15:11:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697580711; x=1698185511; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=puGQJGIHfY5xCphlAFOX5ejKzV6bSAwbB4OV0hfuO3s=; b=D/wblsD3gTkqMDoTebQ6FoZuClYoHJnlUatyCr5KXw11ieFxoQ1NlRzTKY9vnd7hmq D4bF+3bZwYMpOB6+UmJaFNQDrpNetG85O8q+CAPkd4Opn/sIHM9Jnb/2OlOiEwgmufOs 9k/nUrrE4h8uewrRGIyXcSCnM6aPmASDefwwyF5FqZsXuUSVOfV0XI8Zk9kVNb9fW0W1 o+N7shBJyA1zvVyOM5RZycLqmj+wzKz5E4jZ+7qO+rZiEgW/t5VklTWRNX3oJEpX/3Wl dkA7vlcBIqTdll98+KJ+kk4qLS2PeU6aRQCi8K8RURw2AL1jfJG7WC9k5LDYov1hSR6K ebQQ== X-Gm-Message-State: AOJu0Ywf6evxFwbOHmBDzACa5x6+iHAlo/IifJ8F5Dh1+g/X/WQaY2Es jH9/kuU70+3IRSbxDd2m5sOxLtLV10FMq52EmRn9EyerqBA= X-Google-Smtp-Source: AGHT+IGMDE7Qq1n274SSOcl8To0PtiIPReG0u+rMbh+sxHEsIn6qZ7x59uoMNUaDQ1PGA19wHA34c7Jn/9GZNHg4yXU= X-Received: by 2002:ac2:530c:0:b0:500:b53f:fbc2 with SMTP id c12-20020ac2530c000000b00500b53ffbc2mr2593716lfh.26.1697580710330; Tue, 17 Oct 2023 15:11:50 -0700 (PDT) MIME-Version: 1.0 Date: Tue, 17 Oct 2023 15:11:38 -0700 Message-ID: To: FFmpeg development discussions and patches Subject: [FFmpeg-devel] [RFC] A file format to store generic raw image/video files X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Chema Gonzalez via ffmpeg-devel Reply-To: FFmpeg development discussions and patches Cc: Chema Gonzalez , cosmin@cosmin.at, Kyle Swanson Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: # RFC: A file format to store generic raw image/video files Context: Developers and researchers often want to use raw image/video files, as they avoid the effects of encoding. Dealing with raw image/video files is typically done using raw files. This is an operational pain point, as the metadata about the file needs to be carried out on the side. Raw video/image metadata includes (at least) pixel format (aka `pix_fmt`), width, height, framerate, and others. For example, in order to read a specific rgba file that I got from a shader, I need to write: ``` $ ffmpeg -y -f rawvideo -video_size 1920x1080 -pix_fmt rgba -i post_shader.1920x1080.rgba -vf scale="out_color_matrix=bt601" out.png ``` # Y4M: A YUV, Planar Raw File Format This problem has been partially addressed in [y4m](https://wiki.multimedia.cx/index.php/YUV4MPEG2). y4m is a raw (uncompressed), annotated image/video format for YUV content. It supports both images and video (a set of consecutive frames). It also supports framerates, interlaced/progressive content, aspect, location of subsampled chroma planes, and color ranges. The main limitation of y4m is that it only supports a reduced set of planar, 8-bit, YUV formats (444, 422, 420): yuv420p, yuv422p, and yuv444p. # R4M: A Generic Raw File Format We propose a new raw image/video format, R4M, that extends Y4M to support: * (a) every `pix_fmt` that ffmpeg supports. The implementation would support generic AVPixFmtDescriptor's, so any new pixel format based on it will be automatically supported. This includes planar/packed/semiplanar, yuv/rgb/bayer, and others. * (b) different bit sizes (this is part of `pix_fmt`), * (c) strides (pixel, row, plane), to allow devices to just dump their buffers directly, and * (d) flexible color information (h273, ICC, etc.). # Details R4M should support any raw image/video format that can be used by ffmpeg. R4M would keep the current Y4M features. * (1) support all y4m image settings * width * height * progressive/interlaced (top/bottom field first)/mixed mode * aspect ratio * chroma subsampling [implicit in `pix_fmt`] * chroma location * XYSCSS=420JPEG [this is the same than chroma subsampling + chroma location] * color range [implicit in color info] * (2) support all y4m video settings * framerates (as ratio) In terms of implementation, R4M would keep the Y4M format structure: text-based (readable) per-file and per-frame text headers, followed by explicitly-sized, per-frame, binary dumps of each frame. In terms of file format, we would keep the text-based header format (with a different header ID). As for the field list, we would add a human-readable dump of the `AVPixFmtDescriptor` struct, and remove the fields that are implicit in the `AVPixFmtDescriptor` dump (e.g. remove the "C" (colorspace) item). R4M would add the following new features: * (3) support every `pix_fmt` in ffmpeg (including planar/packed/semiplanar, yuv/rgb/bayer, etc.), and in general any `pix_fmt` representable using a `AVPixFmtDescriptor` struct. * planar/packed/semiplanar * yuv/rgb/bayer * (4) support different bitsizes [implicit on `pix_fmt`] * 8/9/10/12/14/16-bit color/other * little-endian/big-endian * float formats * (5) support image strides [implicit on `pix_fmt`] * pixel stride * row stride * plane stride * different values for different planes (e.g. Y, U, V, UV) * (6) support for flexible color information (a set of new fields). * h273: cp, tc, mc, range * ICC profile (restricted/unrestricted) * nclx (jpeg-xr color definition, ISO/IEC 29199-2/T.832) * (7) dynamic per-frame values * goal is to support videos where e.g. the resolution changes in the middle * support per-FRAME settings In terms of implementation, the suggestion is to write a thin library inside ffmpeg that can read from R4M files to ffmpeg buffers, and that can write from ffmpeg buffers to R4M files. An alternative is doing it as an external library. Rationale for in-ffmpeg library is this approach will make the code easier to land and better integrated with ffmpeg. In fact, this is how ffmpeg supports y4m right now (libavformat/yuv4mpeg* instead of any of the y4m libraries around). There is some value in an external library that supports it (e.g. I'd be interested in adding imlib2 support at some point, as I use it for quickly seeing images using feh), but I expect the source of truth to be what ffmpeg has. This is IMO the case for y4m. # Q&A Q1: Would this new format be acceptable in ffmpeg? If so, I could write a more detailed proposal with the R4M format, following the guidelines suggested here. Q2: Anybody has any other suggestion for a feature that should be added for a raw image/video file format? Q3: Anybody is interested in collaborating with the format? I would be more than happy to use a github repo for the format description. Thanks, -Chema _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".