[FFmpeg-devel] [PATCH 0/5] Apple Spatial Metadata

[FFmpeg-devel] [PATCH 0/5] Apple Spatial Metadata

[Derek Buitenhuis]

So I am a bit hesitant to send this, because while some of the boxes are documented
officially by Apple, some of what are created by the Vision Pro / iPhones are not,
because the official PDF is outdated.

It's a call we have to make whether or not we want to push this, or hold out faith
Apple will eventually update their document.

I reverse engienered the rest of the boxes based on the what the AVFoundation APIs
expose for spatial files. See each commit message.

Example file: https://www.dropbox.com/scl/fi/g4b0q6c0e0xb9kwxcsvaa/IMG_1722.MOV?rlkey=az1qik442xut3jwq73jvcau9n&dl=0
Example stream from Apple: https://developer.apple.com/streaming/examples/

I can provide more if needed.

Anyone with an iPhone 15+ or Vision Pro can make some files, too, of course.

Some other notes:
  * I did not implement the 'must' box, as described in [1], since its only purpose
    seems to be to fail input files. I can add this as information-only if people want.
    I have not found any samples with this box, including files made by Apple.
  * Right now I am checking if the side data values are zero, as this is the best I could
    come up with, but comments welcome.
  * I like the idea that the 'dadj' box sits inside a 'cmfy' box, it seems very wholesome.

[1] https://developer.apple.com/av-foundation/HEVC-Stereo-Video-Profile.pdf

Derek Buitenhuis (5):
  avutil/spherical: Add more spherical types
  avutil/stereo3d: Fill out stereo info provided by Vision Pro files
  fftools/ffprobe: Print more Stereo 3D info from side data
  avformat/mov: Add support for exporting Video Extension Usage info
  avformat/mov: Add support for reading and exporting horizontal field
    of view

 fftools/ffprobe.c     |   8 ++
 libavformat/mov.c     | 308 ++++++++++++++++++++++++++++++++++++++++++
 libavutil/spherical.c |   3 +
 libavutil/spherical.h |  16 +++
 libavutil/stereo3d.c  |  52 +++++++
 libavutil/stereo3d.h  |  80 +++++++++++
 libavutil/version.h   |   2 +-
 7 files changed, 468 insertions(+), 1 deletion(-)

-- 
2.43.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 1/5] avutil/spherical: Add more spherical types

[Derek Buitenhuis]

These originate from the Apple Vision Pro, and are documented here:

    https://developer.apple.com/documentation/coremedia/cmprojectiontype

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
---
 libavutil/spherical.c |  3 +++
 libavutil/spherical.h | 16 ++++++++++++++++
 2 files changed, 19 insertions(+)

diff --git a/libavutil/spherical.c b/libavutil/spherical.c
index 800d3459a5..d78f98f945 100644
--- a/libavutil/spherical.c
+++ b/libavutil/spherical.c
@@ -57,6 +57,9 @@ static const char *const spherical_projection_names[] = {
     [AV_SPHERICAL_EQUIRECTANGULAR]      = "equirectangular",
     [AV_SPHERICAL_CUBEMAP]              = "cubemap",
     [AV_SPHERICAL_EQUIRECTANGULAR_TILE] = "tiled equirectangular",
+    [AV_SPHERICAL_HALF_EQUIRECTANGULAR] = "half equirectangular",
+    [AV_SPHERICAL_RECTANGULAR]          = "rectangular",
+    [AV_SPHERICAL_FISHEYE]              = "fisheye",
 };
 
 const char *av_spherical_projection_name(enum AVSphericalProjection projection)
diff --git a/libavutil/spherical.h b/libavutil/spherical.h
index 828ac836da..118fbcc3d1 100644
--- a/libavutil/spherical.h
+++ b/libavutil/spherical.h
@@ -66,6 +66,22 @@ enum AVSphericalProjection {
      * the position of the current video in a larger surface.
      */
     AV_SPHERICAL_EQUIRECTANGULAR_TILE,
+
+    /**
+     * Video frame displays as a 180 degree equirectangular projection.
+     */
+    AV_SPHERICAL_HALF_EQUIRECTANGULAR,
+
+    /**
+     * Video contentframe displays on a flat, rectangular 2D surface.
+     */
+    AV_SPHERICAL_RECTANGULAR,
+
+    /**
+     * Fisheye projection (Apple).
+     * See: https://developer.apple.com/documentation/coremedia/cmprojectiontype/fisheye
+     */
+    AV_SPHERICAL_FISHEYE,
 };
 
 /**
-- 
2.43.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/5] avutil/spherical: Add more spherical types

[James Almer]

On 6/10/2024 3:44 PM, Derek Buitenhuis wrote:
> These originate from the Apple Vision Pro, and are documented here:
> 
>      https://developer.apple.com/documentation/coremedia/cmprojectiontype
> 
> Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
> ---
>   libavutil/spherical.c |  3 +++
>   libavutil/spherical.h | 16 ++++++++++++++++
>   2 files changed, 19 insertions(+)
> 
> diff --git a/libavutil/spherical.c b/libavutil/spherical.c
> index 800d3459a5..d78f98f945 100644
> --- a/libavutil/spherical.c
> +++ b/libavutil/spherical.c
> @@ -57,6 +57,9 @@ static const char *const spherical_projection_names[] = {
>       [AV_SPHERICAL_EQUIRECTANGULAR]      = "equirectangular",
>       [AV_SPHERICAL_CUBEMAP]              = "cubemap",
>       [AV_SPHERICAL_EQUIRECTANGULAR_TILE] = "tiled equirectangular",
> +    [AV_SPHERICAL_HALF_EQUIRECTANGULAR] = "half equirectangular",
> +    [AV_SPHERICAL_RECTANGULAR]          = "rectangular",
> +    [AV_SPHERICAL_FISHEYE]              = "fisheye",
>   };
>   
>   const char *av_spherical_projection_name(enum AVSphericalProjection projection)
> diff --git a/libavutil/spherical.h b/libavutil/spherical.h
> index 828ac836da..118fbcc3d1 100644
> --- a/libavutil/spherical.h
> +++ b/libavutil/spherical.h
> @@ -66,6 +66,22 @@ enum AVSphericalProjection {
>        * the position of the current video in a larger surface.
>        */
>       AV_SPHERICAL_EQUIRECTANGULAR_TILE,
> +
> +    /**
> +     * Video frame displays as a 180 degree equirectangular projection.
> +     */
> +    AV_SPHERICAL_HALF_EQUIRECTANGULAR,
> +
> +    /**
> +     * Video contentframe displays on a flat, rectangular 2D surface.
> +     */
> +    AV_SPHERICAL_RECTANGULAR,

This should ideally be the enum with value 0, but until next major when 
such a change can happen, it would be IMO a good idea if you set 
spherical->projection to AV_SPHERICAL_RECTANGULAR in av_spherical_alloc().

> +
> +    /**
> +     * Fisheye projection (Apple).
> +     * See: https://developer.apple.com/documentation/coremedia/cmprojectiontype/fisheye
> +     */
> +    AV_SPHERICAL_FISHEYE,
>   };
>   
>   /**
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/5] avutil/spherical: Add more spherical types

[Derek Buitenhuis]

On 6/11/2024 7:44 PM, James Almer wrote:
> This should ideally be the enum with value 0, but until next major when 
> such a change can happen, it would be IMO a good idea if you set 
> spherical->projection to AV_SPHERICAL_RECTANGULAR in av_spherical_alloc().

I think setting it to 0 (AV_STEREO3D_2D) is correct rather than this (and is already
done), and I don't think we need to reorder.

I had discussed this with Vittorio and it was decided that it was good to disambiguate
between 2D (not stereoscopic) and coded rectangular.

- Derek

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/5] avutil/spherical: Add more spherical types

[Derek Buitenhuis]

On 6/14/2024 2:03 PM, Derek Buitenhuis wrote:
> I think setting it to 0 (AV_STEREO3D_2D) is correct rather than this (and is already
> done), and I don't think we need to reorder.
> 
> I had discussed this with Vittorio and it was decided that it was good to disambiguate
> between 2D (not stereoscopic) and coded rectangular.

Please disregard this, my brain got two enums mixed up. You're right, and I have made
this change locally.

- Derek
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 2/5] avutil/stereo3d: Fill out stereo info provided by Vision Pro files

[Derek Buitenhuis]

Based on what is in the files themselves, and what the API provides
to users.

URLs:
  * https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_heroeye
  * https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_stereocamerabaseline
  * https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_horizontaldisparityadjustment
  * https://developer.apple.com/documentation/coremedia/kcmformatdescriptionextension_horizontalfieldofview

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
---
 libavutil/stereo3d.c | 52 ++++++++++++++++++++++++++++
 libavutil/stereo3d.h | 80 ++++++++++++++++++++++++++++++++++++++++++++
 libavutil/version.h  |  2 +-
 3 files changed, 133 insertions(+), 1 deletion(-)

diff --git a/libavutil/stereo3d.c b/libavutil/stereo3d.c
index 9c29ab01b5..a40a9439bb 100644
--- a/libavutil/stereo3d.c
+++ b/libavutil/stereo3d.c
@@ -55,6 +55,18 @@ static const char * const stereo3d_type_names[] = {
     [AV_STEREO3D_COLUMNS]             = "interleaved columns",
 };
 
+static const char * const stereo3d_view_names[] = {
+    [AV_STEREO3D_VIEW_PACKED] = "packed",
+    [AV_STEREO3D_VIEW_LEFT]   = "left",
+    [AV_STEREO3D_VIEW_RIGHT]  = "right",
+};
+
+static const char * const stereo3d_primary_eye_names[] = {
+    [AV_PRIMARY_EYE_NONE]  = "none",
+    [AV_PRIMARY_EYE_LEFT]  = "left",
+    [AV_PRIMARY_EYE_RIGHT] = "right",
+};
+
 const char *av_stereo3d_type_name(unsigned int type)
 {
     if (type >= FF_ARRAY_ELEMS(stereo3d_type_names))
@@ -74,3 +86,43 @@ int av_stereo3d_from_name(const char *name)
 
     return -1;
 }
+
+const char *av_stereo3d_view_name(unsigned int view)
+{
+    if (view >= FF_ARRAY_ELEMS(stereo3d_view_names))
+        return "unknown";
+
+    return stereo3d_view_names[view];
+}
+
+int av_stereo3d_view_from_name(const char *name)
+{
+    int i;
+
+    for (i = 0; i < FF_ARRAY_ELEMS(stereo3d_view_names); i++) {
+        if (av_strstart(name, stereo3d_view_names[i], NULL))
+            return i;
+    }
+
+    return -1;
+}
+
+const char *av_stereo3d_primary_eye_name(unsigned int eye)
+{
+    if (eye >= FF_ARRAY_ELEMS(stereo3d_primary_eye_names))
+        return "unknown";
+
+    return stereo3d_primary_eye_names[eye];
+}
+
+int av_stereo3d_primary_eye_from_name(const char *name)
+{
+    int i;
+
+    for (i = 0; i < FF_ARRAY_ELEMS(stereo3d_primary_eye_names); i++) {
+        if (av_strstart(name, stereo3d_primary_eye_names[i], NULL))
+            return i;
+    }
+
+    return -1;
+}
diff --git a/libavutil/stereo3d.h b/libavutil/stereo3d.h
index 3aab959b79..6efbb09b32 100644
--- a/libavutil/stereo3d.h
+++ b/libavutil/stereo3d.h
@@ -158,6 +158,26 @@ enum AVStereo3DView {
     AV_STEREO3D_VIEW_RIGHT,
 };
 
+/**
+ * List of possible primary eyes.
+ */
+enum AVStereo3DPrimaryEye {
+    /**
+     * Neither eye.
+     */
+    AV_PRIMARY_EYE_NONE,
+
+    /**
+     * Left eye.
+     */
+    AV_PRIMARY_EYE_LEFT,
+
+    /**
+     * Right eye
+     */
+    AV_PRIMARY_EYE_RIGHT,
+};
+
 /**
  * Inverted views, Right/Bottom represents the left view.
  */
@@ -185,6 +205,28 @@ typedef struct AVStereo3D {
      * Determines which views are packed.
      */
     enum AVStereo3DView view;
+
+    /**
+     * Which eye is the primary eye when rendering in 2D.
+     */
+    enum AVStereo3DPrimaryEye primary_eye;
+
+    /**
+     * The distance between the centres of the lenses of the camera system,
+     * in micrometers. Zero if unset.
+     */
+    uint32_t baseline;
+
+    /**
+     * Relative shift of the left and right images, which changes the zero parallax plane.
+     * Range -10000 to 10000, mapped to -1.0 to 1.0. Zero if unset.
+     */
+    int32_t horizontal_disparity_adjustment;
+
+    /**
+     * Horizontal field of view in thousanths of a degree. Zero if unset.
+     */
+    uint32_t horizontal_field_of_view;
 } AVStereo3D;
 
 /**
@@ -222,6 +264,44 @@ const char *av_stereo3d_type_name(unsigned int type);
  */
 int av_stereo3d_from_name(const char *name);
 
+/**
+ * Provide a human-readable name of a given stereo3d view.
+ *
+ * @param type The input stereo3d view value.
+ *
+ * @return The name of the stereo3d view value, or "unknown".
+ */
+const char *av_stereo3d_view_name(unsigned int view);
+
+/**
+ * Get the AVStereo3DView form a human-readable name.
+ *
+ * @param name The input string.
+ *
+ * @return The AVStereo3DView value, or -1 if not found.
+ */
+int av_stereo3d_view_from_name(const char *name);
+
+/**
+ * Provide a human-readable name of a given stereo3d primary eye.
+ *
+ * @param type The input stereo3d primary eye value.
+ *
+ * @return The name of the stereo3d primary eye value, or "unknown".
+ */
+const char *av_stereo3d_primary_eye_name(unsigned int eye);
+
+/**
+ * Get the AVStereo3DPrimaryEye form a human-readable name.
+ *
+ * @param name The input string.
+ *
+ * @return The AVStereo3DPrimaryEye value, or -1 if not found.
+ */
+int av_stereo3d_primary_eye_from_name(const char *name);
+
+
+
 /**
  * @}
  */
diff --git a/libavutil/version.h b/libavutil/version.h
index 9d08d56884..bd4c5a2ab3 100644
--- a/libavutil/version.h
+++ b/libavutil/version.h
@@ -79,7 +79,7 @@
  */
 
 #define LIBAVUTIL_VERSION_MAJOR  59
-#define LIBAVUTIL_VERSION_MINOR  21
+#define LIBAVUTIL_VERSION_MINOR  22
 #define LIBAVUTIL_VERSION_MICRO 100
 
 #define LIBAVUTIL_VERSION_INT   AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \
-- 
2.43.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 3/5] fftools/ffprobe: Print more Stereo 3D info from side data

[Derek Buitenhuis]

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
---
 fftools/ffprobe.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/fftools/ffprobe.c b/fftools/ffprobe.c
index 2d38e5dfdc..082cec8a64 100644
--- a/fftools/ffprobe.c
+++ b/fftools/ffprobe.c
@@ -2544,6 +2544,14 @@ static void print_pkt_side_data(WriterContext *w,
             const AVStereo3D *stereo = (AVStereo3D *)sd->data;
             print_str("type", av_stereo3d_type_name(stereo->type));
             print_int("inverted", !!(stereo->flags & AV_STEREO3D_FLAG_INVERT));
+            print_str("view", av_stereo3d_view_name(stereo->view));
+            print_str("primary_eye", av_stereo3d_primary_eye_name(stereo->primary_eye));
+            if (stereo->baseline)
+                print_int("baseline", stereo->baseline);
+            if (stereo->horizontal_disparity_adjustment)
+                print_int("horizontal_disparity_adjustment", stereo->horizontal_disparity_adjustment);
+            if (stereo->horizontal_field_of_view)
+                print_int("horizontal_field_of_view", stereo->horizontal_field_of_view);
         } else if (sd->type == AV_PKT_DATA_SPHERICAL) {
             const AVSphericalMapping *spherical = (AVSphericalMapping *)sd->data;
             print_str("projection", av_spherical_projection_name(spherical->projection));
-- 
2.43.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 3/5] fftools/ffprobe: Print more Stereo 3D info from side data

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 1007 bytes --]

On Mon, Jun 10, 2024 at 07:44:05PM +0100, Derek Buitenhuis wrote:
> Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
> ---
>  fftools/ffprobe.c | 8 ++++++++
>  1 file changed, 8 insertions(+)

this needs a update to fate


TEST    matroska-spherical-mono
--- ./tests/ref/fate/matroska-spherical-mono	2024-06-11 16:53:26.034199997 +0200
+++ tests/data/fate/matroska-spherical-mono	2024-06-11 20:09:25.226770382 +0200
@@ -3,6 +3,8 @@
 side_data_type=Stereo 3D
 type=2D
 inverted=0
+view=packed
+primary_eye=none
 [/SIDE_DATA]
 [SIDE_DATA]
 side_data_type=Spherical Mapping
Test matroska-spherical-mono failed. Look at tests/data/fate/matroska-spherical-mono.err for details.
make: *** [tests/Makefile:311: fate-matroska-spherical-mono] Error 1

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The smallest minority on earth is the individual. Those who deny 
individual rights cannot claim to be defenders of minorities. - Ayn Rand

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 3/5] fftools/ffprobe: Print more Stereo 3D info from side data

[Derek Buitenhuis]

On 6/11/2024 7:09 PM, Michael Niedermayer wrote:
>  side_data_type=Stereo 3D
>  type=2D
>  inverted=0
> +view=packed
> +primary_eye=none

Hmm. This does make me think we need a better way to print 'view' and 'inverted' in 
FFprobe. Perhaps if type!=2D?

- Derek
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 3/5] fftools/ffprobe: Print more Stereo 3D info from side data

[Derek Buitenhuis]

On 6/11/2024 7:43 PM, Derek Buitenhuis wrote:
> Perhaps if type!=2D?

Changed locally to this and updated the test.

- Derek
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 4/5] avformat/mov: Add support for exporting Video Extension Usage info

[Derek Buitenhuis]

This box is provided by files created by the Apple Vision Pro, as well
as the iPhone 15+ when capture for Vision Pro is enabled.

The boxes are a mix of things documented by Apple in some PDFs, their
API docs, and reverse engineering. Ideally we will have a real spec
one day.

Links:
  * https://developer.apple.com/av-foundation/Stereo-Video-ISOBMFF-Extensions.pdf
  * https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_horizontaldisparityadjustment
  * https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_stereocamerabaseline
  * https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_heroeye

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
---
 libavformat/mov.c | 279 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 279 insertions(+)

diff --git a/libavformat/mov.c b/libavformat/mov.c
index 160e9626d7..b164bb0adb 100644
--- a/libavformat/mov.c
+++ b/libavformat/mov.c
@@ -6477,6 +6477,284 @@ static int mov_read_sv3d(MOVContext *c, AVIOContext *pb, MOVAtom atom)
     return 0;
 }
 
+static int mov_read_vexu_proj(MOVContext *c, AVIOContext *pb, MOVAtom atom)
+{
+    AVStream *st;
+    MOVStreamContext *sc;
+    int size;
+    uint32_t tag;
+    enum AVSphericalProjection projection;
+
+    if (c->fc->nb_streams < 1)
+        return 0;
+
+    st = c->fc->streams[c->fc->nb_streams - 1];
+    sc = st->priv_data;
+
+    if (atom.size != 16) {
+        av_log(c->fc, AV_LOG_ERROR, "Invalid size for proj box: %"PRIu64"\n", atom.size);
+        return AVERROR_INVALIDDATA;
+    }
+
+    size = avio_rb32(pb);
+    if (size != 16) {
+        av_log(c->fc, AV_LOG_ERROR, "Invalid size for prji box: %d\n", size);
+        return AVERROR_INVALIDDATA;
+    }
+
+    tag = avio_rl32(pb);
+    if (tag != MKTAG('p','r','j','i')) {
+        av_log(c->fc, AV_LOG_ERROR, "Invalid child box of proj box: 0x%08X\n", tag);
+        return AVERROR_INVALIDDATA;
+    }
+
+    avio_skip(pb, 1); // version
+    avio_skip(pb, 3); // flags
+
+    tag = avio_rl32(pb);
+    switch (tag) {
+    case MKTAG('r','e','c','t'):
+        projection = AV_SPHERICAL_RECTANGULAR;
+        break;
+    case MKTAG('e','q','u','i'):
+        projection = AV_SPHERICAL_EQUIRECTANGULAR;
+        break;
+    case MKTAG('h','e','q','u'):
+        projection = AV_SPHERICAL_HALF_EQUIRECTANGULAR;
+        break;
+    case MKTAG('f','i','s','h'):
+        projection = AV_SPHERICAL_FISHEYE;
+        break;
+    default:
+        av_log(c->fc, AV_LOG_ERROR, "Invalid projection type in prji box: 0x%08X\n", tag);
+        return AVERROR_INVALIDDATA;
+    }
+
+    sc->spherical = av_spherical_alloc(&sc->spherical_size);
+    if (!sc->spherical)
+        return AVERROR(ENOMEM);
+
+    sc->spherical->projection = projection;
+
+    return 0;
+}
+
+static int mov_read_eyes(MOVContext *c, AVIOContext *pb, MOVAtom atom)
+{
+    AVStream *st;
+    MOVStreamContext *sc;
+    int size, flags = 0;
+    int64_t remaining;
+    uint32_t tag, baseline = 0;
+    enum AVStereo3DView view = AV_STEREO3D_VIEW_PACKED;
+    enum AVStereo3DPrimaryEye primary_eye = AV_PRIMARY_EYE_NONE;
+    int32_t horizontal_disparity_adjustment = 0;
+
+    if (c->fc->nb_streams < 1)
+        return 0;
+
+    st = c->fc->streams[c->fc->nb_streams - 1];
+    sc = st->priv_data;
+
+    remaining = atom.size;
+    while (remaining > 0) {
+        size = avio_rb32(pb);
+        if (size < 8 || size > remaining ) {
+            av_log(c->fc, AV_LOG_ERROR, "Invalid child size in eyes box\n");
+            return AVERROR_INVALIDDATA;
+        }
+
+        tag = avio_rl32(pb);
+        switch (tag) {
+        case MKTAG('s','t','r','i'): {
+            int has_right, has_left;
+            uint8_t tmp;
+            if (size != 13) {
+                av_log(c->fc, AV_LOG_ERROR, "Invalid size of stri box: %d\n", size);
+                return AVERROR_INVALIDDATA;
+            }
+            avio_skip(pb, 1); // version
+            avio_skip(pb, 3); // flags
+
+            tmp = avio_r8(pb);
+
+            // eye_views_reversed
+            if (tmp & 8) {
+                flags |= AV_STEREO3D_FLAG_INVERT;
+            }
+            // has_additional_views
+            if (tmp & 4) {
+                // skip...
+            }
+
+            has_right = tmp & 2; // has_right_eye_view
+            has_left = tmp & 1; // has_left_eye_view
+
+            if (has_left && has_right)
+                view = AV_STEREO3D_VIEW_PACKED;
+            else if (has_left)
+                view = AV_STEREO3D_VIEW_LEFT;
+            else if (has_right)
+                view = AV_STEREO3D_VIEW_RIGHT;
+            break;
+        }
+        case MKTAG('h','e','r','o'): {
+            int tmp;
+            if (size != 13) {
+                av_log(c->fc, AV_LOG_ERROR, "Invalid size of hero box: %d\n", size);
+                return AVERROR_INVALIDDATA;
+            }
+            avio_skip(pb, 1); // version
+            avio_skip(pb, 3); // flags
+
+            tmp = avio_r8(pb);
+            if (tmp == 0)
+                primary_eye = AV_PRIMARY_EYE_NONE;
+            else if (tmp == 1)
+                primary_eye = AV_PRIMARY_EYE_LEFT;
+            else if (tmp == 2)
+                primary_eye = AV_PRIMARY_EYE_RIGHT;
+            else
+                av_log(c->fc, AV_LOG_WARNING, "Unknown hero eye type: %d\n", tmp);
+
+            break;
+        }
+        case MKTAG('c','a','m','s'): {
+            uint32_t subtag;
+            int subsize;
+            if (size != 24) {
+                av_log(c->fc, AV_LOG_ERROR, "Invalid size of cams box: %d\n", size);
+                return AVERROR_INVALIDDATA;
+            }
+
+            subsize = avio_rb32(pb);
+            if (subsize != 16) {
+                av_log(c->fc, AV_LOG_ERROR, "Invalid size of blin box: %d\n", size);
+                return AVERROR_INVALIDDATA;
+            }
+
+            subtag = avio_rl32(pb);
+            if (subtag != MKTAG('b','l','i','n')) {
+                av_log(c->fc, AV_LOG_ERROR, "Expected blin box, got 0x%08X\n", subtag);
+                return AVERROR_INVALIDDATA;
+            }
+
+            avio_skip(pb, 1); // version
+            avio_skip(pb, 3); // flags
+
+            baseline = avio_rb32(pb);
+
+            break;
+        }
+        case MKTAG('c','m','f','y'): {
+            uint32_t subtag;
+            int subsize;
+            if (size != 24) {
+                av_log(c->fc, AV_LOG_ERROR, "Invalid size of cmfy box: %d\n", size);
+                return AVERROR_INVALIDDATA;
+            }
+
+            subsize = avio_rb32(pb);
+            if (subsize != 16) {
+                av_log(c->fc, AV_LOG_ERROR, "Invalid size of dadj box: %d\n", size);
+                return AVERROR_INVALIDDATA;
+            }
+
+            subtag = avio_rl32(pb);
+            if (subtag != MKTAG('d','a','d','j')) {
+                av_log(c->fc, AV_LOG_ERROR, "Expected dadj box, got 0x%08X\n", subtag);
+                return AVERROR_INVALIDDATA;
+            }
+
+            avio_skip(pb, 1); // version
+            avio_skip(pb, 3); // flags
+
+            horizontal_disparity_adjustment = (int32_t) avio_rb32(pb);
+
+            break;
+        }
+        default:
+            av_log(c->fc, AV_LOG_WARNING, "Unknown tag in eyes: 0x%08X\n", tag);
+            avio_skip(pb, size - 8);
+            break;
+        }
+        remaining -= size;
+    }
+
+    if (remaining != 0) {
+        av_log(c->fc, AV_LOG_ERROR, "Broken eyes box\n");
+        return AVERROR_INVALIDDATA;
+    }
+
+    if (!sc->stereo3d) {
+        sc->stereo3d = av_stereo3d_alloc();
+        if (!sc->stereo3d)
+            return AVERROR(ENOMEM);
+    }
+
+    sc->stereo3d->flags                           = flags;
+    sc->stereo3d->view                            = view;
+    sc->stereo3d->primary_eye                     = primary_eye;
+    sc->stereo3d->baseline                        = baseline;
+    sc->stereo3d->horizontal_disparity_adjustment = horizontal_disparity_adjustment;
+
+    return 0;
+}
+
+static int mov_read_vexu(MOVContext *c, AVIOContext *pb, MOVAtom atom)
+{
+    int size;
+    int64_t remaining;
+    uint32_t tag;
+
+    if (c->fc->nb_streams < 1)
+        return 0;
+
+    if (atom.size < 8) {
+        av_log(c->fc, AV_LOG_ERROR, "Empty video extension usage box\n");
+        return AVERROR_INVALIDDATA;
+    }
+
+    remaining = atom.size;
+    while (remaining > 0) {
+        size = avio_rb32(pb);
+        if (size < 8 || size > remaining ) {
+            av_log(c->fc, AV_LOG_ERROR, "Invalid child size in vexu box\n");
+            return AVERROR_INVALIDDATA;
+        }
+
+        tag = avio_rl32(pb);
+        switch (tag) {
+        case MKTAG('p','r','o','j'): {
+            MOVAtom proj = { tag, size - 8 };
+            int ret = mov_read_vexu_proj(c, pb, proj);
+            if (ret < 0)
+                return ret;
+            break;
+        }
+        case MKTAG('e','y','e','s'): {
+            MOVAtom eyes = { tag, size - 8 };
+            int ret = mov_read_eyes(c, pb, eyes);
+            if (ret < 0)
+                return ret;
+            break;
+        }
+        default:
+            av_log(c->fc, AV_LOG_WARNING, "Unknown tag in vexu: 0x%08X\n", tag);
+            avio_skip(pb, size - 8);
+            break;
+        }
+        remaining -= size;
+    }
+
+    if (remaining != 0) {
+        av_log(c->fc, AV_LOG_ERROR, "Broken vexu box\n");
+        return AVERROR_INVALIDDATA;
+    }
+
+    return 0;
+}
+
 static int mov_parse_uuid_spherical(MOVStreamContext *sc, AVIOContext *pb, size_t len)
 {
     int ret = 0;
@@ -8595,6 +8873,7 @@ static const MOVParseTableEntry mov_default_parse_table[] = {
 { MKTAG('d','f','L','a'), mov_read_dfla },
 { MKTAG('s','t','3','d'), mov_read_st3d }, /* stereoscopic 3D video box */
 { MKTAG('s','v','3','d'), mov_read_sv3d }, /* spherical video box */
+{ MKTAG('v','e','x','u'), mov_read_vexu }, /* video extension usage */
 { MKTAG('d','O','p','s'), mov_read_dops },
 { MKTAG('d','m','l','p'), mov_read_dmlp },
 { MKTAG('S','m','D','m'), mov_read_smdm },
-- 
2.43.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 4/5] avformat/mov: Add support for exporting Video Extension Usage info

[James Almer]

On 6/10/2024 3:44 PM, Derek Buitenhuis wrote:
> +static int mov_read_vexu(MOVContext *c, AVIOContext *pb, MOVAtom atom)
> +{
> +    int size;
> +    int64_t remaining;
> +    uint32_t tag;
> +
> +    if (c->fc->nb_streams < 1)
> +        return 0;
> +
> +    if (atom.size < 8) {
> +        av_log(c->fc, AV_LOG_ERROR, "Empty video extension usage box\n");
> +        return AVERROR_INVALIDDATA;
> +    }
> +
> +    remaining = atom.size;
> +    while (remaining > 0) {

Maybe this loop should call mov_read_default, with proj and eyes added 
to mov_default_parse_table[]. Although i don't know if eyes may show up 
as child for other parent boxes or not.
At least with proj, i see it can be a child for sv3d, where only prhd is 
expected as a child box in turn. But it shouldn't a problem to add a 
mov_read_proj that handles both prhd and prji for this purpose.

> +        size = avio_rb32(pb);
> +        if (size < 8 || size > remaining ) {
> +            av_log(c->fc, AV_LOG_ERROR, "Invalid child size in vexu box\n");
> +            return AVERROR_INVALIDDATA;
> +        }
> +
> +        tag = avio_rl32(pb);
> +        switch (tag) {
> +        case MKTAG('p','r','o','j'): {
> +            MOVAtom proj = { tag, size - 8 };
> +            int ret = mov_read_vexu_proj(c, pb, proj);
> +            if (ret < 0)
> +                return ret;
> +            break;
> +        }
> +        case MKTAG('e','y','e','s'): {
> +            MOVAtom eyes = { tag, size - 8 };
> +            int ret = mov_read_eyes(c, pb, eyes);
> +            if (ret < 0)
> +                return ret;
> +            break;
> +        }
> +        default:
> +            av_log(c->fc, AV_LOG_WARNING, "Unknown tag in vexu: 0x%08X\n", tag);
> +            avio_skip(pb, size - 8);
> +            break;
> +        }
> +        remaining -= size;
> +    }
> +
> +    if (remaining != 0) {
> +        av_log(c->fc, AV_LOG_ERROR, "Broken vexu box\n");
> +        return AVERROR_INVALIDDATA;
> +    }
> +
> +    return 0;
> +}
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 4/5] avformat/mov: Add support for exporting Video Extension Usage info

[Derek Buitenhuis]

On 6/10/2024 8:38 PM, James Almer wrote:
>> +    remaining = atom.size;
>> +    while (remaining > 0) {
> 
> Maybe this loop should call mov_read_default, with proj and eyes added 
> to mov_default_parse_table[]. Although i don't know if eyes may show up 
> as child for other parent boxes or not.
> At least with proj, i see it can be a child for sv3d, where only prhd is 
> expected as a child box in turn. But it shouldn't a problem to add a 
> mov_read_proj that handles both prhd and prji for this purpose.

Well, the proj box that that's in sv3d and this proj box are unrelated -
they both exist because Apple and Google don't play nice with eachother,
and both defined their own incompatible projection boxes, which exist in
entirely different parts of the file, so it seemed kind of wrong to handle
it that way...

The other reason I left it this way is because it left it easier for us to
implement the 'must' box if we wanted to, which needs to keep track of what
boxes have been seen inside the 'vexu' box, and its inner boxes.

To my knowledge eyes (and indeed all the other boxes inside the 'vexu' box besides
the 'proj' box) only appear in the 'vexu' box.

- Derek
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 5/5] avformat/mov: Add support for reading and exporting horizontal field of view

[Derek Buitenhuis]

These boxes are created by the Apple Vision Pro and the iPhone 15+ when
capture for the Vision Pro is enabled.

Based off of the swift API:
  * https://developer.apple.com/documentation/coremedia/kcmformatdescriptionextension_horizontalfieldofview

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
---
 libavformat/mov.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/libavformat/mov.c b/libavformat/mov.c
index b164bb0adb..a118ade089 100644
--- a/libavformat/mov.c
+++ b/libavformat/mov.c
@@ -6755,6 +6755,34 @@ static int mov_read_vexu(MOVContext *c, AVIOContext *pb, MOVAtom atom)
     return 0;
 }
 
+static int mov_read_hfov(MOVContext *c, AVIOContext *pb, MOVAtom atom)
+{
+    AVStream *st;
+    MOVStreamContext *sc;
+
+    if (c->fc->nb_streams < 1)
+        return 0;
+
+    st = c->fc->streams[c->fc->nb_streams - 1];
+    sc = st->priv_data;
+
+    if (atom.size != 4) {
+         av_log(c->fc, AV_LOG_ERROR, "Invalid size of hfov box: %"PRIu64"\n", atom.size);
+         return AVERROR_INVALIDDATA;
+    }
+
+
+    if (!sc->stereo3d) {
+        sc->stereo3d = av_stereo3d_alloc();
+        if (!sc->stereo3d)
+            return AVERROR(ENOMEM);
+    }
+
+    sc->stereo3d->horizontal_field_of_view = avio_rb32(pb);
+
+    return 0;
+}
+
 static int mov_parse_uuid_spherical(MOVStreamContext *sc, AVIOContext *pb, size_t len)
 {
     int ret = 0;
@@ -8874,6 +8902,7 @@ static const MOVParseTableEntry mov_default_parse_table[] = {
 { MKTAG('s','t','3','d'), mov_read_st3d }, /* stereoscopic 3D video box */
 { MKTAG('s','v','3','d'), mov_read_sv3d }, /* spherical video box */
 { MKTAG('v','e','x','u'), mov_read_vexu }, /* video extension usage */
+{ MKTAG('h','f','o','v'), mov_read_hfov },
 { MKTAG('d','O','p','s'), mov_read_dops },
 { MKTAG('d','m','l','p'), mov_read_dmlp },
 { MKTAG('S','m','D','m'), mov_read_smdm },
-- 
2.43.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".