Coverting CVPixelBuffer 2VUY to a Metal Texture

Question

Created Mar ’25

Replies 3

Boosts 0

Participants 2

I am working on a project for macOS where I am taking an AVCaptureSession's CVPixelBuffer and I need to convert it into a MTLTexture for rendering. On macOS the pixel format is 2vuy, there does not seem to be a clear format conversion while converting to a metal texture. I have been able to convert it to a texture but the color space seems to be off as it is rendering distorted colors with a double image.

I believe 2vuy is a single pane color space and I have tried to account for that, but I am unaware of what is off.

I have attached The CVPixelBuffer and The distorted MTLTexture along with a laundry list of errors.

On iOS my conversions are fine, it is only the macOS 2vuy pixel format that seems to have issues.

My code for the conversion is also attached.

If there are any suggestions or guidance on how to properly convert a 2vuy CVPixelBuffer to a MTLTexture I would greatly appreciate it.

Many Thanks

Conversion_Logs.txt

ConversionCode.swift

Answered by DTS Engineer in 833749022

@NeedleTails,

Thanks for the focused sample, there are a couple of things to note here:

You are receiving '2vuy' pixel buffers from AVCapture. This Core Video pixel format maps to MTLPixelFormat.bgrg422, not MTLPixelFormat.gbgr422.
Your kernel is written to handle subsample, but because you are using one of the "422" pixel formats, Metal takes care of this for you automatically! From MTLPixelFormat.h: "There is no implicit colorspace conversion from YUV to RGB, the shader will receive (Cr, Y, Cb, 1)."

So, what should your kernel look like instead? Something like this:

// When using .bgrg422, shader receives (Cr (x), Y (y), Cb (z), 1)
    
    float y = yuvTexture.read(gid).y; // Y <-> y
    float cb = yuvTexture.read(gid).z; // Cb <-> z
    float cr = yuvTexture.read(gid).x; // Cr <-> x
    
    // YCbCr to RGB conversion matrix taken from the ARKit (Metal) template project.
    const float4x4 ycbcrToRGBTransform = float4x4(
        float4(+1.0000f, +1.0000f, +1.0000f, +0.0000f),
        float4(+0.0000f, -0.3441f, +1.7720f, +0.0000f),
        float4(+1.4020f, -0.7141f, +0.0000f, +0.0000f),
        float4(-0.7010f, +0.5291f, -0.8860f, +1.0000f)
    );
    
    
    float4 rgba = ycbcrToRGBTransform * float4(y, cb, cr, 1.0);

    // Write the RGB values to the output texture
    rgbTexture.write(rgba, gid);

That should get you unblocked :)

-- Greg

Boost

Answer 1

DTS Engineer OP

Apple

Mar ’25

Hello @NeedleTails,

Could you provide a focused sample project that reproduces?

At a glance, I don't think you are handling the GBGR422 texture correctly in your kernel, but that's just a hunch. It would be best if I could examine a focused sample project.

-- Greg

0

Answer 2

NeedleTails OP

Apr ’25

Thank you for your prompt response Greg. I have provided a focused sample project as a link. Please look in the code for the methods in the PreviewViewRenderer, in particular convertYUVToRGB. Basically after the capture receives the sample buffer I feed it in to my swift concurrent environment via a stream and then I start processing frames on the GPU. Thanks in advance for your consideration.

Sample Project https://web.tresorit.com/l/YTeHU#Q8JUzXpd2KplKj-trr60-g

0

Answer 3

DTS Engineer OP

Apple

Apr ’25

Recommended

@NeedleTails,

Thanks for the focused sample, there are a couple of things to note here:

You are receiving '2vuy' pixel buffers from AVCapture. This Core Video pixel format maps to MTLPixelFormat.bgrg422, not MTLPixelFormat.gbgr422.
Your kernel is written to handle subsample, but because you are using one of the "422" pixel formats, Metal takes care of this for you automatically! From MTLPixelFormat.h: "There is no implicit colorspace conversion from YUV to RGB, the shader will receive (Cr, Y, Cb, 1)."

So, what should your kernel look like instead? Something like this:

// When using .bgrg422, shader receives (Cr (x), Y (y), Cb (z), 1)
    
    float y = yuvTexture.read(gid).y; // Y <-> y
    float cb = yuvTexture.read(gid).z; // Cb <-> z
    float cr = yuvTexture.read(gid).x; // Cr <-> x
    
    // YCbCr to RGB conversion matrix taken from the ARKit (Metal) template project.
    const float4x4 ycbcrToRGBTransform = float4x4(
        float4(+1.0000f, +1.0000f, +1.0000f, +0.0000f),
        float4(+0.0000f, -0.3441f, +1.7720f, +0.0000f),
        float4(+1.4020f, -0.7141f, +0.0000f, +0.0000f),
        float4(-0.7010f, +0.5291f, -0.8860f, +1.0000f)
    );
    
    
    float4 rgba = ycbcrToRGBTransform * float4(y, cb, cr, 1.0);

    // Write the RGB values to the output texture
    rgbTexture.write(rgba, gid);

That should get you unblocked :)

-- Greg

0