探索 Metal 4

Hello and welcome. I’m Aaron and I’m excited to share the details of a big update to the Metal API.

Metal is Apple’s low-level graphics and compute API. It has powered multiple generations of complex applications, including the latest games like Cyberpunk 2077, as well as powerful pro apps.

Metal has become central to the way developers think about rendering and compute on Apple platforms.

Building on over a decade of experience Metal 4 takes the API to the next level, enabling developers to deliver the most demanding games and pro apps. Metal 4 is built with the next generations of games, graphics and compute apps in mind. It unlocks the full performance potential of Apple silicon, while also ensuring you’ll find it familiar and approachable if you’re coming from other graphics and compute APIs like DirectX.

Metal 4 is part of the same Metal framework that you may already have in your app and it’s supported by devices equipped with the Apple M1 and later, as well as the A14 Bionic and later. Metal 4 starts with an entirely new command structure with explicit memory management, and it changes the way resources are managed to enable richer and more complex visuals.

Shader compilation is quicker with more options so your app can reduce redundant compilation, and machine learning can now be integrated seamlessly with the rest of your Metal app.

New built-in solutions are also available via MetalFX to boost your app’s performance.

I’ll also show you how to get started with your Metal 4 adoption.

I'll start with how your app encodes and submits commands.

Metal is represented in the system by a Metal device, which the OS provides to your app.

Once you have a Metal device, you can create command queues to send work to the hardware. And command buffers that contain the work to be sent. Command encoders allow you to encode commands into those command buffers. You can take advantage of Metal 4 using the same Metal device that your app uses today.

Metal 4 provides a new but familiar model for encoding commands, and introduces new versions of the rest of these objects.

Those changes start with the new MTL4CommandQueue which your app can obtain using the Metal device. Metal 4 decouples command buffers from the queue that will use them, so your app requests for a MTL4CommandBuffer from the device as well. Command buffers are independent of one another, making it easy for your app to encode them in parallel. Apps take advantage of encoders for different types of commands including draws, dispatches, blits, and building acceleration structures.

Metal 4 consolidates existing command encoders. With the new unified compute encoder, your app can also manage blit and acceleration structure command encoding. This reduces the total number of encoders required. There’s a new MTL4RenderCommandEncoder as well. It features an attachment map that your app can use to map logical shader outputs to physical color attachments. Your app can configure the attachment map of a single render encoder with all the necessary color attachments, and can swap between them on the fly. This saves you the need to allocate additional render encoders, saving your app memory and unnecessary code. Your app can use any combination of the available encoder types to encode commands into command buffers. Command buffers are backed by memory. As your app encodes commands, data is written into this memory.

In Metal 4, this memory is managed by a MTL4CommandAllocator.

Use the device to create a command allocator to take direct control of your app’s command buffer memory use.

Memory management is essential to maximizing what modern apps can fit into their available system resources.

And those apps are using more resources than ever.

Your app manages resources differently in Metal 4.

Metal uses two fundamental types of resources. Metal buffers store generic data formatted by your app and Metal textures store image formatted data.

In the past, applications used fewer resources, sometimes even a single buffer and texture per object. And then applications added more textures to improve the level of surface details. And then, to introduce more variety in their rendering, they also added more geometry.

Modern applications continued the trend, and they pulled in more and more resources to support complex new use cases.

But while the number of resources increased, APIs only exposed a fixed set of resource bind points, to set the resources for each draw or dispatch.

Historically each draw would typically only use a few. For example, this bunny has just a texture and a buffer for its geometry. And then for each draw, each object can change buffers for geometry and textures to alter surface appearance.

However, as draw call counts increased and shaders became more complex, the impact of managing bind points per draw added to the CPU time.

So, applications changed to bindless, where the bound resources are moved to another buffer to store the bindings. That way, the app just needs to bind a single argument buffer for each object, or even the entire scene. And this means you can greatly reduce the number of bind points you need.

To help your app avoid paying for extra bind points, Metal 4 provides a new MTL4ArgumentTable type to store the binding points your app needs. Argument tables are specified for each stage on an encoder, but can be shared across stages. You create tables with a size based on the bind points that your application needs. For example, in the bindless case, the argument table just needs one buffer binding.

The GPU also needs to be able to access all of these resources. This is where residency comes in. Apple silicon provides you with a large, unified memory space. You can use it to store all your app’s resources, but Metal still needs to know what resources to make resident. In Metal 4, your app uses residency sets to specify the resources that Metal should make resident. This is to ensure they are accessible to the hardware.

Residency sets are easy to integrate into per frame encoding and committing of command buffers. You just need to make sure that the residency set contains all the required resources for the command buffer you commit. But since the residency set contents rarely change, populating the residency set can be done at app start up.

And once the set is created, you can add it to a Metal 4 command queue. All command buffers committed to that queue will then include those resources. If you need to make any updates, this should be a much lower cost at runtime. For applications which stream resources in and out on a separate thread, you can move the cost of updating the residency set to that thread and update the residency set in parallel with encoding.

A great example of a game that benefited from residency sets is Control Ultimate Edition. The Technical Publishing Director of Remedy Entertainment, Tuukka Taipalvesi, had this to say Control Ultimate Edition found residency sets easy to integrate. While separating resources into different residency sets based on use and managing resource residency on a background thread, we saw significant reductions in the overheads of managing residency and lower memory usage when ray-tracing is disabled.

Delivering beautiful games like Control requires more resources than ever. You may even require more memory than is available, especially when targeting a range of devices. To get the most out of the available memory, your app can dynamically control how resources use that memory.

This requires fine grained control of how memory is allocated to resources, since not all the resources are required at once.

Apps can adjust quality by controlling the resident level of detail to support the same content across a wider variety of devices. Your app can do this using placement sparse resources.

Metal 4 supports buffers and textures allocated as placement sparse resources.

These resources are allocated without any pages to store their data.

With placement sparse resources, pages come from a placement heap.

Your app allocates pages from the placement heap to provide storage for the resource contents.

With Metal 4's focus on concurrency by default, you need to ensure that updates to resources are synchronized. To simplify synchronization, Metal 4 introduces a Barrier API that provides low overhead, stage-to-stage synchronization that maps well to barriers in other APIs.

You can see barriers in action in the Metal 4 sample, “processing a texture in a compute function”. The app starts with a colored image, applies a compute shader that converts it to greyscale, and then renders that converted texture to the screen.

These steps have a dependency on the shared resource: the output of the texture processing. The sample ensures that resource writes and reads occur in the correct order, using Metal 4 barriers.

Without synchronization, these steps could run in any order, and that will either mean using the wrong texture contents or even worse, if the two steps overlap, that will result in a corrupted output.

To get the order right, the app uses a barrier.

Barriers work stage to stage. So you'll need to consider which stage each operation runs in, based on the stages in the encoder.

Processing the texture will execute on a compute command encoder as a dispatch stage operation.

And the render will be part of a render command encoder, which reads the texture in a fragment operation. So the barrier you need is a dispatch to fragment barrier, that waits for dispatch stage work to complete before letting any fragment work start. Using barriers effectively is important to achieve the best performance in your app, especially with a large number of resources.

In addition to resources, modern apps also manage a massive number of shaders. Those shaders need to be compiled before they can be sent to the hardware, to render and compute for your app. Shaders are written in the Metal Shading Language, and lowered to Metal IR. The IR is then compiled to GPU binaries that can be natively executed by the hardware. As the developer, you have control over when shader compilation occurs. The Metal device provides an interface to send shaders to the OS for compilation by the CPU.

Metal 4 manages compilation with dedicated compilation contexts. The new MTL4Compiler interface is now separate from the device. Your app uses the device to allocate the compiler interface. The interface provides clear, explicit control of when your app performs compilation on the CPU.

You can also leverage the MTL4Compiler to take advantage of scheduling improvements in the shader compilation stack. The MTL4Compiler inherits the quality of service class assigned to the thread requesting compilation.

When multiple threads compile at the same time, the OS will prioritize requests from higher priority threads to ensure your app’s most important shaders are processed first before moving on to other compilations.

Explicit control of shader compilation is important, since modern apps have more shaders than ever.

During pipeline state generation, your app must compile each shader the first time before GPU work can proceed.

Sometimes, pipelines share common Metal IR. For example, your app may apply different color states to render with differing transparency.

And that same situation may apply to other sets of pipelines as well. With Metal 4, you can optimize for this case, so you can reduce the time spent in shader compilation.

Render pipelines can now use flexible render pipeline states to build common Metal IR once.

This creates an unspecialized pipeline. Your app then specializes the pipeline with the intended color state. Metal will re-use the compiled Metal IR to efficiently create a specialized pipeline to execute.

Flexible render pipeline states save compilation time when re-using Metal IR across shader pipelines.

Your app creates an unspecialized pipeline once before specializing it for each of the necessary color states.

You can repeat this process for other pipelines that share Metal IR, and share the compilation results from each to reduce the time your app spends compiling shaders.

On-device compilation still costs CPU time.

The most performant path is still to eliminate on-device compilation entirely. Metal 4 streamlines the harvesting of pipeline configurations. For more information on how to take advantage of Metal 4’s compilation workflow, as well as more details on command encoding, follow-up and Explore Metal 4 games. Metal 4 makes it easier than ever to integrate machine learning, which opens up entirely new possibilities for your app.

Rendering techniques like upscaling, asset compression, animation blending, and neural shading, each benefit from the targeted application of machine learning. To apply these techniques efficiently, apps need to operate on complicated data sets and structures.

Buffers are flexible, but leave much of the heavy lifting to the app and textures aren’t a great fit.

Metal 4 adds support for tensors, a fundamental machine learning resource type, supported across all contexts.

Metal tensors are multi-dimensional data containers.

They are extendable well beyond two dimensions, providing the flexibility to describe the data layout necessary for practical machine learning usage.

Metal 4 integrates tensors directly into the API… as well as into the Metal shading language. Tensors offload the complex indexing into multidimensional data, so your Metal 4 app can focus on using them to apply machine learning in novel ways. To make that even easier, Metal 4 expands the set of supported command encoders.

The new machine learning command encoder enables you to execute large-scale networks directly within your Metal app.

Machine learning encoders function in a similar way to existing Metal encoder types.

Tensors are consumed as resources mapped into an argument table. Encoding is done to the same Metal 4 command buffers, and it supports barriers for synchronization with your other Metal 4 encoders.

Metal 4 machine learning encoders are compatible with networks expressed in the existing CoreML package format. Use the Metal toolchain to convert these into a Metal package, and then feed the network directly to your encoder. The new machine learning encoder is perfect for large networks that need command level interleaving with the rest of your Metal app. If you have smaller networks, Metal 4 also gives you the flexibility to integrate them directly into your existing shader pipelines. For example, neural material evaluation replaces traditional material textures with latent texture data. Your app samples those latent textures to create input tensors, performs inference using the sampled values, and uses the final output to perform shading.

Splitting each step into its own pipeline is inefficient, since each step needs to sync tensors from device memory, operate on them, and sync outputs back for later operations.

To get the best performance, your app should combine these steps into a single shader dispatch, so that each operation can share common thread memory. Implementing inference is a complex task, but Metal 4 has you covered with Metal performance primitives. Metal performance primitives are shader primitives designed to execute complex calculations, and they can natively operate on tensors. Each one is optimized to run blazingly fast on Apple silicon.

Tensor ops are perfect for embedding small networks into your shaders. Your app can take advantage of them as part of the Metal Shading Language, and when you do, the OS shader compiler inlines shader code, optimized for the device being used, directly into your shader.

To learn more about how to use Metal’s new machine learning capabilities, you can watch Combine Metal 4 machine learning and graphics.

Metal 4 provides you with all the tools you need to integrate cutting edge Machine Learning techniques. You can also take advantage of production-ready solutions built into MetalFX. Apple devices have incredible, high resolution screens that are perfect for showcasing your amazing games. MetalFX allows your app to deliver high resolutions at even greater refresh rates when rendering complex scenes with realistic reflections.

Rendering high resolution images can consume the GPU for a significant period of time. Instead, your app can render low resolution images and use MetalFX to upscale them. The combined time to render your final image is reduced and that means your app can save time for each frame it renders. You can use the time saved to render the next frame sooner. And if you want to hit the highest refresh rates without sacrificing quality, MetalFX has a great solution built right in.

This year, MetalFX adds support for frame interpolation.

Your app can use it to generate intermediate frames in much less time than it would take to render each frame from scratch. You can use those intermediate frames to achieve even higher frame rates.

Ray tracing is another technique that apps use to achieve realistic rendering results, by tracing rays from the camera to a light source. However if too few rays are cast, the resulting image will be too noisy to use.

MetalFX also now supports denoising during the upscale process, so your app can remove the noise from images rendered with fewer rays, and deliver full-size results. MetalFX helps you produce high quality results your players will love, at increased refresh rates. Your game can combine it with Metal 4 ray tracing to achieve incredible results. You can learn more about how to use these features in “Go further with Metal 4 games." Your app can combine all of these features to do incredible things. And Metal 4 is designed to make porting both approachable and modular. An app is composed of several key categories of functionality, including: how it compiles shaders, how it encodes and submits commands to the hardware, and finally, its management of resources. Each of these can be approached separately in their own distinct phases. Compilation is perhaps the easiest first step to take. You can allocate a Metal 4 compiler and insert it into your app's compilation flows to improve quality-of-service.

Once you’ve adopted the new compilation interface, your app can integrate flexible render pipelines to speed up render pipeline compilation or take advantage of harvesting improvements to improve ahead of time compilation.

Whether you adopt the new compiler or not, your app can also benefit from generating commands with Metal 4.

With Metal 4’s command encoding and generation model, you can take greater control of your memory allocations. You can also leverage native parallel encoding across encoder types to get encoding done quicker and Metal 4’s completely new set of machine learning capabilities unlock new possibilities for your rendering pipelines.

Your app can adopt the machine learning encoder, or shader ML, based on the network you want to integrate.

You can then take it a step further with Metal 4’s resource management.

Residency sets are an easy win. Integrate them to simplify the process of residency management. Barriers allow your app to synchronize resource access across different passes efficiently And placement sparse resources enable you to build resource streaming into your Metal app. As the developer, you are in the best position to judge how to make your app better. And Metal 4 gives you the flexibility to adopt the new functionality where you need it most. Placement sparse is a great example of a feature that enables a specific use case. Here’s how you can integrate it into your existing Metal app. Your Metal app already commits work using the existing Metal command queue while placement sparse mapping operations require a MTL4CommandQueue.

You can use a Metal event to synchronize work between your app’s Metal and MTL4CommandQueue.

The first signal event call unblocks the MTL4CommandQueue to update placement sparse resource mappings. A second signal event call notifies your app to continue the render work using the placement sparse resources. You can use the same Metal event from before.

Your app should submit work that doesn’t depend on those resources before waiting on the event to ensure the hardware remains fully utilized.

Metal comes with an advanced set of developer tools to help you debug and optimize your Metal apps. And this year, these same tools come with support for Metal 4.

API and Shader Validation save you valuable time by identifying common problems. and the comprehensive Metal Debugger helps you deep dive into your Metal 4 usage. The Metal performance HUD provides a real-time overlay to monitor your app’s performance, and Metal System Trace lets you dig into performance traces from your app.

You can learn about all of these tools and find their documentation on the Apple Developer website. Also, check out "Level up your games" for great techniques to debug and further optimize your games. Xcode 26 also comes with a new Metal 4 template built in. Start a new project and select Game templates before choosing Metal 4 as your game technology. With the built in Xcode template, it’s easy to get a basic render going and start your Metal 4 journey.

Now that you’ve discovered what Metal 4 can do, you’re ready to learn how to apply it to the needs of your app. If you’re developing a game, you can dive In and explore how to use Metal 4’s new command encoding and compilation features.

After that, you can learn how to go further and optimize your game with MetalFX, and discover how to take advantage of Metal 4 ray tracing. Or, you can learn how to use Metal 4 to combine Machine Learning and graphics in your Metal app. Metal 4 enables a new generation of apps and games with an incredible feature set. You’ve only just scratched the surface with the foundation you'll need to dive in. You can start using Metal 4 in your new or existing apps with the upcoming developer beta. The sample code is a great example of how to start your Metal 4 adoption.

It’s available now.

Thanks for watching!