BNNS Graph의 새로운 기능

Hi, my name is Simon Gladman, and I work for the Vector and Numerics Group here at Apple. Our group provides a suite of libraries for high-performance, energy-efficient computation on the CPU. These libraries support, among other things, image and signal processing, linear algebra, and what I’ll be talking about today, machine learning.

Basic Neural Network Subroutines, or BNNS, is our machine learning library. It allows you to add CPU-based inference to your app, and is perfectly suited to real-time and low-latency use cases, such as audio processing. You might use it to implement functionalities such as: separating audio to isolate or remove vocals, segmenting audio into different regions based on content, or applying timbre transfer to make one instrument sound like another.

But BNNS is also perfectly suited to working with other data, such as images. So if your app requires high performance inference, BNNS is for you.

Last year we introduced BNNSGraph, an exciting new API that made BNNS faster, more energy efficient and far far easier to work with. And if you recall we demonstrated BNNSGraph by creating this Bitcrusher audio unit that shows just how easy it is to add your own effects to Logic Pro and Garage Band.

This year, we’re giving BNNSGraph an even better programming interface that allows you to leverage the power of Swift to create small models for inference and graphs of operations for pre- and post-processing. Today, I’ll be talking about our new addition to BNNSGraph, BNNSGraphBuilder. I’ll start today’s session with a quick recap of BNNSGraph. What it is, how it can optimize machine learning and AI models and how you can adopt it. Then I’ll introduce BNNSGraphBuilder. I’ll summarize the super easy workflow you need to implement it and as part of this introduction, I’ll demonstrate how to write a simple graph. After introducing BNNSGraphBuilder, I run through three demonstrations. Firstly preprocessing an image using the Graph Builder. Then using the new API to postprocess data generated by an ML model. And finally I’ll update last year’s Bitcrusher demonstration to build the graph of operations in Swift. And now, without further ado, let’s get started. We recommend BNNSGraph for audio and other latency-sensitive use cases because it gives you control over allocating memory and multithreading. Both of these may incur a context switch into kernel code and lead to defeat of real-time deadlines.

Before last year, you would build BNNS based networks using discrete layers such as convolution, matrix multiplication or activation.

If you wanted to use BNNS to implement an existing model, you’d have to code each layer as a BNNS primitive and write the code for all of the intermediate tensors. BNNSGraph takes an entire graph that consists of multiple layers and the data flow between those layers, and consumes it as a single graph object.

For you, this means you don’t have to write the code for each individual layer. Furthermore, you and your users will benefit from faster performance and better energy efficiency.

Because BNNSGraph is aware of the entire model, it is able to perform a number of optimizations that were not possible before. These optimizations reduce execution time, memory usage and energy cost. And even better, they come for free. Let’s use this small section of a model to explore some of these optimizations. One of the optimizations is a mathematical transformation. For example, reordering a slice operation so that BNNS only needs to compute the subset of elements in that slice. Another optimization is layer fusion. For example, fusing a convolution layer and an activation layer into a single operation. And copy elision avoids copying the data in a slice by referencing that subset of data instead. And by ensuring tensors share memory where possible, BNNSGraph optimizes memory usage and eliminates unnecessary allocations.

Finally, BNNSGraph’s weight repacking optimizations can repack weights to provide better cache locality. You don’t need to write any code to benefit from these optimizations. They happen just like that. To create a graph using the file-based API that we introduced last year, you start off with a CoreML Package. Xcode automatically compiles the package to an mlmodelc file and then you write the code that builds a graph from the mlmodelc file. The last step is to create a context to wrap the graph and it’s that context that performs inference. This approach is still the best way to integrate existing PyTorch models into your project. However, for small models or graphs of operations, you might benefit from a more immediate workflow. What if you could define the individual elements of your graph in Swift? Well, this year we’re introducing a new API that does just that BNNSGraphBuilder.

The new BNNSGraphBuilder API enables you to write graphs of operations using the familiar Swift language. You can write pre- and post-processing routines or small machine learning models directly in Swift. You create a context from that Swift code with a single function call and no intermediate steps. Writing your graphs in Swift and inline with your other code brings some immediate benefits. You use a familiar language and your graph is type-checked when your Swift project is compiled. You can also share values between Swift and your graph that are known at runtime but before you generate the graph. For example, if you know a tensor’s shape at runtime before you initialise the graph, you can pass the shape directly into the graph code, and benefit from the better performance of static sizes over flexible sizes.

BNNSGraphBuilder also allows you to query intermediate tensors for properties such as shape and data type, and this helps you to debug graphs. And type tensors allows Xcode to autocomplete and reduces the chance of runtime errors. Let’s look at the new syntax, type checking, new methods and operators, and how to query intermediate tensors, all features that were not previously available.

This year we’ve added a new type method to the BNNSGraph object, named makeContext. It is this new method that converts your Swift code to a reusable context that you use to execute your graph. You only make the context once, typically while your app is starting up. Then your app executes the context when required, and it benefits from the optimizations that treating the graph holistically brings. Now for some live coding to see makeContext in action.

This method accepts a closure, and it’s that closure that you use to define the series of operations that constitute your graph.

Your graph will typically accept one or more arguments. Here, I’ll use the closure’s builder parameter to specify the two arguments, x and y.

The parameters are both eight element vectors of float values. In this example the graph performs an element-wise multiplication on the two input arguments, and writes the result to the output argument that I have named product. My graph also calculates the mean value of the product and writes that value to a second output argument I have named mean, and finally, the code returns those two output arguments.

To aid debugging, we can print out details of the intermediate and output tensors that BNNSGraphBuilder generates.

In this example, I’ve printed the shape of the mean to ensure it’s got a shape of one, that is, it only contains one element. Once the context is created, you can query it to derive the arguments and their positions. Here, the code queries the context’s argument names and creates an array of tensors that represent the graph’s outputs and inputs. BNNSGraph orders argument names so that the outputs are first and the inputs follow.

The BNNSGraphBuilder API provides a rich set of functions for initializing arguments from different data sources, and either copying or referencing data. In this example, the code initializes the inputs from Swift arrays.

With the input and output arguments allocated, calling execute function executes the graph and populates the two outputs with the results.

And we can print out the results by running the function.

We’re not limited to just multiplying and calculating averages though. Let’s take a quick tour around BNNSGraphBuilder and explore some of the other operations that the new API provides. Here’s just a tiny selection of the primitives included in BNNSGraphBuilder. For example, matrix multiplication and convolution, reduction operations, gather and scatter operations, and operations such as padding, reshape and transpose. Our new API supports many operations as simple, Swift operators. So we have arithmetic operators, multiplication, addition, subtraction and division. We also have element-wise comparison operators including equality, less than, and greater than. And to complete the set, we have element-wise logical operators. OK, that’s a quick introduction to the API itself. Let’s take a deep dive into two features of the GraphBuilder API that will be very familiar Swift developers, starting with strong typing.

Strong typing helps you catch errors at compile time that otherwise may happen at runtime. The BNNSGraphBuilder API ensures that the data type for tensors is correct for a given operation. Let's look at this in action. In this example, the graph returns the element-wise results of floating-point base values raised to the power of integer exponents.

It’s able to perform this calculation by casting the integer exponents to FP16.

Attempting to perform this without the cast would prevent the make context method from compiling. The graph also masks the result by zeroing out elements where the values in mask 0 are less than the corresponding values in mask 1.

Because the tensor elements that the comparison generates are Boolean, the graph performs another cast in order to multiply the results by the condition. Once again, without the cast operation, the make context method wouldn’t compile. And it’s always best to catch this type of error at compile time and before your app is in the hands of your users. That's strong typing in action. Let’s now take a look at BNNSGraphBuilder’s approach to slicing, that is selecting parts with tensor. And this too will be very familiar to Swift developers.

A slice is effectively a window onto a specific part of a tensor. For example, a slice may be a single column or row of a matrix. The great thing about BNNSGraph is that it treats slices as references to existing data, without the overhead of copying that data or allocating more memory. Slicing tensors is a common operation. Our new GraphBuilder API allows you to specify slices of a tensor as the source or destination operations.

For example, you might want to crop an image to a region of interest before passing it to a machine learning model. Let’s look at how easy it is to use the GraphBuilder API to select a square region from the centre of this photograph of a squirrel enjoying its lunch.

We’ll define two vImage pixel buffers. Pixel buffers store in images pixel data, dimensions, bit depth and number of channels. The first buffer, source, contains a photograph of the squirrel. The second pixel buffer, Destination, will contain the square crop. Each pixel buffer is three channels, red, green and blue, and each channel is 32-bit floating point format. You can learn more about vImage pixel buffers in our vImage documentation.

The horizontal and vertical margins ensure that the 640 by 640 crop is centered on the source image. And here’s a graph that uses the slice operation to perform the crop. First, we’ll define the source as an argument and specify the height and width as flexible sizes.

Passing a negative value, minus 1 in this case, tells BNNSGraph that those dimensions might be any value. The final value in the shape 3 specifies that the image contains three channels: red, green and blue.

The BNNSGraphBuilder API uses Swift subscripts to perform a slice operation. In this example, the code uses the new SliceRange structure. The start indices for both the vertical and horizontal dimensions are the corresponding margin values. Setting the end indices as the negative margin value indicates that the end index is the end value of that dimension minus the margin.

In this example we don’t want to crop along the channel dimension, and the code specifies fillAll to ensure that we include all three channels.

This year we’re also introducing a new method for vImage pixel buffers. The withBNNSTensor method allows you to create a temporary BNNSTensor that shares memory and properties such as size and channel count with the pixel buffer. As the code here demonstrates, the new method makes passing images to and from your graph super easy. And because memory is shared, there’s no copying or allocating, and that means you get great performance when working with images.

After the execute function method returns, the destination pixel buffer contains a cropped image. And we can double check our squirrel is safely cropped by creating a core graphics image of the result. In addition to the new GraphBuilder slice range structure, the tensor slicing API supports all the Swift range types. So, whatever your slicing needs, we have you covered. And that is a brief overview of slicing.

Now we’ve taken a look at BNNSGraphBuilder and some of its features, let's dive into some use cases.

BNNSGraphBuilder is a great API to construct graphs of operations for pre- and post-processing data that you pass to or receive from ML models. One pre-processing technique is to threshold an image. That is to convert a continuous tone image to a binary image that contains just black or white pixels. Let’s take a look at how we can implement this using the GraphBuilder.

Before we start writing the graph, we’ll again define some vImage pixel buffers. The first stores the source image, and the second is the destination that receives the thresholded image. Both of these pixel buffers are single channel, 16-bit floating point format.

The first step in the graph is to define the source, which represents the input image. The code passes negative values for the image size to specify the graph supports any image size, but does however specify the FP16 data type that matches the pixel buffers. Calculating the average value of the continuous grayscale pixels is simply a matter of calling the mean method. The element wise is greater than operator populates the thresholded tensor with ones for corresponding pixels of the value greater than the average pixel value, and zeros otherwise.

And finally, the graph casts the Boolean ones and zeros to the bit-depth of the destination pixel buffer. We’ll use the withBNNSTensor method again to pass the two image buffers to the context and execute the function to generate the thresholded image.

After passing this continuous-toned grayscale image of a pair of pelicans in flight to the graph We get this thresholded image where all the pixels are either black or white.

Another use case for the BNNSGraphBuilder API is post-processing the results of a machine learning model.

For example, let’s say we want to apply a softmax function followed by a topK operation to the results of an ML model. Here, the post-process function creates a small graph on the fly. The graph declares an input argument based on the function’s input tensor. It then applies a softmax operation to the input argument, and then calculates its topK values and indices. Note that the k parameter the code passes to the topK function is actually defined outside of the make context closure. Lastly, the graph returns the topK results. After defining the graph, the function declares the tensors that store the results, and passes the output and input tensors to the context. Finally, the makeArray method converts the topK values and indices to Swift arrays and returns those.

There we have some examples of pre- and post-processing data with the GraphBuilder API. Now, let’s take a look at last year’s Bitcrusher demonstration and update the Swift piece to use BNNSGraphBuilder. This code demonstrates how BNNSGraph makes adding ML-based audio effects super easy.

Last year, we demonstrated BNNSGraph by incorporating it into an Audio Unit extension app. This app added a real-time Bitcrusher effect to an audio signal, and the demonstration used Swift to apply the same effect to a sine wave to display a visual representation of the effect in the user interface. For example, given this sine wave, the graph is able to reduce the resolution to add distortion to the sound, and the visual representation shows the sine wave as a discrete series of steps rather than a continuous wave. We also included saturation gain to add a richness to the sound. Last year’s demonstration used a CoreML package that was based on PyTorch code. So, let’s take last year’s code and compare it against the same functionality written in Swift using the new GraphBuilder API. Here’s the PyTorch on the left and the new Swift code on the right. One immediate difference is that Swift uses let and var to define intermediate tensors, so you’re free to decide if these are immutable or mutable. You don’t need any extra imports to access the full range of available operations. Furthermore, operations such as, in this case tanh, are methods on tensors rather than free functions.

Element-wise arithmetic operators are the same with the two different approaches.

Another difference though is that the make context closure can return more than one output tensor, so you always return an array. The Swift GraphBuilder allows you to define input arguments inside the make context closure. If you have an existing PyTorch model, we still recommend using that code and the existing file-based API. But, for new projects, our new GraphBuilder API allows you to write graphs using Swift, and as this comparison demonstrates, with a similar structure to the same set of operations written using PyTorch. Furthermore, we can demonstrate BNNSGraph’s 16-bit support, and how, in this example, it is significantly faster than the FP32 operations. Here’s our Swift code again, but this time using a type alias to specify the precision.

Changing the type alias changes the graph to use FP16 rather than FP32. In this example, FP16 is much faster than the FP32 equivalent.

To wrap up, BNNSGraphBuilder provides an easy to use Swift API that allows you to write high-performance and energy-efficient models and graphs-of-operations. It’s great for real-time and latency-sensitive use cases, but its user-friendly programming interface means it can be used for a wide range of applications. Before I go, please allow me to suggest that you head over to our documentation page where you’ll find plenty of reference material that discusses BNNSGraph and BNNSGraphBuilder. Also, be sure to check out our video from last year, where I talk about the file-based API and using BNNSGraph in C.

Many thanks for your time today and best wishes.