QNE2TransparentProxyMac sample code

QNE2TransparentProxyMac isn’t official sample code. Rather, it’s a test project that I created that I’ve given out to a few developers on a one-to-one basis.

Most of my QNE test projects don’t contain any real code. Rather, they’re primarily focused on packaging issues. That’s very much the case for QNE2TransparentProxyMac. It doesn’t actually do any networking, it just shows how to get a transparent proxy to the point where the various handle-new-flow methods get called.

And these days that’s not so valuable because Xcode has move forward to the point where the NE packaging issues that caused a bunch of grief during the early days of NE have all been resolved.

If you have specific questions about transparent proxies, I’m happy to answer them here.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thanks for getting back to me. I'm afraid my poor editing above confused the issue. The project claims it is based on the QNE2TransparentProxyMac and I've been working on it for a while to finish TLS inspection and filtering. The previous developer had moved on before I joined the project so I don't have whatever code he might have received.

I want to add support for UDP and noticed it's not enabled in the Network Extension and there's no UDP flow copier. Is there some transparent proxy example that includes support for UDP?

I'm following the TCP example to implement UDP support but would prefer not to reinvent it if it already exists. Thanks.

• Peter

In func handleNewTCPFlow(_ flow: NEAppProxyTCPFlow) we create a connection as:

let connection = NWConnection(to: flow.remoteEndpoint.nwEndpoint, using: .tcp)

When converting this to UDP, the NEAppProxyUDPFlow doesn't have a flow.remoteEndpoint so it's unclear to me where I should get this from. Presumably each datagram has a destination which defines which flow it belongs to.

Is there some transparent proxy example that includes support for UDP?

No.

Notably, QNE2TransparentProxyMac doesn’t include a TCP flow copier. Its handle-new-flows methods all return false. I suspect your erstwhile colleague got that code from elsewhere. It might’ve been from Handling Flow Copying.

Presumably each datagram has a destination which defines which flow it belongs to.

Right.

This is confusing because we’re a bit inconsistent about the meaning of the word flow:

• Normally I use the term UDP flow to represent a sequence of datagrams with the some local IP / local port / remote IP / remote port tuple.

• However, that’s not what flow means in the case of NEAppProxyUDPFlow. Rather, it represents a UDP network ‘handle’ used by an app. For example, if the app is using BSD Sockets, the NEAppProxyUDPFlow is equivalent to the UDP socket file descriptor.

So, an NEAppProxyUDPFlow object constrains the local IP / local port part of the tuple but not the remote IP / remote port part. That can vary on a datagram-by-datagram basis. That’s why the various readDatagrams(…) methods return the outgoing datagrams and their associated endpoints.

Note that it is possible for the NEAppProxyUDPFlow object to constrain the remote IP / remote port part as well. That’s what you get when, for example, you use BSD Sockets to create a connected UDP socket. In that case the endpoint will come to you via the initialRemoteFlowEndpoint parameter of the -handleNewUDPFlow:… method.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I’m still debugging my UdpFlowCopier and could use some guidance as some aspects are confusing.

I need to copy datagrams between a provider flow object and an actual UDP socket or “connection”. Can you help me understand the difference between a NetworkExtension.NWEndpoint and Network.NWEndpoint and where I would need to use one versus the other?

Following the TCP example, the PassThroughProviderCore calls handleNewUDPFlow(_ flow: NEAppProxyUDPFlow). There’s no remote endpoint but I can retrieve the local endpoint as

guard let nwEndpoint = flow.localEndpoint?.nwEndpoint else { return }

I’m trying to open a UDP listener on this endpoint as follows but it mostly returns failed to create listener on port 0

private func handleStart() -> State {

    let port = portForEndpoint(self.localEndpoint) ?? 0
    logger.debug("UDPFlowCopier - handleStart copier \(self.osLogID) port \(port.rawValue, privacy: .public)")
    let params = NWParameters.udp
    params.allowFastOpen = true
    self.listener = try? NWListener(using: params, on: port)
    self.listener?.stateUpdateHandler = { update in
        switch update {
        case .ready:
            self.isReady = true
            //self.processEvent(.didOpenConnection)
        case .failed, .cancelled:
            // Announce we are no longer able to listen
            self.listening = false
            self.isReady = false
                logger.debug("UDPFlowCopier - copier \(self.osLogID) failed to create listener on port \(port.rawValue, privacy: .public)")
            self.stop()
        default:
            ()
        }
    }
    self.listener?.newConnectionHandler = { connection in
        self.createConnection(connection: connection)
    }
    self.listener?.start(queue: self.queue)

    return .openingConnection
}

Can you offer any insight? Thanks

Can you help me understand the difference between a NetworkExtension.NWEndpoint and Network.NWEndpoint and where I would need to use one versus the other?

Sure. And that is, indeed, a gnarly edge case.

NetworkExtension.NWEndpoint is an Objective-C class. It was introduced prior to the introduction of Network framework proper. Network framework is a Swift API [1] and, in Swift, it makes sense to represent endpoints as an enum, so Network.NWEndpoint is an enum.

Quoting TN3151 Choosing the right networking API:

Network Extension in-provider networking includes NWUDPSession. While there are some very limited circumstances where this is still useful, in most cases it’s better to use Network framework. For more details, see In-Provider Networking.

So, what are those circumstances? In short:

• If your product supports system prior to the introduction of Network framework, that is, macOS 10.14. That’d be pretty unusual these days.

• There are a few Network Extension APIs that work in terms of NetworkExtension.NWEndpoint. In macOS 15 we added parallel mechanisms to that support Network.NWEndpoint. If you support systems prior to that, you’ll need to either use the in-provider network APIs or implement some sort of shim to get things working on top of Network framework.

The additional of these parallel mechanisms was tricky, involving some exciting use of the .apinotes file [2].

I’m trying to open a UDP listener on this endpoint

Wha? I can’t see any circumstances where that’d be necessary.

I need to copy datagrams between a provider flow object and an actual UDP socket or “connection”.

Doing that with NWConnection is tricky due to a semantic disparity. There isn’t necessarily a one-to-one mapping between NEAppProxyUDPFlow and NWConnection because:

• NEAppProxyUDPFlow can represents a single local endpoint that sends to multiple remote endpoints.

• NWConnection represents a UDP flow, that is, a series of datagrams that all share the same local IP / local port / remote IP / remote port tuple

I see two paths forward:

• Use a different API. Notably, BSD Sockets supports the same model as NEAppProxyUDPFlow.

• Create a different NWConnection for each remote endpoint.

I’m not entirely sure whether the NWConnection option will work though, because I don’t see a good way to create multiple outgoing connections with the same local port [3]. I’m still digging into that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] It also has a low-level C API.

[2] Below is an explanation of this I wrote in another context. It talks about NWPath, not NWEndpoint, but the same process applies for both.

In Swift 6 mode, NetworkExtension.NWPath is published as NetworkExtension.__NWPath so that Network.NWPath can take priority. It does this by API notes. Specifically, NetworkExtension.framework/Versions/A/Headers/NetworkExtension.apinotes has this:

…
- Name: NWPath
  SwiftPrivate: true
…
SwiftVersions:
- Version: 5.0
  …
  - Name: NWPath
    SwiftPrivate: false
  …

In the Swift 6 language mode, SwiftPrivate is true and thus the type gets published as __NWPath. In Swift 5 language mode, SwiftPrivate is false and thus it continues to be published as NWPath.

[3] The obvious option, setting requiredLocalEndpoint, just generates an EADDRINUSE, at least in my case.