When the Network Extension(NETransparentProxyProvider) is installed and enabled, data cannot be sent to the UDP server

I implemented a Network Extension in the macOS, use NETransparentProxyProvider. After installing and enabling it, I implemented a UDP client to test its. I found that the UDP client failed to send the data successfully (via sendto, and it returned a success), and when using Wireshark to capture the network data packet, I still couldn't see this UDP data packet.

The code for Network Extension is like this:

@interface MyTransparentProxyProvider : NETransparentProxyProvider
@end
@implementation MyTransparentProxyProvider
- (void)startProxyWithOptions:(NSDictionary *)options completionHandler:(void (^)(NSError *))completionHandler
{
NETransparentProxyNetworkSettings *objSettings = [[NETransparentProxyNetworkSettings alloc] initWithTunnelRemoteAddress:@"127.0.0.1"];
// included rules
NENetworkRule *objIncludedNetworkRule = [[NENetworkRule alloc] initWithRemoteNetwork:nil
remotePrefix:0
localNetwork:nil
localPrefix:0
protocol:NENetworkRuleProtocolAny
direction:NETrafficDirectionOutbound];
NSMutableArray<NENetworkRule *> *arrIncludedNetworkRules = [NSMutableArray array];
[arrIncludedNetworkRules addObject:objIncludedNetworkRule];
objSettings.includedNetworkRules = arrIncludedNetworkRules;
// apply
[self setTunnelNetworkSettings:objSettings completionHandler:
^(NSError * _Nullable error)
{
// TODO
}
];
if (completionHandler != nil)
completionHandler(nil);
}
- (BOOL)handleNewFlow:(NEAppProxyFlow *)flow
{
if (flow == nil)
return NO;
char szProcPath[PROC_PIDPATHINFO_MAXSIZE] = {0};
audit_token_t *lpAuditToken = (audit_token_t*)flow.metaData.sourceAppAuditToken.bytes;
if (lpAuditToken != NULL)
{
proc_pidpath_audittoken(lpAuditToken, szProcPath, sizeof(szProcPath));
}
if ([flow isKindOfClass:[NEAppProxyTCPFlow class]])
{
NWHostEndpoint *objRemoteEndpoint = (NWHostEndpoint *)((NEAppProxyTCPFlow *)flow).remoteEndpoint;
LOG("-MyTransparentProxyProvider handleNewFlow:] TCP flow! Process: (%d)%s, %s Remote: %s:%s, %s",
lpAuditToken != NULL ? audit_token_to_pid(*lpAuditToken) : -1,
flow.metaData.sourceAppSigningIdentifier != nil ? [flow.metaData.sourceAppSigningIdentifier UTF8String] : "",
szProcPath,
objRemoteEndpoint != nil ? (objRemoteEndpoint.hostname != nil ? [objRemoteEndpoint.hostname UTF8String] : "") : "",
objRemoteEndpoint != nil ? (objRemoteEndpoint.port != nil ? [objRemoteEndpoint.port UTF8String] : "") : "",
((NEAppProxyTCPFlow *)flow).remoteHostname != nil ? [((NEAppProxyTCPFlow *)flow).remoteHostname UTF8String] : ""
);
}
else if ([flow isKindOfClass:[NEAppProxyUDPFlow class]])
{
NSString *strLocalEndpoint = [NSString stringWithFormat:@"%@", ((NEAppProxyUDPFlow *)flow).localEndpoint];
LOG("-[MyTransparentProxyProvider handleNewFlow:] UDP flow! Process: (%d)%s, %s LocalEndpoint: %s",
lpAuditToken != NULL ? audit_token_to_pid(*lpAuditToken) : -1,
flow.metaData.sourceAppSigningIdentifier != nil ? [flow.metaData.sourceAppSigningIdentifier UTF8String] : "",
szProcPath,
strLocalEndpoint != nil ? [strLocalEndpoint UTF8String] : ""
);
}
else
{
LOG("-[MyTransparentProxyProvider handleNewFlow:] Unknown flow! Process: (%d)%s, %s",
lpAuditToken != NULL ? audit_token_to_pid(*lpAuditToken) : -1,
flow.metaData.sourceAppSigningIdentifier != nil ? [flow.metaData.sourceAppSigningIdentifier UTF8String] : "",
szProcPath
);
}
return NO;
}
@end

The following methods can all enable UDP data packets to be successfully sent to the UDP server:

1.In -[MyTransparentProxyProvider startProxyWithOptions:completionHandler:], add the exclusion rule "The IP and port of the UDP server, the protocol is UDP";

2.In -[MyTransparentProxyProvider startProxyWithOptions:completionHandler:], add the exclusion rule "All IPs and ports, protocol is UDP";

3.In -[MyTransparentProxyProvider handleNewFlow:] or -[MyTransparentProxyProvider handleNewUDPFlow:initialRemoteEndpoint:], process the UDP Flow and return YES.

Did I do anything wrong?

Answered by DTS Engineer in 846397022

OK. Then that’s definitely bugworthy. Not only should this work in theory — when you return false from handle-new-flow the system is supposed to deal with the flow as it would if your transparent proxy weren’t installed — but it’s also causing you grief in practice. I encourage you to file a bug about this.

Please post your bug number, just for the record.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Another issue was also discovered. When a UDP client binds a port via bind() (for example, 12345), and then uses sendto() and close(), when it attempts to bind this port again using bind(), it will fail, with errno being 48(at this time, handleNewFlow always returns NO).

There is no such problem if the network extension is disabled.

Can you get this working for TCP?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

For TCP, it seems everything is normal.

It seems to be related to the quick close of the socket after sendto in the UDP client. If you wait for a while and then close the socket again, this problem will not occur.

//
// main.cpp
//
#include <iostream>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/types.h>
#include <sys/socket.h>
int main(int argc, const char * argv[])
{
// arg check
int nLocalPort = -1;
const char *lpcszRemoteIP = "192.168.1.123";
int nRemotePort = 12345;
const char *lpcszData = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ";
if (argc > 1)
{
// arg 1
lpcszRemoteIP = argv[1];
if (argc > 2)
{
const char *lpcszRemotePort = argv[2];
nRemotePort = atoi(lpcszRemotePort);
if (argc > 3)
{
const char *lpcszLocalPort = argv[3];
nLocalPort = atoi(lpcszLocalPort);
if (argc > 4)
{
lpcszData = argv[4];
}
}
}
}
size_t stDataSize = strlen(lpcszData) * sizeof(char);
printf("Role : UDP Client\n");
if (nLocalPort != -1)
printf("Local : local:%d\n", nLocalPort);
printf("Remote: %s:%d\n", lpcszRemoteIP, nRemotePort);
printf("Data : [%ld]%s\n", stDataSize, lpcszData);
int nSocket = -1;
do
{
nSocket = socket(AF_INET, SOCK_DGRAM, 0);
if (nSocket == -1)
{
printf("socket() == -1\n");
break;
}
if (nLocalPort != -1)
{
struct sockaddr_in addrLocal;
addrLocal.sin_family = AF_INET;
addrLocal.sin_port = htons(nLocalPort);
addrLocal.sin_addr.s_addr = htonl(INADDR_ANY);
int nBind = bind(nSocket, (const struct sockaddr *)&addrLocal, sizeof(addrLocal));
if (nBind != 0)
{
printf("bind(%d) error. ret: %d errno: %d\n", nLocalPort, nBind, errno);
break;
}
else
{
printf("bind(%d)\n", nLocalPort);
}
}
struct sockaddr_in addrServer;
addrServer.sin_family = AF_INET;
addrServer.sin_port = htons(nRemotePort);
if (inet_pton(AF_INET, lpcszRemoteIP, &addrServer.sin_addr.s_addr) != 1)
{
printf("inet_pton() != 1\n");
break;
}
// Send Data
ssize_t nSend = sendto(nSocket, lpcszData, stDataSize, 0, (const struct sockaddr *)&addrServer, sizeof(addrServer));
if (nSend < stDataSize)
{
printf("sendto() error. ret: %ld errno: %d\n", nSend, errno);
}
} while (false);
/*
int nTime = 0;
while (true)
{
sleep(1);
nTime++;
if (nTime >= 5)
break;
}
*/
if (nSocket != -1)
{
close(nSocket);
nSocket = -1;
}
return 0;
}

How should it be dealt with in [MyTransparentProxyProvider handleNewFlow:] or [MyTransparentProxyProvider handleNewUDPFlow:initialRemoteEndpoint:]?Perhaps when The error "The peer closed the flow" occurs in [NEAppProxyUDPFlow openWithLocalEndpoint:completionHandler:], should the Network Extension take over this flow instead of returning NO?

Written by DehanZ in 845121022
If you wait for a while and then close the socket again, this problem will not occur.

Which problem are we talking about here? The dropped datagram problem? Or the EADDRINUSE problem?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

OK. In that case I don’t find the result super surprising. NE providers that operate at this level (transparent proxies, app proxies, and content filters) add significantly more asynchronicity to the networking stack, so it’s easy to imagine a situation where closing the socket tears down the flow before you’re able to read the datagram from it.

Is this a problem in practice? It’s pretty unusual for a UDP client to open a second, write a datagram, and then immediate close the socket. Real world UDP clients tend to either send multiple datagrams or wait for a reply, or both.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

It might be regarded as a problem in practice.

In my test environment, there is a UDP client implemented by someone else. This UDP client will send an authentication packet to the gateway device at regular times (that is, quickly close the socket after each sendto), and then it is found that this authentication packet has not been sent successfully.

After the investigation, the problem currently under discussion, "Network Extension causes UDP data packets to fail to be sent successfully", was obtained.

I guess, when [MyTransparentProxyProvider handleNewFlow:] return NO, system to connect to the server's operation, Like call [NEAppProxyUDPFlow openWithLocalEndpoint:completionHandler:], and then got a error("The peer closed the flow"), Then the data of this Flow was not sent out.

Written by DehanZ in 845807022
It might be regarded as a problem in practice.

OK. I just wanted to check because I prefer not to spend time on theoretical issues.

Written by DehanZ in 845807022
This UDP client will send an authentication packet to the gateway device at regular times

So, lemme see if I understand this case correctly:

  1. That UDP client does its thing (opens a socket, sends a datagram, and then immediately closes the socket).

  2. The system calls your transparent proxy’s handle-new-flow method.

  3. That method returns false.

  4. But the datagram never appears on the wire.

Is that correct?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

OK. Then that’s definitely bugworthy. Not only should this work in theory — when you return false from handle-new-flow the system is supposed to deal with the flow as it would if your transparent proxy weren’t installed — but it’s also causing you grief in practice. I encourage you to file a bug about this.

Please post your bug number, just for the record.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I have completed the submission of the bug. It is FB18495149.

In the above discussion, there is also the issue of EADDRINUSE.

After the NE Provider is started, when the port bound by the UDP client is rebound again, the situation of EADDRINUSE will occur for a period of time (just like being in the FIN_WAIT_2 state of TCP). Why is this? Neither netstat nor lsof found the connection of the corresponding port.

This situation will not occur when there is no Network Extension.

When the Network Extension(NETransparentProxyProvider) is installed and enabled, data cannot be sent to the UDP server
 
 
Q