Title: CoreFoundation CFRelease Crash on iOS 18

Please post a full Apple crash report, per the advice in Posting a Crash Report.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Please post a full Apple crash report, per the advice in Posting a Crash Report.

All three crashes have attached. @DTS Engineer

Our customers are experiencing what appears to be the exact same issue. This crash is only happening on devices that have been upgraded to iOS 18. Here are our crash reports:

Right. Let’s start with pouya176.

Your crash reports make it clear that you’re using a third-party crash reporter. For example, see frame 4 in thread 4 in your first crash report. I recommend against that, for reasons I explain in detail in Implementing Your Own Crash Reporter. I recommend that your remove that.

Having said that, I don’t see any direct evidence that the third-party crash reporter has messed up in this case.

All three crash reports show some sort of memory management problem related to CFStream. Such a crash can’t arise spontaneously; something in your app is using CFStream. Your crash report doesn’t include any definitive answers, but if I had to guess I’d say that ChatNetworkManager is using CFStream, or its NSStream wrapper, for running a TCP connection.

If that’s the case I recommend that you change that code to use Network framework. Using CFSocketStream for networking is no longer a recommended approach. See TN3151 Choosing the right networking API.

However, that doesn’t help you in the short term. While CFSocketStream is no longer recommended, it shouldn’t crash. I’ll come back to that below.

robert_williams_ditto, you posted 5 crash reports. Of those, the first four look related to this. The fifth is something different entirely. It doesn’t even mention stream in the crash report. For that one, I recommend that you symbolicate your crash report and then start a separate investigation. See Adding identifiable symbol names to a crash report for info on symbolication.

The first four look very similar to pouya176’s crashes. I’m going to focus on the first, on the assumption that they’re all related.

That crash report has two threads that reference CFStream. Thread 19 is the crashing thread, much like in pouya176’s case. But thread 9 is interesting. It suggests you’re using CFStream for Bluetooth. Given that this is iOS, the only way I can see that making sense is if you’re using External Accessory framework to talk to an MFi accessory. Is that right?

If pouya176 is using the stream for TCP and robert_williams_ditto is using it for Bluetooth, that suggests that this isn’t a problem with the stream implementation but a problem with CFStream itself. That isn’t 100% surprising given that Foundation has been undergoing major refactoring over the last few years [1].

Do either of you have an JSON (.ips) crash report? If so, please post it here. That might allow me to learn more about this issue.

In both case, can you try running exercising this case in your app with the standard memory debugging tools, and especially zombies. Given the nature of this crash, those might more the problem more reproducible.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] If you’re curious, watch the first half of Swift & Interoperability.

https://youtu.be/%77%6e6C_XEv1Mo

robert_williams_ditto, you posted 5 crash reports. Of those, the first four look related to this. The fifth is something different entirely. It doesn’t even mention stream in the crash report.

That's my mistake. Let's just focus on the first 4 crash reports.

That crash report has two threads that reference CFStream. Thread 19 is the crashing thread, much like in pouya176’s case. But thread 9 is interesting. It suggests you’re using CFStream for Bluetooth. Given that this is iOS, the only way I can see that making sense is if you’re using External Accessory framework to talk to an MFi accessory. Is that right?

We use Bluetooth (specifically BLE) to communicate with other iOS devices. After establishing a connection we open an L2CAP channel which provides an input stream and an output stream. So we don't use CFStream directly, but the CBL2CAPChannel type from CoreBluetooth does appear to use CFStream in its implementation.

Thank you for calling my attention to thread 9 in the first crash report. It is actively closing the L2CAP channel at the time of the crash:

Thread 9:
0   libsystem_kernel.dylib              0x00000001d550eb18 __psynch_mutexwait + 8 (:-1)
1   libsystem_pthread.dylib             0x000000020ca672f4 _pthread_mutex_firstfit_lock_wait + 84 (pthread_mutex.c:1414)
2   libsystem_pthread.dylib             0x000000020ca66d08 _pthread_mutex_firstfit_lock_slow + 220 (pthread_mutex.c:1490)
3   CoreFoundation                      0x00000001856cd28c _CFStreamClose + 476 (CFStream.c:312)
4   DittoObjC                           0x0000000107df2274 -[DITL2CAPConnection close] + 88 (DITL2CAPConnection.m:101)
5   DittoObjC                           0x0000000107defe18 -[DITBluetoothPlatform disconnectFromPeripheral:] + 136 (DITBluetoothPlatform.m:980)
6   DittoObjC                           0x0000000107debc0c ble_disconnect_peripheral_cb + 84 (DITBluetoothPlatform.m:92)
7   DittoObjC                           0x000000010809bdf4 ditto_mesh::ble::client_transport::BleClientPeerTransport::request_disconnect::ha7af5922ecc95e18 + 192
8   DittoObjC                           0x00000001075d4694 ditto_mesh::ble::client_transport::BleClientRemotePeer::connect_handshake::_$u7b$$u7b$closure$u7d$$u7d$::hbf7382db8871e8b6 + 11428

I wonder if there may be a race condition, because the CFStream is being closed while it is still in use on another thread. It would be very interesting to confirm whether the lines of code in CoreFoundation where the crash occurs are using resources that are being freed/deallocated on the other thread:

Thread 19 Crashed:
0   CoreFoundation                      0x0000000185680e54 CF_IS_OBJC + 76 (CFRuntime.c:461)
1   CoreFoundation                      0x000000018567ad6c CFRetain + 64 (CFRuntime.c:1169)
2   CoreFoundation                      0x0000000185731540 _signalEventSync + 92 (CFStream.c:609)
3   CoreFoundation                      0x000000018573146c _cfstream_shared_signalEventSync + 392 (CFStream.c:757)
...

So far, I have been unable to confirm my theory because I can't yet reproduce the crash myself. I have only received these crash reports from our customers who use our SDK. Do you have any ideas about how to force this crash to occur so that I can confirm my hypothesis?

For context, here is our code that closes an L2CAP channel:

- (void)close {
    // Closing streams on two threads can race and cause a crash internal to NSStream (#3696)
    // This gets called both when handling stream failures and when we want to deliberately
    // disconnect, and we deliberately disconnect on failure, just to make sure.
    @synchronized (self) {
        [_channel.inputStream setDelegate:nil];
        [_channel.inputStream close]; // line 101
        [_channel.inputStream removeFromRunLoop:self.runLoop forMode:NSDefaultRunLoopMode];
        [_channel.outputStream setDelegate:nil];
        [_channel.outputStream close]; // line 104
        [_channel.outputStream removeFromRunLoop:self.runLoop forMode:NSDefaultRunLoopMode];
    }
}

- (void)dealloc {
    [self close];
} // line 111

(I included some line numbers because they are referenced in the first, third, and fourth crash reports)

And here is our code that sets up an L2CAP channel:

- (instancetype)initWithChannel:(CBL2CAPChannel *)channel handle:(DITTransportHandleWrapper *)handle isServer:(BOOL)isServer uuid:(NSUUID *)uuid runLoop:(NSRunLoop *)runLoop {
    self = [super init];
    if (self) {
        _channel = channel;
        _runLoop = runLoop;
        _handle = handle;
        _isServer = isServer;
        _uuid = uuid;
        _isFailed = NO;

        DITStreamWeakAdaptor *inputAdaptor = [[DITStreamWeakAdaptor alloc] initWithTrueDelegate:self];
        DITStreamWeakAdaptor *outputAdaptor = [[DITStreamWeakAdaptor alloc] initWithTrueDelegate:self];

        [channel.inputStream setDelegate:inputAdaptor];
        [channel.outputStream setDelegate:outputAdaptor];

        // NOTE: we've run into multiple race conditions due to this connection
        // being deallocated while the channel is active. To avoid a whole class
        // of problems, we make sure this delegate remains alive until the
        // channel's streams are properly closed.
        objc_setAssociatedObject(channel.inputStream, (__bridge void *)self, inputAdaptor, OBJC_ASSOCIATION_RETAIN);
        objc_setAssociatedObject(channel.outputStream, (__bridge void *)self, outputAdaptor, OBJC_ASSOCIATION_RETAIN);

        [channel.inputStream scheduleInRunLoop:runLoop forMode:NSDefaultRunLoopMode];
        [channel.outputStream scheduleInRunLoop:runLoop forMode:NSDefaultRunLoopMode];

        [channel.inputStream open];
        [channel.outputStream open];
    }
    return self;
}

Thank you for your help!

@DTS Engineer I have received several .ips crash reports from our customers. (I'm attaching them as .json files because I'm not allowed to upload .ips files to this site.)

Something that's interesting to me is that only one of the files, CashierApp-2024-11-27-162937.json, contains a reference to CoreBluetooth in the Binary Images section. Does this mean that CoreBluetooth was not being used in the other instances of the app that crashed? I believed I had narrowed the cause of the crash to our Bluetooth code, but perhaps it is related to other parts of our networking code as well.

Thanks for those extra crash reports. Unfortunately they didn’t offer any further clues.

Something that's interesting to me is that only one of the files … contains a reference to CoreBluetooth in the Binary Images section. Does this mean that CoreBluetooth was not being used … ?

Not necessarily. For better or worse, Apple’s crash reporter only includes an item in the Binary Images section if there’s a frame in one of the backtraces that references it. And this particular crash doesn’t offer conclusive evidence whether or not this was a Core Bluetooth stream.

Consider the crashing thread backtrace from the first crash report in your most recent post:

0   libobjc.A.dylib         … objc_loadWeakRetained + 156
1   CoreFoundation          … _outputStreamCallbackFunc + 35
2   CoreFoundation          … _signalEventSync + 215
3   CoreFoundation          … _cfstream_shared_signalEventSync + 391
4   CoreFoundation          … __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 27
5   CoreFoundation          … __CFRunLoopDoSource0 + 175
6   CoreFoundation          … __CFRunLoopDoSources0 + 243
7   CoreFoundation          … __CFRunLoopRun + 839
8   CoreFoundation          … CFRunLoopRunSpecific + 587
9   Foundation              … -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 211
10  Foundation              … -[NSRunLoop(NSRunLoop) runUntilDate:] + 63
11  DittoObjC               … ditto_transports_main + 188
12  Foundation              … __NSThread__start__ + 723
13  libsystem_pthread.dylib … _pthread_start + 135
14  libsystem_pthread.dylib … thread_start + 7

Frames 8 through 1 are all generic Core Foundation code that would show up for any CFStream. And frame 0 is in the Objective-C runtime, loading a weak object pointer. The CFStream infrastructure is trying to load up the stream’s delegate, and that’s crashing presumably due to some sort of memory corruption.

So, the only thing that can tell you what strema type is involved is frame 11. I’m presuming that you recognise this symbol and expect it to be working with Core Bluetooth streams?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I'm following up now that we have identified the root cause of the crash and fixed it.

We were accessing CFStream and NSRunLoop from multiple threads. Particularly, we were closing a CFStream (Bluetooth L2CAP channel) on one thread while it was being actively used on the thread running its NSRunLoop. It was a classic race condition.

We fixed the crash by ensuring that all accesses to the CFStream and NSRunLoop happen on the same thread.

This issue has been present in our codebase for 4+ years, but it didn't cause a crash until iOS 18.

@robert_williams_ditto that is great news! when should we expect the fix to be available?