Xcode downloads client crash report with reason "index 0 beyond bounds for empty array" but the stacktraces don't contain any of my app's symbols

All the threads only contain system calls. The crashed thread only contains a single call to my app's code which is main.swift:13.

What could cause such a crash?

So, the direct issue is a combination of two issues:

• The crash itself was caused by an exception being thrown. Note that this sequence is the the standard "boilerplate" the system uses to proces exceptions and doesn't really tell you anything about why you crashed.

Thread 0 Crashed:
0   libsystem_kernel.dylib        	0x0000000193eff720 __pthread_kill + 8 (:-1)
1   libsystem_pthread.dylib       	0x0000000193f37f70 pthread_kill + 288 (pthread.c:1721)
2   libsystem_c.dylib             	0x0000000193e44908 abort + 128 (abort.c:122)
3   libc++abi.dylib               	0x0000000193eee44c abort_message + 132 (abort_message.cpp:78)
4   libc++abi.dylib               	0x0000000193edca40 demangling_terminate_handler() + 348 (cxa_default_handlers.cpp:77)
5   libobjc.A.dylib               	0x0000000193b853e4 _objc_terminate() + 156 (objc-exception.mm:496)
6   libc++abi.dylib               	0x0000000193eed710 std::__terminate(void (*)()) + 16 (cxa_handlers.cpp:59)
7   libc++abi.dylib               	0x0000000193eed6b4 std::terminate() + 108 (cxa_handlers.cpp:88)

• The crash should have included an additional thread stack about thread 0 that was the backtrace of the exception that was thrown. The typical reason it would be absent is that this was a C++ exception (which we can't capture stack traces from), however, there are few other "oddities".

Here's what's "odd":

1)

This line typically comes from the ObjC exception object and is specifically describing an ObjC exception. That would imply that something weirder is going on.

Exception Reason:      *** -[__NSArray0 objectAtIndex:]: index 0 beyond bounds for empty array

2)

This is a "trip wire" we added relatively recently that intended to identify cases where one our background threads interfered with normal exception processing. It doesn't actually point to what went wrong, but it support the ideat that "something" weird was occurring.

Thread 6:
0   HIServices                    	0x000000019a9b0150 SOME_OTHER_THREAD_SWALLOWED_AT_LEAST_ONE_EXCEPTION + 0 (HIExceptions.m:11)
1   Foundation                    	0x00000001951e9444 __NSThread__start__ + 724 (NSThread.m:991)
2   libsystem_pthread.dylib       	0x0000000193f382e4 _pthread_start + 136 (pthread.c:931)
3   libsystem_pthread.dylib       	0x0000000193f330fc thread_start + 8 (:-1)

3)

Finally, there is this thread chunk:

Thread 2:
0   libobjc.A.dylib               	0x0000000193b66a20 getMethodNoSuper_nolock(objc_class*, objc_selector*) + 252 (objc-runtime-new.mm:7188)
1   libobjc.A.dylib               	0x0000000193b6a01c lookUpImpOrForward + 436 (objc-runtime-new.mm:7609)
2   libobjc.A.dylib               	0x0000000193b69b84 _objc_msgSend_uncached + 68
3   UserActivity                  	0x00000001a4266fb0 __39+[UAUserActivityManager defaultManager]_block_invoke_2 + 168 (UAUserActivityManager.m:201)
4   libsystem_trace.dylib         	0x0000000193c8f248 ___os_state_request_for_self_impl_block_invoke + 40 (state.c:222)
5   libdispatch.dylib             	0x0000000193d855b4 _dispatch_client_callout + 20 (object.m:576)
6   libdispatch.dylib             	0x0000000193d94e08 _dispatch_lane_barrier_sync_invoke_and_complete + 56 (queue.c:1104)

Two points here:

1. Specifically crashing inside the ObjC runtime like this isn't common. It typically indicates that this was what actually crashed, however, that doesn't match up with the array indexing issue referenced in #1. The other way it can happen is that there is some kind of "entanglement" with the underlying problem, which is then causing the crash to play out with very specific timing.

2. Looking at our code, the codes that's running here is specifically diagnostic data that's collected by the framework itself. That could indicate that the crash originated in something tied to the activity system, however, the code involved is also not something I can see really crashing.

With all of that context, a few followup questions:

• How often are you seeing this crash and how consistent is the log, particularly when it comes to #3 above. If you're seeing exactly the same pattern in a large number of crashes then it could be a key factor, otherwise it's probably noise.

• Do you have any other information about the crash? With difficult crashes, knowing the larger context is at least as important as the crash log itself.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks for your insights. Unfortunately I don't have any more information about the crash: the crash report downloaded by Xcode is all I have. I think this is the first time I've seen this kind of crash. Whenever I see a crash report that gives me no clue about what the issue is, I have no choice but to assume that it's something unrelated to my own code, but if that's really the case, then in my opinion I shouldn't even get the crash report at all, since I cannot do anything about it. Looking into these reports only to realize that there's nothing I can do about it still takes quite some time when summing them all up. I thought I might still ask if there's anything I can do.

Thanks for your insights. Unfortunately I don't have any more information about the crash: the crash report downloaded by Xcode is all I have. I think this is the first time I've seen this kind of crash. Whenever I see a crash report that gives me no clue about what the issue is, I have no choice but to assume that it's something unrelated to my own code, but if that's really the case, then in my opinion I shouldn't even get the crash report at all, since I cannot do anything about it.

So, there are a few different things I want to say here:

• It's entirely possible for bugs in your app to cause crashes that don't contain any of your code. Simple crash ("your code crashed here") happen because your code did something wrong and immediately failed. Complex crashes (like this one) happen because your code did one or more things which created the circumstances which lead to the final failure. Both case are ultimately caused by "your code", the second case is just more complicated.

• The system have VERY little ability to determine whether or not a particular crash will be "meaningful" to you. That's partly because the basic analysis itself is hard (it's VERY close the solving the halting problem) but it's also because the system doesn't know what other information/knowledge you have.

This idea in particular:

then in my opinion I shouldn't even get the crash report at all, since I cannot do anything about it.

...is one I've very ACTIVELY argued against. At a minimum, giving you a crash log means you are at least aware that "something" is going on. I'll talk more about what your options are below, but the only thing worse than a crash you can't fix is a crash that you don't even know about.

Looking into these reports only to realize that there's nothing I can do about it still takes quite some time when summing them all up. I thought I might still ask if there's anything I can do.

First off, if you haven't already make sure you look at all of the "raw" log data, not sure Xcode's display of it. Crashpoint files are actually file packages, so you can access the raw crash logs directly using "Show Package Contents" in the Finder. While you're looking at that data, don't just look for issues in the stack data itself, but also path attention to things like the crash time or the app path (this works much better in iOS apps). One of the things you can catch this way is cases when a set of crashs logs are actually from a single/limited user and not necessarily a broad problem. As an example, I once looked at set of seemingly unrelated, very low level crashes which seemed concerning but were actually from a single user (based on the UUID in the install path) on a modified device (based on what was in the library list) which had all happened over ~3 hours (one you lined up the timestamps).

I'd also recommend looking at any other crash logs you've received and/or failure reports from users. It's not unusual for a single problem to result in multiple crash patterns (for example, based on the timing between events) and correlating those logs together may help you find the underlying problem.

Finally, think about how you get more/better information. There isn't any fixed approach for that, but it includes things like:

• Implementing app logging so you can determine when was happening in your app if/when your able to connect with a user who's experiencing the crash.

• Changing your apps implementation so that it includes clear indications of it's actual activity in the crash log.

• Making sure you've got a system in place where end users can contact you asking for help.

That last point is critical here. It's often the case that the key to fixing a critical issue isn't any particular code change, but is actually being able to connect with a user who is able to reproduce the problem and is wiling to work with you to try and fix it.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

It's entirely possible for bugs in your app to cause crashes that don't contain any of your code

Ok, but if I don't see any of my code in the stack trace, it can be quite difficult (if not infeasible) to find out what caused the crash.

Complex crashes (like this one) happen because your code did one or more things which created the circumstances which lead to the final failure

Since I don't have the means of understanding what the final failure was, I would expect that your internal code that caused the final failure would throw some meaningful error that would allow me to understand the issue, or that an error is thrown as soon as your internal code detects that the "circumstances" or preconditions are invalid.

Changing your apps implementation so that it includes clear indications of it's actual activity in the crash log.

You mean also in case that the user contacts me, right? Otherwise I wouldn't know how to make this visible in the crash report.

Making sure you've got a system in place where end users can contact you asking for help

How could I make sure of that, or how could I detect that a crash has happened or is about to happen if I have no idea what code causes it?

Ok, but if I don't see any of my code in the stack trace, it can be quite difficult (if not infeasible) to find out what caused the crash.

Yes. This kind of crash can be very difficult to find and fix.

Since I don't have the means of understanding what the final failure was, I would expect that your internal code that caused the final failure would throw some meaningful error that would allow me to understand the issue, or that an error is thrown as soon as your internal code detects that the "circumstances" or preconditions are invalid.

That's certainly what we try to do but, unfortunately, it's simply not possible for us to do that in the truly general case.

You mean also in case that the user contacts me, right? Otherwise I wouldn't know how to make this visible in the crash report.

A lot of this depends on exactly what your app does and how your app works. At the basic level, it's things like ensuring you've named very queue/thread to make sure you're pushing more data into the log. As a more complex solution you can actually use a thread to "publish" information about exactly that your app is going as it moves through it's normal operation. In concrete terms, you intentionally block a thread in a function like "DoingOperationX" and then allow that function to return once "operation X" is finished. It wastes a thread is not particularly elegant, but it can be helpful if you're able to narrow the range of possible failure points and are trying to pin things down more precisely.

It's also worth looking at this issue from the other direction. Thy crashing your app at other "points" in it's lifecycle and compare the state of the process at those time against your crash log. I don't know enough about your app to know if that will be useful or not, but I've worked with apps in that past where the crash location could be narrowed simply because the app didn't "look" like the log most of the time.

How could I make sure of that, or how could I detect that a crash has happened or is about to happen if I have no idea what code causes it?

You wouldn't. The idea here is to have a plan and investigative data in plave if/when a user does contact you. Case in point, having app logging in place so that your able to trace what your app was doing before it crashed.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks for the suggestions. No user has contacted me about these kinds of issues yet, so logging wouldn't help until someone does. I don't think manipulating threads would help me, since even once I knew what thread causes the crash, I would still be looking for a needle in a haystack, without any information about the kind of function call that caused the crash. My app is an AppKit app which spawns some threads here and there with DispatchQueue.

The best (and only feasible) thing, in my opinion, would be for macOS to catch invalid state early and throw a meaningful exception.

For a different app of mine I was thinking of how I can include interesting data in a crash report. One thing that came to my mind, and seems to work, is to set the current thread's name with

Thread.current.name = (Thread.current.name ?? "") + ". Invalid index \(i) for count \(array.count)"

All thread names are included in the crash report.

I don't think this is a good idea (and I'm surprised this even works, thinking of the privacy implications). Am I allowed to do this and is there a better alternative? The messages passed to preconditionFailure and fatalError are not included in the crash report.

Am I allowed to do this and is there a better alternative?

What are you actually trying to do/capture?

Note that I don't think it would work well for either of there cases:

The messages passed to preconditionFailure and fatalError are not included in the crash report.

For both of those cases, your larger app won't "know" a crash is happening until abort is called, at which point you're getting very close to writing your own crash reporter, a terrible idea you can read all about here. Certainly modifying thread names in a ****** handler is a bad idea.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

What are you actually trying to do/capture?

I'm hoping to understand the issue better by seeing what the invalid index is. My idea was to check that the index is valid, and if not, set the thread name to some debug message and then let the app crash. Even if it sounds like a bad idea, I think it's still better than keep letting the app crash without having a clue about the cause and hence being unable to fix it.

Hi,

First of, I need to make sure that it's clear that your crash log does not show the "normal" way an exception should have been processed. Thread 0 show the standard backtrace used to process exceptions (which is normal) but that code path should also have generated an additional thread stack for the exception itself. The two most common reasons this occurs are:

1. The exception being thrown as a C++ exception (not ObjectiveC/Swift). C++ doesn't define a standard mechanism for capturing the backtrace, so the system can't capture it. This is also why none of our frameworks will/should EVER throw a C++ exception "out" of the frameworks own code.

2. The exception hooks (more on that below) or some other exception processing manipulation has destroyed/lost the exception. Note that our frameworks avoid this sort of thing, both because it can be very disruptive in widely used code (like frameworks) and it's not really necessary since we control the underlying runtime (and can make it do "whatever" we want).

I don't know which of those two is the case but #2 does seem the more likely case. My concern is that you're working on the assumption that this is a straightforward issue of a bad index into NSArray and haven't really looked into broader context.

However, this did make me realize that there is one thing that you should specifically test/experiment with, this to try throwing the exception yourself (declare an empty array, then ask for index "0") and "throughout" your app. I'd write a standalone function for this, insert calls to that function at different/"interesting" points throughout my app, and then test all those points to "see what happens". At a minimum, that will either confirm that your app generates the crash log (with the exception backtrace) it "should" under normal circumstances or show that it doesn't (which you can then find an fix). If your lucky, you may also find a point which generates the same/similar log, at which point you've got a much clearer starting point to investigate.

I also want to repeat my point here:

First off, if you haven't already make sure you look at all of the "raw" log data, not sure Xcode's display of it.

This process of looking closely at every log you've received is the single most important step. In particular:

• Most "difficult" crashes involve some degree of timing variance, which means you'll often find that one or more logs include some additional "hint" as to what's actually failing.

• If ALL of the crashes look exactly like this, then you actually have a reasonable clue to start an investigation, which is to focus on "UAUserActivityManager". That's the system class that supports NSUserActivity, so that would indicate this is died to data "entering" your either directly through handoff or through mechanisms like drag and drop.

Moving into the specifics:

I'm hoping to understand the issue better by seeing what the invalid index is.

Just to be clear, that particular data is already in the crash. The message "index 0 beyond bounds for empty array" means "the index of 0 was passed into an empty array".

However:

My idea was to check that the index is valid, and if not, set the thread name to some debug message and then let the app crash.

So, there are two handlers an app can register for uncaught exceptions:

1. The general foundation handler "NSSetUncaughtExceptionHandler", which is the "generic" handler.

2. NSApplication.reportException which is used by AppKit to route it's own exceptions. Note this requires subclassing NSApplication using the process described in "Subclassing Notes".

If you decide to implement those handlers, there are a few notes:

• The you should follow is to do "your" work and then return control to the system. With #1, that means retrieving and saving the previous handler with NSGetUncaughtExceptionHandler, then calling it from your handler. With #2, that means calling "super.reportException".

• While both of these handlers run in the "normal" execution flow (compared to something like a ****** handler), your app is no longer operating "normally" and ANY activity you do should be considered quite dangerous. The standard approach in process crash reporters use is actually to log all the data they want to collect into an already open file handle, then upload/process that data the next time the app runs. If you choose not to use that approach, then my recommendation would be that you entirely avoid Swift/ObjC (both rely on runtime locks) and you rely as much as possible on "existing" state/configuration instead of creating "new" state/configuration. I don't know whether I'd consider this a good approach or not:

set the thread name to some debug message

• ...but, using it as an example, it would be much safer to create a sleeping thread at launch and then rename it with "pthread_setname_np" vs. creating a new thread and naming it via NSThread.

I'll also say that this might be a good point to approach this a customer service/data collection issue instead of just focussing on trying to improve the crash log data. If your app is aware that it previously crashed (which you could use the mechanisms above to detect and log) then you can ask the use to contact/email you and provide the data you collected. This may also create the opportunity to collect more complete data (like a sysdiagnose) directly from the user or by integrating MetricKit.

Even if it sounds like a bad idea, I think it's still better than keep letting the app crash without having a clue about the cause and hence being unable to fix it.

The major risk with any code that interact with the "crash path" is that the code will disrupt/interfere with that process in away that either reduces the quality of the crash log or, even worse, "hide" the crash entirely. For example, if an issue in your crash handler cause your app to hang or stall and the user the force quits your app, that crash has now "vanished" from your crash data even though the users experience is basically the same. The key point here is that your #1 goal with this kind of code has to be ensure that you ALWAYS crash "soon", NOT collecting diagnostic data.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks for your detailed answer. It's not my intention to implement a custom exception handler, I was just wondering about how I could include additional information in the crash report. You're right that the one I included at the beginning already says that the array is empty, I didn't even remember. I was actually thinking about Swift arrays, for which an out of bounds exception causes a crash report with only this information:

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BREAKPOINT (SIGTRAP)
Exception Codes:       0x0000000000000001, 0x00000001a6002c24

Termination Reason:    Namespace ******, Code 5 Trace/BPT trap: 5
Terminating Process:   exc handler [28880]

Hence my idea of putting the array count and the invalid index as the thread name. I got many crash reports recently because of invalid Swift array accesses and I was looking for a way of debugging them without asking the user for help, which can be helpful but I think most users would not be happy to do.

but, using it as an example, it would be much safer to create a sleeping thread at launch and then rename it with "pthread_setname_np" vs. creating a new thread and naming it via NSThread.

I wasn't suggesting to create a new thread and naming it, but simply naming the current thread.

I wasn't suggesting to create a new thread and naming it, but simply naming the current thread.

So, the core issues here are:

1. How do you detect that a crash/problem is occurring so that you can react to it?

2. That information can be safely manipulated at the point you detect a problem has occurred?

The complication here is how these issues become entangled and the details of exactly what you're looking at matter. Case in point:

I was actually thinking about Swift arrays, for which an out of bounds exception causes a crash report with only this information:

Unfortunately, this is actually a much trickier than an ObjectiveC exception. In the Swift array case, those check are implemented through compiler inserted checks which directly raise SIGTRAP. Unfortunately, reacting to those means implementing a ****** handler which is actually MUCH trickier than a custom exception handler (equivalent to implementing your own crash reporter). Note that it is definitely NOT safe to modify a threads name from a ****** handler.

Also, I need to correct this suggestion:

...but, using it as an example, it would be much safer to create a sleeping thread at launch and then rename it with "pthread_setname_np" vs. creating a new thread and naming it via NSThread.

This won't work either as all of our thread naming APIs only allow a thread to change it's own name, NOT to modify the name of another thread. Note that this is true even in cases where the API would APPEAR to allow remote name. For example, Thread.name would appear to allow you to change the name of "another" thread, however, what it actually does is:

• Change the value of an internal string it uses to track it's "own" name.

• Check self==Thread.currentThread and call pthread_setname_np if it's also the current thread. In other words, it will only change a threads pthread name if it's called on that thread.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware