pas_panic_on_out_of_memory_error crash on tvOS 15.4 and 15.4.1

You are right that this does look similar. Two things:

• Have you seen it on the 15.5 betas that we’re currently seeding?

• Please post a full crash report. See Posting a Crash Report for details on how to do that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Am unable to validate if it still happens on tvOS 15.5 betas, since this crash still only happened in production with versions 15.4 and 15.4.1.

Unfortunately was unable to get a full symbolicated crash report from the analytics platform being used. For some reason, this same crash is not showing up in Xcode Organiser section. However, if there's a specific info you're looking for, please let me know, since I may still be able to get it.

I encounter similar issues with our application. In most of cases it crashes with error like this. @DTS Engineer Do you have any hints according that problem?

Crashed: ITMLKit
0  JavaScriptCore                 0x169bf08 pas_panic_on_out_of_memory_error + 46
1  JavaScriptCore                 0x164c9f4 bmalloc_iso_allocate_impl_impl_slow + 18
2  JavaScriptCore                 0x165a808 bmalloc_heap_config_specialized_local_allocator_try_allocate_small_segregated_slow + 5424
3  JavaScriptCore                 0x164bc20 bmalloc_allocate_impl_casual_case + 728
4  JavaScriptCore                 0xdffe10 JSC::LLInt::setupGetByIdPrototypeCache(JSC::JSGlobalObject*, JSC::VM&, JSC::CodeBlock*, JSC::BytecodeIndex, JSC::GetByIdModeMetadata&, JSC::JSCell*, JSC::PropertySlot&, JSC::Identifier const&) + 296
5  JavaScriptCore                 0xdd7790 JSC::LLInt::performLLIntGetByID(JSC::BytecodeIndex, JSC::CodeBlock*, JSC::JSGlobalObject*, JSC::JSValue, JSC::Identifier const&, JSC::GetByIdModeMetadata&) + 2748
6  JavaScriptCore                 0xdd6b8c llint_slow_path_get_by_id + 480
7  JavaScriptCore                 0x16a7c44 pas_reallocation_did_fail + 7218576452
8  JavaScriptCore                 0x169c0b8 pas_reallocation_did_fail + 7218528440
9  JavaScriptCore                 0xc7a92c JSC::Interpreter::executeCall(JSC::JSObject*, JSC::CallData const&, JSC::JSValue, JSC::ArgList const&) + 904
10 JavaScriptCore                 0x11ad884 JSC::ProxyObject::getOwnPropertySlotCommon(JSC::JSGlobalObject*, JSC::PropertyName, JSC::PropertySlot&) + 4944
11 JavaScriptCore                 0xdd72a0 JSC::LLInt::performLLIntGetByID(JSC::BytecodeIndex, JSC::CodeBlock*, JSC::JSGlobalObject*, JSC::JSValue, JSC::Identifier const&, JSC::GetByIdModeMetadata&) + 1484
12 JavaScriptCore                 0xdd6b8c llint_slow_path_get_by_id + 480
13 JavaScriptCore                 0x16a7c44 pas_reallocation_did_fail + 7218576452
14 JavaScriptCore                 0x16c470c pas_reallocation_did_fail + 7218693900
15 JavaScriptCore                 0x16c470c pas_reallocation_did_fail + 7218693900
16 JavaScriptCore                 0x16c31d4 pas_reallocation_did_fail + 7218688468
17 JavaScriptCore                 0x169c0b8 pas_reallocation_did_fail + 7218528440
18 JavaScriptCore                 0xc7a92c JSC::Interpreter::executeCall(JSC::JSObject*, JSC::CallData const&, JSC::JSValue, JSC::ArgList const&) + 904
19 JavaScriptCore                 0xfe8bc4 JSC::runJSMicrotask(JSC::JSGlobalObject*, WTF::ObjectIdentifierGeneric<JSC::MicrotaskIdentifierType, WTF::ObjectIdentifierThreadSafeAccessTraits<unsigned long long>, unsigned long long>, JSC::JSValue, std::__1::span<JSC::JSValue const, 18446744073709551615ul>) + 532
20 JavaScriptCore                 0x126bde0 JSC::VM::drainMicrotasks() + 468
21 JavaScriptCore                 0xfe6e9c JSC::JSLock::willReleaseLock() + 136
22 JavaScriptCore                 0xfe677c JSC::JSLockHolder::~JSLockHolder() + 136
23 JavaScriptCore                 0x48b6ec -[JSValue invokeMethod:withArguments:] + 676
24 ITMLKit                        0x58c48 -[IKJSObject invokeMethod:withArguments:] + 152
25 ITMLKit                        0x9f304 __43-[IKJSXMLHTTPRequest setRequestReadyState:]_block_invoke + 184
26 ITMLKit                        0x44174 -[IKAppContext _doEvaluate:] + 104
27 ITMLKit                        0x43f40 -[IKAppContext _evaluate:] + 80
28 ITMLKit                        0x40c30 __41-[IKAppContext evaluate:completionBlock:]_block_invoke + 40
29 ITMLKit                        0x43d10 -[IKAppContext _sourcePerform] + 204
30 ITMLKit                        0x1a290 -[IKConcurrentEvaluator lockSchedulingForEvaluation:] + 96
31 ITMLKit                        0x43bfc IKRunLoopSourcePerformCallBack + 108
32 CoreFoundation                 0x8399c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
33 CoreFoundation                 0x838e4 __CFRunLoopDoSource0 + 168
34 CoreFoundation                 0x83064 __CFRunLoopDoSources0 + 220
35 CoreFoundation                 0x7d9fc __CFRunLoopRun + 784
36 CoreFoundation                 0x7d280 CFRunLoopRunSpecific + 536
37 ITMLKit                        0x43a44 -[IKAppContext _jsThreadMain] + 276
38 Foundation                     0x716320 __NSThread__start__ + 716
39 libsystem_pthread.dylib        0xf24 _pthread_start + 104
40 libsystem_pthread.dylib        0x9aa8 thread_start + 8

Do you have any hints according that problem?

I was hoping to get a crash report. Can you post one?

See Posting a Crash Report for advice on how to do that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@DTS Engineer sorry, here is the full crashlog

Sadly, that’s not an Apple crash report. I have general problems with non-Apple crash reports — for reasons I explain in Implementing Your Own Crash Reporter — but in this specific case I really need an Apple crash report to offer any meaningful insight.

Ideally that’d be in JSON format (.ips), but one in the human readable format (.crash) would at least get me started.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@DTS Engineer you are right. sorry about that.

Thanks for that.

Annoyingly, that crash report got corrupted in transit. Note how thread 16 runs immediately into the Binary Images section:

Thread 16:
0 JavaScriptCore 0x00000001ad687f08 pas_panic_on_out_of_memory_error + 0 (pas_utils.c:208)
1 JavaScriptCore 0x00000001ad6389f4 pas_allocation_result_crash_on_error + 20 (pas_allocation_result.h:118)
2 JavaScriptCore 0x00000001ad646808 bmalloc_heap_config_specialized_local_allocator_try_allocate_small_segregat…
3 JavaScriptCore 0x00000001ad637c20 bmalloc_allocate_impl_casual_case + 728 (bmalloc_heap_inlines.h:69)
4 JavaScriptCore 0x00000001ac6f13fc WTF::HashTable, WTF::DefaultRefDerefTraits>, WTF::KeyValuePair /var/contain…
        0x104ab4000 -         0x104abffff libobjc-trampolines.dylib arm64   /usr/lib/libobjc-trampolines.dylib
        0x104b88000 -         0x104b8ffff FirebaseCoreExtension arm64   /private/var/containers/Bundle/Applicat…
…

I’m also concerned that thread 2 and thread 16 are both in pas_panic_on_out_of_memory_error, which suggests that there’s more to this than meets the eye. Specifically, a crash report where multiple threads are in the crashing routine is a common symptom of a third-party crash reporter disrupting the Apple crash report [1].

Given that, I recommend that you remove your third-party crash reporter and watch for reports coming in from the Apple crash report. If you manage to get an Apple crash report for this, please post the JSON version here.

In the meantime, can you look at your copy of the Apple crash report you posted above and see if it’s corrupted as well. That’ll let us rule out a DevForums problem [2].

If you can find a non-corrupted copy of the crash report, please try posting it. If the result continues to be corrupted, upload it to your favourite file sharing service and post a link.

Note Tip 14 of Quinn’s Top Ten DevForums Tips explains how to post links.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Implementing Your Own Crash Reporter says:

It’s a good idea to suspend all threads in your process early in your sign-al handler. However, there’s no way to completely eliminate this window.

[2] DevForums, sadly, has a long history of corrupting crash reports.

@DTS Engineer let's try with this one. The previous one was indeed corrupted and not complete for a reason.

https://we.tl/t-vKKjkbT890

Thanks for the crash report link. I downloaded that successfully.

As to why it was corrupted, I ran some tests here and confirmed that this a known bug in the DevForums platform (r. 136655649).

Unfortunately, this is still a human readable crash report (.crash) and I’d ideally like to get a JSON one (.ips). If you find one of those, please share it here.

However, I can still learn something from the report.

First up, you’re calling JavaScriptCore from two threads, thread 2 and thread 16. Are you using different contexts for those? With different virtual machines?

I’m not a JavaScriptCore expert but I believe that nothing good will come from you using the same context simultaneously on different threads. See this comment in the docs.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@DTS Engineer Unfortunately, that crash is our top crash on production and I can’t lock up the ips log files in Xcode Organizer. Our application still relies on TVMLKit so one context is used by me directly. About the second one, I suspect that a third-party framework, such as SmartLib_tvOS, is also using that for its own purposes.

thread 16:

11  JavaScriptCore                	0x00000001ad6880b8 llint_call_javascript + 8
12  JavaScriptCore                	0x00000001acc6692c JSC::Interpreter::executeCall(JSC::JSObject*, JSC::CallData const&, JSC::JSValue, JSC::ArgList const&) + 904 (Interpreter.cpp:1299)
13  JavaScriptCore                	0x00000001ac504224 JSObjectCallAsFunction + 752 (JSObjectRef.cpp:740)
14  JavaScriptCore                	0x00000001ac4770d0 -[JSValue callWithArguments:] + 364 (JSValue.mm:643)
15  SmartLib_tvOS                 	0x00000001056c1848 0x1056b8000 + 38984
16  Foundation                    	0x0000000197a5e320 __NSThread__start__ + 716 (NSThread.m:991)```

I can’t lock up the ips log files in Xcode Organizer.

Understood.

Honestly, I don’t think getting more crash reports is going to help you here, and you’d be better off investing that time in trying to reproduce the issue. Clearly it’s a common problem, and Apple TV has a pretty limited configuration space, so this is more feasible than it would be, say, if this were happening on a Mac.

First, survey your crash reports to see if they tell you anything about the device spread. The crash report you posted is from AppleTV6,2, that is, Apple TV 4K (1st generation). Are they all from that? Or are you seeing crashes from more modern Apple TVs.

Similarly, do this analysis for the tvOS version spread.

Also look at the app lifespan. In your crash the app had been running for 20-ish minutes. Is that typical? A memory problem like this is more likely to show up the longer the app has been running.

Finally, debugging. JavaScriptCore is part of the WebKit open source project, so you can actually see the source code in your backtrace. For example, frame 0 of thread 2 is pas_panic_on_out_of_memory_error, and its source code is here.

This source should give you some idea of the context of the error. Gosh, just the name is a useful hint: pas_panic_on_out_of_memory_error clearly indicates that JavaScriptCore has trapped because it’s run out of memory.

I don’t know enough about how JavaScriptCore is implemented to tell you whether it’s hitting some arbitrary limit within the code, or whether it’s gone to the system memory allocator to get more memory and that’s refused. However, the code is there for you to investigate and learn those answers for yourself.

And those answer might help you reproduce the problem. For example:

• If your process has run out of memory, you could create a build of your app that artificially uses extra memory and see if that triggers a crash like this.

• Alternatively, if JavaScriptCore has run out of its own internal memory, you could investigate how your app causes it to use that memory.

I’m sorry I don’t have an easy answer for you here. Debugging problems that only show up in the field is hard.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@DTS Engineer I started checking the crash again and from what I see all issues are related to specific device - AppleTV6,2. This device seems to be the first apple tv device that supports 4K. No crashes for newer devices so I wonder what could be the difference comparing to the newer generation of apple tv's.

Does this specific device have some more strict limitations about memory usage comparing to second generation?

Unfortunately that's the device that I do not have on my own to check it so maybe that's the reason why I cannot reproduce it.

@DTS Engineer I did some basic check of memory status after browsing a few screens of application and what I see is the high pick of MALLOC_NANO region but I'm not sure if it can cause some issues.

all issues are related to specific device - AppleTV6,2.

OK. That doesn’t come as a huge surprise. While we don’t publish RAM specs for Apple TV, it’s generally the case that newer models have more memory, and so the oldest model that supports 4K is likely to be the most memory constrained. It also has the slowest CPU, which means it takes longer to get through a task, and thus offers more chance of multiple tasks running simultaneously, all using memory.

However, while that useful info, it’s certainly not the full answer.

I think the next step is to beg, borrow, or ‘steal’ [1] one of these devices. You can then start stress testing your app on real hardware. Reproducing this yourself is a key step in creating a reliable fix.

Additionally, I think it’d be worthwhile digging into how JavaScriptCore manages memory. How much of a pool does it start with? How does it garbage collect that pool? And how do you get to the pas_panic_on_out_of_memory_error trap? Is it because the pool was exhausted? Or did JavaScriptCore try to get more memory from the system memory allocator and that failed?

I’m not a JavaScriptCore expert, so I don’t have direct answers for you. However, this is open source, so you can research it yourself. Additionally, the JavaScriptCore team is known to publish info about how their system works. My favourite article from them, Understanding Garbage Collection in JavaScriptCore From Scratch, might not have the info you’re looking for, but it should give you some threads to pull on.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Not literally steal. An Apple representative would never recommend such a thing (-: