How to Symbolicate an Apple Silicon Panic?

First off, please file a bug about the documentation side of this and then post the number back here. We really should do a much better job of explaining exactly what all this involves.

Next, a clarification on this:

That's not even a program instruction - that's within a cstring. No, that cstring isn't involved in anything pertaining to the intentional panic I am expecting to see.

Which gives the following incorrect result:

  Address: KextName[0x0000000000003e40] (KextName.__TEXT.__cstring + 14456)
  Summary: "ffer has %d retains\n"

That's not even a program instruction - that's within a cstring. No, that cstring isn't involved in anything pertaining to the intentional panic I am expecting to see.

What you described here isn't exactly what's going on.

The symbolication process is inherently, for lack of a better word, "stupid". Conceptually, your symbol file is basically just a long list of strings and offset pair, so all symbolication does is find the closest offset "smaller" than the address it was given, then returns "String + remainder". The problem with this line:

  Address: KextName[0x0000000000003e40] (KextName.__TEXT.__cstring + 14456)

...is that you're "remainder" is ENORMOUS. 14456 bytes-> 14Kb. It's not pointing you at a string, it's pointing you at "crazy offset" from some other string. If you work the math out "manually", you can how this all looks very "odd". Here's the basic math:

NOTE: I've thrown out the upper 32 bits, as it makes the numbers easier to work with

Base Address: 0xfffffe001203bfd0 -> 0x1203bfd0

Crashing Address: 0xfffffe001203fe10 -> 0x1203fe10

Crash Offset = Crashing Address - Base Address
0x3E40 = 0x1203fe10 - 0x1203bfd0

Symbolication Offset = "... + 14456" -> 0x3878


Offset of "KextName.__TEXT.__cstring" = Crash Offset - Symbolication Offset

0x5C8 = 0x3E40 - 0x3878

0x5C8 -> 1480 bytes

In other words, the actual location of "KextName.__TEXT.__cstring" is 1480 bytes from the "start" of your library, which actually seems somewhat reasonable.

In other words, the question here isn't "why am I pointing at a c string", it's "why aren't there any other symbols in the ~13,000 bytes after that cstring". Here's what I'd actually do next:

• Symbolicate the other frames you've got. This is a quick way to differentiate between "something is wrong with THIS symbol" vs "something systemic is wrong".

• "dwarfdump" will print the contents of a dsym file, which will either show that the symbols

• Assuming there isn't any issue with the symbol file itself, then this could be because ASLR side your KEXT and hasn't been accounted for correctly. I think that's what this slide actually is:

KernelCache slide: 0x000000000b1c0000

...but I'm not sure if that was already accounted for in your addresses or not. However, if your dsym is otherwise "valid", then you should also be able to see a difference between the offset the kernel returned for the symbol it did find (KextName.__TEXT.__cstring-> 0x5C8) and the offset the dsym lists for that symbol. You can then adjust your address offsets to remove that difference.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Documentation bug reported, as FB14929037.

Yes - the offset from the string is quite large, but I just want useful output such as "ThisFunctionWasInvolved" from "ThisFunctionCausedAPanic.cpp, line 455".

Symbolicate the other frames you've got. This is a quick way to differentiate between "something is wrong with THIS symbol" vs "something systemic is wrong".

Yes, this is a “something systemic is wrong” situation. I was trying to reduce the amount of information for readers to wade through, but I am not getting useful results for any efforts at symbolicating the addresses on the back trace or the PC (program counter).

” dwarfdump" will print the contents of a dsym file, which will either show that the symbols

dwarfdump works on dSYM files, but apparently not on debug-builds-with-symbols-in-the-binary. As such, I rebuilt with separating the symbols into a dSYM file. dwarfdump provides what appears to be correct information for the symbols, but for a sanity check here is a blurb from the start of the dump:

KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME(arm64e):	file format Mach-O arm64

.debug_info contents:
0x00000000: Compile Unit: length = 0x000001f9, format = DWARF32, version = 0x0004, abbr_offset = 0x0000, addr_size = 0x08 (next unit at 0x000001fd)

0x0000000b: DW_TAG_compile_unit
              DW_AT_producer	("Apple clang version 13.1.6 (clang-1316.0.21.2.5)")
              DW_AT_language	(DW_LANG_C99)
              DW_AT_name	("/Users/username/Library/Developer/Xcode/DerivedData/KEXT_NAME-stuff/ArchiveIntermediates/KEXT_NAME/IntermediateBuildFilesPath/KEXT_NAME.build/Release/KEXT_NAME.build/DerivedSources/KEXT_NAME_info.c")
              DW_AT_LLVM_sysroot	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk")
              DW_AT_APPLE_sdk	("MacOSX12.3.sdk")
              DW_AT_stmt_list	(0x00000000)
              DW_AT_comp_dir	("/Users/username/sandbox/codedirectory")
              DW_AT_APPLE_optimized	(true)

Since the binary is different, I triggered a new panic to work with:

The command line utility atos should be helpful as well, but here’s an example of what I get as output from that, when using full dSYM file information:

zsh_prompt% atos -arch arm64e -o KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME  -l 0xfffffe0015423fd0 0xfffffe0015426e2c
0x00002e5c (in KEXT_NAME)

If I manually look up the 0x2e5c offset in the output from dwarfdump, I can see the following:

0x00002e5c:       DW_TAG_formal_parameter
                    DW_AT_type	(0x0000000000009408 "const IOService *")
                    DW_AT_artificial	(true) 

But that's definitely not conveying the one-step-to-symbolication that atos is supposed to be able to perform. It does not appear that I am using it incorrectly, but I am definitely not getting helpful results from it.

Following the idea that perhaps the kernel slide is not being properly dealt with, I followed this link ( https://lists.apple.com/archives/darwin-kernel/2014/Jan/msg00011.html ) for advice on how to deal with kernel slide (below is all commands listed, and lack of useful information provided):

So to answer all of the questions:

1. Systemic issue, I am having trouble interpreting a kernel panic log and getting valid symbols as a result of address lookup for Apple Silicon
2. dSYM file appears to be built properly
3. atos is not providing helpful information, despite having proper dSYM file
4. lldb commands to load images and deal with kernel slides does not yield helpful information

Kevin Elliot, DTS Engineer, solved this in a private code-level support ticket.

There is yet another offset that was not being accounted for, but can be found using the command line tool otool:

zsh_prompt% otool -arch arm64e -l /Library/Extensions/KEXT_NAME.kext/Contents/MacOS/KEXT_NAME

will give a lot of output information, but a section with this information:

      cmd LC_SEGMENT_64
  cmdsize 232
  segname __TEXT_EXEC
   vmaddr 0x0000000000004000
   vmsize 0x0000000000010000

Shows that an address of 0x4000 needs to be accounted for when performing the atos lookups.

By adding that value to the load address of the kext when calling atos, we get the correct / expected answer.

zsh_prompt% atos -arch arm64e -o KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME  -l <load address + (vmaddr of __TEXT_EXEC section)>  0xfffffe0015426e2c

com_company_kext_name::expectedFunction(OSObject*, void*, void*, void*, void*) (in KEXT_NAME) (SourceFile.cpp:1999)

Such a relief! Kudos to Kevin Elliot for finding a solution to this.

Kudos to Kevin Elliot for finding a solution to this.

You're very welcome. atos needs to be updated to address this, probably by adding a new option when importing the base address. However, until then the math above should work fine.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

To any future readers, you can find a more detailed breakdown of this process that also covers kernel (not just KEXT) frames here.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware