Unable to Start macOS VM via Virtualization API in a Sandboxed Launchd Service

Question

Created 2w

Replies 7

Boosts 0

Participants 2

I’m encountering an issue when trying to start a macOS VM using Apple’s Virtualization framework in a sandboxed environment.

When I create a standalone Xcode project, the VM launches successfully. However, when I integrate the same code into my existing project—where the VM is launched by a service started via launchd and running in a sandbox—it fails with the following error:

Internal Virtualization Error: Failed to issue USB HCI sandbox extension

To resolve this, I tried adding the com.apple.security.device.usb entitlement. But after doing that, the app started crashing with the following trace :

Application Specific Signatures:
SYSCALL_SET_USERLAND_PROFILE

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libsystem_secinit.dylib       	       0x19a7141bc _libsecinit_appsandbox.cold.9 + 84
1   libsystem_secinit.dylib       	       0x19a713324 _libsecinit_appsandbox + 2080
2   libsystem_trace.dylib         	       0x18c2326cc _os_activity_initiate_impl + 64
3   libsystem_secinit.dylib       	       0x19a712ab0 _libsecinit_initializer + 80
4   libSystem.B.dylib             	       0x19a72a32c libSystem_initializer + 280
5   dyld                          	       0x18c162efc invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 444
6   dyld                          	       0x18c19f864 invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 324
7   dyld                          	       0x18c1bf58c invocation function for block in mach_o::Header::forEachSection(void (mach_o::Header::SectionInfo const&, bool&) block_pointer) const + 240
8   dyld                          	       0x18c1bc318 mach_o::Header::forEachLoadCommand(void (load_command const*, bool&) block_pointer) const + 208
9   dyld                          	       0x18c1bda58 mach_o::Header::forEachSection(void (mach_o::Header::SectionInfo const&, bool&) block_pointer) const + 124
10  dyld                          	       0x18c19f334 dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 516
11  dyld                          	       0x18c162cb4 dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 176
12  dyld                          	       0x18c16e530 dyld4::PrebuiltLoader::runInitializers(dyld4::RuntimeState&) const + 44
13  dyld                          	       0x18c1848b0 dyld4::APIs::runAllInitializersForMain() + 88
14  dyld                          	       0x18c147e00 dyld4::prepare(dyld4::APIs&, mach_o::Header const*) + 3092
15  dyld                          	       0x18c1471d8 dyld4::start(dyld4::KernelArgs*, void*, void*)::$_0::operator()() const + 236
16  dyld                          	       0x18c146b4c start + 6000

I suspect this might be due to the provisioning profile, which doesn’t seem to include the required entitlement. However, I haven’t found a way to explicitly add this entitlement to the provisioning profile.

My questions are:

How can I add com.apple.security.device.usb to the provisioning profile?
Is there a way to confirm that adding this entitlement would resolve the issue?
Are there recommended steps to debug and test Virtualization framework usage in a sandboxed environment (especially when launched as a service)?
I also tried disabling SIP and AMFI, but the crash still occurs. Is there a better way to work around or test this in development?

Any insights or suggestions would be greatly appreciated!

Answered by DTS Engineer in 842123022

I think the answer here is “Don’t do that.”

Virtualization framework is meant to be used by virtualisation apps. Such apps obviously run in a user context. A launchd agent is sufficiently close to a GUI user context that I fully expect that Virtualization will work there.

Old school Unix-y context switching techniques, like the double fork thing, are generally OK if you limit yourself to Unix-y APIs, but they often cause problems when you use higher-level frameworks. That’s because they result in an inconsistent execution context, where part of the context has switched and part hasn’t. I discuss this ideal in a lot more detail in TN2083 Daemons and Agents (it’s old, but still remarkably relevant).

Having said that, the immediate cause of your crash seems to be an App Sandbox re-initialisation issue. See Resolving App Sandbox Inheritance Problems. That’s not super surprising given your current setup, because this process has inherited its sandbox from your app.

The best path forward depends on the expected lifecycle of this process:

If you want to tie its lifecycle to your app, make it an XPC service.
If you want it to run independently of your app, make it a launchd agent.

Both of these are started by launchd, and thus have full control over their sandbox. You completely avoid the tricky problems associated with sandbox inheritance.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Boost

Answer 1

DTS Engineer OP

Apple

2w

Is this a launchd daemon? Or a launchd agent?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 2

user_29e OP

2w

The parent process is a launchd agent, running as logged in user.

Basically my UI app launches this service, which is a launchd agent, which starts a macOS VM app inside (posix_spawn), then the macOS VM app has the problem. Everything runs inside sandbox.

Just repeating, if I launch this macOS VM independently, it launches. Problem comes only if I launch it under a parent. The macOS VM app does not work if I start as child of my UI app too (sandboxed).

0

Answer 3

DTS Engineer OP

Apple

2w

The parent process is a launchd agent

OK. Does it have a LimitLoadToSessionType property? If so, what’s it set to?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 4

user_29e OP

2w

Actually, the service is started dynamically using QProcess::startDetached, not as a launch agent or launch daemon. Apologies for the confusion earlier — I initially misunderstood the setup (this wasn’t written by me, so my analysis was off).

I see that this service process runs as a child of launchd, and it appears under system/subdomains when I run launchctl print system. Because of that, I can’t access its LimitLoadToSessionType.

Would starting the service as a launch agent help resolve this issue? I also tried launching the macOS VM as a child of my GUI application (which itself is a launch agent with LimitLoadToSessionType set to "Aqua"), but that also failed to start the VM process.

0

Answer 5

DTS Engineer OP

Apple

2w

Hmmm, there’s a lot to unpack here, but let’s start with this:

the service is started dynamically using QProcess::startDetached

That’s not an Apple API. What Apple API is it calling under the covers?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 6

user_29e OP

2w

QProcess::startDetached is from qt, when checked, it uses the double fork method to detach child process (GUI process forks child1, child1 forks child2, then child1 quits - so the child2 becomes child of launchd). Then it starts service using 'execv'.

Now this service runs detached, as a child of launchd, and I'm trying to start the macOS VM process under this.

0

Answer 7

DTS Engineer OP

Apple

1w

Recommended

I think the answer here is “Don’t do that.”

Virtualization framework is meant to be used by virtualisation apps. Such apps obviously run in a user context. A launchd agent is sufficiently close to a GUI user context that I fully expect that Virtualization will work there.

Old school Unix-y context switching techniques, like the double fork thing, are generally OK if you limit yourself to Unix-y APIs, but they often cause problems when you use higher-level frameworks. That’s because they result in an inconsistent execution context, where part of the context has switched and part hasn’t. I discuss this ideal in a lot more detail in TN2083 Daemons and Agents (it’s old, but still remarkably relevant).

Having said that, the immediate cause of your crash seems to be an App Sandbox re-initialisation issue. See Resolving App Sandbox Inheritance Problems. That’s not super surprising given your current setup, because this process has inherited its sandbox from your app.

The best path forward depends on the expected lifecycle of this process:

If you want to tie its lifecycle to your app, make it an XPC service.
If you want it to run independently of your app, make it a launchd agent.

Both of these are started by launchd, and thus have full control over their sandbox. You completely avoid the tricky problems associated with sandbox inheritance.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0