Audio session activation occasionally fails from CarPlay

I'm working on adding CarPlay support to an audio app and am running into an issue. Occasionally, when a user opens the app from CarPlay while the main app scene is either not connected or is currently in the background, I will receive an error when attempting to activate the audio session. The code below mimics my setup:

do {
    try AVAudioSession.sharedInstance().setCategory(.playback, mode: .spokenAudio)
    try AVAudioSession.sharedInstance().setActive(true)
} catch {
    print(error) // NSOSStatusErrorDomain - 560557684: Session activation failed
}

That error code maps to AVAudioSession.ErrorCode.cannotInterruptOthers.

Once in this state, all subsequent attempts to play different pieces of content will fail. However, things will start working normally if the user opens the app on their phone and tries again from CarPlay (while the app is in the foreground on their phone).

I'm not sure why it would behave this way and want to note that I do have the audio background mode capability enabled.

Has anyone else encountered this? Are there any workarounds or changes I could make to prevent this from happening?