How to add and remove child entities to a rigged entity in RealityKit?

I am currently using RealityKit (perspective camera) to render a character in my swiftUI app. The character has customization such as clothing items and hair and all objects are properly weighted to the rig. The way the model is setup in Blender is like so: Groups of objects that will be swapped (ex: Shoes -> Shoes objects) and an armature. I then export it to usdc with all objects active. This is the resulting hierarchy:

Before exporting for the animation (armature modifier applied), I simply had to store the Model entities and swap them in but now when I export with the Armature Modifier applied, so that animations get exported, the ModelComponent gets flattened to the armature and swapping entities and applying new materials to them is no longer as simple.

Here's a demo blend file and usdc export with a setup like mine, having an animated bone to swing a cube and sphere, to be swapped so that only one is visible https://www.dropbox.com/scl/fo/be2q6qcztc83z7c4gj1w0/AMapxWc_ip2KZ8oTOYDUMv8?rlkey=rcdaggcxq06dyen09mw5mqmem&st=bnc0d7j0&dl=0

This is how I'm loading the entity and removing a part, with the demo files

import SwiftUI
import RealityKit

struct SwapDemoView: View {

    var body: some View {
        RealityView { content in
            let camera = PerspectiveCamera()
            camera.transform.translation = SIMD3(x: 0, y: 0.1, z: 3)
            
            guard let root = try? await Entity(named: "simpleSwapDemo") else { fatalError("simpleSwapDemo.usdc is not present") }
            print(root) // Get initial hierarchy
            guard let cube = root.findEntity(named: "Cube") else { fatalError("Entity cube doesn't exist") }
            cube.removeFromParent() // <-- Cube is still visible after removal
            print(root) // Get hierarchy to confirm removal of cube
            
            let resource = root.availableAnimations[0]
            root.playAnimation(resource.repeat())
            
            content.add(root)
            content.add(camera)
        }
        .background(.white)
    }
}

And this is what the entity hierarchy looks like in RealityKit before cube removal

▿ 'root' : Entity, children: 1
  ⟐ SynchronizationComponent
  ⟐ AnimationLibraryComponent
  ⟐ Transform
  ▿ 'Armature' : ModelEntity, children: 2
    ⟐ SynchronizationComponent
    ⟐ ModelComponent
    ⟐ SkeletalPosesComponent
    ⟐ AnimationLibraryComponent
    ⟐ Transform
    ▿ 'Armature' : Entity
      ⟐ SynchronizationComponent
      ⟐ Transform
    ▿ 'Primitives' : Entity, children: 2
      ⟐ SynchronizationComponent
      ⟐ Transform
      ▿ 'Sphere' : Entity, children: 1
        ⟐ SynchronizationComponent
        ⟐ Transform
        ▿ 'Sphere' : Entity
          ⟐ SynchronizationComponent
          ⟐ Transform
      ▿ 'Cube' : Entity, children: 1
        ⟐ SynchronizationComponent
        ⟐ Transform
        ▿ 'Cube' : Entity
          ⟐ SynchronizationComponent
          ⟐ Transform

And here's the hierarchy after removal

▿ 'root' : Entity, children: 1
  ⟐ SynchronizationComponent
  ⟐ AnimationLibraryComponent
  ⟐ Transform
  ▿ 'Armature' : ModelEntity, children: 2
    ⟐ SynchronizationComponent
    ⟐ ModelComponent
    ⟐ SkeletalPosesComponent
    ⟐ AnimationLibraryComponent
    ⟐ Transform
    ▿ 'Armature' : Entity
      ⟐ SynchronizationComponent
      ⟐ Transform
    ▿ 'Primitives' : Entity, children: 1
      ⟐ SynchronizationComponent
      ⟐ Transform
      ▿ 'Sphere' : Entity, children: 1
        ⟐ SynchronizationComponent
        ⟐ Transform
        ▿ 'Sphere' : Entity
          ⟐ SynchronizationComponent
          ⟐ Transform

And this is the result:

What's the best practice here? Should animation be exported separately and then applied to the skeleton? If so, how is that achieved? I'm not really sure how to proceed here.