Inconsistent ornament scale

I am developing an application which make use of 2 ornaments anchored to a volumetric window, one used a toolbar and one to display different views. The problem I am facing consistently is that the ornaments seems to scale up or down after moving the volume using the OS handle or starting a GroupActivity session.

This first image shows the ornaments as soon as I started the app, no dragging nor group activities:

This second images shows them as soon as I join a group activity session:

The map, which might seem smaller, has not been touched and has always the same scale.

In this last image I had just dragged the entire volume using the OS toolbar, resulting in the ornaments scaling down:

This is how the volume and the ornaments are declared:

WindowGroup(id: "CityVolume") {
            let cityVM = CityViewModel(volumeSize: CityView.initialVolumeSize)
            CityView(cityVM: cityVM)
                .ornament(attachmentAnchor: .scene(.bottomFront)) {
                    HStack {
                        TourismChartsButton()
                        LandmarksListButton()
                        CenterMapButton()
                        ToggleImmersiveSpaceButton()
                        TrafficDataButton()
                        BusLinesButton()
                    }
                    .padding()
                    .offset(z: 10)
                    .rotation3DEffect(Angle(degrees: 15), axis: (x: 1.0, y: 0.0, z: 0.0))
                }
                .ornament(attachmentAnchor: .scene(.back)) {
                    ZStack {
                        if AppModel.Instance.tourismVM.isChartViewVisible {
                            TourismChartsView()
                        }
                        if AppModel.Instance.busLinesVM.isDataViewEnabled {
                            BusLineView()
                        }
                    }
                }
                .task(observeGroupActivity)
                .onAppear {
                    appModel.cityVM = cityVM
                }
        }
        .windowStyle(.volumetric)
        .windowResizability(.contentSize)
        .volumeWorldAlignment(.gravityAligned)
        .defaultSize(CityView.initialVolumeSize, in: .meters)

It happens also without starting a SharePlay session, but not as frequently as during SharePlay. Experienced the same behaviour with toolbars. Am I doing something wrong with how I created the ornaments? Am I missing something?