I also have been blocked by a sudden failure to upload macOS archives to App Store Connect the past few days, but finally managed to fix it by forcing XCode 16.3 beta to regenerate the provisioning profile(s) of the offending app extensions that uses the App Groups capability. I deliberately include all the steps I took to provide as much detail as possible for anyone else facing the same issue.
The exact error I got is as follows (redacted for privacy reasons):
ITMS-90286: Invalid code signing entitlements - Your application bundle’s signature contains code signing entitlements that aren’t supported on macOS. Specifically, the “[group.com.xxx]” value for the com.apple.security.application-groups key in “com.xxx.appname.pkg/Payload/appname.app/Contents/PlugIns/AppNameExtension.appex/Contents/MacOS/AppNameExtension” isn’t supported. This value should be a string or an array of strings, where each string is the “group” value or your Team ID, followed by a dot (“.”), followed by the group name. If you're using the “group” prefix, verify that the provisioning profile used to sign the app contains the com.apple.security.application-groups entitlement and its associated value(s).
The XCode managed provisioning profiles can be found in "~/Library/Developer/Xcode/UserData/Provisioning Profiles". Upon inspecting the .provisionprofile for AppNameExtension, I found that there was no value set for the key "com.apple.security.application-groups".
Following the suggestions here, I installed XCode 16.3 beta (16E5104o) as I'd rather not deal with managing profiles & certificates. To force XCode to regenerate provisioning profiles, simply delete/move the files found at the path I mentioned above.
However, validating an archive failed with the same error even after regenerating all profiles! Upon examining the new provisioning profile for AppNameExtension, I realised that here was still no value set for the key "com.apple.security.application-groups".
This was puzzling, and I reluctantly went to the Apple Developer website and tried to generate a profile manually for AppNameExtension. This required me to generate a certificate signing request (CSR) with Keychain Access on my Mac. After doing all this, I downloaded the generated profile and inspected it. Again, there was no value set for the key "com.apple.security.application-groups"!
I was at a lost for what to do, but I gave XCode 16.3 beta another chance and tried regenerating the profiles again. This time, I played around with the App Groups capability under the Signings & Capabilities tab. I deleted the capability, then forced a regeneration by deleting the profile. Of course, there would be no value set for the key "com.apple.security.application-groups", since I disabled App Groups. Then, I added back the capability, selected my app group, and re-generated the profile once more. At last, there was a value set for the key "com.apple.security.application-groups"! And it corresponded to my app group: "group.com.xxx".
After this, I tried validating the same archive and it was successful, and I proceed to upload the archive to App Store Connect, which also succeeded of course.
I hope this long story helps someone solve this, as it was an extremely frustrating experience for me to do such a simple thing as releasing a patch update for my macOS app.