Exactly what problems are you having?
I reviewed those QT instructions. They don't look too bad.
However, your build scripts are significantly different. In particular, you're trying to use brew. Brew can be useful in some cases where people just want to download some open source tool on their Mac and run it without thinking. But if you're going to distribute something, especially on iOS, it should be considered a non-starter.
Trying to build both iOS and Mac versions of a framework using command line scripts can be very challenging. There are lots of subtle problems. You're essentially doing a cross-compile. It sounds like so many people have done this that there's good support for iOS. That's considered the default. So it makes sense you would have trouble with an oddball platform (relatively speaking) like macOS.
If you can get the source into a true Xcode project, it would be much, much easier to build multiple platforms.
For the record, ffmpeg is a challenging project. I did need to incorporate ffmpeg into a project myself a few years ago. Even on a major linux distribution, I still had to make changes to the ffmpeg source to get it to compile.
You seem to be taking the correct approach, just trying to do too much at once. I recommend starting over with the original QT instructions. Go more slowly this time. Don't try to build both X64 and ARM. Build an xcframework just for iOS. Then add macOS Apple Silicon, which should be very similar. I expect all the complications would be from the X86 build. ffmpeg definitely supports that, but many low-level compilation settings, and maybe even source files, will be radically different.
