Notarization service says signature invalid, but codesign says it's fine

At the top of Placing Content in a Bundle you’ll find this warning:

If you put content in the wrong location, you may encounter hard-to-debug code signing and distribution problems. These problems aren’t always immediately obvious. For example, when building a Mac app, incorrectly placed code might work during day-to-day development, but might cause problems during notarization.

And this is exactly what’s come to pass )-:

The best way to solve this problem is to follow the rules in Placing Content in a Bundle. However, that can be tricky when dealing with code structures from other platforms. We talk about this in Embedding nonstandard code structures in a bundle. I also go into the rpath stuff in more more detail in the Dynamic Library Standard Setup for Apps forums post.

If you’re unable to completely rework your structure, see the Use symlinks for gnarly edge cases section of that doc.

Finally, it’s possible to bend these rules a bit, but the more you bend the more problems you run into. For example, code signing and notarisation are both quite happy for you to put code in an area reserved for data. However, it looks like you’re using Xcode and it’s more persnickety. That’s one of the advantages of the symlink approach. As far as Xcode is concerned, you’re just embedding:

• A bunch of pre-built binaries into a location reserved for executables

• A bunch of pre-built libraries into a location reserved for libraries

• A weird directory hierarchy, that happens to include a lot of symlinks, into an area reserved for data

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thank you for your super detailed answer! And, oops, sorry about missing (or probably forgetting about) these warnings in the docs.

I've been looking into various solutions, mostly gravitating towards the “symlinks for gnarly edge cases” solution. However, I ran into something perplexing:

If I create an archive build within Xcode, then submit it using “Direct Distribution”, the process fails with a “The signature of the binary is invalid.” issue. But! If I then submit this same, archived .app, using notarytool, this submission then succeeds!

Is this what you were referring to, when mentioning Xcode being persnickety? I don't understand the implications here—is Xcode performing the notarization submission differently? Is it OK to just rely on the command line for notarization?

Here's how I submit via Terminal. I'm using the same AC profile in both cases:

xcrun notarytool submit --keychain-profile "AC_PASSWORD" --wait ZippedArchivedApp.zip 

Is this what you were referring to, when mentioning Xcode being persnickety?

Yep. The exact structure of a bundle is not well specified, so different subsystems — Xcode, notary, Gatekeeper, and so on — have different ideas about how things should work. The goal of Placing Content in a Bundle is to provide concrete advice as to what should work everywhere. When you don’t follow those rules, it’s hard to provide a coherent explanation as to what behaviour you’ll see in practice.

If I then submit this same, archived .app, using notarytool, this submission then succeeds!

Hmmm. That suggests that the app within your Xcode archive is signed with a Developer ID signing identity. That’s not something I recommend, as I explain in The Care and Feeding of Developer ID.

The standard workflow here is:

1. You use Apple Development signing for day-to-day work.

2. When you do a Product > Archive, the resulting app within Xcode archive is still Apple Development signed.

3. You can’t distribute such an app. Instead, when you use the Xcode organiser to distribute it, Xcode re-signs the app for that distribution channel. So, if it’s going to App Store Xcode uses Apple Distribution and if it’s directly distributed then Xcode uses Developer ID.

And this re-sign process is one of the places you run into problems if you have a wacky bundle structure.

Notably, most folks who have a nonstandard bundle structure don’t use Xcode. Rather, they build and sign their app manually, using tools like CMake. That lets them bend the rules a bit because notary and Gatekeeper are less persnickety [1].

One option here is to follow the Distribute App > Custom > Direct Distribution > Export workflow. That’ll give you a copy of what Xcode would’ve submitted to the notary service. You can submit using notarytool to see if it reproduces the problem. You can then rummage inside to see exactly what Xcode has done.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Well, to be fair, they’re persnickety on different axes (-:

(interesting reads, thank you—just learned a bunch of things)

My previous message was misleading though, sorry: when notarizing through the command line, I was submitting the .app that Xcode re-signed with my Developer ID identity, not the original .app from the archive. Or said differently, I was submitting the .app found in *.xcarchive/Submissions, not *.xcarchive/Products.

Still, the confusion remains! Because that means I'm submitting the exact same .app for notarization; and so in Xcode notarization fails, and using xcrun notarytool it succeeds. So strange.

For now we should be fine notarizing through the command line, but if you do have insight into why there's a difference in the notarization service's behavior, I'm very much interested.

Yeah, this is one of the more strange cases I’ve seen |-:

Lemme see if I have this straight:

1. You have an Xcode project that builds your app.

2. You choose Product > Archive.

3. In the Xcode organiser, you select that archive, click the Distribute App button, and run through the Direct Distribution workflow.

4. That fails with an invalid signature error.

5. Still in the organiser, you control-click on the archive and choose Show in Finder.

6. You navigate to Submissions/<UUID>, where <UUID> is the UUID of the notary request that failed.

7. You package that up into a zip archive.

8. And notarise that using notarytool.

9. The notary service accepts that.

Is that correct?

If so, please post the request UUIDs from step 3 and step 8.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Yes, that's exactly it! I just reproduced the issue, strictly running through your list, to make extra sure.

Here's the Xcode submission UUID: 417D4144-65D6-4C05-A219-4B2B5039AF7E
And the notarytool submission UUID: 035482f3-855c-455f-bd60-6be63ceefd61

And for good measure, a screenshot of the submission failure in Xcode:

Thank for those UUIDs. I asked the notary team for a copy of those submissions, so I could see exactly what the submitted zip archives look like, and that revealed a clear problem.

Consider this file listing of your notarytool submission:

% unzip -t ok-035482f3-855c-455f-bd60-6be63ceefd61.zip         
Archive:  ok-035482f3-855c-455f-bd60-6be63ceefd61.zip
    …
    testing: Wwwwwwww.app/Contents/MacOS/graphviz/bin/gvmap.sh   OK
    testing: __MACOSX/Wwwwwwww.app/Contents/MacOS/graphviz/bin/._gvmap.sh   OK
    …
No errors detected in compressed data of ok-035482f3-855c-455f-bd60-6be63ceefd61.zip.

Note I’ve redacted stuff using my ‘patented’ ‘first letter’ algorithm [1].

First up, the __MACOSX indicates that you’ve sequestered Mac metadata. That doesn’t make sense in this context. I explain why in Extended Attributes and Zip Archives.

However, the real issue is that you have Mac metadata at all! Unpacking the archive I see this:

% xattr Wwwwwwww.app/Contents/MacOS/graphviz/bin/gvmap.sh
com.apple.cs.CodeDirectory
com.apple.cs.CodeRequirements
com.apple.cs.CodeRequirements-1
com.apple.cs.CodeSignature

These code signature extended attributes are a problem. They indicate that you’re signing data as code, and that causes all sorts of weird issues. And the reason why these attributes are present is that your packaging is off: You’ve put data (gvmap.sh) in a location reserved for code (Contents/MacOS).

When I list your Xcode submission I see this:

% unzip -t ng-417d4144-65d6-4c05-a219-4b2b5039af7e.zip | grep '\._'
% 

There are no AppleDouble files, meaning your extended attributes have been dropped. Unpacking the archive I confirm that:

% xattr Wwwwwwww.app/Contents/MacOS/graphviz/bin/gvmap.sh
% 

and that results in a broken code signature:

% codesign --verify --deep --strict Wwwwwwww.app 
Wwwwwwww.app: code object is not signed at all
In subcomponent: /Users/quinn/Administrivia/DevForums work/Notarization service says signature invalid, but codesign says it's fine/ng-417d4144-65d6-4c05-a219-4b2b5039af7e/Wwwwwwww.app/Contents/MacOS/graphviz/bin/gvmap.sh

So, in summary:

1. Your app contains a nonstandard code structure, that is, one that doesn’t follow the rules in Placing Content in a Bundle.

2. This causes codesign to store critical information in extended attributes.

3. The notarytool notarisation path preserves those attributes, and hence notarisation succeeds.

4. The Xcode notarisation path drops those attributes, resulting in a broken code signature and a notarisation failure.

My advice:

• Stop sequestering Mac metadata. That only makes sense when you’re going to send the zip archive to a non-Apple system. Instead, create your zip archive using ditto as explained in Packaging Mac software for distribution.

• In the short term, continue notarising using notarytool.

• In the medium term, rework your app to follow the rules in Placing Content in a Bundle. That’s the only path to long-term compatibility.

• For specific advice on how to do this, see Embedding nonstandard code structures in a bundle.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] See Posting a Crash Report.

Thank you very much for looking into this; this makes perfect sense now.

When I first looked into fixing the file structure to fit the rules, my first instinct was to experiment by submitting slightly modified versions for notarization (i.e. add/delete/move a file, then resign, zip and notarize) until notarization broke again. This was foiled, however, since CLI notarization always succeeded!

Now that I know that I can reproduce this stricter behavior found in Xcode (by dropping extended attributes while zipping), it'll be easy to experiment and check my work. Thank you!