Storing metadata alongside files outside of sandbox

Hello all,

I'm the developer of REHex, a hex editor which I have been distributing as an app bundle outside of the app store for a few years.

REHex allows assigning various bits of metadata (comments, data types, etc) which get stored as filename.rehex-meta alongside the original filename, this works fine when the app is just a standalone bundle, however, when distributed via the app store, sandboxing seems to be mandatory, and there doesn't appear to be any obvious way to get permission to read/write such files.

As fallbacks, I've considered adding support for storing the metadata as an extended attribute instead (which breaks compatibility, and won't translate when the file is on a FAT/etc filesystem or network share), or popping up the save/load dialog a second time for the user to select a .rehex-meta file, adding it to the list of whitelisted files for the application (keeps compatibility, but UX is clunky).

Are there any ways I can work around this, or perhaps other methods I should consider for storing the metadata in an Apple-tolerant manner?

Thanks

Answered by DTS Engineer in 834096022

App Sandbox does have a facility for this sort of thing, but I’m concerned that it might not work for your specific use case. Specifically, it supports the notion of related files (NSIsRelatedItemType). See the discussion in Use related file access to work with groups of files section of Accessing files from the macOS App Sandbox.

This is designed for folks working with specific document types. For example, imagine an app that opens .foomoov files which might have an associated .foomoovsub file containing subtitles. My concern in your case is that your app opens all types of files, and it’s not clear if the related files mechanism will work in that case.

Still, it’s worth you running some tests. My advice on that front is to start with the expected use case, that is, with a declared document type and a declared related file (.foomoov and .foomoovsub in the example above). Once you get that working you can try it out in the general case that you actually care about.

Lemme know if you run into any snags with this stuff. This is quite obscure. I’ve never had cause to create a test project for it, so I don’t have any specifics to share )-:

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

App Sandbox does have a facility for this sort of thing, but I’m concerned that it might not work for your specific use case. Specifically, it supports the notion of related files (NSIsRelatedItemType). See the discussion in Use related file access to work with groups of files section of Accessing files from the macOS App Sandbox.

This is designed for folks working with specific document types. For example, imagine an app that opens .foomoov files which might have an associated .foomoovsub file containing subtitles. My concern in your case is that your app opens all types of files, and it’s not clear if the related files mechanism will work in that case.

Still, it’s worth you running some tests. My advice on that front is to start with the expected use case, that is, with a declared document type and a declared related file (.foomoov and .foomoovsub in the example above). Once you get that working you can try it out in the general case that you actually care about.

Lemme know if you run into any snags with this stuff. This is quite obscure. I’ve never had cause to create a test project for it, so I don’t have any specifics to share )-:

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hi Quinn,

Thanks for the detailed response. I've done a bit of testing with the related files mechanism and can confirm that when I expose a "rehex-foo" and "rehex-meta" file type, I am able to write a foo.rehex-meta file when writing a foo.rehex-foo file using NSFileCoordinator, however I have not been able to get it to work in any other cases - even foo.rehex-foo.rehex-meta is not allowed when writing a foo.rehex-foo file.

The NSFileCoordinator.coordinateWritingItemAtURL() method invokes my callback even in the cases that won't work, with the original path which it still doesn't allow access to.

For reference, here are the relevant sections of the Info.plist I tested with:

<key>UTExportedTypeDeclarations</key>
<array>
<dict>
<key>UTTypeIdentifier</key>
<string>net.solemnwarning.rehex-meta</string>
<key>UTTypeConformsTo</key>
<array>
<string>public.json</string>
</array>
<key>UTTypeDescription</key>
<string>Metadata of a file opened in REHex</string>
<key>UTTypeTagSpecification</key>
<dict>
<key>public.filename-extension</key>
<array>
<string>rehex-meta</string>
</array>
</dict>
</dict>
<dict>
<key>UTTypeIdentifier</key>
<string>net.solemnwarning.rehex-foo</string>
<key>UTTypeDescription</key>
<string>Test file</string>
<key>UTTypeTagSpecification</key>
<dict>
<key>public.filename-extension</key>
<array>
<string>rehex-foo</string>
</array>
</dict>
</dict>
</array>
<key>CFBundleDocumentTypes</key>
<array>
<dict>
<key>CFBundleTypeRole</key>
<string>Editor</string>
<key>LSHandlerRank</key>
<string>Default</string>
<key>LSItemContentTypes</key>
<array>
<string>net.solemnwarning.rehex-foo</string>
</array>
</dict>
<dict>
<key>NSIsRelatedItemType</key>
<true/>
<key>CFBundleTypeRole</key>
<string>Editor</string>
<key>LSHandlerRank</key>
<string>Owner</string>
<key>LSItemContentTypes</key>
<array>
<string>net.solemnwarning.rehex-meta</string>
</array>
</dict>
</array>

What do you need these metadata files for? More specifically, do you need these metadata files or do your users?

I deal with the same thing with GIS files. In my case, I have some open-source libraries that want to automatically write those metadata files. Obviously that fails in the sandbox.

But what I've been able to do is define an alternate location for these sidecar files. In my case, the library already supports a "PROXY_DIR" environment variable for read-only locations. So I just used that. I could have also hacked up the source, but I got lucky.

What this means is what when various legacy software bits want to write a sidecar file, it always succeeds. It writes the sidecar file in an appropriate location inside my app's sandbox container. Then when other bits of the legacy software look for the sidecar data, it's there and always available.

If I needed to export the sidecar files (which I won't), I could do that by allowing the user to specify an entire output directory and write all necessary files under a pre-determined name.

Hi Etresoft,

I'm not really sure what you mean by the "do you need these metadata files or do your users?" question - the metadata files store any comments or other annotations made by the user.

If there is no way to make the <allowed file>.rehex-meta filenames accessible from within the sandbox, I could instead store the metadata within the application sandbox and provide import/export functionality, but I don't like this approach since the metadata is then hidden away rather than in a visible/easily copied place along with the file they are working on. Additionally, if the user moves/deletes the file, then the metadata within the application sandbox would become detached.

Written by solemnwarning in 834124022
I have not been able to get it to work in any other cases

Yeah. That’s pretty much what I expected )-:

Unfortunately that rules out the option of storing your metadata as a related file. The remaining options I see are:

  • Store it in extended attributes.

  • Store it in a database in your sandbox.

Both of these have clear drawbacks.

Regarding the first, you wrote:

Written by solemnwarning in 780237021
I've considered adding support for storing the metadata as an extended attribute instead (which breaks compatibility, and won't translate when the file is on a FAT/etc filesystem or network share),

On volumes without extended attribute support the system will store extended attributes in an AppleDouble file. That’s a public file format, so you could support it on other platforms.

Regarding the second approach, you wrote:

Written by solemnwarning in 834280022
if the user moves/deletes the file, then the metadata within the application sandbox would become detached.

There are steps you can take to avoid problems in the move and rename case. macOS has fairly good facilities to track the identity of files across a move or rename. It’s not perfect though, so you’re going to need some sort of recovery path.

It’s hard to clean up when the file is deleted, but that’s only a problem if this metadata is huge. Most users are unlikely to notice you ‘leaking’ the small amounts of storage used for orphaned metadata.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Storing metadata alongside files outside of sandbox
 
 
Q