public API which allows to get information about APFS

FYI, you also asked about APFS on this thread and I've moved you answer to that question on this thread.

Is it bug or ATTR_VOL_SPACEUSED is unsupported on macOS 11?

I don't know exactly why it was done, but I believe it was an intentional choice while the team was sorting out how APFS should "present" itself to the VFS system. ATTR_VOL_SPACEUSED is actually calculated in the VFS layer (you can see the code here if you're curious) and I suspect that at the time "vs.f_bused" wasn't giving a value that matched well with the "expectation" of the rest of the system. Disabling ATTR_VOL_SPACEUSED was the easiest way to prevent that from causing broader problems.

That leads to here:

Are there any other way to get space which is used on an APFS volume? (C++)

So, the "basic" answer is what's the described in "Checking Volume Storage Capacity". The more complicated answer starts with "What are you actually trying to do?". The big issue here isn't simply that APFS is more complicated (which it definitely is), it's that part of that transition was a broad shift in how the system manage storage.

Historically issues like slow I/O performance (particularly random I/O) and the file system locking architecture limited how aggressively the system relied on the file system. In concrete terms, if "enough" was happening at the same time, overall performance could collapse as random I/O spiked and/or locking bottlenecks in HFS+ hampered the file systems ability to process modifications. This generally wasn't an issue for normal user usage but it did mean that the system tried to limit how much "extra" I/O the system generated.

The transition to SSDs removed the first issue (I/O performance) and APFS largely resolved the second (file system locking), enabling a broad change in how the system manages local storage. The modern approach is built around the idea that unused storage is a "wasted" resource in much the same way that unused memory is. In very broad terms, the system tracks caching across the entire system and dynamically increases/decreases how much data is being cached based on storage demand from the "rest" of the system. It's goal is basically to maintain an empty buffer big enough (several GBs) to immediately handle demand spike and then allow the rest of storage to fill with "useful" cache data.

With all that background, let me move to here:

I am working on a daemon which collects information about disk space usage on macOS.

Why? What are you actually trying to understand from this information? The problem here is that a drive can be physically "full" under completely different circumstances:

1. The user has filled the device with "real" data and the volume really is "full".

2. The user has used some fraction of the device and the system has then filled the rest with "stuff" that it will happily delete as soon as that storage is needed.

Critically, those two cases (or the range in between) all look identical at the filesystem level ("the drive is full"). Differentiating between these cases is why the APIs in "Checking Volume Storage Capacity" exist.

My application must provide disk usage by APFS containers.

Are there any recommended way to get space usage by particular APFS volume?

Are there any recommended way to get free space on particular APFS container?

Are there any recommended way to enumerate APFS containers and volumes?

We don't have public API for any of this, however, the "diskutil" command line tool is intentionally architected to be usable as "semi-API". Note, for example, that most of it's command take a "-plist" argument, making the output easy to programmatically process. While there isn't any formal commitment to this, the engineering team understand the role it serve and it's argument and output format (particularly it's plist output) have been quite stable. Like any command line tool usage, you should be careful when processing it's output and fail gracefully if/when something changes, however, this is a tool that we expect apps be rely on.

I am using Disk Arbitration to get APFS info. However, I get restricted info about space usage because I get get disk usage for mounted volumes only. Are there any public API (daemon-safe) which allows to easily get disk space usage on macOS?

Whether or not the tool is daemon-safe doesn't actually matter* to your daemon, but I expect diskutil to work fine in a daemon context as long as you avoid commands which trigger UI. I can't easily provide a list of all those cases, but most of those cases should be clear from the man page and none of the informational commands should be an issue.

*A tool/command may not work in the daemon context but because it's running in a separate process, any issues it creates shouldn't effect your daemon.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you for such a detailed answer!
Is there any sense in creating an enhancement request for a daemon-safe API (one that can be safely used within a daemon)?
If I need to retrieve statistics once per second (as required by the customer), calling another process does not seem like the best solution.

Thank you for such a detailed answer! Is there any sense in creating an enhancement request for a daemon-safe API (one that can be safely used within a daemon)?

I think most of them are daemon safe. NSURL is daemon safe (it's part of Foundation, which is documented as daemon safe), as are the BSD APIs below it. Expanding on what I said here:

...I expect diskutil to work fine in a daemon context as long as you avoid commands which trigger UI.

This is mostly an issue of looking at the man page and avoiding "obvious" edge cases. For example, diskutil commands to unlock volumes can present UI and/or access the keychain, which won't work in daemon. So "don't do that" and it will work fine. I've been somewhat vague because diskutil is a big command with lots of edge cases and you should test all the details for yourself, however, in practice I don't expect you have any problems.

Shifting to the enhancement request question itself...

Is there any sense in creating an enhancement request for a daemon-safe API (one that can be safely used within a daemon)?

While I'm always in favor of having bugs filed, I don't think there is a huge benefit here. Practically speaking, the reason we've ended up with this "command line tool" approach instead of a true API is:

• The command line tool is necessary no matter what (so we were going to build it anyway).

• Any API we built would either require a complex authorization system or would require exactly the same security configuration as the command line approach.

• Frankly, it's very hard to build elegant API for this sort of thing, particularly over an extended period of time and as the system evolves. There are lots of different commands, most of which can accept very different argument sets with different meanings, all of which tend to make a clean API hard to create.

We certainly could create API for this, but the issues above are why we haven't.

If I need to retrieve statistics once per second (as required by the customer),

So, one thing I need to understand here is the classic question "what are you trying to do". The reason diskutil becomes a factor here is specifically because you're trying to operate at the container level, not the volume, and I'm not sure that will be as useful as you might think. What are you actually trying to learn/understand from this data?

I haven't looked all that closely at EXACTLY how the container data correlates with volume level data (particularly across API layers), but I expect you'll find significant oddities in the data the closer/more frequently you look at the data. The container's role is to distribute storage across it's underlying volumes and I don't think either layer is trying to be perfectly efficient*, which means there's likely to be difference between what the volume layer APIs return and what diskutil returns. Those differences aren't actually meaningful, however, they mean that trying to pin down EXACTLY what these numbers "are" at any given instant isn't all that useful.

*In other words, the container and volumes aren't specifically trying to ensure that EVERY single block on the device is immediately used to store data. They're optimizing around other considerations like overall performance and correctness.

In any case, I'm not sure what information you're trying to learn from the container itself, especially at that kind of frequency.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you for answer!

what are you trying to do?

The customer wants real-time monitoring of SSD/HDD space usage alongside the partition scheme.

The customer wants real-time monitoring of SSD/HDD space usage alongside the partition scheme.

Again, I ask "why". More specifically:

• On most file systems, watching storage from 1s to the next is of limited use, as storage usage generally doesn't meaningfully shift that quickly.

• A CoW (copy on write) file system makes this dynamic even worse, as you're looking at the instantaneous value of an oscillating* value.

*This happens because basically "all" operations temporarily increase storage usage, even operations like file deletion.

• Because this value isn't actually all that useful, APFS doesn't "actively" track it (at least not as a single value in can directly return), so calculating the value is relatively "expensive". I don't think the cost enough that doing it every second will be directly measurable, but that doesn't mean it's worth doing.

Returning to this point:

The customer wants

Presumably, the customer has an underlying purpose for this number beyond having a number they can stare at*. Understanding that underlying goal is what's important here, as that's how you your product can provide real value.

*For example, if their goal is ACTUALLY to have a pretty number they can stare that, then you'd be better off polling every 5 or so seconds and then generating a series of random values that animate between the previous and new values at a much higher frequency (probably 2-4x/sec). They'll be a 5s lag with the "current truth" but it will basically be just as useful and it will look much cooler.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you a lot for the help!

I agree, looks like, there is no any sense to request storage usage so frequently if we are talking about collecting statistic only.