Why is using clonefile for a folder strongly discouraged?

Things went long, so you get a Part 1...

Is FileManager using cloning behind the scenes?

Yes. More specifically, most of our copy APIs are actually built on top of copyfile, and copyfile is what actually handles file cloning.

Or more exactly how guaranteed are we it will use it?

It basically uses it "everywhere it can". That includes both the obvious case of copying within a volume but also includes less obvious cases like copying a hierarchy that includes clone between two APFS volumes. Generally speaking, you should get the same results as what you'd get copying using the Finder*.

*The one exception I can think of is copying files within an SMB server which will definitely be slower and which I believe will also bypass cloning. You can read more about that particular issue here.

(I know it does, I tried manually cping the resources and it was thousands of times slower.)

Just to clarify, I assume you meant manually copying the contents (using open/read/write), not using the "cp" command? cp also uses copyfile, so you should be getting the same results as NSFileManager.

The main one I’m interested in: Why is using clonefile for a folder strongly discouraged?

Expanding a bit on what I described here, you can basically think of cloning a directory as doing two things:

1. Pushing the copy operation into kernel, avoiding the syscall overhead of directory iteration, creation, and individual clonefile calls.

2. Preventing all changes to the source hierarchy while the operation is in progress, making the process atomic.

That second point is what makes this potentially dangerous as, in the worst case, you could theoretically panic the kernel by stalling all activity on critical locations long enough that the kernel "gives up" and panics.

So, if you're going to do this, there are basically two things you need to pay attention to:

• How big is the hierarchy?

The individual operations aren't slow, so you only really have issues once the hierarchy becomes "big enough" to matter.

• Where is the hierarchy and "who" has access to it?

The issue here isn't about the operation itself, it's about what that operation could delay or interfere with. As the most extreme case, if you're doing this on a secondary volume that's "set aside" for your apps own use, cloning even ENORMOUS hierarchies is unlikely to have any noticeable impact on the larger system.

Yes, you're blocking access to a large hierarchy for an extended period of time, but that only matter if someone else wants access to the same hierarchy.

As a part of the video editing app I’m working on, I want to efficiently copy a folder of resources on the same (local) filesystem. Because iOS is on APFS, cloning (CoW) is an option.

Even on macOS there are ways to "isolate" the file your working with well enough that I'd consider directory cloning fairly safe. Assuming your working out of your apps data container, that's about as isolated as it's possible to be. There maybe other factor at work that would make this problematic, but on the whole I don't see any real issue.

Having said that, two things I'd think about as you decide what approach to use here:

• (1) Are you actually looking at this in terms of who your app actually works?

For example, it's very likely that your app is going to be iterating and storing the full hierarchy anyway (so it knows what's there and can show it to the user), so that reduces some of the gain. More broadly, if the more "natural" pattern for your app has it manipulating different files on different threads, you may find trying to force "everything" to a single thread (so you can clone the hierarchy) hurt your overall app design more than it helps performance.

• (2) Absolute performance matters more than relative performance. copyItem maybe slower than clonefile, but that difference may not be worth bother with.

That 10x might sound huge but I suspect that this is case of comparing two numbers that are both REALLY small. The "best case" for an individual item implementation like "copyItem" is basically a slowly growing constant syscall cost (the overhead of the bulk directory iteration) + 1 syscall per object. So for 250 files you're comparing the cost of ~251 syscalls vs. 1 syscall. In that context, 10x is actually a pretty good showing for copyItem at, but regardless it shouldn't be THAT slow unless the file count is VERY high.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

That leads me to Part 2...

I did a small sample project which demonstrates that using clonefile on a folder works correctly and is 10× faster than using FileManager’s copyItem method.

As I've learned over and over again, you need to VERY careful and thoughtful when testing anything file system related, as it's very easy to build tests that don't actually show what you think. Case in point, I actually downloaded your sample project and did a bit of testing and here are the initial numbers I got:

doClone      (0.284348 seconds)
doClone      (0.149546 seconds)
doDuplicate  (0.340530 seconds)
doDuplicate  (0.102446 seconds)

My immediate thought was that those numbers look REALLY big and the inconsistence is odd. The whole point of file cloning is to "be fast" and, just as important, all of this is basically manipulating the file system structures in memory so the time required shouldn't vary much.

SO, the first thing I did was modify your project so that it reused the existing source data (if it already existed) instead of creating it again. Note that this is much closer to how a real world app would work. Here are the numbers from that test:

doClone      (0.008590 seconds)
doClone      (0.007045 seconds)
doClone      (0.007122 seconds)
doDuplicate  (0.070820 seconds)
doDuplicate  (0.068781 seconds)
doDuplicate  (0.068148 seconds)

Note that the performance gap itself has actually widened, but we're now comparing "ludicrously fast" to "fast".

After a bit of flailing around, I figured out what was going and then modified your code to separate source creation from clone/duplicate by pulling the source creation code into it's own function and doing this for both test cases:

Task{
	await doSource()
	Task {
		await doClone()
	}
}

As part of that I also logged how long source creation took, and here is what that logs:

 crtSourceF  (25.729200 seconds)
doClone      (0.007985 seconds)
 crtSourceF  (17.960522 seconds)
doClone      (0.008167 seconds)
 crtSourceF  (25.166749 seconds)
doDuplicate  (0.066928 seconds)
 crtSourceF  (23.373173 seconds)
doDuplicate  (0.067559 seconds)

...exactly the same performance numbers as when I wasn't creating the source data at all. What's going on here is that the previous ~20s of work meant the scheduler was treating your thread as a lower priority I/O thread. That's also why the performance gap was narrower in the first test- copyItem is making lots of very short syscalls, which means the scheduler was increasing it's priority over the time it ran.

Finally, on a unrelated note, I'd appreciate you filing a bug and posting the number bach here asking for a file path specific alternative to this syntax:

try sourceURL.path(percentEncoded: false).withCString{ sourcePath in
	try destURL.path(percentEncoded: false).withCString{ destPath in

The way network and file URls were entangled by CFURL/NSURL is a longstanding API headache that we REALLY need to be sorting out, not propagating. A strongly typed language like Swift should be warning/preventing "unreasonable" code at compile time, which means our API shouldn't be asking you whether or not you want to percent encode a file path. In this particular case, I'd probably suggest using "withUnsafeFileSystemRepresentation()" instead, but that just dodges the question of how you get a FilePath/String when that's what you actually need.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware