libncftp v. macOS Native curl with Secure Transport APIs and Session Reuse

I am working on adding RFC4217 Secure FTP with TLS by extending Mike Gleason's classic libncftp client library. I refactored the code to include an FTP channel abstraction with FTP channel abstraction types for TCP, TLS, and TCP with Opportunistic TLS types. The first implementation of those included BSD sockets that libncftp has always supported with the clear TCP channel type.

I first embarked on extending the sockets implementation by adding TCP, TLS, and TCP with Opportunistic TLS channel abstraction types against the new, modern Network.framework C-based APIs, including using the “tricky” framer technique to employ a TCP with Opportunistic TLS FTP channel abstraction type to support explicit FTPS as specified by RFC4217 where you have to connect first in the clear with TCP, request AUTH TLS, and then start TLS after receiving positive confirmation. That all worked great.

Unfortunately, at the end of that effort, I discovered that many modern FTPS server implementations (vsftpd, pure-ftpd, proftpd) mandate TLS session reuse / resumption across the control and data channels, specifying the identical session ID and cipher suites across the control and data channels. Since Network.framework lacked a necessary and equivalent to the Secure Transport SSLSetPeerID, I retrenched and rewrote the necessary TLS and TCP with Opportunistic TLS FTP channel abstraction types using the now-deprecated Secure Transport APIs atop the Network.framework-based TCP clear FTP channel type abstraction I had just written.

Using the canonical test server I had been using throughout development, test.rebex.net, this Secure Transport solution seemed to work perfectly, working in clear, secure-control-only, and secure-control+data explicit FTPS operation.

I then proceeded to expand testing to include a broad set of Microsoft FTP Service, pure-ftpd, vsftpd, proftpd, and other FTP servers identified on the Internet (a subset from this list: https://gist.github.com/mnjstwins/85ac8348d6faeb32b25908d447943300).

In doing that testing, beyond test.rebex.net, I was unable to identify a single (among hundreds), that successfully work with secure-control+data explicit FTPS operation even though nearly all of them work with secure-control-only explicit FTPS operation.

So, I started regressing my libncftp + Network.framework + Secure Transport implementation against curl 8.7.1 on macOS 14.7.2 “Sonoma":

% which curl; `which curl` --version
/usr/bin/curl
curl 8.7.1 (x86_64-apple-darwin23.0) libcurl/8.7.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.61.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe UnixSockets

I find that curl (also apparently written against Secure Transport) works in almost all of the cases my libncftp does not. This is a representative example:

% ./samples/misc/ncftpgetbytes -d stderr --secure --explicit --secure-both ftps://ftp.sjtu.edu.cn:21/pub/README.NetInstall

which fails in the secure-control+data case with errSSLClosedAbort on the data channel TLS handshake, just after ClientHello, attempts whereas:

% curl -4 --verbose --ftp-pasv --ftp-ssl-reqd ftp://ftp.sjtu.edu.cn:21/pub/README.NetInstall

succeeds.

I took an in-depth look at the implementation of github.com/apple-oss-distributions/curl/ and git/github.com/apple-oss-distributions/Security/ to identify areas where my implementation was, perhaps, deficient relative to curl and its curl/lib/vtls/sectransp.c Secure Transport implementation. As far as I can tell, I am doing everything consistently with what the Apple OSS implementation of curl is doing. The analysis included:

• SSLSetALPNProtocols
  • Not applicable for FTP; only used for HTTP/2 and HTTP/3.
• SSLSetCertificate
  • Should only be relevant when a custom, non-Keychain-based certificate is used.
• SSLSetEnabledCiphers
  • This could be an issue; however, the cipher suite used for the data channel should be the same as that used for the control channel. curl talks about disabling "weak" cipher suites that are known-insecure even though the default suites macOS enables are unlikely to enable them.
• SSLSetProtocolVersionEnabled
  • We do not appear to be getting a protocol version negotiation error, so this seems unlikely, but possible.
• SSLSetProtocolVersionMax
  • We do not appear to be getting a protocol version negotiation error, so this seems unlikely, but possible.
• SSLSetProtocolVersionMin
  • We do not appear to be getting a protocol version negotiation error, so this seems unlikely, but possible.
• SSLSetSessionOption( , kSSLSessionOptionFalseStart)
  • curl does seem to enable this for certain versions of macOS and disables it for others. Possible.
  • Running curl with the --false-start option does not seem to make a difference.
• SSLSetSessionOption( , kSSLSessionOptionSendOneByteRecord)
  • Corresponds to "*****" which seems defaulted and is related to an SSL security flaw when using CBC-based block encryption ciphers, which is not applicable here.

Based on that, further experiments I attempted included:

• Disable use of kSSLSessionOptionBreakOnServerAuth: No impact
• Assert use of kSSLSessionOptionFalseStart: No impact
• Assert use of kSSLSessionOptionSendOneByteRecord: No impact
• Use SSLSetProtocolVersionMin and SSLSetProtocolVersionMax in various combinations: No impact
• Use SSLSetProtocolVersionEnabled in various combinations: No impact
• Forcibly set a single cipher suite (TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, known to work with a given server): No impact
• Employ a SetDefaultCipherSuites function similar to what curl does (filtering out “weak” cipher suites): No impact
  • Notably, I can never coax a similar set of cipher suites that macOS curl does with that technique. In fact, it publishes ciphers that aren’t even in <Security/CipherSuite.h> nor referenced by github.com/apple-oss-distributions/curl/curl/lib/vtls/sectransp.c.
• Assert use of kSSLSessionOptionAllowRenegotiation: No impact
• Assert use of kSSLSessionOptionEnableSessionTickets: No impact

Looking at Wireshark, my ClientHello includes status_request, signed_certificate_timestamp, and extended_master_secret extensions whereas macOS curl's never do--same Secure Transport APIs. None of the above API experiments seem to influence the inclusion / exclusion of those three ClientHello additions.

Any suggestions are welcomed that might shine a light on what native curl has access to that allows it to work with ST for these FTP secure-control+data use cases.