Important item in Keychain seems to have disappeared (after years)

Question

Created Mar ’25

Replies 8

Boosts 0

Participants 2

I had the following code in a program that I used to encrypt some important files. I haven't run it in a few years. It used to work, and now it seems the password is mysteriously gone from my Keychain! The return value is now errSecItemNotFound.

I'm upset with myself for not backing up the key/password somewhere else. Is there anywhere this could be hiding? Did Apple move it somewhere? I know they created this "Passwords" app in recent years, but I don't see anything in there with the "account" string I used. I run the app from Xcode, so maybe it is in the "container" data somewhere? I do see keychain files under ~/Library.

Maybe there is a way to look through old Time Machine backups. Ug. So stressful.

Just looking for pointers on where the data might be, and why it might have disappeared. Unfortunately it was not a "guessable" password, it was a generated 256 bit key, base64 encoded. Perhaps I could crack that with brute force if I'm determined enough...


public static func queryGenericPasswordAsString(account: String) throws -> String {
    let query: [String: Any] = [kSecClass as String: kSecClassGenericPassword,
                                kSecMatchLimit as String: kSecMatchLimitOne,
                                kSecAttrAccount as String: account,
                                kSecReturnAttributes as String: true,
                                kSecReturnData as String: true]
    var item: CFTypeRef?
    let status = SecItemCopyMatching(query as CFDictionary, &item)
    guard status != errSecItemNotFound else { throw KeychainError.noPassword }
    ...    
}

Answered by DTS Engineer in 829202022

I’m presuming this is on the Mac. That’s important, because the keychain is much more complex on the Mac than it is on iOS. See TN3137 On Mac keychain APIs and implementations.

I have some generic advice about how to use the keychain effectively:

Looking at your code, I don’t see anything fundamentally wrong with it.

The code doesn’t opt in to the data protection keychain, so it’s gonna be using the file-based keychain. Moreover, it’s likely that you created the item in the default keychain, which is usually the login keychain. Given that, you should be able to look for the keychain item using Keychain Access. That’ll tell you whether it’s there or not.

If it’s not, you can’t fix that with code. You’ll have to find a keychain with the item in it. I have hints and tips on how to do that, for very different reasons, in The Care and Feeding of Developer ID.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Boost

Answer 1

DTS Engineer OP

Apple

Mar ’25

Recommended

I’m presuming this is on the Mac. That’s important, because the keychain is much more complex on the Mac than it is on iOS. See TN3137 On Mac keychain APIs and implementations.

I have some generic advice about how to use the keychain effectively:

Looking at your code, I don’t see anything fundamentally wrong with it.

The code doesn’t opt in to the data protection keychain, so it’s gonna be using the file-based keychain. Moreover, it’s likely that you created the item in the default keychain, which is usually the login keychain. Given that, you should be able to look for the keychain item using Keychain Access. That’ll tell you whether it’s there or not.

If it’s not, you can’t fix that with code. You’ll have to find a keychain with the item in it. I have hints and tips on how to do that, for very different reasons, in The Care and Feeding of Developer ID.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

1

Answer 2

rnikander OP

Mar ’25

Yes, it's on macOS. I'm trying to piece together exactly what code I was running at the time, a few years ago, but I think the code that created the item may have used the data protection keychain, which you mentioned.

var query: [String: Any] = [kSecClass as String: kSecClassGenericPassword,
                            kSecAttrAccount as String: account,
                            kSecUseDataProtectionKeychain as String: true,
                            kSecValueData as String: password]
if let label = label {
    query[kSecAttrLabel as String] = label
}
let status = SecItemAdd(query as CFDictionary, nil)

A quick test suggests that even if I add it like that, the snippet in my original post, with SecItemCopyMatching, finds the item that was created, even though it doesn't have that kSecUseDataProtectionKeychain key in its query.

A second piece of info: in ~/Library/Keychains, there is a file named login_renamed_1.keychain-db from 2022. I don't know what creates a file like this, but I wonder if it's from a password reset/change that I did on my MacBook. When I try to open it in Keychain Access, it appears in the left nav, but I don't think it's completely open / searchable, because I can't "unlock" it. It's not accepting my current MacBook password. Does that mean it is locked by my previous MacBook password and might contain the missing password item?

If so, it's going to be very difficult for me to remember that password, but maybe not impossible. I almost remember it - it was a certain word or two with a few extra characters thrown in. That might be "crackable", unlike the random 256 bit key I'm trying to find.

0

Answer 3

DTS Engineer OP

Apple

Mar ’25

even though it doesn't have that kSecUseDataProtectionKeychain key in its query.

Right. If you don’t supply a value for the kSecUseDataProtectionKeychain attribute [1], SecItemCopyMatching will query both keychain implementations. And if you request multiple results via kSecMatchLimit, it can return results from different keychain implementations.

Does that mean it is locked by my previous MacBook password and might contain the missing password item?

Possibly. The renamed label means that at some point macOS renamed on old keychain out of the way because it wasn’t usable for some reason. That could be because it didn’t have the unlock password, but it’s also possible there’s some other problem in play (like a file corruption issue).

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] For the older kSecAttrSynchronizable attribute, there is kSecAttrSynchronizableAny.

1

Answer 4

rnikander OP

Mar ’25

I have a few sub-questions that would help me narrow down my search.

1. Can I see every item name in login_renamed_1, without unlocking?

This could help me skip the large effort I'm expending trying to crack my old password. When I open that old login_renamed_1.keychain-db file in Keychain Access, it appears in the left nav under "Custom Keychains" and it is locked. However, I can still click on it and see the names of the items in the main window area to the right. My lost master key item is not there.

Would unlocking it reveal more names, more items? If not, if I can already see the entire content of that keychain, then I don't see a point in continuing my current CPU/GPU-heavy attempts to recover my partially-remembered old laptop password, with the goal of unlocking this keychain.

2. Is the entire data protection keychain in the file keychain-2.db?

One of the links you (Quinn) posted says that the data protection keychain is here, but people say "file-based keychain" to refer to the other (non data protection) keychain. This is confusing me, since this looks to be "file based" as well, if it is contained in the SQLite DB in keychain-2.db file?

3. Could the update to Passwords app have deleted or hidden my SecItem?

Data migrations are dangerous. At some point after I originally called SecAddItem in late 2020, Apple introduced the Passwords app and passwords seemed to move there. Are there any known issues here with SecItems becoming hidden, or lost?

4. Can I query the names of SecItems in keychain-2.db, using sqlite3?

If my item is somehow lost in that database file, I wonder if I might recover it with some manual SQL selects. I can use sqlite3 to view the file contents, but most of the table contents appear... cryptic.

0

Answer 5

DTS Engineer OP

Apple

Mar ’25

I don’t have a lot of expertise in keychain recovery. That’s not really an API issue, which is my focus, but more of a user-level issue, which is something for the folks over on Apple Support Community. However, I can answer some of this:

Can I see every item name in login_renamed_1, without unlocking?

I’d ignore Keychain Access here and instead use the security tool. It’s very focused on file-based keychain.

Is the entire data protection keychain in the file keychain-2.db?

Yes.

Well, the entire data protection keychain for this specific user. Each user has their own data protection keychain.

This is confusing me, since this looks to be "file based" as well

Sure. But all persistent storage on the Mac has to be a file at some point, right?

The point of the file-based keychain is that you can have multiple keychain files. These act like documents, with Keychain Access being the document viewer. You can move files around, put them on an external drive, even move them to another Mac

In contrast, the data protection keychain is a single (per-user) database that you can’t manipulate in any supported way.

Could the update to Passwords app have deleted or hidden my SecItem?

Well, anything is possible, but if you’re asking whether that’s likely then my answer is “No.” The Passwords app is just a new view on to the data protection keychain. The underlying keychain APIs and implementations didn’t change [1].

Can I query the names of SecItems in keychain-2.db, using sqlite3?

Probably not.

While the format of this file is not considered API, it’s easy enough to work out that it’s SQLite and SQLite databases are self describing. Feel free to rummage around using your favourite SQLite tool.

However, you’ll find that many columns in the database are encrypted — that’s kinda the point of the keychain after all — and that’ll limit your ability to find items of interest.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Well, they change all the time, but there was no relevant change that was specifically tied to the introduction of the Passwords app.

1

Answer 6

rnikander OP

Mar ’25

Yeah, this issue is kind of in the gray area between developer-land and user-land. I hope that doesn't lead to me never getting answer about what happened. I will circle back around with normal tech support again today or tomorrow.

Let me try to focus this question on the developer API in the Security framework, and put aside "keychain recovery" for a moment. Because I understand that recovery is a topic that is moving away from developer tech support and into general tech support.

I called the function SecItemAdd. I want to know the exact details of that function's contract. I need to know that, to use the function in the best way possible. I thought I was adding to a database, and that the data would be accessible forever via calls to SecItemCopyMatching, unless I called SecItemDelete on the item, or manually deleted it using an app like Keychain Access (which would itself be call SecItemDelete or equivalent). In other words, I imagined SecItemAdd was like SQL insert, and that as long as I didn't do a delete, I'd have my item. Other linked documents suggest this is close, at least, to the reality.

But apparently my assumption was wrong. There must be some condition under which a "general password" data item disappears or becomes inaccessible. That condition is part of the contract of these functions, and it's something developers need to know in order to use these functions in a highest quality way.

If I can ever understand this, then I could think about possible recovery, if it's even possible, but I'm not even at that stage yet.

0

Answer 7

rnikander OP

Mar ’25

I followed up with Apple Support, and unfortunately it sounds like there is nothing I can do here. Well, short of breaking the AES-GCM encryption, to recover my files without the key that I had in keychain. Haha. That's basically hopeless. Maybe I can recover my photos 30 years from know, before I die, using a quantum computer. :)

Lesson learned. Master keys need to be backed up in multiple places, or constructed from memorable pass phrases. 🤦‍♂️

0

Answer 8

DTS Engineer OP

Apple

Mar ’25

I imagined SecItemAdd was like SQL insert

It’s reasonable to think of things that way. Indeed, that’s how I explain this in SecItem: Fundamentals. If you haven’t read that yet, and its companion, SecItem: Pitfalls and Best Practices, I recommend that you do so.

Indeed, with the data protection keychain that’s how things are actually implemented, with SQLite under the covers. The file-based keychain uses a different database format, but the logic is similar.

as long as I didn't do a delete, I'd have my item.

Sure. As long as your data centre doesn’t burn down…

… and your DBMS has no bugs…

… and your sysops make no mistakes…

… and so on.

I’m not sure what happened in this case but, yeah, if you use strong encryption to protect critical data then you need to plan for disasters like this.

Maybe I can recover my photos

)-:

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0