Failing to run SystemLanguageModel inference with custom adapter

Hi,

I have trained a basic adapter using the adapter training toolkit. I am trying a very basic example of loading it and running inference with it, but am getting the following error:

Passing along InferenceError::inferenceFailed::loadFailed::Error Domain=com.apple.TokenGenerationInference.E5Runner Code=0 "Failed to load model: ANE adapted model load failure: createProgramInstanceWithWeights:modelToken:qos:baseModelIdentifier:owningPid:numWeightFiles:error:: Program load new instance failure (0x170006)." UserInfo={NSLocalizedDescription=Failed to load model: ANE adapted model load failure: createProgramInstanceWithWeights:modelToken:qos:baseModelIdentifier:owningPid:numWeightFiles:error:: Program load new instance failure (0x170006).} in response to ExecuteRequest

Any ideas / direction?

For testing I am including the .fmadapter file inside the app bundle. This is where I load it:

@State private var session: LanguageModelSession? // = LanguageModelSession()
    
    
        
    func loadAdapter() async throws {
        if let assetURL = Bundle.main.url(forResource: "qasc---afm---4-epochs-adapter", withExtension: "fmadapter") {
            print("Asset URL: \(assetURL)")
            let adapter = try SystemLanguageModel.Adapter(fileURL: assetURL)
            let adaptedModel = SystemLanguageModel(adapter: adapter)
            session = LanguageModelSession(model: adaptedModel)
            print("Loaded adapter and updated session")
        } else {
            print("Asset not found in the main bundle.")
        }
    }

This seems to work fine as I get to the log Loaded adapter and updated session. However when the below inference code runs I get the aforementioned error:

func sendMessage(_ msg: String) {
        self.loading = true
        if let session = session {
            Task {
                do {
                    let modelResponse = try await session.respond(to: msg)
                    DispatchQueue.main.async {
                        self.response = modelResponse.content
                        self.loading = false
                    }
                } catch {
                    print("Error: \(error)")
                    DispatchQueue.main.async {
                        self.loading = false
                    }
                }
            }
        }
    }

Hi, your code looks correct!

But there’s a known bug that .fmadapters fail to load on iOS. For now, you can test on macOS.

Sorry for the inconvenience. 😢

Ah, thanks! I was trying to avoid upgrading my main computer to the MacOS beta until it stabilizes but I guess it needs to be done 🙃

Failing to run SystemLanguageModel inference with custom adapter
 
 
Q