Max tokens for Foundation Models

Do we know what a safe max token limit is? After some iterating, I have come to believe 4096 might be the limit on device.

Could you help me out by answering any of these questions:

Is 4096 the correct limit? Do all devices have the same limit? Will the limit change over time or by device?

The errors I get when going over the limit do not seem to say, hey you are over, so it's just by trial and error that I figure these issues out.

Thanks for the fun new toys.

Regards, Rob

Hi @Robby,

You are correct that currently the token limit for Foundation Models framework is around 4,000. There is no guarantee that this will stay the same forever or across devices, however, so we encourage developers to write their code in a way that is ready to handle the context window limit when it arises.

As mentioned in this session, your app can catch the exceededContextWindowSize error and handle accordingly. One suggestion for this is to summarize a session's transcript thus far, and create a new session with the condensed transcript, but the exact implementation will depend on your use-case.

Best,

-J

Max tokens for Foundation Models
 
 
Q