Here is the shocking thing related to context window and AI model billing.

LLMs don't have a memory of previous chats. So say you send a chat to ChatGPT with the message via an API call: "Hi ChatGPT, weather is good today in Brisbane. My name is Tony and how are you doing today?" ChatGPT will respond: "I am good Tony. I wish I am in brisbane right now!"

Then you send another chat message via an API call and in that message you are asking what is your name, testing if ChatGPT ever recalls your name base from your previous chat: "What is my name?" Then you get shocked as ChatGPT responds: "I am sorry, who are you again?" 🤣

The case is different if you are chatting with ChatGPT in via the ChatGPT product in the web because it has access to your whole chat so it can remember you, so it seems 🤣.

The real situation is the API call examples that I mentioned above and the product ChatGPT made a way via the codes so that from users perspective, it looks like ChatGPT remembers your name along the chat.

So in reality, you will have to append your previous chat to your new chat that you want to submit to ChatGPT via an API call. It is the same with other frontier or open source AI models out there.

The implication is that the token growth of input grows proportionally as the chat goes along because under the hood of this AI model chat products, every message you sent, behind the scene it includes the whole chat before your latest one + your latest message and sent to the AI model.

The input tokens grows linearly as the chat moves along, while the processing time grows exponentially.

That means the longer you are talking to an AI model, the input tokens grows linearly although you are unaware of it and it eats your AI bills faster.

We have not talked about the output tokens yet that you are also paying or at least consuming your free tier. The messages that is returned to you during the chat session has an equivalent number of tokens and of course the corresponding price but behind the scenes, the thought process of the AI model before it gives you its response to your previous chat also has tokens.

That means if asked ChatGPT on what is the integral of 1/x and it answers you ln |x| + C, the answer has an equivalent token + the thought process that comes with solving the integral of 1/x is also added to your total output token.

The reason why we see instagram posts, stories, or reels where CEO of some companies are crying now for their AI bill topping billions of dollars to date 🤣

Also the reason why I am looking for ways to find a good or efficient AI model that can run locally so that I don't need to pay, and also the reason why I am inventing a new one which is my baby hobby project 🤣