The thread has little explanation as to what weird thing they’re doing to Codex that is making the default work poorly, and it kind of seems like it’s getting confused about whether it wants to set the caching mode or the breakpoint or both.
In any case, I find the behavior change interesting. It sounds to be like 5.5 and below may have been using a conventional attention scheme where a cached KV sequence can be easily used to restore a prefix of itself, but perhaps 5.6 is using linear attention or LSTM or another recurrent scheme where you cannot rewind the model state by just truncating it.
I have noticed the same thing starting with 5.6 when editing my last prompt inside the vscode codex plugin, I’ve seen the model’s thinking respond to the edit with a remark.
Slightly bummed out about it because in the past you could try different situations during a planning session and it wouldn’t pollute the cache but now it does. I’m not sure if forking the conversation has the same problem.
My most awkward experience was a maintainer commenting on my feature request just to prompt a bot to "explain to issue reporter why this is very hard to implement."
It felt like they were trying to avoid me. They could have simply addressed me and given me the explanation they gave to the bot: it would have been simpler for him and more polite. I did in fact reply without waiting for the bot.
It's as though you're talking to someone and they were said to their 'assistant', "Explain this to this person" and walked away. It doesn't really matter what the explanation is, it's just gross.
> It sounds to be like 5.5 and below may have been using a conventional attention scheme where a cached KV sequence can be easily used to restore a prefix of itself, but perhaps 5.6 is using linear attention or LSTM or another recurrent scheme where you cannot rewind the model state by just truncating it.
I feel like this is the kind of substantial change to your product that you would need to tell your customers about. It would be simply disrespectful to your customers to not disclose this upfront.
The fact that companies with access to SOTA non-public AI keep having these kinds of dumb bugs in something as simple as a chat app helps validate my disbelief of people claiming AI is ready to replace all software development.
Our codex on AWS Bedrock read / write cache ratio was less than 5%. Cache writes are very expensive and they were never being used. This results in codex on Bedrock causing ~10x what it should due to no caching and massive writes.
The workaround in issue resolved for me:
web_search = "disabled"
If you’ve got a workaround, I’d suggest updating the issue description to have it up top there so similarly impacted users can spot it quickly and benefit.
My conspiratorial mind thinks they're doing this deliberately and using the resets to mask things so people can't tell their limits are reduced. The $200 / month plan covers about 2 days of usage for me right now.
Here are the docs:
https://developers.openai.com/api/docs/guides/prompt-caching...
The thread has little explanation as to what weird thing they’re doing to Codex that is making the default work poorly, and it kind of seems like it’s getting confused about whether it wants to set the caching mode or the breakpoint or both.
In any case, I find the behavior change interesting. It sounds to be like 5.5 and below may have been using a conventional attention scheme where a cached KV sequence can be easily used to restore a prefix of itself, but perhaps 5.6 is using linear attention or LSTM or another recurrent scheme where you cannot rewind the model state by just truncating it.
Slightly bummed out about it because in the past you could try different situations during a planning session and it wouldn’t pollute the cache but now it does. I’m not sure if forking the conversation has the same problem.
I mean i use AI too but was taken back when an agent popped up dictating what i should do and so on....felt weird
It felt like they were trying to avoid me. They could have simply addressed me and given me the explanation they gave to the bot: it would have been simpler for him and more polite. I did in fact reply without waiting for the bot.
I feel like this is the kind of substantial change to your product that you would need to tell your customers about. It would be simply disrespectful to your customers to not disclose this upfront.
Users is the product.
The companies you see struggling are ripe for disruption.
The workaround in issue resolved for me: web_search = "disabled"
If enough comments are added to the discussion, it might end up being collapsed.
My conspiratorial mind thinks they're doing this deliberately and using the resets to mask things so people can't tell their limits are reduced. The $200 / month plan covers about 2 days of usage for me right now.
"Random" accidents that always go against you, too biased to be random.
But don't notice that too much, you might start to see patterns here and there that you're not allowed to, might get you banned from places, etc.
I think you are just biased.