I mean if your OS was “smart” as not to send IO to devices that indicate critical failure (e.g. by marking them read-only in the array?), and then thinks all devices have failed critically, wouldn’t this happen in that kind of system as well…

The problem isn’t the tool, it’s the user: they don’t know if they’re getting good code or not, therefore they cannot make the prompt to improve it.

In my view the problems occur when using AI to do something you don’t already know how to do.