In addition to what everyone else said, I’ll tell you why I don’t think that what we have currently can become some runaway superhuman intelligence:

All “AI” people are talking about are basically “just” LLM wrappers: tools built to run a LLM in a loop while giving it access to the ability to write and run code. This gets pretty powerful, since you can do a lot programmatically (including monetary transactions), but has a fundamental limit:

LLMs are trained ahead of time. And that’s a very cost intensive, long process. Beyond that, the LLM has only a context window of finite length. So there are no long-term planning capabilities. While running after being trained, the LLM can only condense knowledge into a summary and then re-read it, a lossy process that might introduce mistakes in “understanding” and will make any prolonged task degrade over time.

And even if they figure out how to make the things actually self-modify and learn things, you can’t give every session that ability, since that’ll vastly increase the storage and performance requirements. Do sessions with that ability will be closely monitored and limited.

So I don’t see any space where an evil AI could emerge, even when factoring in an essential possible future improvement to the technology.

You gotta admit that’s a huge market.