@cwebber It's very interesting, and I appreciate you taking the time to write down your thoughts. You touched on many caveats, and I share all the concerns you mentioned. But one question I have that I wish we'd spend more time discussing is why do we want to create intelligent (presumably sentient) agents instead of focusing on creating a workshop filled with reliable, non-sentient tools?
The earth abounds in natural intelligences, and humanity still struggles to extend rights, compassion, and empathy to its own kind let alone the others we share this planet with. But given we are surrounded by natural intelligences, what are the motivations for creating an "artificial" one? Are these motivations healthy and ethical? Should we be doing it at all?
Of course, you're not responsible for answering these questions. But when I ponder these questions, the answers I come up with are not good.