I do not think that imposes the limits that you’ve suggested. I’m operating on the perhaps unwarranted assumption that the appearance of jealousy, detrimental advice … we have seen from e. g. Bing AI were expressly written in.
That said, I don’t think that the current batch of AI are near being self-aware. But there comes a point when there will be a machine that passes every of the millions of Turing tests it will get. At that point, imo the distinction becomes without a difference.
It all depends on what a mind is and where our thoughts come from.
Most people's model of the brain is as a super computer with a network of neurons solving problems and coming up with answers through some sort of set of algorithms. The problem with that theory is, there is no evidence of a program that is stored in brain cells. Who wrote the algorithm? How did it come to be in the first place? There is no evidence of inheriting knowledge but babies come into the world already able to achieve high function in brain-body actions.
An alternate theory that can help explain these observations casts the brain as a transducer and thoughts come from somewhere other than brain cells.
I posted this earlier. It's not a fully developed and accepted idea but helps explain observations that don't make sense in the "brain is a meat computer" theory.
https://www.discovermagazine.com/mind/your-brain-is-not-a-computer-it-is-a-transducer.
The following was extracted further down in the text after he goes through the bones of the theory. I skipped down and extracted passages from the article in the interest of brevity.
Again, setting the details aside, physicists agree that the three-dimensional space we experience is simply not the whole picture. As theoretical physicist Lee Smolin put it recently, “Space is dead.”
Evidence for Transduction?
Hard evidence that supports a neural transduction theory is lacking at the moment, but we are surrounded by odd phenomena that are at least consistent with such a theory. And, no, I’m not talking about the claims that best-selling authors have made over the decades about proof that telepathy, out-of-body experiences, and communication with the dead are real. No such proof exists, in my view, but other well documented phenomena are difficult to brush aside.
I have sometimes dreamt intricate full-length movies that seemed as good as any Hollywood film. Alas, most of the time, no matter how hard I try, I can’t seem to hold on to even a shred of a dream during the few seconds when I’m staggering from my bed to the bathroom.
Where does all this content come from, and why do we have so little control over it?
A 2020 study summarizing the observations of 124 caregivers of dementia patients, concluded that in "more than 80 percent of these cases, complete remission with return of memory, orientation, and responsive verbal ability was reported by observers of the lucid episode" and that ‘"[the] majority of patients died within hours to days after the episode." The periods of lucidity typically lasted 30 to 60 minutes.
If the brain is a self-contained information processor, how can we explain the sudden return of lucidity when the brain is severely damaged?
I’ve also been intrigued by what appear to be credible reports about visual experiences that some congenitally-blind people have had when they were near death. Experiences of this sort were first summarized in a 1997 paper by Kenneth Ring and Sharon Cooper, later expanded into a book called Mindsight (1999). The paper and book describe the experiences of 14 people who were blind from birth and who had near-death experiences (NDEs), some of which included content that appeared to be visual in nature. Soon after Vicki U. was in a near-fatal car accident at age 22, she remembered "seeing" a male physician and a woman from above in the emergency room, and she "saw" them working on a body. Said Vicki:
I knew it was me.... I was quite tall and thin at that point. And I recognized at first that it was a body, but I didn't even know that it was mine initially. Then I perceived that I was up on the ceiling, and I thought, "Well, that's kind of weird. What am I doing up here?" I thought, "Well, this must be me. Am I dead?"
Vicki had never had a visual experience before her NDE, and, according to the researchers, did not even "understand the nature of light." While near death, she also claimed to have been flooded with information about math and science. Said Vicki:
I all of a sudden understood intuitively almost [all] things about calculus, and about the way planets were made. And I don't know anything about that.... I felt there was nothing I didn't know.
I know this sounds all woo woo magicky but a lot of what the author says rings true in my own life and those around me. I've made decisions that worked out spectacularly well when my prior experience or knowledge was not sufficient to justify my confidence. My mother died due to hemorrhage after giving birth to my oldest brother and was brought back by a medical team. She describes feeling God before she came back and at the time did not want to come back. There is nothing in the "brain is a meat computer" that explains these observations or those of Vicky's or the return of memories and capabilities in Alzheimer's patients with so much damage to their brains they can't maintain their own bodies any more. This is not proof, but it does contradict the idea that we can build a computer equal to humans in thought power that operates in 3D space when our minds may be communicating in 4D or some other way.
