That is why I really dislike the attempts to apply some explanation of religion label non-religious folks. Atheist/Agnostic IMO are just basically slander words.
What's interesting to me is that the basic morals found in religion can also apply to something akin to an "intelligent selfishness", as in, I don't want to be raped, so I won't rape, or I don't want shit stolen from me, so I won't steal.