Very true you can not avoid adding users to the public role. What I've done in the past is create a new roll ( something like AllUsers ) and grant that role the base level access that you would have given public. In fact you can generate a permissions script and do a bit of find and replace to make it easy. Then you give public nothing. Once public has nothing there should be no concerns if a user has public access.
Chuck