When you define the linked server, you can leave the login mapping blank, and select "Be made using the login's current security context"
If other users do not have permissions to access the phsical server (the linked server), then they cannot access data via the linked server too.