If you fire up profiler you will see that the query is passed through to the other server to execute a rowset object is created on the server with the data. You will still have network issues as the data is passed back to the calling machine and other task performed by the calling server. So, both take a hit but the one with the data takes the biggest hit.
Wes