With the help of trawl though the C: drive, I managed to find it. The details are stored in a (not especially readable) XML file "C:\Documents and Settings\MyUserName\Application Data\Microsoft\VisualStudio\9.0\ServerExplorerDefaultView.SEView" (for VS2008).
After a bit of hacking with XMLNotepad, I managed to remove the unwanted references.