I have experienced this problem when the disk on which the backup is being written to is compressed.

I don't see why you can't install in either order 2005 then 2008 or 2008 then 2005. I've only ever added a higher version instance to a server with...