Having CLR enabled and loaded consumes around 100MB of VAS.
Another point to bear in mind is that CLR can be loaded even when disabled - some of the internal MS procedures use it and will force it to be loaded if called...sp_browsereplcmds if one IIRC.
32-bit and SQL Server is not a happy combination, even without hosting the CLR.