Adding memory at any time is safe..removing it will definitely kick things out of the cache, of course, so I wouldn't necessarily do it during peak hours (though I have before with no adverse effects).
I am not a memory guru by any means but about the only way to clear the buffer pool is to reboot, restart the instance, or run dbcc dropcleanbuffers. Changing the memory setting should be fine.