I also did a profiler, and by filtering out large amounts of connection and other data, I found out there is one person is running ssis package on that machine manually at the time when we reproduce the delay, he has been running this for a while daily.
This may explain why the same process sometimes is fine, sometimes is slow.