I'm not sure what data type you are using for you boolean fields, but you could setup something like this:
create table test (b1 bit, b2 bit, b3 bit, b4 bit)
go
create trigger test_trigger on test
for insert, update
as
if update(b1) or update(b2) or update(b3) or update(b4)
begin
if exists (select * from inserted where convert(int,b1) + convert(int,b2) + convert(int,b3) + convert(int,b4) > 1)
rollback
end