There is a back story and context to what has been going on that we haven't been seeing. I am basing this on peoples reactions, comments and emotions to him as well as what they have said.
From the beginning of last nights episode.
Aisha made comments early on also about how he puts himself in awkward situations with the group.
Mike Goldman's voice over said he's been annoying Sandra and the house mates with his inappropriate comments.
Sandra said in the diary room she is getting really annoyed with david. The previous day a lot of people told her their opinion of him, that he doesn't gel or fit in. That he takes a joke too far. She was saying she knows it and being a pair she is responsible for his actions and it makes her resent him.
She was saying she's in a bad position because her friends don't like him and it affects her. She's in a hard place because she also finds him annoying. She also said David see's everyone else as been in the wrong.
David even in the diary room said he annoys and bugs people and doesn't care that other people think of him.
So the annoyed at David is the context and when you are annoyed at someone, it's easier for them to set you off and get more annoyed.