His quip about Estelle waiting for someone to transfer in from another school so she could have a friend was both an accurate observation of what was happening and said in front of just about the whole household. Yet he gets accused of "bitching behind everyone's backs".
Some of the HM's said to Ava, we will leave you alone for 5 mins, so you can unpack etc & Ava thanked them.
When they went to leave, Estelle stayed. (that's why Ben said that comment)