Agree. And imagine if a public interaction gave away something to one or more HM(s) like a twist, a secret mission, voting or eviction details like whether it is going to be a double or not, or worse, which one or more of the HMs are hated by the public and why.
For example, who's not to say that somebody won't yell out "Nobody likes you on the outside, Holly!". Even if such a statement is true, or not, it throws a cat amongst the pidgeons in the house as that individual (in this case, Holly) may not be aware of that and will be affected in some way. They may start behaving differently, get emotional, or complain and whine about it. And if it causes a change to their interactions in the house, then it affects their chances ... and indirectly, other HMs too ... of winning, or not. Who knows. No, it is not a good idea to allow public interaction with the house, in any form.