I don't have any experience in this area, but what I can suggest is to think of it this way: what if F was YOUR child? What would you feel comfortable with? What would you be ok with him being "exposed" to? Would it bother you if his dad's girlfriend stayed over when he visited?, etc. Granted, that is hard to imagine, especially if you don't have kids or if yours aren't that age, but it can give you some perspective.
Just from the 2 things you said, it sounds like you think/feel you should stay away(ie not stay over) & give the boys their time.
I don't know if that helped or not, but food for thought maybe?