I know the general thought is that pot smokers are immature assholes but that's not always the case. I'd try talking to them. Let them know that you don't want your kids exposed to it as well as yourself.

Personally, I don't have a problem with others smoking but having a severe allergy to marijuana (of all things?), I have had to have this conversation on a couple of occasions with good results.