Yup, I'd go with libertarian. Generally speaking, my understanding of libertarian is that it's basically "leave me alone and I'll leave you alone" sort of thing, advocating for smaller government, etc.
I myself think I'm a conservatarian. That is, a conservative-libertarian. So, while I don't agree with gay marriage, I don't think it's the government's job to dictate one way or the other. I don't personally use nor see the point of marijuana, but again, I don't mind those who do so long as it's done in a responsible manner. That sort of thing.