off campus fraternity

I am in an off campus fraternity here and I gotta say, I've been looked down upon because of it. I'll talk to girls here and they'll figure out my fraternity isn't on campus and walk away or say "oh nice" and just quit the conversation from there. Why is that? Do they just automatically think I couldn't get a bid from somewhere else? Or do they think I don't have money back home? I'm not sure but someone please explain this. I joined this fraternity for the people I'd be around. They're all awesome guys and it's great.