What to do if people look at me wrong for not going to church?

Ignore them. As long as you're happy in your beliefs and are not hurting another, it's really none of their business if you go to church or not, is it?