I don't belong to any denomination but get sick of people asking what Church I go to.

If you truly belong to God, then all you have to say is that you belong to Jesus Christ's Church called the Body of Christ. You can remind them of the true meaning of Church and perhaps let them know where you assemble (fellowship) in houses, in the hall down the road, where ever the Spirit leads...

Sometimes when people ask me what church I belong to in order to judge me based on my answer, I sometimes reply with the following question. "Do you follow Paul, Apollos or Cephas"? Of course if the person is well versed in the scriptures, they will know that I am referring to 1 Corinthians 1:10-13 which is all about divisions in the church. If they have no idea what I am talking about, then I can describe to them what this verse means, which gives me an opportunity shed some light.
10 I appeal to you, brothers, in the name of our Lord Jesus Christ, that all of you agree with one another so that there may be no divisions among you and that you may be perfectly united in mind and thought.
11 My brothers, some from Chloe's household have informed me that there are quarrels among you.
12 What I mean is this: One of you says, "I follow Paul"; another, "I follow Apollos"; another, "I follow Cephas"; still another, "I follow Christ."
13 Is Christ divided? Was Paul crucified for you? Were you baptized into the name of Paul?

To some, this may seem like a cheeky or harsh answer, but I want you to consider that we should all be followers of Christ and do all in his name. We should never be divided over denomination, creed or church leader. Also consider the following reasons why we shouldn't judge a person based on the denomination they belong to, (or lack of membership to a denomination).

In conclusion, we who are believers must make every effort to keep the unity with one another, because this is the will of Christ. In fact this is what he prayed to his Father, just before he was arrested and hung on the cross. So it is not only his will, but one of the most important principles that he wants us to live by because our unity will result in the world believing that Jesus came from God.

Also, if we are true followers of Christ, then we should be seekers of truth, because Jesus is the Truth. Therefore we should not actively participate in the Apostasy (the great falling away from the truth). Rather we should speak the truth always and not perpetuate the ideas of man nor the doctrine of demons.

Now if you meet someone who claims to be a believer, then treat him as such. In time you will know him/her by their fruit. If it turns out that the fruit is bad, then you have a chance to convert this person to the way of righteousness. If the person doesn't repent, then depart from this person and tell it to the church.