How to get people to stop judging me just because of where I live?

I'm sorry this is really long only read where you want to its just to explain if you want the details

I lived in my new house for a year now and it's much bigger than most people's houses but because of this people immediately judge me and think I'm rich.before I lived in a really tiny house my room could only just fit a bunk bed in which I used to share with my sister. I lived in my old house for 13 years and had never moved before then but I'm not complaining I loved my old house it was cosy and has many memories. My parents always wanted a bigger house and now we live here and yea it's pretty big but it just annoys me how people assume that my whole family's rich the money wasn't passed down to us or anything my dad had to work for it and what annoys me more is tht they think we're rich when there the ones with like iPhone 5's and really expensive laptops or like they've had an extension or redecorated there room a billion times or have really expensive clothes cars swimming pools caravans branded foods or seem to go abroad every school break we get. I'm sorry if I'm coming across as rude but I just think especially when it's your friends who are judging you when all you've done is been nice to them it's just wrong.yes I do live in a big house but just because my parents worked for it and saved up made sacrifices, instead of going to Disney every summer like some people seem to do, it does not automatically mean that suddenly I have loads of cash.plus I find it rude talking about other people's money as its personal.

how can I get people to stop thinking I'm stinking rich when really we put all our money into our house and we don't have money to throw everywhere
also without being rude as some of these people are my friends

Just agree with them and they will stop commenting. A workmate said to me in front of colleagues, 'xxxxx lives in a mansion, don't you'? I replied, 'Yes I do'. They shut up after that.

Stop worrying about what other people think.

Kids did the same thing to my best friend, she lives in a HUGE house on a giant property but her family worked for years to get the money for it and she doesn't have expensive clothes or ever go on vacation but people tell her she's rich, the best way to deal with it is to ignore them or like the other person said is agree. She eventually started agreeing with them to shut them up and they stopped. I understand though, I live in a tiny apartment but have nice things so people assume I'm rich when really it's usually things I got as presents from my other family or stuff I saved up for. Just remember, it doesn't matter what they think!