Can someone track your address if you give them your cell phone number?

I gave a girl my cell phone number and I am bugging out that it might not be a girl and some freaky dude will show up at my house. HELP!!!