You CAN keep up. I think YOU need to redefine what you think a man should be. If you really believed that you don't need bulky muscles, etc then just do your thing and stop worrying about what other people think,
I have almost no muscle and am constantly mistaken for a girl. Yesterday, I ran errands all day and almost everyone called me a girl. I don't work out, play sports, or anything you're describing. The problem is not other men, it's your insecurity that makes you cave to peer pressure.
If you change gender, you will feel like you're in a nightmare living in the wrong body. You don't even have the slightest idea what transition will do to you. And talk about peer pressure? Women throw it on each other in much heavier doses than men!! And they can be downright cruel about it. You will never make it as a woman looking like those guys you want to look like. THEY'RE NOT WOMEN! They don't even remotely look like women. Yeah some people say they do, but those are just people who associate makeup and long hair with women.
And a therapist will never agree to hormone therapy since you're not a woman. "Living a feminine lifestyle and being thin" is NOT what makes a woman! I have that kind of lifestyle. My hair is really long, and I'm growing to my waist. I groom way more than most men do. My clothes are androgynous. I have a soft voice. Sometimes I wear minimal makeup. And so on.
See this guy here? This is my style.
It is NOT your gender that's the issue here.
You need to experiment with how you look. How do you want your hair to look like? If you want it long, grow it out. If you want to shave your body hair, do it! Take some of BowynAerrow's advice about your diet. See a nutritionist. You'll look awkward in the beginning, kind of like a pre teen girl learning how to do her hair and put on makeup, but that's how you find what's comfortable for you.
I went through the same phase. Granted, I'm a little blessed genetically in that I'm intersex, but I've seen plenty of "normal" guys pull of the andro look.