Idk what to do and I'm desperate for advice..

[Camfer]

UPDATE:
So I just talked to him about it over the phone and we figured it out. I was holding back and I wasn't comfortable because I was afraid to be clingy to him. So he told me that he is happy with me talking to him whenever. That it was him and dealing with his fear and it has nothing to do with me. He keeps bringing his past fucked up relationships over to us and it keeps ruining and complicating things. If things continue this way, I don't think we could make it. I'll try to work things out and I'll do my past but he's gotta do his part too

From what you have told us, I don't think you figured it out here. Holding back is exactly what you should do when you are all crazy infatuated and having a blast with someone new. You being worried about the upcoming weekend together speaks volumes. Slow down. Spend time with him in smaller doses so you can get to know him.

If he's going to freak out to the point of being unable to voice his concern when you give some guy your digits in plain view of him, even when you ask him what's up, well then I say he's the one who has some major work ahead in learning to manage his fears and emotions, and in learning not to manipulate others through his passive aggressive silences. Sorry that's a lot in one sentence.

You weren't being clingy. Don't be the superhero trying to rescue him when he creates his own drama that has nothing to do with you.

It will all come clear in time, so take it slowly. Be cautious of investing too much of your emotions in this too soon.