Idk what to do and I'm desperate for advice..

[verysimple]

So I have been seeing this ITALIAN guy for almost 3 weeks now. I only see him on weekends and we spend the whole weekend together. Anyway, first weekend he witnessed me exchanging my phone number with a guy working at a clothing shop and after we walked out of the store he turned into this whole other person. He was super quiet and cold. I kept asking him what's wrong, only he wouldn't say and he'd tell me nothing so I waited for him to tell me at his own pace. When he did he asked why I exchanged my phone number with that guy, I told him it was because I wanted him to send me some of the remixes I heard at the store and I really liked so I have him my phone number to send them through an app using my number. And then I told him that it was nothing and then out of no where he tells me that he wants to go home and that he isn't feeling it (note that we were heading to a festival together and he was gonna bail on me) so I sat him down somewhere and discussed it. So it turned out he had trust issues and he didn't know me so well. So we resolved it and moved on. The weekend after which was last weekend. We spent it all together and it went amazing. We went to a concert, a sound clash concert. Had breakfast together, went sightseeing in the city together, and did many other fun stuff. Then at the end of the last day which was Saturday, we were going to hang someplace and then when we got there he just said "I wanna go home" which is was the weirdest thing I hear. I asked him why, and he said that he doesn't know me so well and he spends too much time with me (although he didn't seem annoyed or troubled with me. He was having fun) I was really fucking hurt and then I just let him go home and so did I.. The next morning he texts me this long message apologizing and doesn't wanna be rude but he wants some personal space during those weekends which is very logical and I was understanding and I told him I wanna continue dating him and see what happens. Now we are planning to go on a long trip to a sea and spend a couple of days there then I will host him at my home and take him places in the town where I live. I'm scared though.. I keep having this nagging thing in my head that if I had him here he'd say the excuse that he is spending too much time and needs to be away or that he needs to go home when I'll be putting in a lot of my time, effort and money and I'm afraid that It'd all go for nothing. I'm just afraid he'll do what he did the past couple weekends..

I'm so confused and I have been stressing about this lately. I keep holding back and not feel completely comfortable talking to him in the fear that I'll be hurt again He is also starting to notice that I'm not okay through texting me and It's easy to show when I'm really not ok..