Quote:very quiet and I just saw from his eyes that he went trough alot.
And there it is.
Folks who have gone through a lot tend to have a different slant on life. We also tend to deny ourselves a good deal more than average people. We do not allow ourselves to do simple things like fall in love, share our lives, share our woes....
Whilst he may love you, he mayn't be able to express it fully and deeply. You will have to reach a little deeper in yourself to find understanding that this is a can't sort of thing, not an unwillingness - an inability.
So he gets annoyed at the hugs... Understand that there was a time when he most likely needed one and no one gave him one - now he is loath to admit he wants one. Immediate reaction is to feign annoyance, when deep inside is gratitude. Its self defense, not true annoyance - but fear that when he seeks the hugs he won't get them (again?).
Without knowing the exact nature of what he went through, no one can predict how he will act/react to given situations.
However. From what you wrote it seems like he does care, he does have feelings. Yet, due to whatever is there that he carries inside of him, he most likely will not respond quite as you may imagine or hope.
Still, he is trying. I suggest less texting and more conversation - face to face where you can catch the fleeting facial expressions and read his body language. This way you can get to know the real person, not the mask that texting can easily become.
Trust me, one can write hella lies and everyone believes, its difficult to lie face to face, when your whole body tells its truths.