The assumption here is that he doesn't bath. This may not be the case. It may actually be his pheromones or natural scent.
Humans have worked diligently to stomp out the natural odor and replace it with things like Old Spice, Channel, and now Axe. It is a form of lying which no one seems to understand as such. however humans do rely on the sense of smell when picking their partner. It is possible he smells 'stale' to you and only you because you two are not biologically (scent wise) compatible.
This incompatibility on the biological level shouldn't be to big of a problem, as long as you two don't procreate, your offspring will be horrific. :tongue:
The other issue here is age has its own scent:
http://healthland.time.com/2012/05/31/ol...tists-say/
http://web-japan.org/trends00/honbun/tj990921.html
The last link is more important because you described this odor as 'stale' which is typical of the 'old person smell'. Furthermore you are uncertain if its in his clothes, which suggests to me his whole body 'out gasses' this chemistry.
"
A researcher at Shiseido Laboratories has traced the problem to a fatty acid known as palmitoleic acid. He has also learned that the body of a person up to about the age of 30 does not secrete a noticeable amount of this substance, but that once a person--whether male or female--hits 40, the volume rises sharply. The volume of palmitoleic acid released by the human body is 10 times as great among people in their seventies as in their forties."
http://synbio4all.org/Old_people_odor talks about the technomancy science part of chemistry.
Mind this has nothing to do with bathing and showering, it is acid being secreted out of the body, like water vapor constantly escapes the skin, or like body oil which no amount of bathing completely gets rid of.
Chances are high that the older you get the more musty/stale you will smell - you know like old people.
Since you are 57 I assume the guy you are dating is around that age.
Mind I think diet also plays a role, but I'm uncertain as to the links there. Vegans and Vegetarians appear to have this old person smell at earlier ages, so something about the lack of fatty acids in the diet may be linked to this.