It can hurt greatly, but it is simply a fact of trading online.
The best thing you can do is make sure you track your packages, and if someone scams you file the necessary stuff here. You can also pursue someone for mail fraud, but you will need to be able to prove you mailed things to them & that they were delivered (tracking).
For that reason alone, I always track stuff over 10$.
Even with my feedback, I have people who are cautious about trading. I can completely understand their viewpoint, because I was in the same shoes. The best thing you can do is trust the system, be prudent, and be vigilant.
I wouldn't fret yet though. Its still within the reasonable frame of delay.
It wouldn't hurt to send him a message (I'm sure you have already, so disregard if so) asking what's going on.