I think much worse than getting turned down yourself.
If you are an early-stage investor, you are going to not end up doing a lot of good ones. You just are.
The tougher ones are the ones you know are great, that you want to do — but you lose the deal after you offer to invest, provide a term sheet, etc.
Those seem to be the ones that stick with most VCs.