What is most intimidating part of the sexual experience for men?
Performance. It's always performance. Even for the most macho and overbearing guy on the planet. I'll make an exception, though: those who do not think about the woman they are with and to whom it never happened. That mainly includes young men, and idiots. The caveat, though, is that if it happened to them once, however young or brainless, it's over, they'll think about it every time.
You might ask, how does it happen to those uncaring brutes? Well, it takes a woman they really care about or find intimidating in any way (very intelligent, beautiful, confident...), or too many drinks to have an erection, but not enough to forget about it.
One thing you ladies should remember, is that just the fear of impotence, brings on impotence. It's a vicious circle.
Generally speaking, you ladies worry about your appearance, we worry about our performance.