Yes, just from one guy's perspective, it absolutely is!
I have been out for less than a year but do pass, and despite never having been given trouble, it ALWAYS terrifies me to need a restroom. I will avoid drinking when I need to just to minimize the chances I will need a bathroom in public.
Most of this I hope is in my head, but I guess it speaks the perceived intimacy of a truly private, gendered space that absolutely intensifies impostor syndrome and the feeling of 'invading something sacred' that we perceive is not ours by birth.