I don't see this as ridiculous or silly. My advice would be some form of counselling. From what you have described, it sounds like a mild form of PTSD and although you are aware of it, it could get worse if left unexplored.
Another thought would be if something happens that may be slightly disturbing in your life, it will compound the problem, for instance say if you come across a loud argument in the street. Seems innocent enough however given your aroused state it will add to the stressors already heightend in your physical system.
Is this why you are leaving your current job? just curious
