Id think it would be fine, but just to make things less of a hassle, I wouldnt fly with it. When I fly, I dress up somewhat (slacks, shirt, and blazer), I have found that looking respectable tends to get you a little less of a hassle. Not too much, but enough. Besides, I like dressing up
Security is enough of a hassle as it is, anything that could make it moreso I would go without.