It's not so much ownership as an
implied covenant of good faith in marriage.
If custom dictates that marriage usually entails an implied term of fidelity, then, unless expressly disclaimed, couples are entitled to expect fidelity from their spouse.
You can go fuck 30 porn stars and be part of a Swingers group but that's not the norm. If works for you, good on you. But don't force that down another person's throat.
I agree about the physical violence part, though. Just pick up a hobby instead.