You may want to get yourself a plastic divider and isolate the swordfish for a week or so to see if more fish disappear. If you can't do this, I would still bet on the sharks being the culprits. I'm not sure this species in particular, but sharks do love their fish meat. Getting bigger fish may just be the equivalent of buying them a bigger lunch.
Meaner fish - now that's the ticket.