My first time swinging was with my current bf 4 yrs ago. It was actually after a good friend's wedding (I was the maid of honor). My bf and I started flirting with another couple, and things progressed from there. We eventually found ourselves at the bride & groom's house. She got involved for a lil while, but her husband asked her to come to bed. My bf and I and the other couple had an awesome time.
Of course, he & I discussed everything afterwards. There was some jealousy on my part, but we worked through it and discovered that this was something we wanted to continue doing. We're now a host couple for a swing club in Western NY. (If anyone has any questions, PM me. I'd be happy to help in any way I can.)
And yes, a majority of swingers (at least that I've seen) are in their mid 40's-early 50's. Many say that they're just getting to a point in their relationship that they are comfortable enough to even bring up the subject of swinging, much less try it. Another reason is that many are parents and their kids are out of the house or at least old enough to look after themselves for a night.