Is it $500 for the whole family? So $500 for 4 kids plus two adults? And how long is the swimming season? Assuming it's 3 months, and you go about 2x/week, it's about $20 per visit for the whole family, or about $3.50 per person. That doesn't seem all that costly, and maybe you will go more often than 2x per week. I'd hope that this isn't a financial strain.
I would probably say to go for it--swimming is really good for kids, keeps them active, tires them out so they eat/sleep well, gives them something to do and less "I'm bored" whining. At least, that's how my parents justified putting a pool in our backyard when I was a kid. :)