I have several friends with homes like this. Some inherited, some bought along with other members of family and some just bought themselves. I get common comments from all of them:
1: They all say they spend lots of their free time driving to the rental house to fix things, clean things or do other chores they'd rather not be doing between rental weeks.
2: Their successes have made the home a rental and not a vacation home. Why? Because the weeks that are during good weather, high rental season are rented. To leave a week open is too expensive. In essence, those who used to go to the area to rent are priced out of their own house.
3: People are hard on rentals and things are broken that should normally not be broken. Do people really hang on and swing on refrigerator doors enough to bend the hinges? Uh, yah.
One coworker goes to his house in the off season every weekend. His son is a young adult and the coworker goes there to hike and mountain bike with his son. His son, having a family of his own usually has to cancel coming. So my co-worker tends to come back on Monday with stories of great recreational things. Oh wait.....is putting up sheet rock the relaxing recreation you'd dream of doing in 100 degree heat? Sounds fun to me to just watch on TV from my living room.
Have you done the math? Sure, rentals are expensive, but if you want to save money one year, you can simply not go if you're renting. If you have a mortgage, you have to pay every month, even if a big rent cancels and you're left holding their $200 deposit for a $4000 rental.
Obviously, I don't own a rental (or a time share). When I go on vacation, I rent and I get to do it anywhere in the world I want to.