I started saving for retirement after my divorce at 36. I had no debt, but only a few thousand in my IRA. I made some costly financial mistakes along the way, but now at age 52, I am about 10 months away from retiring debt-free, with no mortgage. I got real serious and ramped up the savings about 5 years ago and got more serious about reducing my spending (which was never really that high) 2 years ago and I'm still tweaking it even now.
The man I was dating at 36 is now my husband. He was very bad with money back then (credit card balances, student loans, upside down on car loans, ate out all the time, drove gas guzzlers, bought books instead of using the library, etc, etc), but I've worked on him bit by bit, and today he's pretty damn awesome financially. He is 8 years younger than I, but he will retire with me at the end of this year at age 45. He actually has a bit bigger nest egg than I do, thanks to a couple of small inheritances and some company stock grants he received.
It really doesn't matter when you start. No matter what, it will bring your retirement date closer than it would have been otherwise. And while you're getting there, you will enjoy much more financial security than most people do.