Author Topic: FIRECalc/cFIRESim: confidence in longer timelines  (Read 2495 times)

Bird In Hand

  • Pencil Stache
  • ****
  • Posts: 842
FIRECalc/cFIRESim: confidence in longer timelines
« on: February 23, 2020, 10:17:26 AM »
FIRECalc/cFIRESim/etc. have a limited set of data to operate on.  Since the data starts in 1871, there have been 118 30-year periods, 108 40-year periods, 98 50-year periods, and so on.  As the period increases, the confidence that a future period will resemble a past period (or some central tendency of all same-length past periods) probably decreases.

What's the best way to account for this using tools like FIRECalc?  If you're interested in a 50 year timeline, is it better to:

1) run the analysis for all 98 of the 50-year periods since 1871?
2) run the analysis for all 108 40-year periods with the condition that there must be a certain amount remaining in the portfolio for the remaining 10 years?
3) run the analysis for all 118 30-year periods with the condition that there must be a certain amount remaining in the portfolio for the remaining 20 years?
4) something else

Metalcat

  • Senior Mustachian
  • ********
  • Posts: 17235
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #1 on: February 23, 2020, 10:26:06 AM »
Would it really have any predictive power?

nereo

  • Senior Mustachian
  • ********
  • Posts: 17472
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #2 on: February 23, 2020, 10:28:49 AM »
I find it far more informative to look at where each of the simulations end up after 30 year runs.  cFireSim allows you to download the actual simulation runs and further analyze the results.

In brief, I look at how many simulations resulted in a net portfolio increase, and how many were some fraction of the original retirement amount.  If most of the simulations showed a net increase I know those would likely be fine for a few more decades.  Then I look at the reduced-porfolio simulations and look at how depleted they were (how far below the starting value) and where the problems began (it's overwhelmingly within the first 6 years, aka SORR).

Others will use Monte Carlo simulations to boot-strap 10 or 20 year periods together, but that comes with some extreme and frankly improbably scenarios (e.g. two great depressions running back-to-back, or an economic expansion that runs 15+ years).  So if you use this method it's best to ditch the statistical outliers on both tails of the distribution, and focus only those within 1.5x of the inner quartile (25% thru 75%).

In the end though it's important to understand that predicting your life 50 years into the future is an exercise in futulity.  Both your life and your community around you is unlikely to resemble your current situation - and its maddeningly hard to predict what the future will look like beyond 20-30 years from now. 

Bird In Hand

  • Pencil Stache
  • ****
  • Posts: 842
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #3 on: February 23, 2020, 02:40:57 PM »
Would it really have any predictive power?

Are you asking whether FIRECalc/etc. have predictive power, or whether, in my opinion, one of the options (1-3) have relatively more predictive power than the others?

I think most of us ascribe to these tools various levels of predictive power, commensurate with our belief and/or hope that the future market behavior will resemble past market behavior.  However, as the timeline gets larger, I start to wonder whether the lack of historical data (smaller sample size) lessens the predictive value, statistically speaking.  And if so, what to do about it?

Bird In Hand

  • Pencil Stache
  • ****
  • Posts: 842
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #4 on: February 23, 2020, 02:56:52 PM »
I find it far more informative to look at where each of the simulations end up after 30 year runs.  cFireSim allows you to download the actual simulation runs and further analyze the results.

In brief, I look at how many simulations resulted in a net portfolio increase, and how many were some fraction of the original retirement amount.  If most of the simulations showed a net increase I know those would likely be fine for a few more decades.  Then I look at the reduced-porfolio simulations and look at how depleted they were (how far below the starting value) and where the problems began (it's overwhelmingly within the first 6 years, aka SORR).

That's similar to how I've been thinking about it.  I run 30 year simulations and look at the ending balances.  If most the ending balances are higher than some threshold (chosen to put a floor on my remaining portfolio between 30 years from now and end-of-life), I'm happy.  Basically if I have $1 million of today's spending power left in my 70's, I feel comfortable enough believing that I'll never run out of money.

Quote
In the end though it's important to understand that predicting your life 50 years into the future is an exercise in futulity.  Both your life and your community around you is unlikely to resemble your current situation - and its maddeningly hard to predict what the future will look like beyond 20-30 years from now.

Of course.  And yet most of us do want some yardstick to measure our FI/RE-ability.  Whether it's 3-5% SWR, or FIRECalc/cFIRESim, or something else.

Retire-Canada

  • Walrus Stache
  • *******
  • Posts: 8659
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #5 on: February 23, 2020, 03:15:49 PM »
I don't bother running periods longer than 40 years on cFIREsim - which takes me to age 90. I can get to 100% success over that period not counting any variation in spending or home equity. If I need to take corrective action due to sequence of returns risk in the first 10 years of FIRE I will. Beyond that I don't see the point in worrying about it now.

nereo

  • Senior Mustachian
  • ********
  • Posts: 17472
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #6 on: February 23, 2020, 03:32:01 PM »
I find it far more informative to look at where each of the simulations end up after 30 year runs.  cFireSim allows you to download the actual simulation runs and further analyze the results.

In brief, I look at how many simulations resulted in a net portfolio increase, and how many were some fraction of the original retirement amount.  If most of the simulations showed a net increase I know those would likely be fine for a few more decades.  Then I look at the reduced-porfolio simulations and look at how depleted they were (how far below the starting value) and where the problems began (it's overwhelmingly within the first 6 years, aka SORR).

That's similar to how I've been thinking about it.  I run 30 year simulations and look at the ending balances.  If most the ending balances are higher than some threshold (chosen to put a floor on my remaining portfolio between 30 years from now and end-of-life), I'm happy.  Basically if I have $1 million of today's spending power left in my 70's, I feel comfortable enough believing that I'll never run out of money.

Quote
In the end though it's important to understand that predicting your life 50 years into the future is an exercise in futulity.  Both your life and your community around you is unlikely to resemble your current situation - and its maddeningly hard to predict what the future will look like beyond 20-30 years from now.

Of course.  And yet most of us do want some yardstick to measure our FI/RE-ability.  Whether it's 3-5% SWR, or FIRECalc/cFIRESim, or something else.
Not really what I meant.
IMO, you can have a decent financial plan for 20-30 years. Beyond that, you are unlikely to predict - or be able to prevent - the thingd which are most likely to kill a portfolio. So execute a plan that’s likely to leave you in good standing in 20-30 years. In the meantime, stay vigilant and adaptable. As Malkynn put it, “your can’t bogelgesd your way to complete financial independence”

Bird In Hand

  • Pencil Stache
  • ****
  • Posts: 842
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #7 on: February 23, 2020, 03:41:04 PM »
Not really what I meant.
IMO, you can have a decent financial plan for 20-30 years. Beyond that, you are unlikely to predict - or be able to prevent - the thingd which are most likely to kill a portfolio. So execute a plan that’s likely to leave you in good standing in 20-30 years. In the meantime, stay vigilant and adaptable.

I dunno, I think we're talking about +/- the same thing.  A "decent financial plan for 20-30 years" is a lot like my approach: "see how my portfolio is likely to look after 30 years.  If I project that there will be a good chunk of $$ leftover, it seems like a decent financial plan".  Of course SORR can make things interesting -- but as you say, that's something you can course-correct for early on.

nereo

  • Senior Mustachian
  • ********
  • Posts: 17472
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #8 on: February 23, 2020, 03:58:00 PM »
Possibly?
In your OP you mentioned how running cFireSIm and FIRECalc for more than 30 year periods resulted in dramatically fewer periods to draw inference from. The reason that I don’t run these simulations beyond 30 years isn’t because of the reduced sample size, but because WR simply aren’t very informative more than a few decades out.

Bird In Hand

  • Pencil Stache
  • ****
  • Posts: 842
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #9 on: February 23, 2020, 04:28:28 PM »
Possibly?
In your OP you mentioned how running cFireSIm and FIRECalc for more than 30 year periods resulted in dramatically fewer periods to draw inference from. The reason that I don’t run these simulations beyond 30 years isn’t because of the reduced sample size, but because WR simply aren’t very informative more than a few decades out.

Your definition of 'dramatically' must be quite different from mine!

It sounds like we are both not using > 30 year periods for different reasons, but we're still doing basically the same thing with the tools.

Hopefully it won't require several more volleys of replies to confirm that we're not actually disagreeing.    :D

nereo

  • Senior Mustachian
  • ********
  • Posts: 17472
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #10 on: February 23, 2020, 05:13:33 PM »
Possibly?
In your OP you mentioned how running cFireSIm and FIRECalc for more than 30 year periods resulted in dramatically fewer periods to draw inference from. The reason that I don’t run these simulations beyond 30 years isn’t because of the reduced sample size, but because WR simply aren’t very informative more than a few decades out.

Your definition of 'dramatically' must be quite different from mine!

It sounds like we are both not using > 30 year periods for different reasons, but we're still doing basically the same thing with the tools.

Hopefully it won't require several more volleys of replies to confirm that we're not actually disagreeing.    :D

I tend to overexplain things in threads.  And in real life.  Habit of mine. 
I’m not arguing with you - just clarifying what I meant and why I do things the way I do. 
You are right that while our philosophies might differ, the underlying approach is similar and will likely end us both in a similar place.

That better? :-)

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #11 on: February 23, 2020, 05:17:20 PM »
Very few people will blindly spend exactly the same 4% for 30 years, let alone 40 or 50 years.

Simulation models get less accurate the more fuzziness there is about the reliability of their assumptions. There's enough fuzziness in the normal 30-year simulation; anyone that thinks they can remotely predict their spending 50 years out is... uh, crazy.


Metalcat

  • Senior Mustachian
  • ********
  • Posts: 17235
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #12 on: February 24, 2020, 04:19:09 AM »
Would it really have any predictive power?

Are you asking whether FIRECalc/etc. have predictive power, or whether, in my opinion, one of the options (1-3) have relatively more predictive power than the others?

I think most of us ascribe to these tools various levels of predictive power, commensurate with our belief and/or hope that the future market behavior will resemble past market behavior.  However, as the timeline gets larger, I start to wonder whether the lack of historical data (smaller sample size) lessens the predictive value, statistically speaking.  And if so, what to do about it?

FIRECalc perform only very rough calculations based on very rough assumptions and a future spending pattern that doesn't represent anyone's actual reality.

So no, I don't see it as particularly predictive. I see it as a reasonable enough simulation with which to guide financial decisions in the present, as it's a decent enough tool for comparing the relative impact of different approaches.

I think getting that granular with FIRECalc, etc, is where it completely loses it's utility.

Bird In Hand

  • Pencil Stache
  • ****
  • Posts: 842
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #13 on: February 24, 2020, 06:10:17 AM »
I’m not arguing with you - just clarifying what I meant and why I do things the way I do. 
You are right that while our philosophies might differ, the underlying approach is similar and will likely end us both in a similar place.

That better? :-)

Yeah, I guess I'll take that.  :D

FIRE 20/20

  • Pencil Stache
  • ****
  • Posts: 758
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #14 on: February 24, 2020, 06:54:00 AM »
These tools are excellent examples of the difference between precision and accuracy.  They are extremely precise - they give an exact number like 93% success, but as precise as they are they cannot be assumed to be accurate.  They can't be expected to accurately represent future returns and interest rates because we have such small sample sizes and the future may not match the past.  They also are unlikely to be accurate because real people won't behave anywhere close to how they assume we will behave. 
I've been a math and physics tutor at multiple times throughout my life, and my career was in building computer simulations.  These are very difficult concepts to grasp for people who aren't exposed to them every day.

With that said, as long as you understand their limitations, they're incredibly useful.  Using them to get a feeling for the set of possible futures and relative if not absolute likelihoods is great.  I think that playing around with each option the OP presents and that people have discussed is a good idea.  But after that it's time to put them down and see what the real world presents us with.

Metalcat

  • Senior Mustachian
  • ********
  • Posts: 17235
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #15 on: February 24, 2020, 07:21:22 AM »
Very few people will blindly spend exactly the same 4% for 30 years, let alone 40 or 50 years.

Simulation models get less accurate the more fuzziness there is about the reliability of their assumptions. There's enough fuzziness in the normal 30-year simulation; anyone that thinks they can remotely predict their spending 50 years out is... uh, crazy.

Not "very few", absolutely no one will spend exactly X%+projected inflation year over year. No one has ever spent so precisely, it doesn't make any sense, and yet it's the cornerstone assumption of the calculation.

Life will simply never cost the same amount (+projected inflation) every single year. It's an insane assumption in terms of precision prediction of the future, but it's a great assumption for looking at broad strokes impact of decision making in the present.

Plus the "projected spend" is a very rough estimate, often including significant fat and padding.

Fat= optional spending line items that could be easily trimmed, such as travel, restaurants, subscriptions, memberships, collectibles, luxury services, etc.
Padding= an added margin to projected spending just to be safe (rounding up error), which effectively lowers WR without calculating it as a lower WR, which totally throws off the entire output of the calculation.

The calculation is based off of completely unrealistic assumptions and entirely made up inputs. Person A could input unrealistically low spending numbers and get 100% output, while person B could have a ton of rounding up error in their inputs, half of their budget allocated to fat, and get an 85% output from the calculator.

100% success is a comforting illusion, it's not a prediction of safety.

nereo

  • Senior Mustachian
  • ********
  • Posts: 17472
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #16 on: February 24, 2020, 09:08:05 AM »
These tools are excellent examples of the difference between precision and accuracy.  They are extremely precise - they give an exact number like 93% success, but as precise as they are they cannot be assumed to be accurate.
Not exactly:
Accuracy is how close an estimate is to the true value.  Accuracy does NOT take into account the amount of spread (or deviation) there is in your estimates.

Precision is how tightly clustered (or confident) your estimate is.  Precision does NOT take into account how close you are to the 'real' number.

You can be accurate but not terribly precise (cFIRESim outputs are very accurate - they just have an enormous spread)
You can be precise but not at all accurate
You can be both precise and accurate
You can be neither precise nor accurate


To put it graphically



Terminology matters if we're going to understand each other.  Assuming the worst and best periods in the past are roughly no worse nor better than in the future, cFIRESim is very a very accurate predictor - it's just not very precise
signed:  Someone who teaches statistics

FIRE 20/20

  • Pencil Stache
  • ****
  • Posts: 758
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #17 on: February 24, 2020, 12:56:21 PM »
Thanks for the corrections, @nereo .  I've found that my industry used terms in unusual ways before, and this may be another case of that.  I'm happy to be shown to be wrong, because that's the only way to learn.  In some cases in my industry we used a more proper word than in common usage (decimate as originally intended to mean to reduce by one tenth) and in other cases the word was used horribly incorrectly (Use Case was horribly misused throughout my career).   When we used a simulation that always gave the same answer for a set of inputs or one that had an unreasonably small variance but gave an incorrect answer as one that was precise but inaccurate.  It was like the first image above "low accuracy / high precision".  That's how I see these tools.  If you feed them the same inputs, they will return the same answer over and over again, but I don't think there's enough tuning data to believe that the result reflects what will happen in the future.  I don't believe there is a large enough sample to believe that when CFireSim gives you 95% that your future chances of success are actually 95%.  However, if you run it repeatedly you'll get that exact same answer.   

So what I meant, not using the terms "accurate" or "precise" is that CFireSim and other similar tools will always give the same answer when you feed it the same inputs, but I don't think that answer is actually correct.  If I have a 3.5% withdrawal rate, those tools might say - every time - that I have a 100% chance of success.  I don't believe that is the correct answer.  The answers I get are so tightly clustered that they split each dart, but the darts are probably off the board somewhere. 


ixtap

  • Magnum Stache
  • ******
  • Posts: 4553
  • Age: 51
  • Location: SoCal
    • Our Sea Story
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #18 on: February 24, 2020, 01:00:08 PM »
I always thought decimate should be to reduce TO 10%, rather than by 10%. It is just so much more dramatic my way.

Retire-Canada

  • Walrus Stache
  • *******
  • Posts: 8659
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #19 on: February 24, 2020, 01:02:32 PM »
I always thought decimate should be to reduce TO 10%, rather than by 10%. It is just so much more dramatic my way.

Definitely more dramatic, but not so awesome if you killed 9 out of 10 of your soldiers to make a point and then had to fight a battle soon after. ;-)

Retire-Canada

  • Walrus Stache
  • *******
  • Posts: 8659
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #20 on: February 24, 2020, 01:06:07 PM »
Talk of FIREcalc and cFIREsim as "predictive" makes me cringe. They are backwards looking programs and don't say anything about the future. We make the logical leap from 90% success vs. historical data being a good indication that we'll be successful in the future, but those programs don't say anything like that.

Bird In Hand

  • Pencil Stache
  • ****
  • Posts: 842
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #21 on: February 24, 2020, 01:21:47 PM »
Talk of FIREcalc and cFIREsim as "predictive" makes me cringe. They are backwards looking programs and don't say anything about the future. We make the logical leap from 90% success vs. historical data being a good indication that we'll be successful in the future, but those programs don't say anything like that.

I think we all understand this.  Again, we choose to ascribe predictive value to the extent that we believe the future will resemble the past.  Nobody (that I'm aware of) believes these tools actually predict a specific future.

nereo

  • Senior Mustachian
  • ********
  • Posts: 17472
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #22 on: February 24, 2020, 01:21:59 PM »

So what I meant, not using the terms "accurate" or "precise" is that CFireSim and other similar tools will always give the same answer when you feed it the same inputs, but I don't think that answer is actually correct.  If I have a 3.5% withdrawal rate, those tools might say - every time - that I have a 100% chance of success.  I don't believe that is the correct answer.  The answers I get are so tightly clustered that they split each dart, but the darts are probably off the board somewhere.

Here I think we run into the problem of people looking only at the "headline number" that these simulators spit out and assuming that's the data point.  In fact what these programs will say is something like this:  119 cycles, minimum ending balance = $87,183; maximum balance $6,132,262; median balance =  $2,275,955.  Then it goes on to list all 119 cycles of the model.

This is what I mean about it likely being accurate, but not precise.  Unless future conditions are either significantly worse or significantly better than the worst or best 30 year investment period over the last 120+ years, an account with $1MM today and a blind 3.5% WR will have somewhere between $87k and 6.13MM in 30 years.  The 'true value' will fall somewhere in that range.  The fact that there is such a large range means the model output is not terribly precise.

As for whether these are predictive - as Retire-Canada indicates, you have to make the assumption that the historical inputs will represent future conditions.  As I indicate, if you focus on the tails, rather than the median values, the assumptions you are really making is that the future will be no worse than the worst period, and no better than the best period.  To me, that seems a fair place to start.  But I won't place my faith and future financial security blindly into these calculators. Layers of safety, perpetual vigilance, adaptability and all that...

Definitely more dramatic, but not so awesome if you killed 9 out of 10 of your soldiers to make a point and then had to fight a battle soon after. ;-)

Ok, another lexiconic tangent - until rather recently 'awesome' meant "evoking fear and awe".  I suppose if I were a foot soldier I was think my commanders were awesome to shoot 9 of my platoon-mates just before a battle.... but not what most of us mean by that word today.

Retire-Canada

  • Walrus Stache
  • *******
  • Posts: 8659
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #23 on: February 24, 2020, 01:24:45 PM »
Nobody (that I'm aware of) believes these tools actually predict a specific future.

They don't predict any future.

Retire-Canada

  • Walrus Stache
  • *******
  • Posts: 8659
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #24 on: February 24, 2020, 01:27:52 PM »
Ok, another lexiconic tangent - until rather recently 'awesome' meant "evoking fear and awe".  I suppose if I were a foot soldier I was think my commanders were awesome to shoot 9 of my platoon-mates just before a battle.... but not what most of us mean by that word today.

Your army would not be awesome [to the enemy] if you killed 9 out of 10 soldiers to make disciplinary point. It would be pretty awesome from the point of view of the soldiers left.

Bird In Hand

  • Pencil Stache
  • ****
  • Posts: 842
Re: FIRECalc/cFIRESim: confidence in longer timelines
« Reply #25 on: February 24, 2020, 01:28:35 PM »
Nobody (that I'm aware of) believes these tools actually predict a specific future.

They don't predict any future.

Ok, since it's super fun and productive to argue semantics online, I-- wait...nevermind.  I'm out.  :D