Author Topic: The forum is back up!  (Read 23932 times)

jinga nation

  • Magnum Stache
  • ******
  • Posts: 2696
  • Age: 247
  • Location: 'Murica's Dong
Re: The forum is back up!
« Reply #50 on: September 27, 2019, 10:34:00 AM »
it's ironic that on Sept 20 Mr MM tweeted kudos to his web dev and hosting folks for 4 years of zero downtime.

i assume that while the site is up, the terrible read/write times may be due to RAID being rebuilt or resync or some underlying storage issue.

if i use a diff browser to log in, i get time outs. i have to use a browser that has cached credentials. so maybe same/diff problem.

ETA around 1245 ET today site is snappy and fast. no issues observed.
« Last Edit: September 27, 2019, 10:52:08 AM by jinga nation »

arebelspy

  • Administrator
  • Senior Mustachian
  • *****
  • Posts: 28444
  • Age: -997
  • Location: Seattle, WA
Re: The forum is back up!
« Reply #51 on: September 27, 2019, 11:18:16 AM »
Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)
I am a former teacher who accumulated a bunch of real estate, retired at 29, spent some time traveling the world full time and am now settled with three kids.
If you want to know more about me, this Business Insider profile tells the story pretty well.
I (rarely) blog at AdventuringAlong.com. Check out the Now page to see what I'm up to currently.

jinga nation

  • Magnum Stache
  • ******
  • Posts: 2696
  • Age: 247
  • Location: 'Murica's Dong
Re: The forum is back up!
« Reply #52 on: September 27, 2019, 11:25:58 AM »
@arebelspy It would be good to hear about the root cause of the issue.
The next time it happens, we can say "Hey <problem> is happening again, go check <component> or <service>".

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #53 on: September 27, 2019, 11:58:08 AM »
Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

Do you want me to add your email to the alerts sent when there's downtime?

They have been very accurate so far to total outages.


dragoncar

  • Walrus Stache
  • *******
  • Posts: 9922
  • Registered member
Re: The forum is back up!
« Reply #54 on: September 28, 2019, 12:31:52 AM »
Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

If I can’t load the forum, how can I send a PM?

Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

Do you want me to add your email to the alerts sent when there's downtime?

They have been very accurate so far to total outages.



This would be sweet and also a bot that tweet at mmm “forum is down” and also update www.isthemrmoneymustacheforumdown.com
« Last Edit: September 28, 2019, 12:33:59 AM by dragoncar »

solon

  • Handlebar Stache
  • *****
  • Posts: 2359
  • Age: 1823
  • Location: OH
Re: The forum is back up!
« Reply #55 on: September 28, 2019, 07:22:38 AM »
Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

If I can’t load the forum, how can I send a PM?

Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

Do you want me to add your email to the alerts sent when there's downtime?

They have been very accurate so far to total outages.



This would be sweet and also a bot that tweet at mmm “forum is down” and also update www.isthemrmoneymustacheforumdown.com

I clicked it. I wish it was a real link.

Dicey

  • Senior Mustachian
  • ********
  • Posts: 22319
  • Age: 66
  • Location: NorCal
Re: The forum is back up!
« Reply #56 on: September 28, 2019, 08:19:29 AM »
Still slow and timing out today.

BigMoneyJim

  • Bristles
  • ***
  • Posts: 368
  • Age: 53
  • Location: Nomadic retiree in the Rockies
    • Jim's Personal Finance Blog
Re: The forum is back up!
« Reply #57 on: September 28, 2019, 08:21:28 AM »
it's ironic that on Sept 20 Mr MM tweeted kudos to his web dev and hosting folks for 4 years of zero downtime.

That was specifically in reference to the MMM blog & site which is separate from the forums and presumably much, much busier and more profitable.

RWD

  • Walrus Stache
  • *******
  • Posts: 6529
  • Location: Arizona
Re: The forum is back up!
« Reply #58 on: September 28, 2019, 09:41:48 AM »
This would be sweet and also a bot that tweet at mmm “forum is down” and also update www.isthemrmoneymustacheforumdown.com

I clicked it. I wish it was a real link.

Real link:
https://downforeveryoneorjustme.com/forum.mrmoneymustache.com

lutorm

  • Pencil Stache
  • ****
  • Posts: 831
  • Location: About the middle of Sweden
Re: The forum is back up!
« Reply #59 on: September 28, 2019, 02:46:14 PM »
Yeah something's definitely not right. Even when the forum is up, I'm frequently experiencing a minute of response time for loading pages. It makes browsing the forum so painful so as to be effectively impossible.

Roadrunner53

  • Magnum Stache
  • ******
  • Posts: 3570
Re: The forum is back up!
« Reply #60 on: September 28, 2019, 04:51:20 PM »
I am in CT and have been having issues too. What is going on?

G-dog

  • Senior Mustachian
  • ********
  • Posts: 19097
Re: The forum is back up!
« Reply #61 on: September 28, 2019, 04:56:42 PM »
@arebelspy and @FrugalToque - how should we notify someone when the forum is down? When it is down, I can’t get to the PM function.

The forum has been flakey since the initial incident earlier.

2Birds1Stone

  • Walrus Stache
  • *******
  • Posts: 7916
  • Age: 1
  • Location: Earth
  • K Thnx Bye
Re: The forum is back up!
« Reply #62 on: September 28, 2019, 06:29:18 PM »
The forum is running like dogshit right now (8:27 PM ET) and was down for the past 30+ minutes.

This will definitely hurt long term traffic, especially regulars.

DadJokes

  • Handlebar Stache
  • *****
  • Posts: 2360
Re: The forum is back up!
« Reply #63 on: September 28, 2019, 06:56:09 PM »
Yeah, I hate to complain about something I access for free, but it's been hot garbage lately. It's probably a good excuse to take a break though. I spend a bit too much time reading through posts here.

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #64 on: September 28, 2019, 06:58:49 PM »
I just realized that I can't actually add a CC email address without paying a bunch.

I'm using SiteUptime to do a super basic check against https://forum.mrmoneymustache.com/index.php (http) -- it should be really easy to setup their basic monitor, if anyone wants to.

InSearchOfFire

  • 5 O'Clock Shadow
  • *
  • Posts: 83
Re: The forum is back up!
« Reply #65 on: September 28, 2019, 09:44:33 PM »
Hi,

@arebelspy, @FrugalToque

Sorry to complain, and I do appreciate all that you all do to keep this place running, but I've been having some really slow page-loading since the initial incident - including trying to load this page tonight and trying to post this...

Also been having some pages time-out since the initial incident.

Thanks!

dragoncar

  • Walrus Stache
  • *******
  • Posts: 9922
  • Registered member
Re: The forum is back up!
« Reply #66 on: September 28, 2019, 10:26:47 PM »
Yeah, I hate to complain about something I access for free

Holup

You guys aren’t paying for Mmm premium elite?

FiveSigmas

  • Pencil Stache
  • ****
  • Posts: 597
Re: The forum is back up!
« Reply #67 on: September 28, 2019, 11:18:25 PM »
This would be sweet and also a bot that tweet at mmm “forum is down” and also update www.isthemrmoneymustacheforumdown.com

I clicked it. I wish it was a real link.

Time for www.istheisthemrmoneymustacheforumdowndown.com ?

P.S. Forum’s back to its pokey self. :-/

jpdx

  • Pencil Stache
  • ****
  • Posts: 760
Re: The forum is back up!
« Reply #68 on: September 29, 2019, 01:35:23 AM »
It's after midnight PST and the site is currently slow as molasses.

Dicey

  • Senior Mustachian
  • ********
  • Posts: 22319
  • Age: 66
  • Location: NorCal
Re: The forum is back up!
« Reply #69 on: September 29, 2019, 05:14:27 AM »
It would be great to hear from the mods about what the problem is. Clearly it's ongoing and crazy frustrating for all involved.

katsiki

  • Handlebar Stache
  • *****
  • Posts: 2015
  • Age: 43
  • Location: La.
Re: The forum is back up!
« Reply #70 on: September 29, 2019, 08:52:52 AM »
DDOS is an interesting theory.  I wonder if one of the mods will chime in at some point once (fully?) resolved.

I noticed that the user counts on the forum's main page seemed up significantly.  I had been away from the forums for a couple of weeks but the totals per day and max total ever stats seemed significantly higher than what I recalled seeing in the past.  I can't be sure, just going from memory.

Does anyone know how guests are counted?  A few minutes ago (9:45 AM CT), the forum home page showed 7,600+ guests online now.  Could these guests be bots or something nefarious?  If not, has the forum just rocketed into a new level of activity for some reason?  Normally, I recall 2000+ users online at a time to be 'normal'.

@arebelspy   FYI if this helps the admin or mods figure this out.

maizefolk

  • Walrus Stache
  • *******
  • Posts: 7400
Re: The forum is back up!
« Reply #71 on: September 29, 2019, 01:43:22 PM »
Katsiki, that is an intriguing observation.

We're at ~7,300 users right now, but the forum set its all time record for connected users just today at 8,820.



That would be consistent with either a DDoS attack or a sudden spike in traffic causing the general forum lag (it's been essentially unusable for me).

Has MMM gotten any recent extremely large press/online coverage?

It is also possible something with the overload of the server is causing it to lose track or double/triple count unregistered users, which would mean the extremely high guest counts would be an effect rather than a cause of the problem.

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #72 on: September 29, 2019, 03:13:15 PM »
An easy fix would be to restrict the forum to logged in users only, too, that would likely cut down on server load depending on what those guests are doing

2Birds1Stone

  • Walrus Stache
  • *******
  • Posts: 7916
  • Age: 1
  • Location: Earth
  • K Thnx Bye
Re: The forum is back up!
« Reply #73 on: September 29, 2019, 06:24:12 PM »
Down yet again for the past 20 minutes =D @MMM  double you tee eff mate?
« Last Edit: September 29, 2019, 06:28:38 PM by 2Birds1Stone »

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #74 on: September 29, 2019, 06:27:32 PM »
My alert is firing on and off all day today...

I can't really do anything about it.

If more $$$ would help avoid this in the future I think many of us would be willing to chip in some.

SwordGuy

  • Walrus Stache
  • *******
  • Posts: 8955
  • Location: Fayetteville, NC
Re: The forum is back up!
« Reply #75 on: September 29, 2019, 08:01:54 PM »
Sometimes I get this message when the forum won't come up, sometimes not:

"Can’t connect securely to this page
This might be because the site uses outdated or unsafe TLS security settings. If this keeps happening, try contacting the website’s owner."

Microsoft Edge and also Firefox.

dragoncar

  • Walrus Stache
  • *******
  • Posts: 9922
  • Registered member
Re: The forum is back up!
« Reply #76 on: September 30, 2019, 01:25:17 AM »
An easy fix would be to restrict the forum to logged in users only, too, that would likely cut down on server load depending on what those guests are doing

Psh.  Just restrict access to walrus stache or above.  I’ll generate enough content for everyone

dragoncar

  • Walrus Stache
  • *******
  • Posts: 9922
  • Registered member
Re: The forum is back up!
« Reply #77 on: September 30, 2019, 01:26:26 AM »
Content not guaranteed

Le Poisson

  • CM*MW 2024 Attendees
  • Senior Mustachian
  • *
  • Posts: 16256
Re: The forum is back up!
« Reply #78 on: September 30, 2019, 04:56:50 AM »
Content not guaranteed

Quality of content also not guaranteed.

I speak often, but not well.

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #79 on: September 30, 2019, 06:45:22 AM »
An easy fix would be to restrict the forum to logged in users only, too, that would likely cut down on server load depending on what those guests are doing

Psh.  Just restrict access to walrus stache or above.  I’ll generate enough content for everyone

I better get posting then! :-)

Dicey

  • Senior Mustachian
  • ********
  • Posts: 22319
  • Age: 66
  • Location: NorCal
Re: The forum is back up!
« Reply #80 on: September 30, 2019, 07:32:34 AM »
It would be great to hear an update from the mods, since the problem has not been resolved.

arebelspy

  • Administrator
  • Senior Mustachian
  • *****
  • Posts: 28444
  • Age: -997
  • Location: Seattle, WA
Re: The forum is back up!
« Reply #81 on: September 30, 2019, 08:36:05 AM »
We've been looking into it; those positing a DDOS attack appear to be on the right track. We're looking into potential cloudflare solutions.
I am a former teacher who accumulated a bunch of real estate, retired at 29, spent some time traveling the world full time and am now settled with three kids.
If you want to know more about me, this Business Insider profile tells the story pretty well.
I (rarely) blog at AdventuringAlong.com. Check out the Now page to see what I'm up to currently.

Dicey

  • Senior Mustachian
  • ********
  • Posts: 22319
  • Age: 66
  • Location: NorCal
Re: The forum is back up!
« Reply #82 on: September 30, 2019, 10:37:29 AM »
We've been looking into it; those positing a DDOS attack appear to be on the right track. We're looking into potential cloudflare solutions.
Thanks for the reply. Pardon my ignorance, but what's the point of a DDOS? Are they looking for ransom money? Trying to drive traffic elsewhere? Or just doing it because they can?

rantk81

  • Pencil Stache
  • ****
  • Posts: 905
  • Age: 42
  • Location: Chicago
Re: The forum is back up!
« Reply #83 on: September 30, 2019, 11:00:40 AM »
That really sucks if this is the deliberate act of someone.


rantk81

  • Pencil Stache
  • ****
  • Posts: 905
  • Age: 42
  • Location: Chicago
Re: The forum is back up!
« Reply #84 on: September 30, 2019, 11:05:45 AM »
We've been looking into it; those positing a DDOS attack appear to be on the right track. We're looking into potential cloudflare solutions.
Thanks for the reply. Pardon my ignorance, but what's the point of a DDOS? Are they looking for ransom money? Trying to drive traffic elsewhere? Or just doing it because they can?

Could be a ransom (We would probably never get confirmation from MMM if that were the case.)  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum.  Could be random collateral damage of a DDOS against another website hosted on the same equipment or same data-center.  Could be someone just screwing around.  We'll probably never know why.  People can be jerks for no reason.

frugalnacho

  • Walrus Stache
  • *******
  • Posts: 5055
  • Age: 41
  • Location: Metro Detroit
Re: The forum is back up!
« Reply #85 on: September 30, 2019, 11:35:26 AM »
I just want to pile on and say the forum has been working like hot garbage for the last 1-2 weeks for me as well.  I tried to make a post/send a PM about it last Friday, but have been unable to load access the site until now.  Took 8 attempts to access this thread and make a reply...hopefully goes through this time. 

I have been frantically hitting the refresh button every few seconds but it doesn't seem to be solving the problem

dragoncar

  • Walrus Stache
  • *******
  • Posts: 9922
  • Registered member
Re: The forum is back up!
« Reply #86 on: September 30, 2019, 12:40:23 PM »
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?

solon

  • Handlebar Stache
  • *****
  • Posts: 2359
  • Age: 1823
  • Location: OH
Re: The forum is back up!
« Reply #87 on: September 30, 2019, 03:14:18 PM »
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?

Maybe sol and boarder42 met up IRL and hatched this diabolical scheme?

2Birds1Stone

  • Walrus Stache
  • *******
  • Posts: 7916
  • Age: 1
  • Location: Earth
  • K Thnx Bye
Re: The forum is back up!
« Reply #88 on: September 30, 2019, 05:35:43 PM »
The forum top is in.

norajean

  • Pencil Stache
  • ****
  • Posts: 602
Re: The forum is back up!
« Reply #89 on: September 30, 2019, 06:46:19 PM »
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?

Someone said the Sol person wasn't banned, just got angry and left. Something about his wife forcing him back to work.

Dicey

  • Senior Mustachian
  • ********
  • Posts: 22319
  • Age: 66
  • Location: NorCal
Re: The forum is back up!
« Reply #90 on: September 30, 2019, 11:05:41 PM »
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?

Someone said the Sol person wasn't banned, just got angry and left. Something about his wife forcing him back to work.
I miss him. Paging @ Sol. Your absence is felt, dude!

golfreak12

  • Bristles
  • ***
  • Posts: 365
Re: The forum is back up!
« Reply #91 on: October 01, 2019, 10:23:16 AM »
This forum has been working like garbage for me too the for the last 2-3 weeks. Couldn't access it last night.

Lulee

  • Bristles
  • ***
  • Posts: 252
  • Location: NH
  • "We'll jump off that bridge when we come to it."
Re: The forum is back up!
« Reply #92 on: October 01, 2019, 10:28:13 AM »
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?

Someone said the Sol person wasn't banned, just got angry and left. Something about his wife forcing him back to work.
I miss him. Paging @ Sol. Your absence is felt, dude!

He is missed even if he is a bit of an acquired taste.  Whatever his faults (and yes, I found him frequently grating as he didn't always seem to grasp the emotional content/impact of his words) I wouldn't believe for a second that he or any other long time board member would actually be party to a DDOS attack making the forum unusable to the rest of us.  I suspect someone got a bug up their backside reading about the FIRE movement and is hassling us.  Just wish there was a way to help the  moderators in fixing this.

Jon_Snow

  • Magnum Stache
  • ******
  • Posts: 4458
  • Location: An Island in the Salish Sea (or Baja)
  • I am no man’s chair.
Re: The forum is back up!
« Reply #93 on: October 01, 2019, 10:44:24 AM »
Forum performance is suddenly snappy for me.

RWD

  • Walrus Stache
  • *******
  • Posts: 6529
  • Location: Arizona
Re: The forum is back up!
« Reply #94 on: October 01, 2019, 10:56:32 AM »
MMM says the offending IP addresses are from China:
https://twitter.com/mrmoneymustache/status/1178760887777120257

dragoncar

  • Walrus Stache
  • *******
  • Posts: 9922
  • Registered member
Re: The forum is back up!
« Reply #95 on: October 01, 2019, 11:20:20 AM »
MMM says the offending IP addresses are from China:
https://twitter.com/mrmoneymustache/status/1178760887777120257

That doesn’t even sound distributed... maybe it’s a changing subnet but seems easy to filter 255 addresses.  This is probably a result of the trump tariffs on chinese lentils

frugalnacho

  • Walrus Stache
  • *******
  • Posts: 5055
  • Age: 41
  • Location: Metro Detroit
Re: The forum is back up!
« Reply #96 on: October 01, 2019, 11:28:07 AM »
So Sol went back to work in China and is hacking the forum? I fucking knew it.

arebelspy

  • Administrator
  • Senior Mustachian
  • *****
  • Posts: 28444
  • Age: -997
  • Location: Seattle, WA
Re: The forum is back up!
« Reply #97 on: October 01, 2019, 11:44:29 AM »


MMM says the offending IP addresses are from China:
https://twitter.com/mrmoneymustache/status/1178760887777120257

That doesn’t even sound distributed... maybe it’s a changing subnet but seems easy to filter 255 addresses.  This is probably a result of the trump tariffs on chinese lentils


You would think.

But adding in an IP ban slowed everything even more, as every request had to be checked against the filter.

Per the admin:
Spoiler: show
I got an ip range to block, and attempted to load it into the web server to block everything, but that tripled the total server load, as every single request was being checked by the web server against the blacklist.  

I’ve just reloaded the web server with the block list being done on the server’s firewall (firewallD) which does it on the first connection not every web request.  However it’s still having to be processed on the server end so the load is still very high.  [Web hosting company] does have a cloud firewall option for free but each block would have to be done by hand, and the ip block list I received has around 6700 entries for the known china range.


Still trying different solutions, basically.

Sorry for the hot garbage. Trash collector is working on it.
I am a former teacher who accumulated a bunch of real estate, retired at 29, spent some time traveling the world full time and am now settled with three kids.
If you want to know more about me, this Business Insider profile tells the story pretty well.
I (rarely) blog at AdventuringAlong.com. Check out the Now page to see what I'm up to currently.

PoutineLover

  • Handlebar Stache
  • *****
  • Posts: 1570
Re: The forum is back up!
« Reply #98 on: October 01, 2019, 11:57:53 AM »
Posting to follow. I didn't realize how much I checked the forum until I couldn't anymore. Hope everything gets resolved for good soon.

deborah

  • Senior Mustachian
  • ********
  • Posts: 15960
  • Age: 14
  • Location: Australia or another awesome area
Re: The forum is back up!
« Reply #99 on: October 01, 2019, 12:17:49 PM »
It’s much faster now!