The Money Mustache Community

General Discussion => Forum Information & FAQs => Topic started by: RWD on September 20, 2019, 10:02:37 AM

Title: The forum is back up!
Post by: RWD on September 20, 2019, 10:02:37 AM
Looks like it was down for about 22 hours, based on the last post. What happened?
Title: Re: The forum is back up!
Post by: ender on September 20, 2019, 10:21:54 AM
It's been on/off for a while according to my monitoring.

Quote
The system confirmed 18 failed checks at 30 minute intervals starting at September 20, 2019 03:17:50

It also failed intermittently yesterday.
Title: Re: The forum is back up!
Post by: chemistk on September 20, 2019, 10:46:15 AM
It's very strange, seeing nearly a day of no new activity. A ghost town for sure.

I still seem to be having some significant issues with long load times but nice to see it back for now.

It will be interesting to learn what happened.
Title: Re: The forum is back up!
Post by: DadJokes on September 20, 2019, 10:51:21 AM
Every now and then, it struggles mightily to load, including since it came back up today.

Maybe the potato of a server hosting these forums needs to be replaced.
Title: Re: The forum is back up!
Post by: nereo on September 20, 2019, 11:28:58 AM
Oh thank god.
Without the forum to peruse, I had to fill my time in other ways.  So I refinished a kitchen trestle table I found on CL.
Good to know I can go back to being unproductive again...
Title: Re: The forum is back up!
Post by: norajean on September 20, 2019, 12:05:21 PM
Didn’t notice. Maybe it should be down once in a while to give people a break.
Title: Re: The forum is back up!
Post by: ender on September 20, 2019, 12:24:50 PM
Didn’t notice. Maybe it should be down once in a while to give people a break.

I only noticed because I have a monitor setup to check if the forum is up every 30 minutes.

It was down for 9 hours until it came back up, so I posted here ;-)
Title: Re: The forum is back up!
Post by: FireLane on September 20, 2019, 07:22:31 PM
I can connect to the forum again today, but it's very slow and frequently times out. Maybe the server is taking an early retirement.
Title: Re: The forum is back up!
Post by: dragoncar on September 20, 2019, 10:38:34 PM
Didn’t notice. Maybe it should be down once in a while to give people a break.

I only noticed because I have a monitor setup to check if the forum is up every 30 minutes.

It was down for 9 hours until it came back up, so I posted here ;-)

I only noticed because I check the forum every 5-10 seconds for... uh... quality assurance purposes

Went on twitter to see if it was only down for me, and enjoyed reading his shout-out to his awesome web host
Title: Re: The forum is back up!
Post by: Dicey on September 21, 2019, 01:42:03 AM
Didn’t notice. Maybe it should be down once in a while to give people a break.

I only noticed because I have a monitor setup to check if the forum is up every 30 minutes.

It was down for 9 hours until it came back up, so I posted here ;-)

I only noticed because I check the forum every 5-10 seconds for... uh... quality assurance purposes

Went on twitter to see if it was only down for me, and enjoyed reading his shout-out to his awesome web host
Lol, I did too. In the interest of, uh, quality assurance, I decided to send the fabulous webmaster a message and...it timed out. Maybe that's where the problem lies.

BTW, it's still slow as hell.
Title: Re: The forum is back up!
Post by: Travis on September 21, 2019, 03:00:45 AM
Back up...ish. 0200 Pacific time it took me 15 minutes of reloading the page to write this.
Title: Re: The forum is back up!
Post by: soccerluvof4 on September 21, 2019, 03:29:37 AM
yea this has been frustrating. If i get it to load up its worse than the old dial up days. Have the time it seem like i have to try 2 or 3 xs. Not sure what the deal is but I only have so much time to sit around on this site but I enjoy it and wish they would get it fixed. Hitting post so hope it goes!
Title: Re: The forum is back up!
Post by: maizefolk on September 21, 2019, 06:05:29 AM
Yes, seems like something is still quite wrong. I'm curious what the root cause will turn out to have been.

If it was increased load on the server from more people using the forum I would have expected it to gradually get worse over time. If a software or hardware problem I would have expected it to start suddenly, but then go away entirely when it was fixed.
Instead it came on suddenly and completely (so doesn't seem like a traffic issue), and then somewhat but not completely resolved (doesn't seem like a piece of hardware/software breaking).
Title: Re: The forum is back up!
Post by: ender on September 21, 2019, 07:59:02 AM
(https://i.imgur.com/7RlfRnK.png)

The average response time is also much higher in September, too. As all you human robots have confirmed this as well :-)
Title: Re: The forum is back up!
Post by: Dicey on September 21, 2019, 10:30:12 AM
Nope. Still wonky. FWP, I know, but still aggravating.  Grrr, need my forum fix the way some people need coffee. Or beer ;-)
Title: Re: The forum is back up!
Post by: RWD on September 21, 2019, 11:24:06 AM
My guess is DDoS. But MMM is aware of the issue per a comment on the main blog.
Title: Re: The forum is back up!
Post by: maizefolk on September 21, 2019, 11:50:15 AM
Good point RWD. Unlike any of the causes I could come up with DDoS would fit the pattern of symptoms we've seen so far.

Thanks for posting the comment showing that MMM and team know about the problem. If it is a DDoS attack mitigation is probably a painful and messy process for those involved.
Title: Re: The forum is back up!
Post by: katsiki on September 23, 2019, 02:23:14 PM
DDOS is an interesting theory.  I wonder if one of the mods will chime in at some point once (fully?) resolved.

I noticed that the user counts on the forum's main page seemed up significantly.  I had been away from the forums for a couple of weeks but the totals per day and max total ever stats seemed significantly higher than what I recalled seeing in the past.  I can't be sure, just going from memory.
Title: Re: The forum is back up!
Post by: dragoncar on September 23, 2019, 04:00:35 PM
DDOS was my first thought... financial samurai has to pay $100k/year for his kids somehow!
Title: Re: The forum is back up!
Post by: nereo on September 23, 2019, 05:13:11 PM
can someone explain DDoS/DDOS to me in laymans terms?
Title: Re: The forum is back up!
Post by: RWD on September 23, 2019, 05:16:51 PM
can someone explain DDoS/DDOS to me in laymans terms?

A Distributed Denial of Service (DDoS) attack is where a bunch of computers (typically compromised systems, but could also be a coordinated multi-person effort) all repeatedly access the same server to overload it and prevent it from responding to normal users.
Title: Re: The forum is back up!
Post by: solon on September 23, 2019, 05:25:11 PM
@arebelspy, or another mod - can anyone tell us what happened last week?
Title: Re: The forum is back up!
Post by: nereo on September 23, 2019, 05:31:34 PM
can someone explain DDoS/DDOS to me in laymans terms?

A Distributed Denial of Service (DDoS) attack is where a bunch of computers (typically compromised systems, but could also be a coordinated multi-person effort) all repeatedly access the same server to overload it and prevent it from responding to normal users.

Some people are dicks.
Title: Re: The forum is back up!
Post by: secondcor521 on September 23, 2019, 05:36:09 PM
Another possibility - although I'm not very knowledgeable in these areas so consider this spitballing - would be a bad software or hardware or database upgrade that they had trouble with and then had trouble rolling back.  I think you're always supposed to have a rollback plan and it's common practice to announce these things to end users, but maybe in this case it was viewed as a low risk thing that they could roll out silently.  I've also seen upgrades go bad and then the rollback doesn't go as planned (who tests rollback plans anyway?), and you get in a mess for a few days until you restore from a backup or something.

Title: Re: The forum is back up!
Post by: maizefolk on September 23, 2019, 07:01:59 PM
can someone explain DDoS/DDOS to me in laymans terms?

A Distributed Denial of Service (DDoS) attack is where a bunch of computers (typically compromised systems, but could also be a coordinated multi-person effort) all repeatedly access the same server to overload it and prevent it from responding to normal users.

Some people are dicks.

Can be done either because you're a dick and you can. Because you asked for protection money NOT to DDoS a website and were denied. Or politically/strategically to knock a competing business offline.

They appear to be surprisingly common and the huge number of compromised Internet-of-Things devices (and clever/evil methods like DNS amplification) is making them easier and cheaper to launch.
Title: Re: The forum is back up!
Post by: nereo on September 24, 2019, 05:08:23 AM

Can be done either because you're a dick and you can. Because you asked for protection money NOT to DDoS a website and were denied. Or politically/strategically to knock a competing business offline.

To me, all of those reasons would make the perpetrator a dick.  Whether you are doing it simply because you can, or for extortion, or to knock a rival offline... all dick moves IMO.
Title: Re: The forum is back up!
Post by: arebelspy on September 24, 2019, 08:57:49 AM
@arebelspy, or another mod - can anyone tell us what happened last week?

Yes. Thanks for the @ when you need my attention. :)

It was not a DDOS, just a mistake.

Per the forum admin, the issue was "10% software 90% [admin] error." Server was inaccessible (due to software error), was manually shut down, then life issues came up for the admin, who neglected to turn it back on. MMM noticed after about a day, PM'd admin, who went "whoops!" and server was successfully rebooted.

I was out of the country in Germany from Wed - Monday, not on the forums at all, so I only heard about it via CC'd emails from them. Had I been home, likely I'd have noticed earlier and ping'd him.

Human, made mistake, we all hopefully went outside a bit more that day. Shouldn't be an ongoing issue.

Cheers!
Title: Re: The forum is back up!
Post by: solon on September 24, 2019, 09:03:01 AM
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.
Title: Re: The forum is back up!
Post by: nereo on September 24, 2019, 09:50:23 AM
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.
+1 @arebelspy @FrugalToque - other than the 'report to moderator' button (which I assume just flags a post as being in violation of the forum rules somehow) there isn't an obvious way to alert whichever moderator(s) are available to say "hey, here's something you might want to be aware of"
Title: Re: The forum is back up!
Post by: ender on September 24, 2019, 01:18:12 PM
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.
+1 @arebelspy @FrugalToque - other than the 'report to moderator' button (which I assume just flags a post as being in violation of the forum rules somehow) there isn't an obvious way to alert whichever moderator(s) are available to say "hey, here's something you might want to be aware of"

I could add them to my monitoring email that I get anytime the forum is down ;-)
Title: Re: The forum is back up!
Post by: dragoncar on September 24, 2019, 01:49:38 PM
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.
+1 @arebelspy @FrugalToque - other than the 'report to moderator' button (which I assume just flags a post as being in violation of the forum rules somehow) there isn't an obvious way to alert whichever moderator(s) are available to say "hey, here's something you might want to be aware of"

I could add them to my monitoring email that I get anytime the forum is down ;-)

No, imma need a personal cell phone or at least Snapchat

(https://i.imgur.com/OihGqz8.jpg)
Title: Re: The forum is back up!
Post by: Le Poisson on September 24, 2019, 03:51:38 PM
Awww man, I thought it was the incredible traffic load from folks checking in on their way to Camp Mustache Toronto on Thurs/Fri that took down the server.

(https://media1.giphy.com/media/jLejCQ45gm3WU/giphy.gif?cid=790b76114712a1470b653e6414ea6a2f6f55c01fae7cfc1b&rid=giphy.gif)
Title: Re: The forum is back up!
Post by: nereo on September 25, 2019, 04:19:49 AM
Awww man, I thought it was the incredible traffic load from folks checking in on their way to Camp Mustache Toronto on Thurs/Fri that took down the server.


I'll be there in spirit, my friend.
Title: Re: The forum is back up!
Post by: 2Birds1Stone on September 25, 2019, 07:34:51 AM
You would figure Pete could afford better hosting.
Title: Re: The forum is back up!
Post by: ender on September 25, 2019, 11:38:52 AM
You would figure Pete could afford better hosting.

yeah, I wish I had a way to understand this type of issue.

I know in the past there was a similar problem and someone did <something> to fix it and until recently was in a lot better shape.
Title: Re: The forum is back up!
Post by: jinga nation on September 26, 2019, 08:02:09 AM
I know nothing about Forum software hosting etc. Or where this site is being served from (server in someone's home/office, in a co-location facility, in Ze Kloud)
I work on cloud infrastructure and automation and services.
If there's a database problem, would it help to move it to a cloud-native database like AWS RDS? Along with webservers sitting behind a load balancer? If wanting to get too fancy, use CloudFront to cache and deliver the site content.
Obviously, I don't know how much the current site setup costs and how much an AWS or Azure hosted site would.
Just an idea...
Title: Re: The forum is back up!
Post by: Dollar Slice on September 26, 2019, 12:07:39 PM
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.
+1 @arebelspy @FrugalToque - other than the 'report to moderator' button (which I assume just flags a post as being in violation of the forum rules somehow) there isn't an obvious way to alert whichever moderator(s) are available to say "hey, here's something you might want to be aware of"

For example, if we wanted to make sure the mods knew that the forum has been slow, timing out, and generally terrible ever since it came back from the outage? :-|  Seems odd if the problem was just "needs a reboot."
Title: Re: The forum is back up!
Post by: arebelspy on September 26, 2019, 12:39:51 PM
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!
Title: Re: The forum is back up!
Post by: FiveSigmas on September 26, 2019, 12:53:17 PM
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

@arebelspy The forum is still pretty pokey for me today (page loads take maybe 5 secs each).
Title: Re: The forum is back up!
Post by: 2Birds1Stone on September 26, 2019, 01:17:14 PM
I'm still experiencing very slow load times and time-outs every 1/10 page loads.
Title: Re: The forum is back up!
Post by: FIRE@50 on September 26, 2019, 01:27:32 PM
As someone who comes here mostly to waste time, the forum being down half the time is a major drag.
Title: Re: The forum is back up!
Post by: ender on September 26, 2019, 01:36:40 PM
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

Forum has gone down entirely multiple different times today.
Title: Re: The forum is back up!
Post by: nereo on September 26, 2019, 01:56:44 PM
As someone who comes here mostly to waste time, the forum being down half the time is a major drag.

But doesn't an abnormally pokey forum just take MORE of your time, thereby accomplishing your goal?
Hmmm.....
Title: Re: The forum is back up!
Post by: katsiki on September 26, 2019, 08:49:45 PM
I think I figured it out...  This is a MMM ploy to get us all to go outside, ride our bikes, collect leaves, etc :>
Title: Re: The forum is back up!
Post by: RWD on September 27, 2019, 06:45:00 AM
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

@arebelspy  The forum has already timed out for me once this morning.
Title: Re: The forum is back up!
Post by: jinga nation on September 27, 2019, 07:09:39 AM
As someone who comes here mostly to waste time, the forum being down half the time is a major drag.
This is a Mustachian Person's Problem, perhaps the ultimate MPP.
We're all in the same boat.
Title: Re: The forum is back up!
Post by: habanero on September 27, 2019, 07:16:25 AM
Issue is definately not fixed. Forum has gone from being pretty stable to being a lottery with highly varying reply times. Occationally it's quite snappy like it was, but over the last few days (including today and yesterday) response times are very slow.
Title: Re: The forum is back up!
Post by: jinga nation on September 27, 2019, 07:42:30 AM
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

@arebelspy  The forum has already timed out for me once this morning.

I've noticed the time out does not occur if I have a single window/tab open for MMMForums. Page load times are in excess of 10 seconds.
The moment I right-click and open, say General Discussion, in a new tab, I get the time out.
Seems that 2 GET requests from same source IP can't be resolved. Or that the first GET takes so long that subsequent GETs time out, which may mean very slow DB reads.
It's terrible to have read and writes to the same DB; at work, writes (from reply box) go to the master DB, while queries/searches/browsing are served by a replica DB.
Title: Re: The forum is back up!
Post by: ender on September 27, 2019, 08:35:04 AM
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

@arebelspy  The forum has already timed out for me once this morning.

I've noticed the time out does not occur if I have a single window/tab open for MMMForums. Page load times are in excess of 10 seconds.
The moment I right-click and open, say General Discussion, in a new tab, I get the time out.
Seems that 2 GET requests from same source IP can't be resolved. Or that the first GET takes so long that subsequent GETs time out, which may mean very slow DB reads.
It's terrible to have read and writes to the same DB; at work, writes (from reply box) go to the master DB, while queries/searches/browsing are served by a replica DB.

I have a monitor that pings the index page of the forum and it has been frequently failing for all of this week.

It's not just two tabs, it's something on the forum itself.
Title: Re: The forum is back up!
Post by: Dicey on September 27, 2019, 10:23:51 AM
Boo-hoo, I'm having the same problems too. Wah!
Title: Re: The forum is back up!
Post by: jinga nation on September 27, 2019, 10:34:00 AM
it's ironic that on Sept 20 Mr MM tweeted kudos to his web dev and hosting folks for 4 years of zero downtime.

i assume that while the site is up, the terrible read/write times may be due to RAID being rebuilt or resync or some underlying storage issue.

if i use a diff browser to log in, i get time outs. i have to use a browser that has cached credentials. so maybe same/diff problem.

ETA around 1245 ET today site is snappy and fast. no issues observed.
Title: Re: The forum is back up!
Post by: arebelspy on September 27, 2019, 11:18:16 AM
Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)
Title: Re: The forum is back up!
Post by: jinga nation on September 27, 2019, 11:25:58 AM
@arebelspy It would be good to hear about the root cause of the issue.
The next time it happens, we can say "Hey <problem> is happening again, go check <component> or <service>".
Title: Re: The forum is back up!
Post by: ender on September 27, 2019, 11:58:08 AM
Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

Do you want me to add your email to the alerts sent when there's downtime?

They have been very accurate so far to total outages.

Title: Re: The forum is back up!
Post by: dragoncar on September 28, 2019, 12:31:52 AM
Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

If I can’t load the forum, how can I send a PM?

Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

Do you want me to add your email to the alerts sent when there's downtime?

They have been very accurate so far to total outages.



This would be sweet and also a bot that tweet at mmm “forum is down” and also update www.isthemrmoneymustacheforumdown.com
Title: Re: The forum is back up!
Post by: solon on September 28, 2019, 07:22:38 AM
Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

If I can’t load the forum, how can I send a PM?

Thanks for alerting me again. Think it is fixed again.

Any time something like this occurs, please don't assume it's normal, we'd like everything running smoothly.

If I acknowledge it and it still persists days later, assume I don't realize it isn't fixed (sometimes it seems it is, but is not) and feel free to contact again. PM is quicker than @ if it's really bad. :)

Do you want me to add your email to the alerts sent when there's downtime?

They have been very accurate so far to total outages.



This would be sweet and also a bot that tweet at mmm “forum is down” and also update www.isthemrmoneymustacheforumdown.com

I clicked it. I wish it was a real link.
Title: Re: The forum is back up!
Post by: Dicey on September 28, 2019, 08:19:29 AM
Still slow and timing out today.
Title: Re: The forum is back up!
Post by: BigMoneyJim on September 28, 2019, 08:21:28 AM
it's ironic that on Sept 20 Mr MM tweeted kudos to his web dev and hosting folks for 4 years of zero downtime.

That was specifically in reference to the MMM blog & site which is separate from the forums and presumably much, much busier and more profitable.
Title: Re: The forum is back up!
Post by: RWD on September 28, 2019, 09:41:48 AM
This would be sweet and also a bot that tweet at mmm “forum is down” and also update www.isthemrmoneymustacheforumdown.com

I clicked it. I wish it was a real link.

Real link:
https://downforeveryoneorjustme.com/forum.mrmoneymustache.com
Title: Re: The forum is back up!
Post by: lutorm on September 28, 2019, 02:46:14 PM
Yeah something's definitely not right. Even when the forum is up, I'm frequently experiencing a minute of response time for loading pages. It makes browsing the forum so painful so as to be effectively impossible.
Title: Re: The forum is back up!
Post by: Roadrunner53 on September 28, 2019, 04:51:20 PM
I am in CT and have been having issues too. What is going on?
Title: Re: The forum is back up!
Post by: G-dog on September 28, 2019, 04:56:42 PM
@arebelspy and @FrugalToque - how should we notify someone when the forum is down? When it is down, I can’t get to the PM function.

The forum has been flakey since the initial incident earlier.
Title: Re: The forum is back up!
Post by: 2Birds1Stone on September 28, 2019, 06:29:18 PM
The forum is running like dogshit right now (8:27 PM ET) and was down for the past 30+ minutes.

This will definitely hurt long term traffic, especially regulars.
Title: Re: The forum is back up!
Post by: DadJokes on September 28, 2019, 06:56:09 PM
Yeah, I hate to complain about something I access for free, but it's been hot garbage lately. It's probably a good excuse to take a break though. I spend a bit too much time reading through posts here.
Title: Re: The forum is back up!
Post by: ender on September 28, 2019, 06:58:49 PM
I just realized that I can't actually add a CC email address without paying a bunch.

I'm using SiteUptime (https://siteuptime.com) to do a super basic check against https://forum.mrmoneymustache.com/index.php (http) -- it should be really easy to setup their basic monitor, if anyone wants to.
Title: Re: The forum is back up!
Post by: InSearchOfFire on September 28, 2019, 09:44:33 PM
Hi,

@arebelspy, @FrugalToque

Sorry to complain, and I do appreciate all that you all do to keep this place running, but I've been having some really slow page-loading since the initial incident - including trying to load this page tonight and trying to post this...

Also been having some pages time-out since the initial incident.

Thanks!
Title: Re: The forum is back up!
Post by: dragoncar on September 28, 2019, 10:26:47 PM
Yeah, I hate to complain about something I access for free

Holup

You guys aren’t paying for Mmm premium elite?
Title: Re: The forum is back up!
Post by: FiveSigmas on September 28, 2019, 11:18:25 PM
This would be sweet and also a bot that tweet at mmm “forum is down” and also update www.isthemrmoneymustacheforumdown.com

I clicked it. I wish it was a real link.

Time for www.istheisthemrmoneymustacheforumdowndown.com ?

P.S. Forum’s back to its pokey self. :-/
Title: Re: The forum is back up!
Post by: jpdx on September 29, 2019, 01:35:23 AM
It's after midnight PST and the site is currently slow as molasses.
Title: Re: The forum is back up!
Post by: Dicey on September 29, 2019, 05:14:27 AM
It would be great to hear from the mods about what the problem is. Clearly it's ongoing and crazy frustrating for all involved.
Title: Re: The forum is back up!
Post by: katsiki on September 29, 2019, 08:52:52 AM
DDOS is an interesting theory.  I wonder if one of the mods will chime in at some point once (fully?) resolved.

I noticed that the user counts on the forum's main page seemed up significantly.  I had been away from the forums for a couple of weeks but the totals per day and max total ever stats seemed significantly higher than what I recalled seeing in the past.  I can't be sure, just going from memory.

Does anyone know how guests are counted?  A few minutes ago (9:45 AM CT), the forum home page showed 7,600+ guests online now.  Could these guests be bots or something nefarious?  If not, has the forum just rocketed into a new level of activity for some reason?  Normally, I recall 2000+ users online at a time to be 'normal'.

@arebelspy   FYI if this helps the admin or mods figure this out.
Title: Re: The forum is back up!
Post by: maizefolk on September 29, 2019, 01:43:22 PM
Katsiki, that is an intriguing observation.

We're at ~7,300 users right now, but the forum set its all time record for connected users just today at 8,820.

(https://imgpile.com/images/137P9o.png) (https://imgpile.com/i/137P9o)

That would be consistent with either a DDoS attack or a sudden spike in traffic causing the general forum lag (it's been essentially unusable for me).

Has MMM gotten any recent extremely large press/online coverage?

It is also possible something with the overload of the server is causing it to lose track or double/triple count unregistered users, which would mean the extremely high guest counts would be an effect rather than a cause of the problem.
Title: Re: The forum is back up!
Post by: ender on September 29, 2019, 03:13:15 PM
An easy fix would be to restrict the forum to logged in users only, too, that would likely cut down on server load depending on what those guests are doing
Title: Re: The forum is back up!
Post by: 2Birds1Stone on September 29, 2019, 06:24:12 PM
Down yet again for the past 20 minutes =D @MMM  double you tee eff mate?
Title: Re: The forum is back up!
Post by: ender on September 29, 2019, 06:27:32 PM
My alert is firing on and off all day today...

I can't really do anything about it.

If more $$$ would help avoid this in the future I think many of us would be willing to chip in some.
Title: Re: The forum is back up!
Post by: SwordGuy on September 29, 2019, 08:01:54 PM
Sometimes I get this message when the forum won't come up, sometimes not:

"Can’t connect securely to this page
This might be because the site uses outdated or unsafe TLS security settings. If this keeps happening, try contacting the website’s owner."

Microsoft Edge and also Firefox.
Title: Re: The forum is back up!
Post by: dragoncar on September 30, 2019, 01:25:17 AM
An easy fix would be to restrict the forum to logged in users only, too, that would likely cut down on server load depending on what those guests are doing

Psh.  Just restrict access to walrus stache or above.  I’ll generate enough content for everyone
Title: Re: The forum is back up!
Post by: dragoncar on September 30, 2019, 01:26:26 AM
Content not guaranteed
Title: Re: The forum is back up!
Post by: Le Poisson on September 30, 2019, 04:56:50 AM
Content not guaranteed

Quality of content also not guaranteed.

I speak often, but not well.
Title: Re: The forum is back up!
Post by: ender on September 30, 2019, 06:45:22 AM
An easy fix would be to restrict the forum to logged in users only, too, that would likely cut down on server load depending on what those guests are doing

Psh.  Just restrict access to walrus stache or above.  I’ll generate enough content for everyone

I better get posting then! :-)
Title: Re: The forum is back up!
Post by: Dicey on September 30, 2019, 07:32:34 AM
It would be great to hear an update from the mods, since the problem has not been resolved.
Title: Re: The forum is back up!
Post by: arebelspy on September 30, 2019, 08:36:05 AM
We've been looking into it; those positing a DDOS attack appear to be on the right track. We're looking into potential cloudflare solutions.
Title: Re: The forum is back up!
Post by: Dicey on September 30, 2019, 10:37:29 AM
We've been looking into it; those positing a DDOS attack appear to be on the right track. We're looking into potential cloudflare solutions.
Thanks for the reply. Pardon my ignorance, but what's the point of a DDOS? Are they looking for ransom money? Trying to drive traffic elsewhere? Or just doing it because they can?
Title: Re: The forum is back up!
Post by: rantk81 on September 30, 2019, 11:00:40 AM
That really sucks if this is the deliberate act of someone.

Title: Re: The forum is back up!
Post by: rantk81 on September 30, 2019, 11:05:45 AM
We've been looking into it; those positing a DDOS attack appear to be on the right track. We're looking into potential cloudflare solutions.
Thanks for the reply. Pardon my ignorance, but what's the point of a DDOS? Are they looking for ransom money? Trying to drive traffic elsewhere? Or just doing it because they can?

Could be a ransom (We would probably never get confirmation from MMM if that were the case.)  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum.  Could be random collateral damage of a DDOS against another website hosted on the same equipment or same data-center.  Could be someone just screwing around.  We'll probably never know why.  People can be jerks for no reason.
Title: Re: The forum is back up!
Post by: frugalnacho on September 30, 2019, 11:35:26 AM
I just want to pile on and say the forum has been working like hot garbage for the last 1-2 weeks for me as well.  I tried to make a post/send a PM about it last Friday, but have been unable to load access the site until now.  Took 8 attempts to access this thread and make a reply...hopefully goes through this time. 

I have been frantically hitting the refresh button every few seconds but it doesn't seem to be solving the problem
Title: Re: The forum is back up!
Post by: dragoncar on September 30, 2019, 12:40:23 PM
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?
Title: Re: The forum is back up!
Post by: solon on September 30, 2019, 03:14:18 PM
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?

Maybe sol and boarder42 met up IRL and hatched this diabolical scheme?
Title: Re: The forum is back up!
Post by: 2Birds1Stone on September 30, 2019, 05:35:43 PM
The forum top is in.
Title: Re: The forum is back up!
Post by: norajean on September 30, 2019, 06:46:19 PM
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?

Someone said the Sol person wasn't banned, just got angry and left. Something about his wife forcing him back to work.
Title: Re: The forum is back up!
Post by: Dicey on September 30, 2019, 11:05:41 PM
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?

Someone said the Sol person wasn't banned, just got angry and left. Something about his wife forcing him back to work.
I miss him. Paging @ Sol. Your absence is felt, dude!
Title: Re: The forum is back up!
Post by: golfreak12 on October 01, 2019, 10:23:16 AM
This forum has been working like garbage for me too the for the last 2-3 weeks. Couldn't access it last night.
Title: Re: The forum is back up!
Post by: Lulee on October 01, 2019, 10:28:13 AM
  Could be someone spiteful who was banned from the board.  Could be someone who got into a heated argument on the forum. 

sol confirmed?

Someone said the Sol person wasn't banned, just got angry and left. Something about his wife forcing him back to work.
I miss him. Paging @ Sol. Your absence is felt, dude!

He is missed even if he is a bit of an acquired taste.  Whatever his faults (and yes, I found him frequently grating as he didn't always seem to grasp the emotional content/impact of his words) I wouldn't believe for a second that he or any other long time board member would actually be party to a DDOS attack making the forum unusable to the rest of us.  I suspect someone got a bug up their backside reading about the FIRE movement and is hassling us.  Just wish there was a way to help the  moderators in fixing this.
Title: Re: The forum is back up!
Post by: Jon_Snow on October 01, 2019, 10:44:24 AM
Forum performance is suddenly snappy for me.
Title: Re: The forum is back up!
Post by: RWD on October 01, 2019, 10:56:32 AM
MMM says the offending IP addresses are from China:
https://twitter.com/mrmoneymustache/status/1178760887777120257
Title: Re: The forum is back up!
Post by: dragoncar on October 01, 2019, 11:20:20 AM
MMM says the offending IP addresses are from China:
https://twitter.com/mrmoneymustache/status/1178760887777120257

That doesn’t even sound distributed... maybe it’s a changing subnet but seems easy to filter 255 addresses.  This is probably a result of the trump tariffs on chinese lentils
Title: Re: The forum is back up!
Post by: frugalnacho on October 01, 2019, 11:28:07 AM
So Sol went back to work in China and is hacking the forum? I fucking knew it.
Title: Re: The forum is back up!
Post by: arebelspy on October 01, 2019, 11:44:29 AM


MMM says the offending IP addresses are from China:
https://twitter.com/mrmoneymustache/status/1178760887777120257

That doesn’t even sound distributed... maybe it’s a changing subnet but seems easy to filter 255 addresses.  This is probably a result of the trump tariffs on chinese lentils


You would think.

But adding in an IP ban slowed everything even more, as every request had to be checked against the filter.

Per the admin:
Spoiler: show
I got an ip range to block, and attempted to load it into the web server to block everything, but that tripled the total server load, as every single request was being checked by the web server against the blacklist.  

I’ve just reloaded the web server with the block list being done on the server’s firewall (firewallD) which does it on the first connection not every web request.  However it’s still having to be processed on the server end so the load is still very high.  [Web hosting company] does have a cloud firewall option for free but each block would have to be done by hand, and the ip block list I received has around 6700 entries for the known china range.


Still trying different solutions, basically.

Sorry for the hot garbage. Trash collector is working on it.
Title: Re: The forum is back up!
Post by: PoutineLover on October 01, 2019, 11:57:53 AM
Posting to follow. I didn't realize how much I checked the forum until I couldn't anymore. Hope everything gets resolved for good soon.
Title: Re: The forum is back up!
Post by: deborah on October 01, 2019, 12:17:49 PM
It’s much faster now!
Title: Re: The forum is back up!
Post by: G-dog on October 01, 2019, 12:23:03 PM
It’s much faster now!

Yes, for the last few hours. Though it did go down in the morning, once it came back up it has been very responsive and fast.

Thanks to all that have been working on this.

I wonder what would happen if China caught FIRE!?
Title: Re: The forum is back up!
Post by: InSearchOfFire on October 01, 2019, 12:58:06 PM
Seems to be working great today.  I really missed it!

Many thanks to MMM and the mods and all who helped get this show back up and running!

Thank you!  Thank you!  Thank you! 
Title: Re: The forum is back up!
Post by: OurTown on October 01, 2019, 01:30:11 PM
The forum being down is an international conspiracy to force me to be productive at work.  Boooooooo!
Title: Re: The forum is back up!
Post by: Le Poisson on October 01, 2019, 01:58:25 PM
Woo Hoo! Baby, we're BACK!

Thanks to all concerned.
Title: Re: The forum is back up!
Post by: dragoncar on October 01, 2019, 03:21:09 PM
I thought IP filtering was typically performed at the router level for speed
Title: Re: The forum is back up!
Post by: secondcor521 on October 01, 2019, 05:34:28 PM
I thought IP filtering was typically performed at the router level for speed

Not an expert.  Yes, routers can do IP filtering at their level.  But if China is sending web traffic MMM's way, it's going along Internet backbones and not going through routers at that point.

Thanks, mods, admins, MMM, etc.  Quite fast for me currently.
Title: Re: The forum is back up!
Post by: Dicey on October 01, 2019, 05:48:53 PM
Hi my name is Dicey and I may have a slight MMM Forum addiction. It's been a rough week. Thanks to our might mod team, especially @arebelspy, who is never foiled by bad guys.
Title: Re: The forum is back up!
Post by: Le Poisson on October 01, 2019, 06:53:57 PM
The forum being down sucked, but it was good that eh glitch began while so many of us were together FOR CM*TO... at least we had friends to share the misery with.
Title: Re: The forum is back up!
Post by: Travis on October 01, 2019, 07:14:24 PM
The forum being down is an international conspiracy to force me to be productive at work.  Boooooooo!

Two words: You Tube.
Title: Re: The forum is back up!
Post by: dragoncar on October 02, 2019, 12:42:36 AM
I thought IP filtering was typically performed at the router level for speed

Not an expert.  Yes, routers can do IP filtering at their level.  But if China is sending web traffic MMM's way, it's going along Internet backbones and not going through routers at that point.

Thanks, mods, admins, MMM, etc.  Quite fast for me currently.

Not an expert either but I think routers are integral to this whole Internet thing and can’t really be bypassed
Title: Re: The forum is back up!
Post by: secondcor521 on October 02, 2019, 01:28:25 AM
I thought IP filtering was typically performed at the router level for speed

Not an expert.  Yes, routers can do IP filtering at their level.  But if China is sending web traffic MMM's way, it's going along Internet backbones and not going through routers at that point.

Thanks, mods, admins, MMM, etc.  Quite fast for me currently.

Not an expert either but I think routers are integral to this whole Internet thing and can’t really be bypassed

At first I thought you were referring to Internet backbone sites (which I thought had routing devices that were called switches).

But after thinking about it some more, I guess you may be referring to the router between the MMM forum server and it's upstream Internet backbone site.  You're right, that router can't be bypassed if you want to get to the MMM server.

My little home-quality cable modem / wifi router isn't powerful or configurable enough to handle the kind of problem being discussed here, but it's certainly conceivable that commercial-quality routers could.

ETA:  Actually, it used to be that you could connect a computer directly to an Internet backbone site with a Ethernet network card and a hardcoded IP address (of course you should own that IP address, otherwise you can cause problems), so *technically* you didn't need a router to be on the Internet.  I did it myself back in the mid-90's at a Silicon Valley-based Fortune 500 company.  In limited cases this is probably true today - I bet there are some Google IT guys' computers set up that way just so then don't have to have a router slowing things down ;-).  Obviously, though, the vast, vast majority of users these days are connected via routers, get dynamically assigned IP addresses, etc.
Title: Re: The forum is back up!
Post by: dragoncar on October 02, 2019, 07:19:22 AM
I thought IP filtering was typically performed at the router level for speed

Not an expert.  Yes, routers can do IP filtering at their level.  But if China is sending web traffic MMM's way, it's going along Internet backbones and not going through routers at that point.

Thanks, mods, admins, MMM, etc.  Quite fast for me currently.

Not an expert either but I think routers are integral to this whole Internet thing and can’t really be bypassed

At first I thought you were referring to Internet backbone sites (which I thought had routing devices that were called switches).

But after thinking about it some more, I guess you may be referring to the router between the MMM forum server and it's upstream Internet backbone site.  You're right, that router can't be bypassed if you want to get to the MMM server.

My little home-quality cable modem / wifi router isn't powerful or configurable enough to handle the kind of problem being discussed here, but it's certainly conceivable that commercial-quality routers could.

ETA:  Actually, it used to be that you could connect a computer directly to an Internet backbone site with a Ethernet network card and a hardcoded IP address (of course you should own that IP address, otherwise you can cause problems), so *technically* you didn't need a router to be on the Internet.  I did it myself back in the mid-90's at a Silicon Valley-based Fortune 500 company.  In limited cases this is probably true today - I bet there are some Google IT guys' computers set up that way just so then don't have to have a router slowing things down ;-).  Obviously, though, the vast, vast majority of users these days are connected via routers, get dynamically assigned IP addresses, etc.

Could be, but I think you are reading too much into the different terminology of router v switches.  That could be how it’s used in industry but these days the distinction is very gray.  You need a device capable of routing packets for internet.  A switch that decides where to send packets is performing a routing function as far as I’m concerned
Title: Re: The forum is back up!
Post by: maizefolk on October 02, 2019, 07:44:11 AM
Thanks for the peak behind the curtain @arebelspy and I'm really happy and surprised the problem turned out to be a single /16 block.

Forum is nice and snappy. Like any short term discomfort I find myself appreciating near instant page loading here in a way I never would have before a week or two of "open each link in a new tab, go do something else, and come back to see if I have something read or the connection timed out" performance.

A big thank you to both the mod and the IT teams!

sol confirmed?

Someone said the Sol person wasn't banned, just got angry and left. Something about his wife forcing him back to work.

I disappear for a few months and I miss all sorts of forum drama. Lucky timing on my part.
Title: Re: The forum is back up!
Post by: rantk81 on October 02, 2019, 07:55:43 AM
Thanks to any and all who took the time to fix what needed fixing!
Title: Re: The forum is back up!
Post by: RWD on October 02, 2019, 08:16:26 AM
sol confirmed?

Someone said the Sol person wasn't banned, just got angry and left. Something about his wife forcing him back to work.

I disappear for a few months and I miss all sorts of forum drama. Lucky timing on my part.

CliffsNotes: Heated discussion in a very political thread made sol decide to leave the forum.

I'm done.  I'm sure the forum will be fine without me.
Title: Re: The forum is back up!
Post by: secondcor521 on October 02, 2019, 12:40:17 PM
I thought IP filtering was typically performed at the router level for speed

Not an expert.  Yes, routers can do IP filtering at their level.  But if China is sending web traffic MMM's way, it's going along Internet backbones and not going through routers at that point.

Thanks, mods, admins, MMM, etc.  Quite fast for me currently.

Not an expert either but I think routers are integral to this whole Internet thing and can’t really be bypassed

At first I thought you were referring to Internet backbone sites (which I thought had routing devices that were called switches).

But after thinking about it some more, I guess you may be referring to the router between the MMM forum server and it's upstream Internet backbone site.  You're right, that router can't be bypassed if you want to get to the MMM server.

My little home-quality cable modem / wifi router isn't powerful or configurable enough to handle the kind of problem being discussed here, but it's certainly conceivable that commercial-quality routers could.

ETA:  Actually, it used to be that you could connect a computer directly to an Internet backbone site with a Ethernet network card and a hardcoded IP address (of course you should own that IP address, otherwise you can cause problems), so *technically* you didn't need a router to be on the Internet.  I did it myself back in the mid-90's at a Silicon Valley-based Fortune 500 company.  In limited cases this is probably true today - I bet there are some Google IT guys' computers set up that way just so then don't have to have a router slowing things down ;-).  Obviously, though, the vast, vast majority of users these days are connected via routers, get dynamically assigned IP addresses, etc.

Could be, but I think you are reading too much into the different terminology of router v switches.  That could be how it’s used in industry but these days the distinction is very gray.  You need a device capable of routing packets for internet.  A switch that decides where to send packets is performing a routing function as far as I’m concerned

Fair enough.  After my initial disagreeable post, I was mostly just trying to understand where the disconnect was.  Honestly how the Internet actually works is not my main area of knowledge.
Title: Re: The forum is back up!
Post by: Just Joe on October 03, 2019, 11:11:46 AM
The forum being down is an international conspiracy to force me to be productive at work.  Boooooooo!

No, someone just wants you to spend-spend-spend! Can't be sharing the kind of wise info this forum teaches.
Title: Re: The forum is back up!
Post by: ender on October 04, 2019, 01:43:34 PM
fwiw when there is a major issue, the email address that is listed as the contact is invalid:

Quote
There was a problem delivering your message to webmaster@MrMoneyMustache.com. See the technical details below.