Author Topic: The forum is back up!  (Read 23943 times)

RWD

  • Walrus Stache
  • *******
  • Posts: 6530
  • Location: Arizona
The forum is back up!
« on: September 20, 2019, 10:02:37 AM »
Looks like it was down for about 22 hours, based on the last post. What happened?

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #1 on: September 20, 2019, 10:21:54 AM »
It's been on/off for a while according to my monitoring.

Quote
The system confirmed 18 failed checks at 30 minute intervals starting at September 20, 2019 03:17:50

It also failed intermittently yesterday.

chemistk

  • Handlebar Stache
  • *****
  • Posts: 1739
  • Location: Mid-Atlantic
Re: The forum is back up!
« Reply #2 on: September 20, 2019, 10:46:15 AM »
It's very strange, seeing nearly a day of no new activity. A ghost town for sure.

I still seem to be having some significant issues with long load times but nice to see it back for now.

It will be interesting to learn what happened.

DadJokes

  • Handlebar Stache
  • *****
  • Posts: 2360
Re: The forum is back up!
« Reply #3 on: September 20, 2019, 10:51:21 AM »
Every now and then, it struggles mightily to load, including since it came back up today.

Maybe the potato of a server hosting these forums needs to be replaced.

nereo

  • Senior Mustachian
  • ********
  • Posts: 17500
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: The forum is back up!
« Reply #4 on: September 20, 2019, 11:28:58 AM »
Oh thank god.
Without the forum to peruse, I had to fill my time in other ways.  So I refinished a kitchen trestle table I found on CL.
Good to know I can go back to being unproductive again...

norajean

  • Pencil Stache
  • ****
  • Posts: 602
Re: The forum is back up!
« Reply #5 on: September 20, 2019, 12:05:21 PM »
Didn’t notice. Maybe it should be down once in a while to give people a break.

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #6 on: September 20, 2019, 12:24:50 PM »
Didn’t notice. Maybe it should be down once in a while to give people a break.

I only noticed because I have a monitor setup to check if the forum is up every 30 minutes.

It was down for 9 hours until it came back up, so I posted here ;-)

FireLane

  • Handlebar Stache
  • *****
  • Posts: 1329
  • Age: 41
  • Location: NYC
Re: The forum is back up!
« Reply #7 on: September 20, 2019, 07:22:31 PM »
I can connect to the forum again today, but it's very slow and frequently times out. Maybe the server is taking an early retirement.

dragoncar

  • Walrus Stache
  • *******
  • Posts: 9923
  • Registered member
Re: The forum is back up!
« Reply #8 on: September 20, 2019, 10:38:34 PM »
Didn’t notice. Maybe it should be down once in a while to give people a break.

I only noticed because I have a monitor setup to check if the forum is up every 30 minutes.

It was down for 9 hours until it came back up, so I posted here ;-)

I only noticed because I check the forum every 5-10 seconds for... uh... quality assurance purposes

Went on twitter to see if it was only down for me, and enjoyed reading his shout-out to his awesome web host

Dicey

  • Senior Mustachian
  • ********
  • Posts: 22322
  • Age: 66
  • Location: NorCal
Re: The forum is back up!
« Reply #9 on: September 21, 2019, 01:42:03 AM »
Didn’t notice. Maybe it should be down once in a while to give people a break.

I only noticed because I have a monitor setup to check if the forum is up every 30 minutes.

It was down for 9 hours until it came back up, so I posted here ;-)

I only noticed because I check the forum every 5-10 seconds for... uh... quality assurance purposes

Went on twitter to see if it was only down for me, and enjoyed reading his shout-out to his awesome web host
Lol, I did too. In the interest of, uh, quality assurance, I decided to send the fabulous webmaster a message and...it timed out. Maybe that's where the problem lies.

BTW, it's still slow as hell.

Travis

  • Magnum Stache
  • ******
  • Posts: 4219
  • Location: California
Re: The forum is back up!
« Reply #10 on: September 21, 2019, 03:00:45 AM »
Back up...ish. 0200 Pacific time it took me 15 minutes of reloading the page to write this.

soccerluvof4

  • Walrus Stache
  • *******
  • Posts: 7161
  • Location: Artic Midwest
  • Retired at 50
    • My Journal
Re: The forum is back up!
« Reply #11 on: September 21, 2019, 03:29:37 AM »
yea this has been frustrating. If i get it to load up its worse than the old dial up days. Have the time it seem like i have to try 2 or 3 xs. Not sure what the deal is but I only have so much time to sit around on this site but I enjoy it and wish they would get it fixed. Hitting post so hope it goes!

maizefolk

  • Walrus Stache
  • *******
  • Posts: 7400
Re: The forum is back up!
« Reply #12 on: September 21, 2019, 06:05:29 AM »
Yes, seems like something is still quite wrong. I'm curious what the root cause will turn out to have been.

If it was increased load on the server from more people using the forum I would have expected it to gradually get worse over time. If a software or hardware problem I would have expected it to start suddenly, but then go away entirely when it was fixed.
Instead it came on suddenly and completely (so doesn't seem like a traffic issue), and then somewhat but not completely resolved (doesn't seem like a piece of hardware/software breaking).

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #13 on: September 21, 2019, 07:59:02 AM »


The average response time is also much higher in September, too. As all you human robots have confirmed this as well :-)

Dicey

  • Senior Mustachian
  • ********
  • Posts: 22322
  • Age: 66
  • Location: NorCal
Re: The forum is back up!
« Reply #14 on: September 21, 2019, 10:30:12 AM »
Nope. Still wonky. FWP, I know, but still aggravating.  Grrr, need my forum fix the way some people need coffee. Or beer ;-)

RWD

  • Walrus Stache
  • *******
  • Posts: 6530
  • Location: Arizona
Re: The forum is back up!
« Reply #15 on: September 21, 2019, 11:24:06 AM »
My guess is DDoS. But MMM is aware of the issue per a comment on the main blog.

maizefolk

  • Walrus Stache
  • *******
  • Posts: 7400
Re: The forum is back up!
« Reply #16 on: September 21, 2019, 11:50:15 AM »
Good point RWD. Unlike any of the causes I could come up with DDoS would fit the pattern of symptoms we've seen so far.

Thanks for posting the comment showing that MMM and team know about the problem. If it is a DDoS attack mitigation is probably a painful and messy process for those involved.

katsiki

  • Handlebar Stache
  • *****
  • Posts: 2015
  • Age: 43
  • Location: La.
Re: The forum is back up!
« Reply #17 on: September 23, 2019, 02:23:14 PM »
DDOS is an interesting theory.  I wonder if one of the mods will chime in at some point once (fully?) resolved.

I noticed that the user counts on the forum's main page seemed up significantly.  I had been away from the forums for a couple of weeks but the totals per day and max total ever stats seemed significantly higher than what I recalled seeing in the past.  I can't be sure, just going from memory.

dragoncar

  • Walrus Stache
  • *******
  • Posts: 9923
  • Registered member
Re: The forum is back up!
« Reply #18 on: September 23, 2019, 04:00:35 PM »
DDOS was my first thought... financial samurai has to pay $100k/year for his kids somehow!

nereo

  • Senior Mustachian
  • ********
  • Posts: 17500
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: The forum is back up!
« Reply #19 on: September 23, 2019, 05:13:11 PM »
can someone explain DDoS/DDOS to me in laymans terms?

RWD

  • Walrus Stache
  • *******
  • Posts: 6530
  • Location: Arizona
Re: The forum is back up!
« Reply #20 on: September 23, 2019, 05:16:51 PM »
can someone explain DDoS/DDOS to me in laymans terms?

A Distributed Denial of Service (DDoS) attack is where a bunch of computers (typically compromised systems, but could also be a coordinated multi-person effort) all repeatedly access the same server to overload it and prevent it from responding to normal users.

solon

  • Handlebar Stache
  • *****
  • Posts: 2359
  • Age: 1823
  • Location: OH
Re: The forum is back up!
« Reply #21 on: September 23, 2019, 05:25:11 PM »
@arebelspy, or another mod - can anyone tell us what happened last week?

nereo

  • Senior Mustachian
  • ********
  • Posts: 17500
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: The forum is back up!
« Reply #22 on: September 23, 2019, 05:31:34 PM »
can someone explain DDoS/DDOS to me in laymans terms?

A Distributed Denial of Service (DDoS) attack is where a bunch of computers (typically compromised systems, but could also be a coordinated multi-person effort) all repeatedly access the same server to overload it and prevent it from responding to normal users.

Some people are dicks.

secondcor521

  • Walrus Stache
  • *******
  • Posts: 5503
  • Age: 54
  • Location: Boise, Idaho
  • Big cattle, no hat.
    • Age of Eon - Overwatch player videos
Re: The forum is back up!
« Reply #23 on: September 23, 2019, 05:36:09 PM »
Another possibility - although I'm not very knowledgeable in these areas so consider this spitballing - would be a bad software or hardware or database upgrade that they had trouble with and then had trouble rolling back.  I think you're always supposed to have a rollback plan and it's common practice to announce these things to end users, but maybe in this case it was viewed as a low risk thing that they could roll out silently.  I've also seen upgrades go bad and then the rollback doesn't go as planned (who tests rollback plans anyway?), and you get in a mess for a few days until you restore from a backup or something.


maizefolk

  • Walrus Stache
  • *******
  • Posts: 7400
Re: The forum is back up!
« Reply #24 on: September 23, 2019, 07:01:59 PM »
can someone explain DDoS/DDOS to me in laymans terms?

A Distributed Denial of Service (DDoS) attack is where a bunch of computers (typically compromised systems, but could also be a coordinated multi-person effort) all repeatedly access the same server to overload it and prevent it from responding to normal users.

Some people are dicks.

Can be done either because you're a dick and you can. Because you asked for protection money NOT to DDoS a website and were denied. Or politically/strategically to knock a competing business offline.

They appear to be surprisingly common and the huge number of compromised Internet-of-Things devices (and clever/evil methods like DNS amplification) is making them easier and cheaper to launch.

nereo

  • Senior Mustachian
  • ********
  • Posts: 17500
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: The forum is back up!
« Reply #25 on: September 24, 2019, 05:08:23 AM »

Can be done either because you're a dick and you can. Because you asked for protection money NOT to DDoS a website and were denied. Or politically/strategically to knock a competing business offline.

To me, all of those reasons would make the perpetrator a dick.  Whether you are doing it simply because you can, or for extortion, or to knock a rival offline... all dick moves IMO.

arebelspy

  • Administrator
  • Senior Mustachian
  • *****
  • Posts: 28444
  • Age: -997
  • Location: Seattle, WA
Re: The forum is back up!
« Reply #26 on: September 24, 2019, 08:57:49 AM »
@arebelspy, or another mod - can anyone tell us what happened last week?

Yes. Thanks for the @ when you need my attention. :)

It was not a DDOS, just a mistake.

Per the forum admin, the issue was "10% software 90% [admin] error." Server was inaccessible (due to software error), was manually shut down, then life issues came up for the admin, who neglected to turn it back on. MMM noticed after about a day, PM'd admin, who went "whoops!" and server was successfully rebooted.

I was out of the country in Germany from Wed - Monday, not on the forums at all, so I only heard about it via CC'd emails from them. Had I been home, likely I'd have noticed earlier and ping'd him.

Human, made mistake, we all hopefully went outside a bit more that day. Shouldn't be an ongoing issue.

Cheers!
I am a former teacher who accumulated a bunch of real estate, retired at 29, spent some time traveling the world full time and am now settled with three kids.
If you want to know more about me, this Business Insider profile tells the story pretty well.
I (rarely) blog at AdventuringAlong.com. Check out the Now page to see what I'm up to currently.

solon

  • Handlebar Stache
  • *****
  • Posts: 2359
  • Age: 1823
  • Location: OH
Re: The forum is back up!
« Reply #27 on: September 24, 2019, 09:03:01 AM »
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.

nereo

  • Senior Mustachian
  • ********
  • Posts: 17500
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: The forum is back up!
« Reply #28 on: September 24, 2019, 09:50:23 AM »
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.
+1 @arebelspy @FrugalToque - other than the 'report to moderator' button (which I assume just flags a post as being in violation of the forum rules somehow) there isn't an obvious way to alert whichever moderator(s) are available to say "hey, here's something you might want to be aware of"

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #29 on: September 24, 2019, 01:18:12 PM »
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.
+1 @arebelspy @FrugalToque - other than the 'report to moderator' button (which I assume just flags a post as being in violation of the forum rules somehow) there isn't an obvious way to alert whichever moderator(s) are available to say "hey, here's something you might want to be aware of"

I could add them to my monitoring email that I get anytime the forum is down ;-)

dragoncar

  • Walrus Stache
  • *******
  • Posts: 9923
  • Registered member
Re: The forum is back up!
« Reply #30 on: September 24, 2019, 01:49:38 PM »
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.
+1 @arebelspy @FrugalToque - other than the 'report to moderator' button (which I assume just flags a post as being in violation of the forum rules somehow) there isn't an obvious way to alert whichever moderator(s) are available to say "hey, here's something you might want to be aware of"

I could add them to my monitoring email that I get anytime the forum is down ;-)

No, imma need a personal cell phone or at least Snapchat

« Last Edit: September 24, 2019, 01:57:31 PM by dragoncar »

Le Poisson

  • CM*MW 2024 Attendees
  • Senior Mustachian
  • *
  • Posts: 16257
Re: The forum is back up!
« Reply #31 on: September 24, 2019, 03:51:38 PM »
Awww man, I thought it was the incredible traffic load from folks checking in on their way to Camp Mustache Toronto on Thurs/Fri that took down the server.


nereo

  • Senior Mustachian
  • ********
  • Posts: 17500
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: The forum is back up!
« Reply #32 on: September 25, 2019, 04:19:49 AM »
Awww man, I thought it was the incredible traffic load from folks checking in on their way to Camp Mustache Toronto on Thurs/Fri that took down the server.


I'll be there in spirit, my friend.

2Birds1Stone

  • Walrus Stache
  • *******
  • Posts: 7916
  • Age: 1
  • Location: Earth
  • K Thnx Bye
Re: The forum is back up!
« Reply #33 on: September 25, 2019, 07:34:51 AM »
You would figure Pete could afford better hosting.

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #34 on: September 25, 2019, 11:38:52 AM »
You would figure Pete could afford better hosting.

yeah, I wish I had a way to understand this type of issue.

I know in the past there was a similar problem and someone did <something> to fix it and until recently was in a lot better shape.

jinga nation

  • Magnum Stache
  • ******
  • Posts: 2696
  • Age: 247
  • Location: 'Murica's Dong
Re: The forum is back up!
« Reply #35 on: September 26, 2019, 08:02:09 AM »
I know nothing about Forum software hosting etc. Or where this site is being served from (server in someone's home/office, in a co-location facility, in Ze Kloud)
I work on cloud infrastructure and automation and services.
If there's a database problem, would it help to move it to a cloud-native database like AWS RDS? Along with webservers sitting behind a load balancer? If wanting to get too fancy, use CloudFront to cache and deliver the site content.
Obviously, I don't know how much the current site setup costs and how much an AWS or Azure hosted site would.
Just an idea...

Dollar Slice

  • Walrus Stache
  • *******
  • Posts: 9614
  • Age: 46
  • Location: New York City
Re: The forum is back up!
« Reply #36 on: September 26, 2019, 12:07:39 PM »
Another quick question. Is there a way to signal all the mods at once? Something like @mods, or @overlords, or something? I signaled you because you're the only one I know.
+1 @arebelspy @FrugalToque - other than the 'report to moderator' button (which I assume just flags a post as being in violation of the forum rules somehow) there isn't an obvious way to alert whichever moderator(s) are available to say "hey, here's something you might want to be aware of"

For example, if we wanted to make sure the mods knew that the forum has been slow, timing out, and generally terrible ever since it came back from the outage? :-|  Seems odd if the problem was just "needs a reboot."

arebelspy

  • Administrator
  • Senior Mustachian
  • *****
  • Posts: 28444
  • Age: -997
  • Location: Seattle, WA
Re: The forum is back up!
« Reply #37 on: September 26, 2019, 12:39:51 PM »
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!
I am a former teacher who accumulated a bunch of real estate, retired at 29, spent some time traveling the world full time and am now settled with three kids.
If you want to know more about me, this Business Insider profile tells the story pretty well.
I (rarely) blog at AdventuringAlong.com. Check out the Now page to see what I'm up to currently.

FiveSigmas

  • Pencil Stache
  • ****
  • Posts: 597
Re: The forum is back up!
« Reply #38 on: September 26, 2019, 12:53:17 PM »
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

@arebelspy The forum is still pretty pokey for me today (page loads take maybe 5 secs each).

2Birds1Stone

  • Walrus Stache
  • *******
  • Posts: 7916
  • Age: 1
  • Location: Earth
  • K Thnx Bye
Re: The forum is back up!
« Reply #39 on: September 26, 2019, 01:17:14 PM »
I'm still experiencing very slow load times and time-outs every 1/10 page loads.

FIRE@50

  • Pencil Stache
  • ****
  • Posts: 553
  • Age: 46
  • Location: Maryland
Re: The forum is back up!
« Reply #40 on: September 26, 2019, 01:27:32 PM »
As someone who comes here mostly to waste time, the forum being down half the time is a major drag.

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #41 on: September 26, 2019, 01:36:40 PM »
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

Forum has gone down entirely multiple different times today.

nereo

  • Senior Mustachian
  • ********
  • Posts: 17500
  • Location: Just south of Canada
    • Here's how you can support science today:
Re: The forum is back up!
« Reply #42 on: September 26, 2019, 01:56:44 PM »
As someone who comes here mostly to waste time, the forum being down half the time is a major drag.

But doesn't an abnormally pokey forum just take MORE of your time, thereby accomplishing your goal?
Hmmm.....

katsiki

  • Handlebar Stache
  • *****
  • Posts: 2015
  • Age: 43
  • Location: La.
Re: The forum is back up!
« Reply #43 on: September 26, 2019, 08:49:45 PM »
I think I figured it out...  This is a MMM ploy to get us all to go outside, ride our bikes, collect leaves, etc :>

RWD

  • Walrus Stache
  • *******
  • Posts: 6530
  • Location: Arizona
Re: The forum is back up!
« Reply #44 on: September 27, 2019, 06:45:00 AM »
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

@arebelspy  The forum has already timed out for me once this morning.

jinga nation

  • Magnum Stache
  • ******
  • Posts: 2696
  • Age: 247
  • Location: 'Murica's Dong
Re: The forum is back up!
« Reply #45 on: September 27, 2019, 07:09:39 AM »
As someone who comes here mostly to waste time, the forum being down half the time is a major drag.
This is a Mustachian Person's Problem, perhaps the ultimate MPP.
We're all in the same boat.

habanero

  • Handlebar Stache
  • *****
  • Posts: 1145
Re: The forum is back up!
« Reply #46 on: September 27, 2019, 07:16:25 AM »
Issue is definately not fixed. Forum has gone from being pretty stable to being a lottery with highly varying reply times. Occationally it's quite snappy like it was, but over the last few days (including today and yesterday) response times are very slow.

jinga nation

  • Magnum Stache
  • ******
  • Posts: 2696
  • Age: 247
  • Location: 'Murica's Dong
Re: The forum is back up!
« Reply #47 on: September 27, 2019, 07:42:30 AM »
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

@arebelspy  The forum has already timed out for me once this morning.

I've noticed the time out does not occur if I have a single window/tab open for MMMForums. Page load times are in excess of 10 seconds.
The moment I right-click and open, say General Discussion, in a new tab, I get the time out.
Seems that 2 GET requests from same source IP can't be resolved. Or that the first GET takes so long that subsequent GETs time out, which may mean very slow DB reads.
It's terrible to have read and writes to the same DB; at work, writes (from reply box) go to the master DB, while queries/searches/browsing are served by a replica DB.

ender

  • Walrus Stache
  • *******
  • Posts: 7402
Re: The forum is back up!
« Reply #48 on: September 27, 2019, 08:35:04 AM »
I am aware those issues existed yesterday. I believe they have been fixed.

Let me know if they persist today and beyond.

Cheers!

@arebelspy  The forum has already timed out for me once this morning.

I've noticed the time out does not occur if I have a single window/tab open for MMMForums. Page load times are in excess of 10 seconds.
The moment I right-click and open, say General Discussion, in a new tab, I get the time out.
Seems that 2 GET requests from same source IP can't be resolved. Or that the first GET takes so long that subsequent GETs time out, which may mean very slow DB reads.
It's terrible to have read and writes to the same DB; at work, writes (from reply box) go to the master DB, while queries/searches/browsing are served by a replica DB.

I have a monitor that pings the index page of the forum and it has been frequently failing for all of this week.

It's not just two tabs, it's something on the forum itself.

Dicey

  • Senior Mustachian
  • ********
  • Posts: 22322
  • Age: 66
  • Location: NorCal
Re: The forum is back up!
« Reply #49 on: September 27, 2019, 10:23:51 AM »
Boo-hoo, I'm having the same problems too. Wah!