Server Reboot Tonight

I am planning to reboot the server tonight to see if that helps alleviate some of the recent slowness.

I will be rebooting the server tonight at:

Midnight Eastern
11:00 PM Central
10:00 PM Mountain
9:00 PM Pacific

Expect 5 to 10 minutes of downtime around then. The Wizard's other sites will also be unavailable during this time.

Ironically, the site has been great for the past 30 minutes. 45 minutes ago it was still acting like a dog.

It has been a real dog for me the past 2 days...... Taking 4 or 5 seconds to load a page.

It's gotten a lot better in the last few hours.

The craps players finally settled down....

Quote: djatc

The craps players finally settled down....

No its the bac players :-)
Site was lagging, great night now.

Its the ghost in the machine. As soon as you typed reboot, everything is now good :-)

Quote: djatc

The craps players finally settled down....

+1

no need for haters ;-)

Any chance the site was under a small DoS attack by a presumably disgruntled ex-member? It is definitely much better now, after 48 hours, and apparently before any overt action was taken.

Quote: RaleighCraps

Any chance the site was under a small DoS attack by a presumably disgruntled ex-member? It is definitely much better now, after 48 hours, and apparently before any overt action was taken.

Lem66

I also have near-instant paging.

Thanks, JB! Really appreciate you doing the maintenance!

Quote: beachbumbabs

I also have near-instant paging.

Thanks, JB! Really appreciate you doing the maintenance!

I wish I could take credit but I haven't done anything yet. T minus 23 minutes until the reboot...

Right in the middle of posting, my message got (*HB#UB$RD

Unlikely, I haven't started it yet. But I am about to...

Quote: JB

Unlikely, I haven't started it yet. But I am about to...

Oh, alright. That sound fai$424(*&98 f 8 a

Well, doesn't seem like this worked. Pages still loading very slowly.

Quote: RaleighCraps

Any chance the site was under a small DoS attack by a presumably disgruntled ex-member? It is definitely much better now, after 48 hours, and apparently before any overt action was taken.

DoS attacks should be fairly obvious from the logs, or even a graph of traffic.

although after the reboot it was still slow, today it seems like it is back to normal.

Didn't we have a member that threatened to attack? Was it my favorite, Varmenti?

Quote: AxiomOfChoice

DoS attacks should be fairly obvious from the logs, or even a graph of traffic.

Yes. I didn't want to just come out and say, Check the logs for DoS, but when the site started performing better before JB had done the reboot, I was trying to guess at what could possibly explain that. My first thought, which I believe you also put forth, had been a possible issue with the database. However, it seemed unlikely that a db issue would have cleared itself up before JB took any action, so I was looking for another explanation.

I had not thought about a drive problem, until you mentioned RAID drives. I'm assuming RAID technology now is capable of rebuilding data on the fly? RAID 5 was the last technology that I dealt with, with data and CRC values striped across 4 or 5 HD drives. You could lose any one HD, and the remaining drives had enough info to let you rebuild the data that was on the destroyed drive. That was really cool technology back then. I think that was around 1995??

I believe someone else mentioned the WOO site was performing fine. Now I would guess that the two sites are probably on the same server hardware, but that would not have to be true. I would bet though. they are with the same hosting service, so I was ruling out any hosting site networking issue.

In any event, the site has improved, although I still get the occasional lag even now, although not nearly as bad as it was last week

Quote: RaleighCraps

I had not thought about a drive problem, until you mentioned RAID drives. I'm assuming RAID technology now is capable of rebuilding data on the fly? RAID 5 was the last technology that I dealt with, with data and CRC values striped across 4 or 5 HD drives. You could lose any one HD, and the remaining drives had enough info to let you rebuild the data that was on the destroyed drive. That was really cool technology back then. I think that was around 1995??

Yes, most modern server class RAID controllers readily handle on-the-fly rebuilds and hot swap replacement. This has been the case for at least 15 years. The downside is that while you have a failed drive in (and during the rebuild), performance is degraded noticeably.

If you're running a database on it, however, it's more likely to be RAID 10 or RAID 100 (vs RAID 5 or 6), as this allows multiple independent seeks simultaneously, and vastly higher performance.

Having a hot spare in is a "good thing", but some chassis don't have enough bays to allow a hot spare and enough drive capacity. Design tradeoff decision, do you want to spend $10k* on another drive cage, or are you willing to take the occasional performance hit when a drive fails?

*Arbitrary price for purpose of discussion. It's rarely an insurmountable number for a critical application, but it's high enough to discourage casually buying vast amounts of spare capacity.

Thanks for the RAID primer lesson. I couldn't believe it had been 19 years when I was trying to figure out how long it had been since my RAID 5 days. It is a testament to the technology that it is still being used now, albeit in newer forms.

Seems improved but intermittently slows down.

Quote: RaleighCraps

Thanks for the RAID primer lesson. I couldn't believe it had been 19 years when I was trying to figure out how long it had been since my RAID 5 days. It is a testament to the technology that it is still being used now, albeit in newer forms.

I built a new desktop/server for home use a few months ago and picked RAID5 based on the drives I had available and the case. Then I found out that RAID5 has some pretty serious problems with today's consumer drives with huge capacities. Thankfully my drive capacities are small enough that I'm not too worried. But, my next build will very likely be RAID1. Space is cheap, just mirror that s#$!.

Assuming the Wiz uses some third party cloud hosting, I highly doubt it is RAID5.

Quote: AcesAndEights

Space is cheap, just mirror that s#$!.

Assuming the Wiz uses some third party cloud hosting, I highly doubt it is RAID5.

... and do some type of monitoring, so that you know when the array degrades and it's time to slot a new drive.