Suggestion for speeding up the forums

I've just noticed that the way this forum renders pages when a user is logged in is much slower than when logged out. I think I know why. When logged out, the server just needs to determine the most recent thread in each forum (for the forum landing, wizardofvegas.com/forums) or overall (for the recent-threads list, wizardofvegas.com/recent). However, when logged-in, the user's view of the forum list or the recent-threads list is custom-rendered with links to the earliest unread page/post of each thread. That means for both the forum landing and the recent threads list, the server has to do a large query of the user's read/unread state data and custom-generate many URLs. It's on the order of 40 for the recent threads and 130 for the forum landing. However, all but one of those URLs are discarded when the user navigates into a thread.

Instead, these pages should be generated as if the user were logged out, and the earliest-unread state should only be queried when the user navigates to a particular thread.

In other words, the algorithm for a logged-in user is:
a) Small DB query to get recent threads;
b) Big DB query to get read/unread state for all those threads
c) Render page with user-custom URL for each thread
d) User clicks one and goes to earliest unread post in that thread.

Instead, it should be
a) Small DB query to get recent threads;
b) Render page with non-custom URL for each thread
c) User clicks one
d) Server queries read/unread state for that specific thread and redirects user to the earliest unread post in that thread.

If I'm right, that would cut down custom URL generation by at least 40x.

Food for thought.

I agree. After a read a thread, and then hit the "back" button to go back to the "recent threads" page, it takes FOREVER.

I am frittering my life away waiting for Wizard of Vegas to load!

Quote: teddys

I agree. After a read a thread, and then hit the "back" button to go back to the "recent threads" page, it takes FOREVER.

I am frittering my life away waiting for Wizard of Vegas to load!

This started for me a few months so.

Also, anytime I try to click "reply", "quote", or advance to the next page, it clicks me on one of the advertisement links for the online casinos. I absolutely hate that!!

Quote: teddys

I agree. After a read a thread, and then hit the "back" button to go back to the "recent threads" page, it takes FOREVER.

I am frittering my life away waiting for Wizard of Vegas to load!

Right-click > Open link in new tab
is your friend. :)

Quote: MathExtremist

Right-click > Open link in new tab
is your friend. :)

Ctrl+Click > ( Right-Click + "open link in new tab" )

Quote: RS

Ctrl+Click > ( Right-Click + "open link in new tab" )

But I can't do that while holding my whiskey. :)

Quote: Ibeatyouraces

This started for me a few months so.

Also, anytime I try to click "reply", "quote", or advance to the next page, it clicks me on one of the advertisement links for the online casinos. I absolutely hate that!!

Hate to be a stick in the mud, but,
I click a lot of stuff and nothing takes me to ads, well a couple of times I went to ads but I figured out that I hit the bottom for the ad instead of what I meant to hit. Did that make sense, cents?

Quote: TwoFeathersATL

Hate to be a stick in the mud, but,
I click a lot of stuff and nothing takes me to ads, well a couple of times I went to ads but I figured out that I hit the bottom for the ad instead of what I meant to hit. Did that make sense, cents?

These buttons are not even close to the links for the casinos. Doesn't happen all of the time, just 99.54%

Quote: Ibeatyouraces

These buttons are not even close to the links for the casinos. Doesn't happen all of the time, just 99.54%

+1

I rather have a "read first unread post" then the current system. It works rather well on 2+2.

Quote: MathExtremist

I've just noticed that the way this forum renders pages when a user is logged in is much slower than when logged out. I think I know why. When logged out, the server just needs to determine the most recent thread in each forum (for the forum landing, wizardofvegas.com/forums) or overall (for the recent-threads list, wizardofvegas.com/recent). However, when logged-in, the user's view of the forum list or the recent-threads list is custom-rendered with links to the earliest unread page/post of each thread. That means for both the forum landing and the recent threads list, the server has to do a large query of the user's read/unread state data and custom-generate many URLs. It's on the order of 40 for the recent threads and 130 for the forum landing. However, all but one of those URLs are discarded when the user navigates into a thread.

Instead, these pages should be generated as if the user were logged out, and the earliest-unread state should only be queried when the user navigates to a particular thread.

In other words, the algorithm for a logged-in user is:
a) Small DB query to get recent threads;
b) Big DB query to get read/unread state for all those threads
c) Render page with user-custom URL for each thread
d) User clicks one and goes to earliest unread post in that thread.

Instead, it should be
a) Small DB query to get recent threads;
b) Render page with non-custom URL for each thread
c) User clicks one
d) Server queries read/unread state for that specific thread and redirects user to the earliest unread post in that thread.

If I'm right, that would cut down custom URL generation by at least 40x.

Food for thought.

I think this is great advice, and I hope they use it. I know there's a major rebuild of the site in progress, but I know nothing else as to timelines, format, expectations. Perhaps the rebuild is already either considering this aspect or will render it moot.

The only way to navigate the forum is
to stay not logged in until you want to
post. It can take 30sec sometimes to
refresh. In that time I can look at
4 threads not logged in.

I haven't been noticing it being super slow until lately. Is the site reaching some breaking point? Or was a blissfully ignorant for a long time?

Quote: MathExtremist

Right-click > Open link in new tab
is your friend. :)

After you right-click on the link in Chrome, choose "Open in incognito window."

Must be your computer, super speedy at this end.

Quote: neverquitwhenup

Must be your computer, super speedy at this end.

The more threads you read while logged in, the more the index page slows down.

It's been 15-30 seconds to load a page for me at times today. Other times, pretty quick.


ZCore13

Someone else mentioned this, I think it might be related to the number of threads and users you have blocked.

I am betting the database just needs to be cleaned up. In the world of databases, the little queries we are doing here are insignificant. I suspect the database just needs to be optimized.

But having said that, MathExtremist sounds like he has a better way of approaching the queries.
I have no one blocked, and subscribe/block zero threads.
It takes a LONG time for my screen to load........ :-(

I have no threads blocked and no users blocked, and I've had several incidences of slow loading, here on the computer, and especially on my Samsung (Note3, 4G service, Android) where it sat over a minute, then refused to load about 6 times out of about 10, so I gave up and moved to the computer.

I don't think it's the blocking part.

Quote: Dalex64

Someone else mentioned this, I think it might be related to the number of threads and users you have blocked.

I have no people and only 2 threads blocked. It can't be that.


ZCore13

I'm not a PC database expert but have also noticed it can go slower than the old site; previously I had never seen 10 second response times. I know from my mainframe days that sometimes one had to "re-organise" the data or rebuild the index. Best of luck finding the problem!

I bet the database could be reduced a fair amount by purging any user that hasn't posted in over a year. If it's been that long I don't see a need to keep it a tive. Casinos archive or delete card holders after a certain time. Many other businesses do to.


ZCore13

Quote: Zcore13

I bet the database could be reduced a fair amount by purging any user that hasn't posted in over a year. If it's been that long I don't see a need to keep it a tive. Casinos archive or delete card holders after a certain time. Many other businesses do to.


ZCore13

I hope they don't do this. We have numerous former members who still log in but can't post (for good reason), and we need to keep the data trail.

Quote: MathExtremist

In other words, the algorithm for a logged-in user is:
a) Small DB query to get recent threads;
b) Big DB query to get read/unread state for all those threads
c) Render page with user-custom URL for each thread
d) User clicks one and goes to earliest unread post in that thread.

Instead, it should be
a) Small DB query to get recent threads;
b) Render page with non-custom URL for each thread
c) User clicks one
d) Server queries read/unread state for that specific thread and redirects user to the earliest unread post in that thread.

If I'm right, that would cut down custom URL generation by at least 40x.

Food for thought.

While I had no part in the redesign of the forum, I feel the need to chime in about the database queries.

In the old version, unread threads appeared in bold on the main page. The query logic above would not allow for that, since the server wouldn't know to generate bold links without knowing if the thread has unread posts for the logged-in user. However, I don't think the new version bolds the links for threads that have unread posts anyway, so that may be a moot point.

However, your query would disregard whether the threads being returned have been blocked by the user. If the 10 most recent threads were all blocked by the user, showing them would defeat the purpose of having blocked them. But you also can't do it in two steps: (1) get the 10 latest threads, and (2) skip showing those which the user has blocked, because then it might not show any threads.

When I built the old version, I used a single query with multiple joins and subqueries to retrieve a compact list of the most recent threads where neither the thread itself nor its parent topic were blocked by the user, and a flag for each such thread indicating whether or not it has unread posts for the logged-in user. (A different, simpler query is used if nobody is logged in.)

Also, on the main forum page, I chose not to generate customized "latest post" links for each of the 100+ topics, but to instead just store the HTML for the actual (global) most recent post directly in the database and show that instead, at the expense of occasionally linking to a post which is either in a thread that was blocked by the user, or was written by a member that the user has blocked.

I also had the luxury of building the forum from scratch and being able to take as long as I needed to test everything (that I could think of) myself, so most of the bugs were worked out prior to launch. The new crew does not/did not have this luxury; they are building upon my code, and have been figuring things out as they go without my help. Maintaining code that someone else wrote is rarely easy, so considering the circumstances I think they have done very well.

Quote: beachbumbabs

I hope they don't do this. We have numerous former members who still log in but can't post (for good reason), and we need to keep the data trail.

To me that history is not that important. There is no such thing as banning someone if they decide they want to come back. I think the best thing to do is base everything off each individual post. If it doesn't meet the standards, it goes. If the account doesn't meet the standard of being a member it goes too.

If the person comes back under a new name and acts like an adult, then who cares. If they post something that doesn't meet the standards it goes again.

Worrying about the past doesn't help the now much. Especially on the internet.


ZCore13

Quote: JB

While I had no part in the redesign of the forum, I feel the need to chime in about the database queries.

In the old version, unread threads appeared in bold on the main page. The query logic above would not allow for that, since the server wouldn't know to generate bold links without knowing if the thread has unread posts for the logged-in user. However, I don't think the new version bolds the links for threads that have unread posts anyway, so that may be a moot point.

However, your query would disregard whether the threads being returned have been blocked by the user. If the 10 most recent threads were all blocked by the user, showing them would defeat the purpose of having blocked them. But you also can't do it in two steps: (1) get the 10 latest threads, and (2) skip showing those which the user has blocked, because then it might not show any threads.

When I built the old version, I used a single query with multiple joins and subqueries to retrieve a compact list of the most recent threads where neither the thread itself nor its parent topic were blocked by the user, and a flag for each such thread indicating whether or not it has unread posts for the logged-in user. (A different, simpler query is used if nobody is logged in.)

Also, on the main forum page, I chose not to generate customized "latest post" links for each of the 100+ topics, but to instead just store the HTML for the actual (global) most recent post directly in the database and show that instead, at the expense of occasionally linking to a post which is either in a thread that was blocked by the user, or was written by a member that the user has blocked.

I also had the luxury of building the forum from scratch and being able to take as long as I needed to test everything (that I could think of) myself, so most of the bugs were worked out prior to launch. The new crew does not/did not have this luxury; they are building upon my code, and have been figuring things out as they go without my help. Maintaining code that someone else wrote is rarely easy, so considering the circumstances I think they have done very well.

I think they should have just paid a programmer to find a way to save and convert every post and topic here into MyBb or phpBB. Those formats have so many features and addons and they are 100% free. Tens of thousands of dollars worth of software and extra features. Features that would blow people away here.


ZCore13

Quote: Zcore13

To me that history is not that important. There is no such thing as banning someone if they decide they want to come back.

Having administered elsewhere, the data is very useful, and is often extremely helpful at quickly curtailing problems that keep popping up.

... and I don't even know what they're storing and showing.


In my opinion, purging off 90% of the user database won't help. The problem would seem to be combining a user's thread-reading history with the current list of threads, to make a useful dynamically generated page.


Edit: Maybe a per-user option for "fast thread index", where users have the option to select a thread list that uses the logged-out, "fast" algorithm.