Well, this happened in March/April, but now that the blog system works, I can finally write about it! To think that 9 years ago to this period, I started SR along with assistance of 4 friends and the help of my parents immediately after I graduated from college… It brings back memories. We’d sit in my inexpensive and unfurnished apartment, above this lunatical old woman, thinking of ideas for SR & how it was going to improve the whole college process, while she pounded the ceiling with her broom and came up to accuse of doing all sorts of rather… ah, unsavory things. What an imagination she had… (use your imagination).
Lately many of you have noticed that SR has slowed down a lot, and that lots of graphs have been falling out of date. Thank you for letting us know! The reason this has been happening is because the traffic to SR has grown way past what our 3 servers can handle. We made some improvements this academic year that multiplied the overall speed by 10, but the traffic (at least on peak days) quickly saturated the newfound capacity. To overcome this, we employed 5 levels of disk caching (built up over the past 10 yrs), which both fill up the disks and create a huge amount of disk-read-load. Without getting boring, suffice it to say that every time an update to the site is made, basically everything has to be rewritten, re-analyzed, copied, driving the number of users that we can handle from 100,000 down to like 10. Not 10,000… just ten. It gets that slow. It seems bad, but that’s the tradeoff we made to be able to handle everyone within our current setup & budget. As long as things don’t change every 5 minutes, we gain a whole lot of speed.
Unf, the time that is an ok has passed. To support the systems, we’ve had to drop a lot of things over the years that made SR awesome, like dynamic PDFs, dynamic analysis, etc. Since those were important things, we will soon be moving from 3 dedicated, geo-located servers to co-located high bandwidth servers of our own design. The new servers are… for lack of a better word… awesome. I tried to understand and characterize how awesome they are, but they are beyond my comprehension (even though I designed them). Each one is more than 10x faster than the sum of our original 3, and comes with satellite distribution nodes… I’ll upload pictures when they are finished being constructed!