Setting up the new Server: Murphy’s Fifth
Hi everyone, I’d like to apologize for all the problems you may have had with SR in the last month. I have been working on building a new co-located server cluster for SR to handle the traffic, and while that was going fine, I planned on installing it and then slowly migrating SR from the old servers to the new cluster as time permitted (i.e. in an orderly way). The old SR main server has been running for 3-4 years continuously, so I wasn’t too worried that everything would be fine.
Of course, Murphy had something to say about that, so while the building was taking place, the old SR main server hard drive crashed (in August). I was able to get all the data off, but the new server wasn’t fully put together yet. So SR then automatically switched over to the backup servers, which generally are OK for a week or so, but then they started to crash under the load as well. This was responsible for all of those `can’t connect’ database errors we received emails about. Thanks for the notes — it really helps to know that people use SR and to hear from people! 
So the new server did not come online at the new co-location provider until around Sept 15, which left us with about a month of flaky-downtime that we could not really do anything about. And its funny, all the systems I put in place over the last 2 years to prevent exactly this problem from occurring (prompted by an explosion at our old provider several years ago) only barely held up the growth in traffic.
You know, I idly wonder if while Murphy is playing his game, he also listens to `Walter Murphy: A Fifth of Beethoven’… I certainly would, dancing around, poking at cables, kicking peoples’ disks. Whoops! I didn’t mean to knock over that water into your server!
Server taking slightly longer than expected :P
So, it was completely my goal to have the new server in place by mid june, but let me tell you, receiving lots of dead parts totally put a cramp in that goal :P That said, the new server is really fast and almost ready to go. I mean… *really fast*. Well, I look forward to sending it in next week!
Updates!
A few weeks ago, we added the ability to upload entire galleries of campus photos! This is a big improvement over the single image upload that we had for the past 9 years or so :P Also, we rebuilt all the pie chart graphs and rebuilt the SAT/ACT graphs!
AND we added graphs of distributions of ratings to the university breakdown page. So now, lets say that you click on “Educational Quality”, it will break down to show you a distribution of the number of ratings from A-F for educational quality at a particular school — this can be broken down even further by major!
The next update is updates to all of the tuitions for the schools!
StudentsReview is 9 Years old!
Well, this happened in March/April, but now that the blog system works, I can finally write about it! To think that 9 years ago to this period, I started SR along with assistance of 4 friends and the help of my parents immediately after I graduated from college… It brings back memories. We’d sit in my inexpensive and unfurnished apartment, above this lunatical old woman, thinking of ideas for SR & how it was going to improve the whole college process, while she pounded the ceiling with her broom and came up to accuse of doing all sorts of rather… ah, unsavory things. What an imagination she had… (use your imagination). Read the rest of this entry »
Major Upgrades Coming
Lately many of you have noticed that SR has slowed down a lot, and that lots of graphs have been falling out of date. Thank you for letting us know! The reason this has been happening is because the traffic to SR has grown way past what our 3 servers can handle. We made some improvements this academic year that multiplied the overall speed by 10, but the traffic (at least on peak days) quickly saturated the newfound capacity. To overcome this, we employed 5 levels of disk caching (built up over the past 10 yrs), which both fill up the disks and create a huge amount of disk-read-load. Without getting boring, suffice it to say that every time an update to the site is made, basically everything has to be rewritten, re-analyzed, copied, driving the number of users that we can handle from 100,000 down to like 10. Not 10,000… just ten. It gets that slow. It seems bad, but that’s the tradeoff we made to be able to handle everyone within our current setup & budget. As long as things don’t change every 5 minutes, we gain a whole lot of speed.
Unf, the time that is an ok has passed. To support the systems, we’ve had to drop a lot of things over the years that made SR awesome, like dynamic PDFs, dynamic analysis, etc. Since those were important things, we will soon be moving from 3 dedicated, geo-located servers to co-located high bandwidth servers of our own design. The new servers are… for lack of a better word… awesome. I tried to understand and characterize how awesome they are, but they are beyond my comprehension (even though I designed them). Each one is more than 10x faster than the sum of our original 3, and comes with satellite distribution nodes… I’ll upload pictures when they are finished being constructed!
Members Section Restored
Thank you to everyone who let us know that our members section was down and that you couldn’t log in. Especially those who helped us debug it and kept retrying over and over. It turns out that when we added the blogging system first, it was vulnerable to spam, and after a million spam-users injected into the WordPress tables, it brought down the entire database that the members section was hosted on. This problem got worse over time, as the members table was locked from new insertions, it got internally out of sync, so we basically had to replay all of our log files to restore the members table to operation. It was also hard to track down because we had setup a backup-failover circumstance, so when the members table wouldn’t repond, the members section requested user data from an old table, making (me) personally think it was working better than it actually was, since well, my member profile was in the old backup table. But it’s working again now, whew, that was hard!!!!
Video Upload!
After many years of not having videos, besides having more of everything else, Video upload is almost complete! We’ve always wanted to have videos (from Day 1, back in 2000), and have sequestered people to make them in the past, but the problem is that video delivery costs a lot in bandwidth-data fees, which gets very expensive, very very quickly, so we just couldn’t afford putting them up in the past. That’s changed a lot in the past year or two with Youtube/etc, driving down bandwidth costs, so now that we can actually afford to host them, we are!
New Blogging System!
Welcome to the new StudentsReview blogging system! After many months (almost a year) of unnecessarily hard work, I have finally been able to integrate Wordpress MU into SR, so that everyone can create their own blog about their school experiences, application headaches, school visit observations, or just what it is like being you! In a few days I’ll put up spam filtering, and then open this up to everyone to register and post! How exciting!
And for those of you who say that Wordpress MU (multiuser) is easy to integrate, you’re wrong. 
I Super Suck
At Life. So I built in this whole blogging system, made it work, then put in a few posts of my own — about the updates to SR, all proud of myself, but forgot to turn on the spam filter. Then today, while deleting the spam-posts (200 fake blogs appeared overnight), I managed to delete the SR blog and all the posts I worked so hard to write. I wonder what kind of mistake I’ll make tomorrow