After all the chips fell here is what we can best decipher happened to the internet yesterday. Long and short – It outgrew design expectations and broke in certain areas.
The details are this: major ISP’s routers ran out of space for the routes they needed to keep track of. The limits on some routers was 512K routes. Today internet traffic exceeded that number and thus the routers failed. This took down a lot of the northeastern portion of the United States. ISP’s such as AT&T, Time Warner, Verizon and Comcast were affected by the outage. A lay person would say, “Go buy a new router”. Logical – yes, but cheap – no. A card, not even the whole router but just a card for routing in the router can cost upwards of $76k for one of these major routers. It is not cheap nor easy to go and replace such hardware.
Here are a couple of articles on the subject:
This Arstechnica user saw it coming last month: http://arstechnica.com/civis/viewtopic.php?f=10&t=1248917
Here is an article on what happened today: http://www.reddit.com/r/sysadmin/comments/2dcol3/the_internet_hit_512k_bgp_routes_today_causing/
And for the visual learners in the classroom, here is a video explaining the routing behind it all: https://www.youtube.com/watch?v=R1mGglKlEBY
Leave a Reply