Posted on Thursday, May 14th, 2009 | Bookmark on del.icio.us

The Great GoogleLapse

by Craig Labovitz

Web sites go down. Circuits fail. Network engineers goof router configs. And few of these outages ever make the nightly news…

But if you happen to be Google and your content constitutes up to 5% of all Internet traffic, people notice.  Network engineers around the world frantically email traceroutes to mailing lists. IRC channels fill with speculation (“definitely was a DDoS attack”, “no, a worm”, “it was ISP xxx’s  fault!”). And end users Twitter (a lot).

So what does it look like when 5% of the Internet disappears on an otherwise uneventful Thursday morning? The below graph shows average traffic across 10 tier1/2 ISPs in North America from Google’s network (ASN 15169). Outage began roughly at 10:15am and lasted through 12:15pm EDT.

Looking at the data, most large transit providers appear to have been impacted (e.g., Level3, AT&T, etc.). Other providers (e.g. large consumer DSL / Cable) showed no drop in traffic from/to Google.

Looking at BGP (below snapshot is from Arbor’s Routeviews Servers) we see a lot of churn in Google’s BGP routes around the outage timeframe — one prefix I choose at random flaps across half a dozen providers before getting withdrawn.

In a recent official company blog post, Google blamed some combination of airplanes and BGP for the outage.

Reblog this post [with Zemanta]

Popularity: 6% [?]

52 Responses | Add your own



Comment Post by: Stat Shot: Google’s Circumnavigation Edition — May 14th, 2009 @ 5:28 pm EST  Reply

[...] Networks, which provides telecommunications gear and network security information, noted on its blog that Google comprises up to 5 percent of the web’s traffic, and when that traffic disappeared [...]

Comment Post by: Fenixnordic Group » Blog Archive » When Google Goes Down, It Falls Hard — May 14th, 2009 @ 6:29 pm EST  Reply

[...] via » The Great GoogleLapse · Security to the Core | Arbor Networks Security. [...]

Comment Post by: Whither Google, so goes the Web | AERS - Advanced Ecommerce Research Systems — May 14th, 2009 @ 6:56 pm EST  Reply

[...] even when it’s just a simple routing error, as it was this morning.  This diagram from Arbor Networks, whose servers are positioned at key NAPs and ISPs globally, portrays the [...]

Comment Post by: When Google fails, the Internet fails » VentureBeat — May 14th, 2009 @ 8:46 pm EST  Reply

[...] Deal. Above, you can see how much data was transferred by 10 different Internet service providers monitored by security company Arbor Networks. Below, you can see traffic to sites run by MerchantCircle, which manages the Internet presence for [...]

Comment Post by: Matt’s Blog » Blog Archive » Google Routing Fail — May 14th, 2009 @ 9:11 pm EST  Reply

[...] all over the place, which means that they have some pretty good insight into things. As they posted about today, Google apparently fudged a route and it propagated out, routing most of their traffic through some [...]

Comment Post by: Searching for Google « Rich’s Random Walks — May 14th, 2009 @ 11:15 pm EST  Reply

[...] SANS links seem to be working now.   There is also a very interesting, somewhat tongue-in-cheek blog post from Arbor Networks, showing the impact of the Google problem on Internet traffic. Possibly related [...]

Comment Post by: Google we have a problem — May 14th, 2009 @ 11:34 pm EST  Reply

[...] Una fotografia del calo del traffico Your web sites go down. Circuits fail. Network engineers goof router configs. And mostly you don’t worry about making the nightly news. But if you happen to be Google and your content constitutes up to 5% of all Internet traffic, people notice.  Network engineers around the world email traceroutes to each other. IRC channels fill with speculation (”definitely was a DDoS attack”, “no, a worm”, “it was ISP xxx’s  fault!”). And end users Twitter (a lot). [...]

Comment Post by: 评论:Google 打个喷嚏 整个 Internet 都感冒 « 每日IT新闻,最新IT资讯,聚合多站点消息,保证你与世界同步 — May 15th, 2009 @ 5:00 am EST  Reply

[...] 新闻来源:VentureBeat Google 的全球访问故障已经告一段落,经查是 Google 错误地将部分流量转到了亚洲地区导致阻塞,虽然尘埃落定,然而从这件事却可以发现 Google 对全球互联网的影响,Arbor Networks 的流量统计分析了故障期间10个不同 ISP 的流量数据,下面这张流量图很明显地说明了问题。 [...]

Comment Post by: ODWGOOG — May 15th, 2009 @ 5:41 am EST  Reply

Whether we like it or not,Google is almost synonymous with Internet. And this behavior is bad, bad for all of us. For more information check our blog, too.

Comment Post by: Cuando Google se cae, se cae bien. Impresionante | Pateando Piedras — May 15th, 2009 @ 9:44 am EST  Reply

[...] gráfica muestra el descenso de tráfico en la red cuando los servicios de Google se caen. Esto demuestra lo enorme que es Google, le daría [...]

Comment Post by: Bill Hartzer — May 15th, 2009 @ 10:01 am EST  Reply

Interesting data and certainly that’s an interesting chart. As people were reporting that Google was down, luckily I was on one of those providers that didn’t see a Google outage. I didn’t have any issues at all.

Comment Post by: Repubblica.it - Blog - Scene Digitali » Blog Archive » Google-botto. E due in pochi mesi... e la rete ne subisce le conseguenze — May 15th, 2009 @ 11:31 am EST  Reply

[...] la più accreditata è quella che indica in due ore circa la fase di arresto: come documentato da un centro per la sicurezza della rete.  Del resto l’azienda si è assunta la responsabilità del disservizio,  spiegando che circa [...]

Comment Post by: Andrew P. — May 15th, 2009 @ 12:21 pm EST  Reply

It’s the best argument against Web 2.0 that we’ve seen so far. I’ve always believed and continue to believe, that individuals and enterprises need to own the software that is critical to their activities and have it installed on local machines that can continue to function without the Internet.

Comment Post by: The great Google collapse | Digital Prank — May 15th, 2009 @ 12:25 pm EST  Reply

[...] at 12:48am Australian time and the slowdown peaked around morning in US, mid-afternoon in Europe. Arbor Networks which monitors the web usage said that traffic from Google servers to major North American Internet [...]

Comment Post by: Terry Slattery — May 15th, 2009 @ 12:54 pm EST  Reply

It would be interesting to know how Google determined that they had goofed and how they determined that it was a routing mistake. I think a lot of us could learn from it.

Comment Post by: jeremy c — May 15th, 2009 @ 2:02 pm EST  Reply

Ha! AT&T has had an outage here for the past 2 days so I never noticed google being down!

Comment Post by: Google Cloud - Jak Łatwo Zawisnąć W Chmurach | SEO Blog — May 15th, 2009 @ 2:25 pm EST  Reply

[...] ale jak pokazuje wykres jednej z amerykańkich firm zajmujących się bezpieczeństwem w sieci (Arbor Networks), niewiele brakowało do zastoju w ruchu pośród największych firm przesyłowych w [...]

Comment Post by: Charles Bronson — May 15th, 2009 @ 2:35 pm EST  Reply

Can you please clarify the description of the traffic represented by your graph? I want to make sure I understand what it is relating.

Is the graph representing *all* of the average traffic across 10 tier 1 and 2 ISP’s or is it *only* the average of traffic originating from ASN 15169 as seen across 10 tier 1 and 2 ISP’s?

Thanks

Comment Post by: The Great Google Depression of 2009 Happened Thursday, May 14, at 7:48am PST | Rev2.org — May 15th, 2009 @ 4:52 pm EST  Reply

[...] graph from Arbor Networks shows the down time.  See that big post-hole?  That’s Google falling off the Internet for a [...]

Comment Post by: Craig Labovitz — May 15th, 2009 @ 6:54 pm EST  Reply

The latter — average only includes traffic originating or transiting AS15169.

Comment Post by: Ike’s World – Weblog Edition — May 15th, 2009 @ 10:48 pm EST  Reply

[...] By ichristo Google outage takes out 5% of the internet traffic. http://asert.arbornetworks.com/2009/05/the-great-googlelapse/ [...]

Comment Post by: When Google Goes Down, So Does 5 Percent Of The Internet? « Multan POST — May 16th, 2009 @ 12:39 am EST  Reply

[...] the statistic offered by Arbor Networks’ Craig Labovitz, who in a Thursday blog post and Web traffic graph noted that between 10:15 a.m. and 12:15 p.m. Eastern Daylight Time, Thursday, [...]

Comment Post by: Finance Geek » Google’s Outage Caused 5% Internet Traffic Decline — May 16th, 2009 @ 6:15 am EST  Reply

[...] Networks chief scientist Craig Labovitz observed in a blog post that failures happen. “[B]ut if you happen to be Google (NSDQ: GOOG) and your content [...]

Comment Post by: 鱼儿在线 » 评论:Google 打个喷嚏 整个 Internet 都感冒 — May 16th, 2009 @ 9:04 am EST  Reply

[...] Google 的全球访问故障已经告一段落,经查是 Google 错误地将部分流量转到了亚洲地区导致阻塞,虽然尘埃落定,然而从这件事却可以发现 Google 对全球互联网的影响,Arbor Networks 的流量统计分析了故障期间10个不同 ISP 的流量数据,下面这张流量图很明显地说明了问题。 [...]

Comment Post by: TheTradingReport » Blog Archive » links for 2009-05-16 — May 16th, 2009 @ 9:26 am EST  Reply

[...] Craig Labovitz: The Great GoogleLapse [...]

Comment Post by: TheTradingReport » Blog Archive » Google’s Outage Caused 5% Internet Traffic Decline — May 16th, 2009 @ 9:27 am EST  Reply

[...] Networks chief scientist Craig Labovitz observed in a blog post that failures happen. “[B]ut if you happen to be Google (NSDQ: GOOG) and your content [...]

Comment Post by: 墨尔本 — May 16th, 2009 @ 9:27 am EST  Reply

I’m glad it comes back to normal!

Comment Post by: La Gran Caída de Google -- Reikuaa.com | Tu blog — May 16th, 2009 @ 10:40 am EST  Reply

[...] Craig Labovitz Traducción: Raúl Batista Fuente: Arbor Networks « Cómo salvar un móvil que ha caído al [...]

Comment Post by: Every Cloud has a silver lining? | New Beltane — May 16th, 2009 @ 7:00 pm EST  Reply

[...] was quite significant, with @ 5% of traffic disappearing between 10.15 and 12.15 EDT according to Arbor Networks Security. According to their investigation, the big boys were effected significantly. If wasn’t [...]

Comment Post by: Wenn Google in Nordamerika 2 Stunden nicht verfügbar ist « Der Jupp — May 17th, 2009 @ 2:45 am EST  Reply

[...] from Google’s network (ASN 15169). Outage began roughly at 10:15am and lasted through 12:15pm EDT.Arbor Networks: Security to the Core, May [...]

Comment Post by: ZitZot » Google’s outage: internet traffic went down by 5% — May 17th, 2009 @ 2:00 pm EST  Reply

[...] the assertion of Arbor Networks’ Craig Labovitz, who in a Thursday blog post and Web traffic graph noted that when Google has an hour and a half outage this week, Internet [...]

Comment Post by: A.H.Banen — May 18th, 2009 @ 3:12 am EST  Reply

According to the Google Apps Status Dashboard – http://www.google.com/appsstatus# – there was no disruption of service…

Comment Post by: Google jumitti verkon : Viimeisellä Rannalla — May 18th, 2009 @ 4:13 am EST  Reply

[...] voi olla valtava vaikutus koko infrastruktuurin toimintaan, sillä Googlen järjestelmien ongelmat 14.5. hidastivat verkkoliikennettä merkittävästi kautta maailman. Useat verkkopalvelut [...]

Comment Post by: Google’s Downtime Affected 5% of the Internet — May 18th, 2009 @ 6:32 am EST  Reply

[...] doubt about that.  And one of the most interesting observations/reactions was perhaps made by the Arbor Networks Security Blog which stated that  5% of overall Internet traffic was affected by the Google [...]

Comment Post by: Google’s Downtime Affected 5% of the Internet | Mujtaba-K.com - Daily Latest Technology News Buzz — May 18th, 2009 @ 11:49 am EST  Reply

[...] doubt about that.  And one of the most interesting observations/reactions was perhaps made by the Arbor Networks Security Blog which stated that  5% of overall Internet traffic was affected by the Google [...]

Comment Post by: A lesson from Google: Don’t get stuck in a cloud | Poc Network - Blog — May 18th, 2009 @ 1:01 pm EST  Reply

[...] important to today’s Internet? Well, a quick look at this graph from the Web security company Arbor Networks shows a canyon-sized hole in North American Internet traffic during the G-outage. With a wide [...]

Comment Post by: More on the Great Google Crash of 2009! — Brian Hancock — May 18th, 2009 @ 11:57 pm EST  Reply

[...] The Great GoogleLapse [...]

Comment Post by: Rational Survivability » Google Gaffe - The Cloud Needs a Snuggie…Or a Wedgie — May 19th, 2009 @ 6:01 pm EST  Reply

[...] By now you’ve undoubtedly heard that Google had a little operational hiccup.  I particularly enjoyed Craig Labovitz’s (arbor) account of “The Great GoogleLapse” [...]

Comment Post by: Lennie — May 21st, 2009 @ 4:55 am EST  Reply

@Andrew P.

It’s not an argument against Web 2.0 (which is mostly just a certain kind of web-application/-page), it is however an argument against cloud-computing, utility-computing, SAAS, etc.

You could do Web 2.0 on your intranet as well.

An even better argument against cloud-computing, etc. is Microsoft Azure, which was down for 23 hours if I remember correctly.

Comment Post by: El día que Google falló « ESNOG — May 21st, 2009 @ 5:16 pm EST  Reply

[...] manera directa en el funcionamiento de Internet tal y como se refleja en el análisis realizado por ArborNetworks donde se aprecia una caída de hasta 15Gbps de tráfico en el tráfico acumulado de unos 10 [...]

Comment Post by: Online Outages, Outrage and Ordeals | A Second Opinion — May 22nd, 2009 @ 10:52 am EST  Reply

[...] But to people who rely on Google for their e-mail, their calendars and their documents, seeing them suddenly drop offline must have felt as if somebody had carved a chunk out of the Internet. [...]

Comment Post by: Kai — May 24th, 2009 @ 10:50 am EST  Reply

Its good also to compare to any other web properties when it occurs.
“we see a lot of churn in Google’s BGP routes around the outage timeframe”

Comment Post by: .commEurope » Crolla Google, crolla la Rete — May 24th, 2009 @ 3:45 pm EST  Reply

[...] la Rete L’incidente di percorso occorso a Google qualche giorno fa, consistente nell’irraggiungibilità di un paio d’ore dei suoi servizi, ha spinto molti esperti di tecnologia a rendersi conto di ciò che i loro colleghi che si occupano [...]

Comment Post by: Joe Wilcox » Antitrust Primer: Google and Microsoft — May 24th, 2009 @ 11:00 pm EST  Reply

[...] over Internet commerce extends much farther. Last week’s four-hour Google outage affected as much as 5 percent of Internet traffic. Google standards influence search and advertising, and much more, not just on the desktop but [...]

Comment Post by: Bruno Kerouanton » Le SSTIC, et autres sujets de recherche — May 25th, 2009 @ 7:20 am EST  Reply

[...] Je suis certain qu’il y a un énorme besoin de faire des travaux de recherche sur le domaine. Certes, c’est un tantinet moins technique que le piratage des signaux TNT, mais c’est intéressant car les décideurs (bref, ceux qui ont les sous…) n’en sont pas encore conscients (des soucis) et on pourrait bien aller droit à la catastrophe si par malheur tout passe par Google et que ce dernier plante, par exemple… [...]

Comment Post by: Antitrust Primer: Google and Microsoft — October 9th, 2009 @ 4:46 pm EST  Reply

[...] over Internet commerce extends much farther. Last week’s four-hour Google outage affected as much as 5 percent of Internet traffic. Google standards influence search and advertising, and much more, not just on the desktop but [...]

Comment Post by: aimClear’s 2009 Daily Training Link Library » aimClear Search Marketing Blog — January 1st, 2010 @ 6:32 pm EST  Reply

[...] The Great GoogleLapse, asert.arbornetworks.com [...]

Comment Post by: Reported cloud outages for Amazon, Google, Microsoft and Salesforce.com in 2008 and 2009 « Muon Cloud — January 31st, 2010 @ 11:51 am EST  Reply

[...] arbornenetworks [...]

Comment Post by: Google Downtime,First In Over 3 years — August 11th, 2010 @ 10:06 pm EST  Reply

[...] 8 min of downtime. And one of the most interesting observations/reactions was perhaps made by the Arbor Networks Security Blog which stated that  5% of overall Internet traffic was affected by the Google downtime, this is [...]

Comment Post by: Antitrust Primer: Google and Microsoft « 5 minutes with Joe — November 13th, 2010 @ 9:59 pm EST  Reply

[...] over Internet commerce extends much farther. Last week’s four-hour Google outage affected as much as 5 percent of Internet traffic. Google standards influence search and advertising, and much more, not just on the desktop but [...]

Comment Post by: Phantom Link Cloaker — September 5th, 2011 @ 11:27 pm EST  Reply

affiliate link cloaker…

The Great GoogleLapse | Security to the Core | Arbor Networks Security…

Leave a Comment