Regardless of your political feelings, seeing Al Gore on stage was electifying. Funny too!

Gore spoke about emerging green tech and specifically about CO₂ and the fact that it’s invisible. It’s invisible to see and smell and invisible to the markets and market control since there aren’t any economic incentives to control it.

He talked about asking the right questions to solve these problems and related a story of an electronics company that, while trying to find a replacement to CFCs used in cleaning circuit boards, asked the question “why did they need to be cleaned in the first place?” and came up with the “no-clean” process.

What resonated most to me was his call to use information technology to “make the invisible visible.”  That’s not something I had thought of.  Feels like I should put some more thought into that.

ps.  He came across as very country, very Tennessee.  Even wore boots.  Totally worth playing hookie for a couple hours this afternoon!

Update: CNET has an excellent article on Gore’s talk and does a better job covering it than I did.

View’s awesome but weather’s been cloudy/raining so no cool pictures.  Takes some getting used to too.

To check this I ran two weeks’ worth of web server logs through awstats with the geoip plugin.  From this, I’ve learned the following:

1. I was wrong.  In this post I said that “connectivity to mainland China can often be congested and can induce a lot of latency.”  Round-trip times between San Jose and China are only 30ms more than they are to Amsterdam (195ms vs. 166ms).  I don’t see any of the congestion that I saw from either of the two hotels I stayed or at that I see from the China office.
2. We’re serving a large portion of the Asia-Pacific region out of our Beijing, China colo. Far more than I think any of us had thought and we’re doing it remarkably well (even Gomez says so)!  I’ve even received emails from folks in that region wondering why www.mozilla.com is so much quicker/closer than any of Mozilla’s other web properties.

Looking at those log hits through awstats and pushing those results into Numbers gets me this:

(The 5% USA appears to be a combination of errors in the allocation database (IP is registered to a US entity but traceroute shows otherwise) and a few US-based ISPs that, for whatever reason, keep hitting China.  Other is a lot of smaller hits to Asia-Pacific countries and some IP addresses that didn’t automatically resolve to a country.)

Since the color seperation might be hard to read, the raw numbers are here.

I’m generally very happy with these results. In the next few weeks we should be adding addons.mozilla.org and spinning up our geo-dns setup to start pushing releases.mozilla.org out of China too.  Stay tuned.

Even to the untrained eye it’s pretty obvious when I made the change!

As I mentioned before, if you inadvertantly find yourself hitting the China data center, please file a bug so we can investigate.

Otherwise, I’m hoping there are a lot of happy Chinese users!

In December I was back in China to install and deploy that data center and promised that “in the next several weeks we’ll be pulling up Mozilla websites in China.”

This past week[1] we did two important things for our Chinese users -

1. Integrated www.mozillaonline.com into our svn repo and pushed out its content to all static web clusters in Amsterdam, San Jose and China (it used to be hosted off a fileserver in the China office!). This site’s currently served out of San Jose and China.
2. Setup a mirror of releases.mozilla.org in China. This is currently known as releases.mozillaonline.com (until we get a more geo-aware DNS solution in place for releases.mozilla.org you’ll have to use that address).

This week we’ll make another important change.

During Tuesday night’s maintenance window (02/12 @ 8pm Pacific) I’ll be adding the China datacenter into www.mozilla.com’s global load balancing (which I talked about here). As I mentioned in my last post, we’ll also be watching the server logs to make sure GSLB isn’t sending users outside of China to China when they’d be better served out of San Jose or Amsterdam. If you find yourself in this category, please let me know (email or file a bug).

All of this effort has been to get content as close as possible to users. From my two trips to China I know getting to Mozilla content is often painfully slow, reminiscent of my pre-ISDN days. Updating Minefield nightlies every morning was often a couple hour process!

[1] I started this post with the intention of posting it January 28th but got side tracked when the host that holds all web content (before pushing out to the web servers) and all of the releases.mozilla.org mirror content lost it’s storage shelf, halting all of these plans. Because of this storm it took longer than usual to get resolved.

The algorithm first checks to see if there’s a match in the static maps and then falls back to proximity metrics. If that’s missing, it’ll round-robin through all the GSLB sites (effectively three - San Jose, Amsterdam and China - though technically it’s six and actually it’ll only round-robin through sites that have that service/site defined).

In practice, this round-robin fallback hasn’t been terribly user impacting mostly. You’d only ever hit the round-robin fallback if no one in your network (/16 in this case, but based on the source address of whatever name server you used) had ever accessed any of Mozilla’s web properties. And while that’s possible, the performance his is really minimal. But then up until now, we’ve only had two GSLB sites to send users to and San Jose and Amsterdam are well connected.

China changes things. Connectivity to mainland China can often be congested and can induce a lot of latency (which matters less for downloads than it does for interactive sites). If you’re in New York and fall through to round-robin, you very likely don’t want to end up taking 80ms to the west coast and another 300ms-400ms to China. This is a bad user experience.

If you don’t believe me, look at my trip times from San Jose to some Mozilla gear in China:

mrz@boris [~/] 15> ping mozcn01
PING mozcn01 (10.86.35.203) 56(84) bytes of data.
64 bytes from mozcn01 (10.86.35.203): icmp_seq=6 ttl=237 time=389 ms
64 bytes from mozcn01 (10.86.35.203): icmp_seq=9 ttl=237 time=439 ms

mrz@boris [~/] 17> ping releases.mozillaonline.com
PING releases-cn.glb.mozilla.com (59.151.50.34) 56(84) bytes of data.
64 bytes from 59.151.50.34: icmp_seq=0 ttl=231 time=206 ms
64 bytes from 59.151.50.34: icmp_seq=1 ttl=231 time=212 ms
64 bytes from 59.151.50.34: icmp_seq=2 ttl=231 time=205 ms

Since December I’ve been working on ways to exclude China from the fallback round-robin method while making sure our Chinese users don’t goto San Jose or Amsterdam.  The latter was easier - using MaxMind’s GeoLite database I’ve built a static proximity location map assigning all IP addresses in China to the China datacenter.  The former, excluding China from round-robin, took longer to figure out.

One of the GSLB methods the Netscalers have is a “weighted round-robin” and while it’s not documented, it turns out that weights apply even to the fallback round-robin method.  Weights range from 1 to 100.  So now I can do something like:

bind gslb vserver glb-mozcom -serviceName mozcom-80-sj -weight 100
bind gslb vserver glb-mozcom -serviceName mozcom-80-cn01
bind gslb vserver glb-mozcom -serviceName mozcom-80-cn02
bind gslb vserver glb-mozcom -serviceName mozcom-80-nl01 -weight 50
bind gslb vserver glb-mozcom -serviceName mozcom-80-nl02 -weight 50

which will send 200 requests  to either San Jose or Amsterdam before sending -1- request to China.

It’ll be interesting to see what the actual user impact of this is and if anyone not in mainland China ends up hitting China.  We’ll be testing this for a week, starting next Tuesday, with www.mozilla.com and looking at the web server logs to see where users are coming from.

I’d also be interested in feedback from you if you find yourself in China and you should be elsewhere or your find yourself in China and www.mozilla.com is quicker for you!

[1]GSLB works by looking up the IP address of the host that made a DNS request to somehost.glb.mozilla.com and determining which data center is closest.   This is a fairly common method to do global load balancing.

The actual start time may be up for debate. I started my timer from the moment the gear (finally) showed up at the datacenter, not the half hour it took to get there or the half hour Jia and I spent waiting for it once we were there.

I’m also not including the glacial time it took to get shipped out of the USA or out of Chinese customs, both of which would simply boggle the mind.

I should note that for most of that time, we had no less than four guys either helping or watching.

The gear we shipped from California arrived in a shipping crate but the datacenter had no way to get it out of the delivery van. Jia and I, along with a bunch of guys from the datacenter, uncrated everything while still in the van and manhandled the HP BladeSystem box out of the van and onto two carts. Once upstairs in the lobby, Jia and I had to unpack all the boxes and unpack anything that was in a plastic bag (which mean all the ethernet cables too).

Testing was easier than I thought it’d be - I plugged my laptop in and before I could do anything, Thunderbird had spewed a bunch of growl notifications all over my screen and had already grabbed a couple hours’ worth of email so obviously a significant amount was working.

In the next several weeks we’ll be pulling up Mozilla websites in China and spinning this site up.

On a personal note, the amount of planet-shrinking technology I have with me is awesome. My VoIP client works exceedingly well and Skype works no less than it does in the US. Both provide endless entertainment for my kids!

From my two trips to Beijing, China feels like a land of extremes. August was hot and humid. December is freezing. Both remind me why I moved away from Chicago

Here’s the dilemma - 50 power bricks converting AC to DC is a waste of heat and space (and my time managing Apple’s 6′ power cables) and makes me having to find a PDU with 50 outlets.

Can someone make me some electrical system  that takes one 110V/30A circuit and powers 25-50 Mac Minis over DC?

Nearly any “dynamic” web application we have is backed by MySQL and we have several “clusters” using MySQL’s replication . This gives a master read-write node and a read-only slave (and in some cases, multiple slaves) and all the apps need to know about this split. This works but has some nasty side effects - kernel upgrades require downtime, overloaded masters/slaves delay replication and impacts other databases on that pair.

And this doesn’t scale without end especially in instances were my gut says we could use more read-write heads to the same database or where we need a new slave right now but can’t take the downtime hit to rsync all of /var/lib/mysql (or whatever).

What I want is to be able to throw more machines into a cluster to increase I/O, CPU or number of client connections & queries I can handle without a lot of effort. On paper, MySQL Cluster looks like it’ll do that but I’m left with a bunch of questions:

1. What do I do if disk I/O is my contention? CPU?
2. If I need more capacity on my read-write heads, do I just add more SQL nodes? Do my apps just need to know of multiple heads?
3. How are simultaneous database writes (updates/inserts) handled? Will I run into locking issues?
4. How critical is the management node? If it stops running, what breaks?

I’m hoping someone out there’s been through these issues and can point me in the right directions. Ping me offline - mrz at mozlla .

We met Mi Jai after breakfast (and after a quick run to Starbucks) who escorted us over to Mozilla Online’s office. We quickly realized we’d need to take a trip to pick up ethernet cables and a few other items. Mi took us to what I can only describe as a 7 story Frys Electronics with hundreds of vendors all selling nearly the same wares and all vying for our attention. It’s probably not a place I would have done on my own.

The rest of the day was spent setting up the frontend router and wifi and what will become the Asterisks server and occasionally complaining about how hot it was.

A couple notes of interest:

1. Please smoke a lot. Mi told us over dinner that China produces 30% of the world tobacco but consumes about 70%.
2. Beds in China are much harder than anything else I’ve ever slept on except the floor. In fact, I could almost argue that my carpeted floor is softer than the bed I’m on.
3. People drive with seemingly disregard for pedestrian traffic. This is in sharp contrast to California where most folks yield to pedestrians.
4. I can manage okay with chopsticks but must be wearing some sign on my back - during lunch the waitress brought out silverware for only Justin and me.

Anyways, more photos are over at this Flickr stream.