eelco.com

I was browsing through my video sources yesterday and i found something i totally forgot to put online: our corvette testdrive! Enjoy!

The movie says it all!

Update: Arghhh.. why did they pick this frame as startframe?!!!

I should actually be ashamed.. a long long time ago I designed and created my own e-mail system called DBmail. The general idea was that e-mail and e-mail provisioning is structured data and structured data should be stored in a database. This way maintenance, searching, scalability, backup and data consistency should be better covered than using a filesystem for storage (all depending on the database of course, but DBmail supports many). This, combined with the fact that I have easy access to my own online servers should have actually resulted in me using DBmail for all my e-mail needs. Well, since a couple of days, this is no longer the case.

A few months ago I talked my dad into transferring his e-mail (and domain) to Google by subscribing to Google Apps. He seems very happy since then (i’m measuring by the number of help requests me and my brothers are receiving). Since i was doing my own e-mail i was also doing my own spamfighting which started to take a lot of my time. FYI, my domain eelco.com is receiving about 20.000 unwanted messages per day. My wife’s domain marloes.info does about the same so i need to block about 40.000 messages per day just for my wife and me. Then i’m also hosting e-mail for a couple of domains for friends, so add another 20.000 to that. If i’m not blocking those messages they (the friends and family) start telling me i’m letting to much spam through. An effective remedy is called greylisting. Greylisting uses a feature of the SMTP protocol to request a resend when a message is being delivered the first time from an unique sender. If the sender is using a regular fully SMTP compliant mailserver the mail will be resend. If the sender is a spammer he probably uses a bulk sender which does not fully comply with SMTP and therefore will not send the message again. So the spam message is blocked. Problem with this approach is that a lot of messages that are sent for the first time will have a arbitrary delay.. which kind of sucks when you’re waiting for a certain subscription message to come through.

The spamfighting is actually the biggest reason i switched to Google Apps for my e-mail. Google uses it’s uge Gmail userbase to identify spam; every time you click the report spam button the Gmail system learns about your spam message and prevents it from being delivered in the future, also for other Gmail users. This works great, i’m getting even less spam then on my own server.

Another really nice feature of the Google mail platform is the search feature. It’s extremely quick and uses Google search technology to index your e-mail messages. So with this feature in mind i wanted use import my complete digital life into the Google mail datbase. Google has a nice option which uses imap to import from other mail accounts. The problem is my e-mail dates back to 1991 (still got all of it in backup) and i really wanted to be able to import that as well. First i tried a dirty hack by ‘resending’ my old e-mail to an intermediate account which i could then import using Google imap import. That didn’t work and quite a few people received really strange bounces, sorry about that folks

Then i discovered the Google E-mail API method for importing e-mail. This is an XML based system which accept standard rfc822 based message in an XML envelope accompanied by a one or more labels for inclusion into the Google e-mail database. Since a lot of my old e-mail is in the Cyrus DB or Maildir format i needed to create a little script that recurses through those storage formats, identify e-mail and send them through the API into the Google db. I wanted to fresh up (actually re-learn) my Python skills so i did the whole thing in a Python script.

You can download the script (GoogleMailPy) here. It’s quite dirty but it works and it supports the Google response messages (Google doesn’t like it when a lot of e-mail is being pushed within a certain timeframe. The script takes this in to account and uses the Google Double Back strategy to Play Nice™). The script defaulty adds two labels: GoogleMailPy (to identify all mail processed by the script) and the directory path in which the original e-mail message was found (with cyrus and Maildir this is also the folder path in your e-mail so it makes kind of sense to use this as temporary label until you find a better one).

GoogleMailPy uses no external Python libs so it should work out-of-the box. I’ve put in some comments in the source if you’d like to put in some feature of your own. I’ve just imported about 32000 messages and everything seems to be working ok. Using the script is of course totally at your own risk and it’s probably stuffed with bugs

Usage is easy:

1. first set the right user credentials in the script. Check for the SETTING comments (There are three, one for turning certain labeling features on or off, one for the user credentials and one for the right userpath).
2. Just call the script with a Maildir of Cyrus DB directory path.

Please let me know about your experiences in the comments of this post.
I’m now checking out the other Google APPS (especially the wordprocessor and spreadsheet). We might connect it to our new client portal at the office to generate and share reports. More on that later.

So, what do you do if you’re a nerd, stuck in an airplane but not close enought to you nerd-friend to have a nice chat. Well, easy. You just put up an ad-hoc network and start ichatting.

Yesterday we’ve been to Muir Park, which is a sequoia forrest just above San Francisco. I’ve just added all the pictures to my flickr account (shortcut is to the right). I’m still working on the Alcatraz movie. I hope to put it online tomorrow morning (so it can render tonight). I need to find a good place to upload it though, i really dislike the Youtube quality.

We’ll be leaving for the airport at around 11:30 and we’re flying at 15:10. Keep you posted!

Update: i’ve redone the clip in vimeo, which is 30 times better than Youtube.

Ofcourse i’ll write some more about Velocity later on (we had some more nice sessions to write about). However, i just created a little clip of Bart and me renting a Ford Focus (see below). I’m not going to get some sleep. Tomorrow i’ll change hotels to one close to Fisherman’s Wharf.

In short: today is already much better then yesterday! First we had a great talk from Steve Sounders about HTML / JS optimization. The thing i didn’t know is that the order and methods used in creating HTML and JS based sites is very determend for the actual speed of your site in a browser. For example: if you have a image load after a javacscript section the browser will (by default) not preload the image (the js will cause a blocking situation). So Steve mentioned a lot of methods to get around this. The thing is, for me it still feels like working around a bigger problem. The fact that browsers still have different interperations for the same type of source is irritating to say the least. I also not understand that the above situation occurs. Why not start a massive pre-fetch operation after parsing the HTML? I can understand that such an operation is useless when followed by a script that decides wether or not the script actually needs to be loaded. However, this also can be scanned by a parser.

Steve was followed up by Adam Bechtel from Yahoo. He compared major infrastrutures with plumbing. An extremely good presentation. Not specificallly because it was new information, but more because the way of thinking while desiging a infrastructure. Since at IC&S we do this often for our customers, this was very refreshing.

David Ulevitch (everydns.net, opendns.com) has a good story about this method called anycast which is basically, using some loopholes in BGP to create real network load balacing and automatic failover. By assigning a /24 subnet to a single server or a cluster of servers those /24 can be assigned it’s own BGP route. This route will be announced all the way to your connection provider. If this is done at multiple locations traffic will be routed to the nearest costpath location (which is the location which has the lowest networking path cost). So, automatic balancing occurs. When one of those locations has a fall out, the route is deleted and all traffic will ‘failover’ to one of the alternate locations. Why is this kind of dirty? Well, flapping can occur, in which case the BGP routing entry will be created and deleted in routers all over the world multiple times (this can lead to damping in which case the uplink providers will delete the route). Also,  because the smallest subnet that can be routed with BGP on the internet is a /24. So your server will use up this complete subnet of ipnumbers. This problem can ofcourse be fixed by using multiple hosts on this cluster and duplicate it completly somewhere else.

After this dark scheme of BGP hacking Adam Jacob came on stage to tell us about building automated infrastructures. This was a very cool talk. He basically explained what tools to use to create a completly automated infrastructure (hence the title of his talk). What was so nice about it is how easy this actually is. Ofcourse, at we do stuff like that already at IC&S but we learned a lot of new stuff. The part about how to deploy a platform on EC2 and extent it when needed with just a few click was very interesting. Parts of it we’ll most certainly use for our Jitscale service.

Currently i’m listing to Peter Zaitsev about mysql scaling and performance. Since his english is quite hard to understand and he’s - as he’s telling us himself - not providing a silver bullit for performance solutions on mysql the talk is not really interesting. Let’s see what’s next!

Another day at Velocity has started! We’re now looking at how Faceball should be done correctly. This is the howto video. Everybody get their thanks (as if the conference is over already!) More later!

So, we’re still alive and kicking here at Velocity.. altough the jetlag is getting to us right now and people around us have to start talking slowly for u to be able to understand

As a roundup; it was a nice day, a few good speaker, some not so good. I think i’m a bit spoiled by TED at which every speaker was marvelous there were no dull moments to be found. Anyways, i’m looking forward to tomorrow which, looking at the schedule, has some nice sessions. At this time we’re waiting for the Ignite sessions to start. These sessions feature startups that can present their idea’s to an expert audience (us). Tomorrow morning one of the ignitees wil be the winner. No idea what they’ll be winning though.

Below is a little clip i shot on our way from Velocity to Beni Hana’s. Bart and Justin conclude the first day of Velocity in this clip. Have fun!