Hardware and Software
For flexibility and performance, Lavabit uses a multi-tier architecture. Each tier is responsible for one small, but critical, component of the service. To ensure reliability each tier is completely redundant with multiple machines working in concert. This strategy has allowed Lavabit to grow quickly and avoid the problems that hinder other free Internet e-mail services.
At the border, Lavabit uses separate clusters for HTTP traffic, incoming SMTP and POP traffic, and outgoing SMTP traffic. We use a load balancer to make each border cluster appear as a single machine to our users. Internally, Lavabit has independent clusters dedicated for file storage and running our database server. For security and performance, we use a separate physical network to access the internal clusters.
All of the Lavabit servers use CentOS 4.4. The application servers are Dell PowerEdge 1650s with dual 1.4 GHz Pentium III processors, 4 GB of RAM and 36 GB SCSI hard drives. The database servers have dual Opteron 246 processors with 8 GB of memory and six 146 GB SCSI drives in a RAID 5 configuration. The storage servers have dual Opteron 242 processors, 2 GB of RAM, a 36 GB SATA boot drive, and 12 250 GB SATA drives in a RAID 5 configuration. The load balancer is an Alteon AD4. For our network, we use unmanaged Linksys gigabit switches.
For incoming SMTP and POP connections, we use a custom e-mail server developed by the Lavabit programming team. The e-mail server is written entirely in C and compiled using GCC. See our features page for a complete listing of everything our e-mail server supports. Developing our own e-mail server enabled us to engineer a system that’s controlled by a database and scales horizontally by clustering commodity servers.
For a handful of critical features, our e-mail server relies on a number of open source libraries. We use ClamAV for virus scanning, DSPAM for spam filtering and OpenSSL to support SSL. Lavabit is in debt to the open source code projects we rely upon. We hope to someday repay this debt by releasing our e-mail server back to the community.
For outgoing messages, users will initially connect to the incoming SMTP cluster that runs our custom e-mail server. This incoming server authenticates the user, enforces any limits associated with the account and scans the message for viruses. Once these steps are complete, the custom e-mail server relays the message to a server in the outgoing e-mail cluster. The outgoing e-mail servers run Postfix. Postfix handles the tricky business of delivering e-mail messages. If Postfix is unable to deliver a message, the message is returned to our custom e-mail server with a bounce notification.
The Lavabit HTTP cluster uses Apache to serve web requests. These servers also host an array of custom web applications used to handle common tasks. The software used to register for a new account or the software used to adjust your preferences are good examples of the web applications we’ve created. We write all of our web applications in C or Perl.
On the internal side of our network, the database cluster uses MySQL version 4.1. Our MySQL servers use replication and are configured to use a master/master relationship. We exclusively use the InnoDB storage engine because it supports transactions and row-level locking. For file storage, we use a custom application that compresses e-mail messages and stores them on disk in large 32 GB chunks. This chunking method allows our servers to avoid the performance limitations of the ext3 file system.
Every application server also runs a distributed caching program. This program was inspired by memcached, but offers a handful of custom features specific to our needs. This caching program stores frequently requested information in memory across all the servers. Our custom e-mail server checks the appropriate caching server for user information before using the database. This caching strategy has allowed us to expand our service with an economy of scale that allows us to offer free accounts. Like our e-mail server, we hope to offer our caching software to the community some day soon.
We also use BIND as our DNS server, Nagios for monitoring availability and MRTG for collecting SNMP information and generating our pretty graphs.
Now that you know more about our network, you might want to visit our graphs page to see how the different components are performing. You can also visit our statistics page to see exactly how much information our network processes.
Of course the best way to experience the reliability, flexibility and speed of Lavabit is to use it! If you’re not already using our service for e-mail, sign up today!
