News
Lavabit has been through a lot. This page is a place for our team to write about the challenges we’ve overcome. Like most human endeavors Lavabit didn’t start out perfect. The reliability, flexibility and performance our current services provide only came as a result of our engineering team’s unequaled dedication and hard work. With the growing pains behind us, Lavabit now owns one of the best e-mail platforms on the Internet. Continue to watch this page as the Lavabit Team adds new and exciting features to our service offerings.
October 1st, 2007 - We've decided to push the launch of IMAP back to November 1st, 2007. Were not comfortable with the current state of the code base, and feel the additional time is needed for testing, and bug fixing.
August 4th, 2007 - We have updated the list of real time blacklists we use. Currently we are using the Spamcop, Spamhaus Zen and UCE Protect Level 1 blacklists. You can always disable these blacklists via our Preferences portal; set the RBL feature to disabled. If you have suggestions or experience problems in regards to our blacklist policy, please let us know.
July 31st, 2007 - We've tentatively scheduled a maintenance windows for the evening of October 1st. This window will involve downtime as we make major hardware and software upgrades to our network. We apologize in advance for the inconvenience. We hope to launch our new IMAP server during this window, but only if we can finish eliminating all of the bugs.
January 9th, 2006 - We will be taking the system down for maintenance the evening of February 1st. The outage should last less than six hours. We will post an update here once all of the maintenance activities are complete.
December 11th, 2006 - Our development team has spent the better part of the last six months refactoring the code for our SMTP and POP3 servers. We have also added support for IMAP into latest version. Look for the refactored code to enter production in the next two or three months, once it has been throughly beta tested.
For those with feedback on where we can improve, now would be a good time to tell the Postmaster. We are specifically looking for feedback on what spam filter we should deploy? We are currently evaluating DSPAM, CRM114 and DCC. Keep in mind that we now have over 60,000 users, and we process more than 500,000 messages a day, so whichever solution we elect will need to scale gracefully.
June 15th, 2006 - We will be taking the system down this evening to perform additional maintenance on the database server. The outage should last between 2 and 4 hours, and start around 11pm ET.
June 14th, 2006 - As promised, everything is back online. The spam filter has been disabled temporarily while we continue to purge the stale token data. Once the purge is complete we will reactivate the spam filter. Thank you for your patience.
If you emailed the support team about this problem, know that we have your message and will do our best to respond, but that because of the volume of email we normally recieve, not everyone will get a response.
June 14th, 2006 - We are currently experiencing problems with our database. We have already identified the issue and are executing a series of purges to eliminate the issue. To prevent this problem from occurring again, we intend add a purge query to our nightly maintenance script.
The issue relates to our use of a personalized statistical spam filter. The growing popularity of this filter has led to increased stress on our database. This combined with the fact that we neglected to periodically purge the statistical data combined to generate a load which bogged down database server and led to errors. The currently running purge should cull the data back down to a maintainable size. Going forward our database will only hold spam signatures for seven days. You will not be able to train the filter with messages older than seven days.
We sincerely pologize for the trouble and hope to have our services restored shortly.
May 4th, 2006 - Our ISP has fixed the reverse DNS issues that have plagued us for the last two weeks. As such, Excite, Comcast, AOL, GoDaddy, and Outblaze should start accepting mail from us again over the next 72 hours. If you continue to experience problems, please let us know.
April 20th, 2006 - We will be taking the system down for maintenance the evening of April 28th to upgrade our network and power infrastructure. We anticipate the outage lasting between 2 and 4 hours. This should be the last outage related to our recent move to Vericenter.
April 18th, 2006 - After migrating data centers, AOL, Comcast and Excite have started blocking emails that originate on our servers. We have since resolved the issue with AOL, but it may take up to two days for their block to completely clear. We are trying to contact someone at Comcast and Excite, but have so far been unable. We will continue to try.
January 6th, 2006 - The preferences application is now online. Use it to adjust your settings, including those related to your spam and virus protection. Setup blacklist or whitelist rules. Enable greylisting, or just change your password. Works best in Firefox.
The Nerdshack will be going down for maintenance the evening of February 4th. During that time we will be moving data centers, upgrading hardware, and installing new versions of the software we use. We anticipate the outage lasting between 8 and 12 hours.
October 9th, 2005 - We have revamped our support for international character sets. Please test that your particular international character set is functioning correctly. If you have messages which are not passing through our SMTP servers correctly, please send the Postmaster a sample message as an attachment.
September 12th, 2005 - Several spammers have begun e-mail campaigns using the nerdshack.com address. Unfortunately, because these e-mails do not pass through our network, we are unable to stop them.
As a result of these campaigns, you may have received bounce messages indicating that your account was used to send out spam. These bounce messages are inaccurate. Your account is secure, and our servers have not been compromised. We are in the process of notifying the affected ISP"s to the presence of a spammer on their respective networks, in the hope that they will kick these miscreants off the Internet.
If these bounce messages become a problem, we are capable of blocking them at the server for your specific account. Note that blocking these bounce messages will also block legitimate bounce messages from reaching you.
August 6th, 2005 - We have narrowed our name search to four possible candidates, in no particular order, lavabit.com, bexi.com, zivas.com and fogbit.com. Our goal is to find a name which is generalized, as we hope to someday offer more than just mail services. If you have an opinion on any of these names, please e-mail the Postmaster.
After the name change, existing users will have a choice between using nerdshack.com, mailshack.com or the new name. New users will be forced to use the new name.
July 12th, 2005 - We are now enforcing SPF on all incoming mail. If you would like your account to NOT use SPF, please notify the Postmaster. SPF is an anti-spam technology which most users will want to have enabled.
In other news, we are considering a possible name change in order to attract a more mainstream audience. If you have any suggestions, please make them to the Postmaster. We are only considering names which are either available, or available for sale at a reasonable price.
June 12th, 2005 - An unknown source is sending our users e-mails claiming to be from the Nerdshack staff. These e-mails have an attachment that contains a virus. If you receive an e-mail like this, please delete it immediately. We are working on a method to block these malicious messages at the server level.
May 21st, 2005 - After many delays, and much hard work, we are FINALLY ready to accept new users. Please use any one of the many registration links to sign up.
April 25th, 2005 - We just upgraded yet again. This new version contains a few minor bug fixes that should make Outlook work more reliably with our servers. The most important thing to note about this upgrade is that we are now running atop CentOS 4 and Linux Kernel version 2.6.
April 16th, 2005 - We released a new version of our software this evening, resulting in a few minutes of downtime. This release should fix all of the known bugs. The only new feature is support for SMTP sessions over SSL on port 465. Previously we only supported SSL using the STARTTLS command.
If this release remains stable, we will start accepting new users approximately one week from today.
April 4th, 2005 - Since the upgrade a week ago, things have been running relatively solid. We are still experiencing one crash per twenty-four hour period, so if you get disconnected, try again. The load balancer should redirect you to an available server. We have traced the source of the crashes to bugs in ClamAV and OpenSSL libraries, and are working on ways to avoid the issues in our code. If you have e-mailed in for support, we will respond! With all of the problems lately, we"ve been a little overwhelmed trying to fix the bugs, and respond. We are working through the queue now. We appreciate your patience in sticking with us during our growing pains. We hope to make this service worth the trouble you"ve gone through!
March 29th, 2005 - It is about three in the morning, and things are just wrapping up. We should be back online in less than an hour with all the latest and greatest software. The hope is that things will start to run stable.
March 28th, 2005 - We will be taking the systems down for software upgrades this evening. Expect between two and three hours of downtime.
March 20th, 2005 - We have moved back over to the new system, and are going to spend the next couple of weeks working out the bugs. Rest assured we are working as hard as humanly possible to build a stable, and reliable platform for your e-mail. Once we are confident all of the bugs have been worked out, we will start accepting new users.
March 3rd, 2005 - It is a sad day. We switched back to the old system, because we just couldn"t get the new one to run stable. Our database server crashed four times this past week, and the problem finally has been narrowed down the underlying hardware. We have ordered replacement parts, but don"t expect them to arrive until Monday or Tuesday. As a result, we have pointed our DNS records back at the old system, and will continue to use it until sometime next week, when we are ready to make a second attempt at a cutover. Bear with us. We are trying.
February 27th, 2005 - The cutover that has been talked about for the last six or so months finally occurred last night. We are now running on the new, and much improved system. As a result, we will likely start accepting new users in about a week. The registration scripts still need to be changed so that they take advantage of the new system.
There are a couple of things to note about the new system. We no longer support authentication using APOP/CRAM/DIGEST or any other secure mechanism. We do support the use of SSL, and we strongly recommend its use. The reason for this is simple, in order to support secure authentication we needed to store your password in an unencrypted fashion on the server. With this new system, we no longer do that, hence the change.
We also no longer support webmail. This is temporary, as we are now working on creating an IMAP server, followed closely by a new webmail system. The hope is to have them soon. Experience has taught us better.
January 14th, 2005 - Development, testing, and deployment have taken about three months longer than anticipated. However, it appears that there will be a hard cutover to the new system by the end of January. If all goes well, it will occur sometime during the weekend of the 21st. The cutover should be transparent to our users, with the exception that you will see a signifcant performance increase, and a significant spam reduction. We appreciate your patience as we continue to work hard to upgrade your user experience.
October 26th, 2004 - Just a quick note to let everyone know we are still working furiously to upgrade our infrastructure. The new servers have been installed, and are in the process of being configured. Our custom backend system is also nearing completion, with over 25,000 lines of C source code having been written in the last two months. Look for the upgrade to come any day now. The only thing holding us up is our commitment to delivering a flawless product.
August 18th, 2004 - We know that some of you have been experiencing errors when trying to download your e-mail. This occurred if your e-mail was moved from a defective hard drive, as permissions were corrupted during the move. We have since reset permissions on all of the accounts, in the hope that it has finally fixed our problems.
August 16th, 2004 - Over the past week, the hard drives have locked up three separate times. As a result, we replaced all of the drives this afternoon. The change took about two hours, and seems (for now) to have fixed our problems. Note however that it could take the next twenty-four hours for our mail server to catch up on processing all of the queued mail. Note that no mail was lost in the transition, only delayed.
We originally planned to replace our already overloaded servers in September. As a result of the reliability problems, and the current load, this upgrade will likely take place before the end of August. We will keep you posted on our plans.
We sincerely apologize for our downtime, rest assured that we are doing our best to keep this service as reliable, and fast as possible given our limited resources.
July 25th, 2004 - Over the weekend, a hard drive locked up, and caused the system to go down. We restarted the server, and resolved problem once we were notified. We sincerely apologize for the downtime incurred. Current plans call for more reliable hardware to be deployed in September.
July 20th, 2004 - We are growing, and fast. More than three thousand people now use the Nerdshack for e-mail. As a result, we will be taking the system down for upgrades August 1st. We expect the outage to last several hours as we move equipment to a data center with more bandwidth. We also intend to upgrade the hardware to improve performance.
We are also looking for advertisers. If you know anyone, have them contact : sales@nerdshack.com.
June 23rd, 2004 - Yesterday, the Nerdshack e-mail servers were upgraded to Postfix 2.1.3. In the process, SMTP authentication was temporarily mis-configured. The problem has since been corrected, and we apologize for the inconvenience.
In our continuing effort to make this service better, we ask that if you find problems, please immediately notify the Postmaster so that we may correct them.
SPF data has also been published for both nerdshack.com and mailshack.com.
June 21st, 2004 - Our service is now advertised in the Free e-Mail Guide.
June 15th, 2004 - E-mail is now protected using a combination of ClamAV & amavisd-new. Presently I have SpamAssassin deployed through amavisd-new but plans call for me to deploy DSPAM when I can finish working out the kinks. SPF is also on the way.
June 10th, 2004 - Quotas are now enforced. I have also resolved some of the bugs in our limiting software. Users may now only send e-mail from addresses which match their user account. Every user has a one hundred megabyte soft quota, which they may exceed for three days before they are no longer able to receive mail. No user may exceed two hundred megabytes.
I have downloaded DSPAM, and ClamAV, and hope to deploy them soon. Combined they are what we feel to be the best ant-spam, anti-virus programs available. More to follow.
June 1st, 2004 - The Nerdshack comes online. We presently are hosting this service using a P4, with one gigabyte of ram, and one terabyte of hard drive space. This machine, affectionately named staci is hosted on a T1. Staci uses White Box Enterprise Linux, Postfix 2.1.1, Qpopper 4.0.5, and Apache 2.0.46. The registration framework is written in perl, and uses MySQL to manage user accounts. I"ve also written a Postfix policy server in C that limits each user"s outgoing mail to one hundred message per day. See our questions section for more on that.
Over the coming weekend I intend to finish installing amavisd-new which will protect users from spam and virii. I also intend to setup SPF. Stay tuned for more information.
