Tuesday, December 31, 2013

Lavabit - There's a sucker born every minute

P.T. Barnum was a master showman. His name has fallen from public use, but his methods haven't.  You may be familiar with his name as part of the Ringling Bros. and Barnum & Bailey Circus.  He's notably remembered for promoting many hoaxes, and associated with the phrase "There's a sucker born every minute".  It likely wasn't his phrase, but when I see a monkey head sewn to a fish, that's the man I think of.

That's also the phrase that comes to mind when I hear about Lavabit. The company was founded specifically as an answer to Gmail's business model of using your email content to deliver ads.  The idea was to encrypt incoming email so it wasn't viewable. They released a whitepaper touting their encryption, but what it came down to was doing a lot of busy work to locally encrypt and decrypt your email. Everything sent over the wire was plain text - albeit via an encrypted tunnel. The service was billed as "so secure that even our administrators can’t read your e-mail". Unfortunately their users aren't InfoSec experts, and didn't understand this statement to be completely false.

What most users don't understand, and I've run into this when speaking with Lavabit supporters, is that there are basically two types of encryption.  There is encryption at rest, and encryption in transit.  SSL provides encryption in transit, the data sent between two nodes is not viewable by a third party. Lavabit provided encryption at rest.  If you weren't using it, it was encrypted. When you sent or received email, it was decrypted (if read from disk) and sent via SSL.  SSL is essentially a 'tunnel' - anyone on either end can see what's going in and out, when the data is in plain text.  A solution to that problem is PGP - encrypt your data before it's sent, and it will be encrypted at rest and in transit (whether it's via SSL or not).

What's curious is this apparently 'successful service' actually stopped accepting new users in May, 2013. That's very odd. Someone may think that's related to the FBI Probe, but that didn't start until June 10th. Now, I run own and operate VFEmail.net - the subpoena on pages 2 and 3 is nothing out of the ordinary. They always have a 'gag order' attached. The FBI is just as paranoid about the target finding out about the order, as many users are paranoid about the NSA reading their email.

On June 28th, Lavabit received a wiretap order.  This is the 'pen/trap' device. Having received this sort of order in the past, using a 3rd party device IS NOT REQUIRED.  In fact, Page 7 states use of a trap and trace device or process ("pen/trap device")   The ('pen/trap device') is a legal notation to say that from here on, this entire sentence will be referred to as 'pen/trap device' - not that a physical device absolutely will be installed.

At that time, Lavabit could have implemented it's own data duplication system.  An email archive feature, and grepping of the log files is all that's needed to comply.

On June 28th (Page 12), we see The representative of Lavabit indicated that Lavabit had [he technical capability to decrypt the information but that Lavabit did not want to 'defeat [its] system.' . Here Levison admits to being able to decrypt data, but is hiding behind Company Policy to state "We just don't do that".  I dare you to find anyone in InfoSec who thinks Company Policy is a valid security method.

On July 9th, (Page 20) we see that Levison is compelled to appear before the court to explain why he simply can't decrypt email for a single account - or for those of us who know how email works - why mail received via SMTP for/by the target account simply wasn't archived prior to encryption.

On Page 31 we see the actual summons, Levison is to appear in the Virginia court on July 16th - this summons includes the request for the SSL key.  This is because Levison is defying a lawful order to simply archive SMTP communication - which is in clear text on his system.

On July 16th, (Page 34), we see all the dirty details of what the FBI needs to do to accomplish a simple thing that Levison could have done. 

We also see on Page 46 - Levison had caved to this simple request, but demands $2000 to provide the information requested, and then another $1500 after 60 days. We also see that the government has paid for his expenses to travel to VA.
(edit:) Why didn't the FBI pay $3500 for that, and instead opt for the SSL key?  Here's an example of how to use Capsa Professional to save emails. Once that content is saved, it's trivial to grep each text file for the target's address. I'm sure the FBI has software that includes the filtering. This isn't magic or hocus pocus, it's simple network sniffing that anyone with a basic background in InfoSec can do. There's no reason, other than complete lack of familiarity about how it's done, that Levison couldn't provide the requested information at a reasonable reimbursement rate.

Beyond that it's all about the SSL key.  The entire episode was CAUSED by Levison's failure, and flat-out incompetence, to implement a simple SMTP archive feature and then his attempted fleecing of the American taxpayer by charging $2000 to provide that information.

(Edit) Some people have said the 'archive' is very complex to implement or the email would have been encrypted.  Actually, neither are true. First, Levison was already 'up to date' by manually decrypting the mailbox, just new mail needed to be sent in an automated fashion. Second, SMTP is very simple - it's basically 4 commands.  You connect, say HELO, MAIL FROM, RCPT TO, DATA.  What you see in an email all follows DATA. MAIL FROM is only for return mail, and RCPT TO is the destination. A simple archive feature would merely read the RCPT TO, do a text match, and programatically add 'fbi' as another RCPT TO.  For outgoing mail, you just do the match on the AUTH (or MAIL FROM) line instead. Levison claims to have written the entire Lavabit code, so adding this option should have been a trivial exercise.(/Edit)

But it's not over.  He's taken his half of the story, the "NSA is attacking the constitution" to the public. I'm no fan of the NSA, but other than the probable target being Ed Snowden, the NSA violations are in no way related to this case.  Surprisingly, even after admitting to the NYT that the entire issue was caused because he couldn't provide real time archiving, people STILL think he's full of integrity.  Quoting the article - He would log the target’s communications, unscramble them with the encryption keys and upload them to a government server once a day. The F.B.I. told him that was not enough. It needed his target’s communications “in real time,” he said.
If you think that's bad - remember Levison's original version of 'security': "We just don't do that".  Apparently that's a good enough answer for him, but when the FBI requested his SSLKeys, all of a sudden 'we apply filters per policy' is no longer an acceptable answer.  So now, supposedly, everyone's data is at risk.

Levison wasn't able to make $3500 off the FBI, but the gullibility of the public has earned him over $200,000 on Kickstarter, and $100,000 on Rally.  Who knows how much he's taken in on Bitcoin, Paypal or other direct methods.  He may not be much of a tech, but he's good at marketing.

VFEmail has no interest in supporting this person or DarkMail - another marketing ploy to target the non-technical public. If someone does come out with an untainted open source, encrypted, SMTP replacement that still allows AV Scanning, Spam Scanning, and User Filtering on the server - we'd love to hear about it.

Who knows why the media won't cover this angle of the story.  I understand USA Today or NYT not having proper staff to understand tech issues at this level, but Ars?  It's a sad state of affairs.  Frankly, I've thought about shutting down VFEmail because of Lavabit. Not because of 'NSA hacking', or 'Subpoenas for SSLKeys' but because Levison and Silent Circle are trying to manipulate the public into using a solution that won't ever be able to meet their demands. 
(Edit 2/24/14) I was subscribed to the Dark Mail Alliance email list - JUST IN CASE they were to release something of substance. Apparently it's nothing more than a marketing list for Silent Circle's products.(/Edit)

If you would like to support me or VFEmail, you may donate to paypal@vfemail.net or via Bitcoin to:


Thursday, June 20, 2013

VFEmail.net Upgrades

VFEmail is in the final stages of a set of whirlwind upgrades.  In the end, we'll have upgraded hardware, software, and a better overall design.

VFEmail also runs Openmail.cc.  These two sites share hardware and core infrastructure, but currently operate mostly independently.  Only the user authentication has been merged.  Webmail is still a separate database and install, though the same webservers are used. See - www.vfemail.net vs openmail.cc

Over the last 10 years VFEmail has slowly grown from a single PC to multiple systems.  We've survived the 'free email' arena by not outspending our competition, we upgrade when it is affordable. 
Now just about each service has a dedicated server. We have:

2 SMTP sending servers
3 incoming MX servers
2 Web Servers
2 DNS Servers
2 Spam Servers
2 SQL Servers
1 'Misc' admin server
1 offsite backup NAS
2 Load Balancing routers in master/slave config

Any servers of which there are at least 2, except the load balancers, are Virtual Machines.  These are backed up to the primary NAS weekly and replicated to the offsite backup NAS.  In the future, we plan to replicate data as it is written to provide even better service.

Each web server points to a VIP internally for SMTP and SQL access.  That allows us to provide 100% failover for either service.   Incoming Web and SMTP are equally load balanced.   At this time POP and IMAP are directly hosted on the NAS to increase response time.  The idea being that since the NAS is not 'hot replicated', if there is an issue it will affect POP/IMAP anyways.  In the future, POP/IMAP will be extracted and load balanced as the backend data is replicated between two hosts.  Other providers do this with DRDB, but VFEmail uses ZFS on the backend, so that's not an option.  ZFS does provide for regular snapshots and easy replication, backups, and restores -but not live replication.   AVS + ZFS would provide that, but I was never able to get that functional, so I'm keeping a close eye on FileReplication Pro as a possible alternative.

Next up, new User Control Panel, Horde 5.1, and Roundcube upgrades.

Friday, May 31, 2013

FreePBX/Asterisk Failover using IAX2 and bandwidth.com

I have a site that REQUIRES zero dropped calls.  These calls are sales, and bring in revenue. If the office cannot be contacted, not only could we lose that referral, but the referer may not bother to send more sales our way.
I am offsite.
In fact, I'm out of state. Too far to quickly physically resolve an issue.

I've been using Asterisk for almost 10 years, and it just keeps getting better.  I used it to failover a dieing ACD via T1 crossover cable, and now I've had the opportunity to do it with a pure Asterisk solution.

Detail -
Bandwidth.com provides SIP trunks for $25/month/line.  They use SRV records to provide high availability.  SRV records work like MX records.  Higher priority hosts get traffic first.  My site has an afterhours answering service, so the first step was to create an Amazon Cloud instance of FreePBX and two SRV records - one record for the office Asterisk machine, and the second for the Amazon Cloud one.  The Amazon one only forwarded to the answering service.
This ensured us that no matter what occurred, all calls would be answered.

Unfortunately after a bouncing circuit caused complete havoc with our phones - calls were dropping /cutting in and out, and I wasn't able to remotely access the PBX to shut the trunk down and force fail over - we realized the Amazon Cloud solution wasn't enough.  These referrals are time sensitive.  If we don't have a way to reach our email to retrieve the answering service notification, and call the customer in a very short amount of time, then the sale is still lost.  The only real solution is 100% uptime during business hours.

Going All Out
I procured a CradlePoint MBR1400 firewall/router with Verizon 4G modem and static IP service from Verizon.  The CradlePoint has the capability of doing failover between interfaces. For example, if your cable provider goes out, it can automatically switch to the 4G service.  That's great for web browsing, but doesn't work so well for SIP trunks.  Asterisk has to know it's external IP Address, and I'm not sure if STUN servers work for SIP trunks.  If the external IP is incorrect in Asterisk, we lose the audio stream.  Therefore I hard code it.  In addition, using the 4G modem allows us to be sure a fibre cut in the street doesn't affect us.

Once the CradlePoint was installed and tested on the PBX VLAN, I installed a 2nd instance of FreePBX on the same VLAN, and added my bandwidth.com trunks.
The BackupPBX uses the CradlePoint as the default route, and the NAT settings are set to the Verizon static IP.  
Then I created an IAX2 trunk between the primary and failover PBXs:

On the Backup PBX -
Trunk Name:  PrimaryPBX
PEER Details:

On the Primary PBX -
Trunk Name: BackupPBX
PEER Details:

Note the context on the BackupPBX Trunk, from-pstn.  This is because we want to push pstn sourced calls from the Backup into the Primary.
Also, if you use a dial prefix, make sure you add that to the Trunk settings on the Primary.  Likely, you may strip a '9' from the beginning of the dial string normally, so you would need to add it back in when sending the call via the backup PBX.  

Then on the backup, create an IncomingRoute with the destination 'Trunk' 'PrimaryPBX' and create an OutgoingRoute that matches your Outgoing Route settings on the Primary.

On the Primary, add the BackupPBX Trunk as the last Trunk in your Outgoing Route.

Now we have a complete backup.  If the Internet were to go down, the SRV record will take care of incoming call fail over, and the BackupPBX Trunk/route will take care of outgoing call fail over.  Completely seamless to the user.
Obviously this doesn't cover a power outage, but due to the nature of the business, most of these sales are local and if there is a major power issue that would exceed the UPS times, then nobody is making those calls anyways.  The Amazon Cloud PBX will be shutdown, and I will run a '3rd tier backup' PBX in a VM on my local servers for the answering service.