Spam via Google? Apparently Google has been hacked by the Chinese for months now

So, much like many other email providers, Spammers try and abuse VFEmail servers by creating thousands of account to use for SMTP sending.  We set daily limits per account, which forces this 'many accounts' issue - but we also have other throttling in place that prevents the Spam from actually leaving our systems.

What's odd is that Chinese spammers have lately been coming FROM GOOGLE.  Now, it's not uncommon for ASPs and other hosting environments to be the source of Spam - just like any email provider, we have no idea what the account holder will do until they've done it. But in this case, the Spam is coming from Google, using the VFEmail accounts for SMTP Auth.  That's really odd, because I wasn't aware of Google allowing that sort of hosting or sending with a remote sites credentials. 

[edit 3/3/14] Upon closer inspection, the DKIM header shows a Gmail hosted domain, 1e100.net as the source.  Google, fix it already.

Here's the relevant line -
Received: from unknown (HELO mail-ie0-f178.google.com) (YXNkLmxva0B2ZmVtYWlsLm5ldA==@209.85.223.178)
  by mail.vfemail.net with ESMTPA; 24 Jan 2014 02:42:16 -0000

According to whois, 209.85.223.178 is a Google IP.  Unfortunately, I'm only getting automated replies to my abuse emails.  So we'll see if this info gets their attention.

Here are the full headers -

Received: (qmail 88632 invoked by uid 89); 24 Jan 2014 15:18:43 -0000
Received: from localhost (HELO freequeue.vfemail.net) (127.0.0.1)
  by localhost with SMTP; 24 Jan 2014 15:18:41 -0000
Received: (qmail 35975 invoked by uid 89); 24 Jan 2014 02:42:17 -0000
Received: by simscan 1.3.1 ppid: 35953, pid: 35972, t: 0.0210s
         scanners:none
Received: from unknown (HELO smtp102-2.vfemail.net) (172.16.100.62)
  by FreeQueue with SMTP; 24 Jan 2014 02:42:17 -0000
Received-SPF: softfail (FreeQueue: transitioning SPF record at vfemail.net does not designate 172.16.100.62 as permitted sender)
Received: (qmail 631 invoked by uid 89); 24 Jan 2014 02:42:16 -0000
Received: by simscan 1.4.0 ppid: 625, pid: 628, t: 0.0694s
         scanners:none
Received: from unknown (HELO mail-ie0-f178.google.com) (SMTPAuthName@209.85.223.178)
  by mail.vfemail.net with ESMTPA; 24 Jan 2014 02:42:16 -0000
Received: by mail-ie0-f178.google.com with SMTP id x13so2137073ief.23
        for ; Thu, 23 Jan 2014 18:42:16 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=f5GsZjfdse0Y8c+H8IzzmEM6HLaZXgOiXDMgsohQc54=;
        b=aR7dijI0xORpdCVjWz5/mWPYv4B7zAXZdZfhRqrvDU0NhN0x0if/xgms8NOKgwLBiS
         iaygwYd8yTE0RXg2TX0HMarSsNwpe1LtAJtETILK5dgdu+f9QhUsKEnfmph986EWNCja
         zah5hWJLOD59MzqyIL8xvRAaw7Yav473gnjn2/pVs+8MZfa5iMy+LxV15hgczNjQ/TbG
         ZeBKbydJVYFy6yRMA8+l2P5bcqk3S29oQMDD2Bt7M2h9Hynk8K+Qzy1W1SwO2qpMI8qS
         B+nb//gPGvGACtlCy3O2KZrmtJSLaS/2RwSdbtMmiKx6tHkycwY+XjQi4Hqk61lyhfcc
         +IcA==
MIME-Version: 1.0
X-Received: by 10.42.64.17 with SMTP id e17mr8876684ici.26.1390531336210; Thu,
23 Jan 2014 18:42:16 -0800 (PST)
Received: by 10.64.229.8 with HTTP; Thu, 23 Jan 2014 18:42:16 -0800 (PST)
Date: Fri, 24 Jan 2014 10:42:16 +0800
Message-ID:
Subject:
From: chang chun
To: uxnr@email.com.cn, v6@email.com.cn, vacancy@email.com.cn,
        vagabond@email.com.cn, vagrant@email.com.cn, vance@email.com.cn,
        vanquish@email.com.cn, variation@email.com.cn
Content-Type: multipart/alternative; boundary=90e6ba3fcdab58111304f0ae4e1c

Update 2/1/14 - still nothign from Google.  They're totally hacked.

 --------------
MESSAGE NUMBER 4875719
 --------------
Received: (qmail 80547 invoked by uid 89); 1 Feb 2014 16:41:32 -0000
Received: from localhost (HELO freequeue.vfemail.net) (127.0.0.1)
  by localhost with SMTP; 1 Feb 2014 16:41:32 -0000
Received: (qmail 80286 invoked by uid 89); 1 Feb 2014 16:41:15 -0000
Received: by simscan 1.3.1 ppid: 80273, pid: 80280, t: 0.0295s
         scanners:none
Received: from unknown (HELO smtp102-2.vfemail.net) (172.16.100.62)
  by FreeQueue with SMTP; 1 Feb 2014 16:41:15 -0000
Received-SPF: softfail (FreeQueue: transitioning SPF record at vfemail.net does not designate 172.16.100.62 as permitted sender)
Received: (qmail 6407 invoked by uid 89); 1 Feb 2014 16:41:15 -0000
Received: by simscan 1.4.0 ppid: 6398, pid: 6402, t: 0.0851s
         scanners:none
Received: from unknown (HELO mail-vb0-f43.google.com) (SMTPAuthName@209.85.212.43)
  by mail.vfemail.net with ESMTPA; 1 Feb 2014 16:41:15 -0000
Received: by mail-vb0-f43.google.com with SMTP id p5so3789171vbn.30
        for ; Sat, 01 Feb 2014 08:41:14 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=7u8pwYRxgVJr58KdRsQpYbJ+VS6xSKmQescj2zcM4WY=;
        b=QEfLb4w2QT9/jNotKqT7WDvrFCMzLQiM96WprG3LDnAULvMbg2R98yVzC8At9Pomf3
         3rNoWLNp+XOy358ul8IvQsloHDCCUvmMDNiNC8A/G6rlzUA63y2qWiKTs98ALoV0PTZv
         GmNW+xR9fowGnQE/q59J64OXH8JoTKBGe9SUyMyZGsK0qo3LJ7DJc2PHgucLIGDOV81s
         uRO57Jbv4BYlv0BeG6e2WNo0PxmIWqrJ65DwFXRqOotTVyn8QwIFeaqfcbCSjdu74nWm
         FLRgmDvSUVUYtIXPimp1LidLTCmeMulp/YqBxPb9XhibpitT7l4UI4nVJI+L+PEsnCCl
         xvQA==
MIME-Version: 1.0
X-Received: by 10.52.61.168 with SMTP id q8mr6816vdr.40.1391272874597; Sat, 01
 Feb 2014 08:41:14 -0800 (PST)
Received: by 10.58.65.69 with HTTP; Sat, 1 Feb 2014 08:41:14 -0800 (PST)
Date: Sun, 2 Feb 2014 00:41:14 +0800
Message-ID:
Subject:
From: chang chun
To: xiaoyaojian@email.com.cn, xiaoyaoke@email.com.cn, xiaoyaoxu@email.com.cn,
        xiaoyaoye@email.com.cn, xiaoyaozi@email.com.cn, xiaoye@email.com.cn
Content-Type: multipart/alternative; boundary=001a1136b37479e2f104f15af5fa

--001a1136b37479e2f104f15af5fa
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: base64

xPq6w6OhDQoNCsfrz8LU2Li9vP61vcT6tcS158TUo6wNCg0Ky6u796Osv8m/tLLY19bKr8PFxrHN
vMaso7sNCg0K1NrOxLz+w/vJz7XjyvOx6iDT0rz8o6y14yDW2MP8w/sgo6wNCr2rzsS8/rXEwKnV
ucP7o6guanBno6m4/LjEzqogLnJhciC686OsDQrU2dPDw9zC66O6dG16ZyC08r+qo6wNCr/J1MS2
wdPrxPrPos+iz+C52LXEusPOxNXCoaMNCg0KDQogMTGhomN6c3RtemctcGRmLmpwZzxodHRwczov
L2RvY3MuZ29vZ2xlLmNvbS9maWxlL2QvMEI1R2dSOF93M3AyOFZqTjJjbFpYVTNoc2QyOC9lZGl0
P3VzcD1kcml2ZV93ZWI+DQoNCiC547SrzfgucmFyPGh0dHBzOi8vZG9jcy5nb29nbGUuY29tL2Zp
bGUvZC8wQjVHZ1I4X3czcDI4Tmxkdk9VSkxPUzF2UXpnL2VkaXQ/dXNwPWRyaXZlX3dlYj4NCg0K
--001a1136b37479e2f104f15af5fa
Content-Type: text/html; charset=GB2312
Content-Transfer-Encoding: quoted-printable

=C4=FA=BA=C3=A3=A1

=C7=EB=CF=C2=D4=D8=B8=BD=BC=FE=
=B5=BD=C4=FA=B5=C4=B5=E7=C4=D4=A3=AC

=CB=AB=BB=F7=A3=AC=BF=C9=BF=B4=
=B2=D8=D7=D6=CA=AF=C3=C5=C6=B1=CD=BC=C6=AC=A3=BB

=D4=DA=CE=C4=BC=FE=
=C3=FB=C9=CF=B5=E3=CA=F3=B1=EA =D3=D2=BC=FC=A3=AC=B5=E3 =D6=D8=C3=FC=C3=FB =
=A3=AC
=BD=AB=CE=C4=BC=FE=B5=C4=C0=A9=D5=B9=C3=FB=A3=A8.jpg=A3=A9=B8=FC=
=B8=C4=CE=AA .rar =BA=F3=A3=AC
=D4=D9=D3=C3=C3=DC=C2=EB=A3=BAtmzg =B4=F2=
=BF=AA=A3=AC
=BF=C9=D4=C4=B6=C1=D3=EB=C4=FA=CF=A2=CF=A2=CF=E0=B9=D8=B5=
=C4=BA=C3=CE=C4=D5=C2=A1=A3

e=3D"width:396px;height:18px;max-height:18px;background-color:#f5f5f5;paddi=
ng:5px;color:#222;font-family:arial;font-style:normal;font-weight:bold;font=
-size:13px;border:1px solid #ddd">
?usp=3Ddrive_web" target=3D"_blank" style=3D"display:inline-block;overflow:=
hidden;text-overflow:ellipsis;white-space:nowrap;text-decoration:none;paddi=
ng:1px 0px;border:medium none;width:100%">tom; border: none;" src=3D"https://ssl.gstatic.com/docs/doclist/images/icon=
_11_image_list.png"> tion:none;vertical-align:bottom">11=A1=A2czstmzg-pdf.jpg

ight:18px;background-color:#f5f5f5;padding:5px;color:#222;font-family:arial=
;font-style:normal;font-weight:bold;font-size:13px;border:1px solid #ddd">
?usp=3Ddrive_web" target=3D"_blank" style=3D"display:inline-block;overflow:=
hidden;text-overflow:ellipsis;white-space:nowrap;text-decoration:none;paddi=
ng:1px 0px;border:medium none;width:100%">tom; border: none;" src=3D"https://ssl.gstatic.com/docs/doclist/images/icon=
_9_archive_list.png"> ation:none;vertical-align:bottom">=B9=E3=B4=AB=CD=F8.rar

--001a1136b37479e2f104f15af5fa--
 

NetRange:       209.85.128.0 - 209.85.255.255
CIDR:           209.85.128.0/17
OriginAS:
NetName:        GOOGLE
 

Ok does this really mean Google is hacked?  No, it shows there's at least one system at Google that is a spam source, but it could just be a few botnet members. I understand this sort of thing happens to everyone, but frankly, I'm sick and tired of cleaning this shit up.


Update - 3/3/14.   After blocking the above CIDR for a month, the spammers are now using a new one -

Received: (qmail 76016 invoked by uid 89); 3 Mar 2014 03:02:15 -0000
Received: by simscan 1.3.1 ppid: 75998, pid: 76006, t: 0.2433s
         scanners:none
Received: from unknown (HELO smtp101-2.vfemail.net) (172.16.100.61)
  by FreeQueue with SMTP; 3 Mar 2014 03:02:15 -0000
Received: (qmail 26663 invoked by uid 89); 3 Mar 2014 03:02:15 -0000
Received: by simscan 1.4.0 ppid: 26615, pid: 26641, t: 0.1460s
         scanners:none
Received: from unknown (HELO mail-we0-f179.google.com) (aThqN2subWZndEB2ZmVtYWlsLm5ldA==@74.125.82.179)
  by mail.vfemail.net with ESMTPA; 3 Mar 2014 03:02:15 -0000
Received: by mail-we0-f179.google.com with SMTP id x48so2491161wes.38
        for <817 qq.com="">; Sun, 02 Mar 2014 19:02:14 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=pEGGLF9NMqHn49abWF4r9h3T2oZ2p/fE63y4TvKB2o0=;
        b=F5wEnRP4ZwkQgcMtnxzyAGW6Xim0JmOhGBYLBbMVaL3IbmAqmmOjI3ibYcIb+duG7m
         PAq2uyFKKBUcgPTYb9cCLDbBw0pU7ShZsFrpHFm6PrMpfXgxN8CPsoXp2zu7T8Klm7MH
         QVAz+shWX1yHMRyxiIJEK3YHOZud+DrBqSYyRS5w4gLNFJM1VWg1ITu8sqirrvkiAkZL
         GR0VXgIOb0US6hmjW2HA32GaijeoReXpKwt1cm86ugc3F2MSDOYDNuV0G0B1w2ax/aW8
         GilCb0FUImZDKuW0mMr+DPFLLQgk6psYu8ZgRYBJA7A4V1pbq8PyvI9YJKqKo0O/6w/l
         qVdg==
MIME-Version: 1.0
X-Received: by 10.194.2.70 with SMTP id 6mr13500638wjs.25.1393815732134; Sun,
 02 Mar 2014 19:02:12 -0800 (PST)
Received: by 10.216.9.1 with HTTP; Sun, 2 Mar 2014 19:02:12 -0800 (PST)
Date: Mon, 3 Mar 2014 11:02:12 +0800
Message-ID:
Subject: =?GB2312?B?19TTycPFIM7evec=?=
From: tryu werd
To: undisclosed-recipients:;
Content-Type: multipart/alternative; boundary=047d7b3a817498a7dd04f3ab0383
Bcc: 817@qq.com

--047d7b3a817498a7dd04f3ab0383
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: base64

zbvGxs34wue34sv4tcTGxs34yO28/tfU08nDxSDO3r3ntcTPwtTYtdjWt6GjDQoNCrXnxNSw5iBo
dHRwOi8vd3d3Lndpa2lmb3J0aW8uY29tLzY5NjUzMy8NCrCy17+w5iBodHRwOi8vd3d3Lndpa2lm
b3J0aW8uY29tLzgxODYwOC8NCg0K08PT2rCy17/K1rv6u/LN+MLnu/q2pbrQOg0KaHR0cHM6Ly9n
b28uZ2wvbVpNR3E5DQpodHRwczovL2dvby5nbC9yc05UMUYNCg0K08PT2rXnxNQ6DQpodHRwczov
L2dvby5nbC8zSlRYZw0KaHR0cHM6Ly9nb28uZ2wvMHZLcjMNCg==
--047d7b3a817498a7dd04f3ab0383
Content-Type: text/html; charset=GB2312
Content-Transfer-Encoding: quoted-printable



--047d7b3a817498a7dd04f3ab0383--
NetRange:       74.125.0.0 - 74.125.255.255
CIDR:           74.125.0.0/16
OriginAS:
NetName:        GOOGLE

Apologies to users are Google.

Lavabit - There's a sucker born every minute

P.T. Barnum was a master showman. His name has fallen from public use, but his methods haven't.  You may be familiar with his name as part of the Ringling Bros. and Barnum & Bailey Circus.  He's notably remembered for promoting many hoaxes, and associated with the phrase "There's a sucker born every minute".  It likely wasn't his phrase, but when I see a monkey head sewn to a fish, that's the man I think of.

That's also the phrase that comes to mind when I hear about Lavabit. The company was founded specifically as an answer to Gmail's business model of using your email content to deliver ads.  The idea was to encrypt incoming email so it wasn't viewable. They released a whitepaper touting their encryption, but what it came down to was doing a lot of busy work to locally encrypt and decrypt your email. Everything sent over the wire was plain text - albeit via an encrypted tunnel. The service was billed as "so secure that even our administrators can’t read your e-mail". Unfortunately their users aren't InfoSec experts, and didn't understand this statement to be completely false.

What most users don't understand, and I've run into this when speaking with Lavabit supporters, is that there are basically two types of encryption.  There is encryption at rest, and encryption in transit.  SSL provides encryption in transit, the data sent between two nodes is not viewable by a third party. Lavabit provided encryption at rest.  If you weren't using it, it was encrypted. When you sent or received email, it was decrypted (if read from disk) and sent via SSL.  SSL is essentially a 'tunnel' - anyone on either end can see what's going in and out, when the data is in plain text.  A solution to that problem is PGP - encrypt your data before it's sent, and it will be encrypted at rest and in transit (whether it's via SSL or not).

What's curious is this apparently 'successful service' actually stopped accepting new users in May, 2013. That's very odd. Someone may think that's related to the FBI Probe, but that didn't start until June 10th. Now, I run own and operate VFEmail.net - the subpoena on pages 2 and 3 is nothing out of the ordinary. They always have a 'gag order' attached. The FBI is just as paranoid about the target finding out about the order, as many users are paranoid about the NSA reading their email.

On June 28th, Lavabit received a wiretap order.  This is the 'pen/trap' device. Having received this sort of order in the past, using a 3rd party device IS NOT REQUIRED.  In fact, Page 7 states use of a trap and trace device or process ("pen/trap device")   The ('pen/trap device') is a legal notation to say that from here on, this entire sentence will be referred to as 'pen/trap device' - not that a physical device absolutely will be installed.

At that time, Lavabit could have implemented it's own data duplication system.  An email archive feature, and grepping of the log files is all that's needed to comply.

On June 28th (Page 12), we see The representative of Lavabit indicated that Lavabit had [he technical capability to decrypt the information but that Lavabit did not want to 'defeat [its] system.' . Here Levison admits to being able to decrypt data, but is hiding behind Company Policy to state "We just don't do that".  I dare you to find anyone in InfoSec who thinks Company Policy is a valid security method.

On July 9th, (Page 20) we see that Levison is compelled to appear before the court to explain why he simply can't decrypt email for a single account - or for those of us who know how email works - why mail received via SMTP for/by the target account simply wasn't archived prior to encryption.

On Page 31 we see the actual summons, Levison is to appear in the Virginia court on July 16th - this summons includes the request for the SSL key.  This is because Levison is defying a lawful order to simply archive SMTP communication - which is in clear text on his system.

On July 16th, (Page 34), we see all the dirty details of what the FBI needs to do to accomplish a simple thing that Levison could have done. 

We also see on Page 46 - Levison had caved to this simple request, but demands $2000 to provide the information requested, and then another $1500 after 60 days. We also see that the government has paid for his expenses to travel to VA.
(edit:) Why didn't the FBI pay $3500 for that, and instead opt for the SSL key?  Here's an example of how to use Capsa Professional to save emails. Once that content is saved, it's trivial to grep each text file for the target's address. I'm sure the FBI has software that includes the filtering. This isn't magic or hocus pocus, it's simple network sniffing that anyone with a basic background in InfoSec can do. There's no reason, other than complete lack of familiarity about how it's done, that Levison couldn't provide the requested information at a reasonable reimbursement rate.


Beyond that it's all about the SSL key.  The entire episode was CAUSED by Levison's failure, and flat-out incompetence, to implement a simple SMTP archive feature and then his attempted fleecing of the American taxpayer by charging $2000 to provide that information.

(Edit) Some people have said the 'archive' is very complex to implement or the email would have been encrypted.  Actually, neither are true. First, Levison was already 'up to date' by manually decrypting the mailbox, just new mail needed to be sent in an automated fashion. Second, SMTP is very simple - it's basically 4 commands.  You connect, say HELO, MAIL FROM, RCPT TO, DATA.  What you see in an email all follows DATA. MAIL FROM is only for return mail, and RCPT TO is the destination. A simple archive feature would merely read the RCPT TO, do a text match, and programatically add 'fbi' as another RCPT TO.  For outgoing mail, you just do the match on the AUTH (or MAIL FROM) line instead. Levison claims to have written the entire Lavabit code, so adding this option should have been a trivial exercise.(/Edit)

But it's not over.  He's taken his half of the story, the "NSA is attacking the constitution" to the public. I'm no fan of the NSA, but other than the probable target being Ed Snowden, the NSA violations are in no way related to this case.  Surprisingly, even after admitting to the NYT that the entire issue was caused because he couldn't provide real time archiving, people STILL think he's full of integrity.  Quoting the article - He would log the target’s communications, unscramble them with the encryption keys and upload them to a government server once a day. The F.B.I. told him that was not enough. It needed his target’s communications “in real time,” he said.
If you think that's bad - remember Levison's original version of 'security': "We just don't do that".  Apparently that's a good enough answer for him, but when the FBI requested his SSLKeys, all of a sudden 'we apply filters per policy' is no longer an acceptable answer.  So now, supposedly, everyone's data is at risk.

Levison wasn't able to make $3500 off the FBI, but the gullibility of the public has earned him over $200,000 on Kickstarter, and $100,000 on Rally.  Who knows how much he's taken in on Bitcoin, Paypal or other direct methods.  He may not be much of a tech, but he's good at marketing.

VFEmail has no interest in supporting this person or DarkMail - another marketing ploy to target the non-technical public. If someone does come out with an untainted open source, encrypted, SMTP replacement that still allows AV Scanning, Spam Scanning, and User Filtering on the server - we'd love to hear about it.

Who knows why the media won't cover this angle of the story.  I understand USA Today or NYT not having proper staff to understand tech issues at this level, but Ars?  It's a sad state of affairs.  Frankly, I've thought about shutting down VFEmail because of Lavabit. Not because of 'NSA hacking', or 'Subpoenas for SSLKeys' but because Levison and Silent Circle are trying to manipulate the public into using a solution that won't ever be able to meet their demands. 
 
(Edit 2/24/14) I was subscribed to the Dark Mail Alliance email list - JUST IN CASE they were to release something of substance. Apparently it's nothing more than a marketing list for Silent Circle's products.(/Edit)

If you would like to support me or VFEmail, you may donate to paypal@vfemail.net or via Bitcoin to:

1MgWNPqZqwqhUR38cx2UVJQsDSckpDxYAt 

VFEmail.net Upgrades

VFEmail is in the final stages of a set of whirlwind upgrades.  In the end, we'll have upgraded hardware, software, and a better overall design.

VFEmail also runs Openmail.cc.  These two sites share hardware and core infrastructure, but currently operate mostly independently.  Only the user authentication has been merged.  Webmail is still a separate database and install, though the same webservers are used. See - www.vfemail.net vs openmail.cc

Over the last 10 years VFEmail has slowly grown from a single PC to multiple systems.  We've survived the 'free email' arena by not outspending our competition, we upgrade when it is affordable. 
Now just about each service has a dedicated server. We have:

2 SMTP sending servers
3 incoming MX servers
2 Web Servers
2 DNS Servers
2 Spam Servers
2 SQL Servers
1 NAS
1 'Misc' admin server
1 offsite backup NAS
2 Load Balancing routers in master/slave config

Any servers of which there are at least 2, except the load balancers, are Virtual Machines.  These are backed up to the primary NAS weekly and replicated to the offsite backup NAS.  In the future, we plan to replicate data as it is written to provide even better service.

Each web server points to a VIP internally for SMTP and SQL access.  That allows us to provide 100% failover for either service.   Incoming Web and SMTP are equally load balanced.   At this time POP and IMAP are directly hosted on the NAS to increase response time.  The idea being that since the NAS is not 'hot replicated', if there is an issue it will affect POP/IMAP anyways.  In the future, POP/IMAP will be extracted and load balanced as the backend data is replicated between two hosts.  Other providers do this with DRDB, but VFEmail uses ZFS on the backend, so that's not an option.  ZFS does provide for regular snapshots and easy replication, backups, and restores -but not live replication.   AVS + ZFS would provide that, but I was never able to get that functional, so I'm keeping a close eye on FileReplication Pro as a possible alternative.

Next up, new User Control Panel, Horde 5.1, and Roundcube upgrades.

FreePBX/Asterisk Failover using IAX2 and bandwidth.com

I have a site that REQUIRES zero dropped calls.  These calls are sales, and bring in revenue. If the office cannot be contacted, not only could we lose that referral, but the referer may not bother to send more sales our way.
I am offsite.
In fact, I'm out of state. Too far to quickly physically resolve an issue.


I've been using Asterisk for almost 10 years, and it just keeps getting better.  I used it to failover a dieing ACD via T1 crossover cable, and now I've had the opportunity to do it with a pure Asterisk solution.

Detail -
Bandwidth.com provides SIP trunks for $25/month/line.  They use SRV records to provide high availability.  SRV records work like MX records.  Higher priority hosts get traffic first.  My site has an afterhours answering service, so the first step was to create an Amazon Cloud instance of FreePBX and two SRV records - one record for the office Asterisk machine, and the second for the Amazon Cloud one.  The Amazon one only forwarded to the answering service.
This ensured us that no matter what occurred, all calls would be answered.

Unfortunately after a bouncing circuit caused complete havoc with our phones - calls were dropping /cutting in and out, and I wasn't able to remotely access the PBX to shut the trunk down and force fail over - we realized the Amazon Cloud solution wasn't enough.  These referrals are time sensitive.  If we don't have a way to reach our email to retrieve the answering service notification, and call the customer in a very short amount of time, then the sale is still lost.  The only real solution is 100% uptime during business hours.

Going All Out
I procured a CradlePoint MBR1400 firewall/router with Verizon 4G modem and static IP service from Verizon.  The CradlePoint has the capability of doing failover between interfaces. For example, if your cable provider goes out, it can automatically switch to the 4G service.  That's great for web browsing, but doesn't work so well for SIP trunks.  Asterisk has to know it's external IP Address, and I'm not sure if STUN servers work for SIP trunks.  If the external IP is incorrect in Asterisk, we lose the audio stream.  Therefore I hard code it.  In addition, using the 4G modem allows us to be sure a fibre cut in the street doesn't affect us.

Once the CradlePoint was installed and tested on the PBX VLAN, I installed a 2nd instance of FreePBX on the same VLAN, and added my bandwidth.com trunks.
The BackupPBX uses the CradlePoint as the default route, and the NAT settings are set to the Verizon static IP.  
Then I created an IAX2 trunk between the primary and failover PBXs:

On the Backup PBX -
Trunk Name:  PrimaryPBX
PEER Details:
username=BackupPBX
secret=password
host=172.16.200.12
type=friend
context=from-internal
qualify=yes
qualifyfreqok=25000
transfer=no
trunk=yes
forceencryption=yes
encryption=yes
auth=md5

On the Primary PBX -
Trunk Name: BackupPBX
PEER Details:
username=PrimaryPBX
secret=password
host=172.16.200.18
type=friend
context=from-pstn
qualify=yes
qualifyfreqok=25000
transfer=no
trunk=yes
forceencryption=yes
encryption=yes
auth=md5

Note the context on the BackupPBX Trunk, from-pstn.  This is because we want to push pstn sourced calls from the Backup into the Primary.
Also, if you use a dial prefix, make sure you add that to the Trunk settings on the Primary.  Likely, you may strip a '9' from the beginning of the dial string normally, so you would need to add it back in when sending the call via the backup PBX.  

Then on the backup, create an IncomingRoute with the destination 'Trunk' 'PrimaryPBX' and create an OutgoingRoute that matches your Outgoing Route settings on the Primary.

On the Primary, add the BackupPBX Trunk as the last Trunk in your Outgoing Route.

Now we have a complete backup.  If the Internet were to go down, the SRV record will take care of incoming call fail over, and the BackupPBX Trunk/route will take care of outgoing call fail over.  Completely seamless to the user.
Obviously this doesn't cover a power outage, but due to the nature of the business, most of these sales are local and if there is a major power issue that would exceed the UPS times, then nobody is making those calls anyways.  The Amazon Cloud PBX will be shutdown, and I will run a '3rd tier backup' PBX in a VM on my local servers for the answering service.

Adding an internal root/CA Cert to Firefox

I swear I added a CA cert to Firefox 1.5 via Netware back in the day, but the more I think about it, the more I recall Netware's application deployment was far better than Microsoft's.  In any case I've found a lot of info and some bad scripts, which I'm compiling and fixing for your benefit.

I have to add that batch file writing skills are sorely lacking these days.  Wow.  A lot of copy/pasting without really understanding what basic variables are.

If you're starting from scratch, you'll probably just want the ESR release of Firefox, then the CCK Wizard to add your cert and any other changes and create an XPI Extension.  Then goto FrontMotion and throw it all together in an MSI.

Firefox uses it's own certificate store.  The reason being is that Mozilla has stricter requirements than Microsoft does when it comes to allowing Root certificates to be deployed.  Ok.  I'll buy that. Mozilla also argues that Windows Update can deploy certificates without user's warning.  Hrm... I don't really buy that outside of home use (where a user would have no clue ANYWAYS).
Typically in a Corporate environment updates are controlled and pushed to the end user.  We're already running Microsoft software, I don't quite see any additional harm in allowing Microsoft to push new certs as well.  In fact, I've been on a Security Team that was blamed for certificate issues in a large company when the Windows team held back a certificate update, and everyone's root expired :/

In any case, in Firefox we're stuck with manually updating the certificate store on every instance. At least you can use FrontMotion to install and update Firefox (and the latest Flash).  Supposedly FrontMotion will install a Base64 cert if you create your own package, but that only worked once for me and while the cert was imported it wasn't trusted for anything :/

In order to distribute new/internal root certs to your users, you need to gather some tools and write a batch file.
 First -  certutil.zip  This is the exe's and dll's needed to modify the Mozilla certifiate store
 Second -  create your batch file:

if not exist k:\ exit
 Set CERTDIR=k:\gpinstall\certutil
REM Set CERTDIR=c:\Temp\CertImport
Set FFProfdir=%Appdata%\mozilla\firefox\profiles
DIR /A:D /B %FFProfdir% > "%Temp%\FFProfile.txt"
 
FOR /F "tokens=*" %%i in (%Temp%\FFProfile.txt) do (
 
CD /d "%FFProfDir%\%%i"
COPY cert8.db cert8.db.orig /y
%CertDir%\certutil.exe -A -n "Cert1Name" -i "%CertDir%\Cert1.crt" -t "TCu,TCu,TCu" -d .

%CertDir%\certutil.exe -A -n "Cert2Name" -i "%CertDir%\Cert2.crt" -t "TCu,TCu,TCu" -d .
)
DEL /f /q "%Temp%\FFProfile.txt"


Make sure CERTDIR is set to a mapped share location.  It should contain the contents of certutil.zip, your certificate(s) and your batch file. Temporarily map that share if you need to, or get more creative and copy all the files from a share location to a local temp folder.  I leave that exercise to the reader.

Make sure "Cert1.crt" refers to your actual certificate's file name, and "Cert1" is the real certificate name.  If you are unsure of the certificate name, manually import the certificate into Firefox and find it under Tools, Options, Encryption.

That's it.  Run your batch file, and it should import.  Add it to your login script to import your root cert for everyone.

Enabling Exchange subaddressing or extension addresses

Another shortcoming with Exchange - it doesn't handle subaddressing.  (you know, name-this@domain.com or name+this@domain.com) So I have give all these slimeballs my 'real' email address with no way to track or filter out that particular slimeball.   My solution is to rewrite the address in Qmail, and put the subaddress in the Subject before forwarding to Exchange.

Since I already front-end my Exchange server with a qmail server for SpamAssassin, enabling support for subaddressing was pretty straightfoward.  You may want to follow a toaster or Life With Qmail to get a basic working install with procmail included.

Assuming you have a working install -

First, you must create a 2nd qmail install if you haven't already.  The reason for this is that to keep things from being too confusing, we'll need to deliver our domain mail locally and then re-forward to the same domain.  If you try and forward within the same qmail structure, you'll end up in a loop.

Again, I already have a 2nd qmail install.  The reason for this is to normal allow outbound mail to have a short queuelifetime (a couple hours), but provide longer lifetimes to other domains if necessary.  Here, qmail-smtpd listens on localhost, and control/smtproutes just directs that domain to localhost to inject mail into the 2nd queue.

Second, create the file structures:
/usr/local/domains/bin
/usr/local/domains/domain.com

Third, create a user.  If you do not have a users/assign file, you'll need to create one in the format of:

+domain.com-:domain.com:64012:64010:/usr/local/domains/domain.com:-::
.

Now run /var/qmail/bin/qmail-newu to 'activate' the user.
Edit /var/qmail/control/virtualdomains and add:
domain.com:domain.com

This will instruct qmail-send (after we HUP it) to deliver all mail for domain.com to /usr/local/domains/domain.com .   In addition, it will use the uid/gid of the qmail user/group.  Make sure those are correct for your system.

Fourth, create a .qmail-default in your /usr/local/domains/domain.com directory:
 |/usr/local/domains/bin/filter.sh
That's a script that will call our procmail script to rewrite the To: and Subject: lines.
So put the following into /usr/local/domains/bin/filter.sh - and make it executable.
#!/bin/sh
`/var/qmail/bin/preline /usr/bin/procmail -p -m //usr/local/domains/bin/procmail.rc`
EXITCODE=$?
cat > /dev/null
if [ $EXITCODE -ne 0 ]; then
       exit 0
else
       #Exited with 0, delivered, so set 99 to stop vdelivermail
        exit 99
fi


As you can see, it calls for /usr/local/domains/bin/procmail.rc. Your procmail.rc file should look like:

SHELL="/bin/sh"
# If you need to debug your configuration of ackmail.rc, just "touch
# ackmail.rc.log" and the recipe below will log the ackmail.rc activity
# to ackmail.rc.log.  When you are satisfied with your configuration,
# simply remove ackmail.rc.log and logging will stop.
# http://www.herring.org/tips/procmail/ackmail.rc
:0
* ? test -f /var/log/procmail.log
{
   LOGFILE=/var/log/procmail.log
   VERBOSE=yes
   LOGABSTRACT=all
}

###Verify EXT User is not an alias
##Will be changed from user-ext@/user-ext-ext to user@
## Assuming no user-user usernames
:0
* EXT ?? -
{
  EXT=`echo $EXT $HOST | awk -F- '{print $1}'|awk '{print $1}'`
  #SUBJECT: our variable with the original subject, for writing the new subject
  SUBJECT=`/usr/bin/formail -zxSubject:`

  :0 f
  | /usr/bin/formail -I "Subject: [$EXT2] $SUBJECT"
}

:0w
| /var/qmail2/bin/forward $EXT@$HOST 
 Bascially, the procmail.rc only touches emails with '-' in the To: ($EXT) address, and only runs formail to modify the subject when necessary.  $EXT2 represents everything after the first EXT up to the @.
So username-ibm@domain.com turns into domain.com-username-ibm@domain.com (funky, but that's how the routing works), which creates the following qmail variables:
$USER-$EXT-$EXT2-$EXT3-$EXT4@$HOST
NOTE: The qmail2 at the end of the file.  This is your 2nd qmail install with an smtproute to your Exchange server for your domain.   If you use the same qmail install as the forward destination, the mail will just loop.


 Just for completeness, here's the smtproute.  I suggest you ensure this is working on your first email install prior to these changes:
smtproutes:
domain.com:192.168.1.100 

 That's it, restart qmail-send (via svc -t /service/send if you used daemontools) or use whatever method you want to send qmail-send a HUP signal to re-read the config files.  
 

 

Restarting Exchange

For whatever reason my new Exchange 2010 server does not properly update everyone's offline addressbook in Outlook.  I've given up trying to fix it.  Here's a batch file to restart all the services, so I don't have to reboot the entire damn thing.

@Echo Off
Echo 'Stopping Microsoft Exchange Services'
net stop MSExchangeAB
net stop MSExchangeADTopology
net stop MSExchangeAntispamUpdate
net stop MSExchangeEdgeSync
net stop MSExchangeFBA
net stop MSExchangeFDS
net stop MSExchangeIS
net stop MSExchangeMailboxAssistants
net stop MSExchangeMailboxReplication
net stop MSExchangeMailSubmission
net stop MSExchangeProtectedServiceHost
net stop MSExchangeRepl
net stop MSExchangeRPC
net stop MSExchangeSA
net stop MSExchangeSearch
net stop MSExchangeServiceHost
net stop MSExchangeThrottling
net stop MSExchangeTransport
net stop MSExchangeTransportLogSearch
Echo 'Starting Microsoft Exchange Services'
net start MSExchangeAB
net start MSExchangeADTopology
net start MSExchangeAntispamUpdate
net start MSExchangeEdgeSync
net start MSExchangeFBA
net start MSExchangeFDS
net start MSExchangeIS
net start MSExchangeMailboxAssistants
net start MSExchangeMailboxReplication
net start MSExchangeMailSubmission
net start MSExchangeProtectedServiceHost
net start MSExchangeRepl
net start MSExchangeRPC
net start MSExchangeSA
net start MSExchangeSearch
net start MSExchangeServiceHost
net start MSExchangeThrottling
net start MSExchangeTransport
net start MSExchangeTransportLogSearch
net start msexchangepop3
net start msexchangeimap4
End