“Aye, fight and you may die. Run, and you’ll live… at least a while. And dying in your beds, many years from now, would you be willin’ to trade ALL the days, from this day to that, for one chance, just one chance, to come back here and tell our enemies that they may take our lives, but they’ll never take… OUR FREEDOM!” — William Wallace, on the traditional 9 to 5 office hour orthodoxy versus emergent ROWE workplace policies.

What’s this all about? See our past articles on ROWE.

Here at Centresource, we have worked hard to continually be improving. You are either moving forward or backward in business, and we’ve made it a goal to continually be progressing and pushing boundaries. This year we could not be more proud to have our CEO, Evan Owens, nominated for Entrepreneur of the Year in the Technology category for the 2012 NEXT Awards. Since his arrival at Centresource, he has steadily grown the reputation of both Nashville and our organization. Most recently he’s led us to break new ground on the launch of three new business units dedicated to Healthcare, Startups and Non-Profits.

“The entire Centresource team is to thank for this nomination… without them we would not be innovating,” Owens commented. “Not only have we added jobs in Nashville, we’ve increased the magnetism of the Nashville Startup and Technology community. I couldn’t be more humbled and honored to be nominated for this award and know Centresource will continue to take risks and break new ground.”

This award nomination comes only weeks after an announcement that Evan was nominated by the Nashville Business Journal as one of Nashville’s Most Admired CEOs. So the question is…will he sweep the series? We sure hope so :)

Congrats to Evan, and on behalf of the Centresource team, we thank him for his leadership.

So what’s at the center of this debate? A cadre of Nashville’s more technically proficient programmer/engineer types feel that Nashville’s Barcamp is overwhelmed by “non-technical” people: entrepreneurs, business-owners, and self-proclaimed social media experts that have organized the event such that it’s hardly in keeping with the original spirit of Barcamp. In a nutshell: too many business cards, not enough neckbeards. More specifically, as Rick Bradley put it:

< @rickbradley> @cwage my (and others’) annual complaints would be silenced by (a) changing the name or (b) following the rules: http://t.co/TGRaeDm

Similarly, @BCN_Critic (an account created on twitter specifically to voice complaints about Barcamp Nashville) objects to the rigid scheduling coupled with the random session selection, among other things.

To be honest, I can understand this sentiment. When you read about the history and make-up of the original barcamps, Nashville’s event bears little to no resemblance. But Nashville has never resembled other cities. Nashville is not Palo Alto, nor Portland. In all likelihood if we “followed the rules” to maintain the true barcamp spirit, Nashville wouldn’t have a barcamp at all, because it would have never happened: the demand (and resources) weren’t there. We didn’t have the pool of nerderiffic talent in Nashville to pull it off — certainly not in 2007, and arguably not now. Nashville does have “open, participatory workshop-events” suitable for the level we’re at: it’s called JJ’s Market on Thursday afternoons.

To their credit, Barcamp Nashville has never pretended it was anything more than what it is: a uniquely Nashville event. The description on the website says it plainly enough, describing BCN as “new-media focused”. It’s very heavy on the social media aspect, yes, but there’s plenty of technical meat. Last year I saw a very good session covering the basics of Arduino. Your average ruby hacker might think banging out WordPress sites is beneath them, but it’s not non-technical. The worst offense you can justifiably level at Barcamp Nashville is that they’ve co-opted the name. Oh well. That which we call a rose by any other name would smell as sweet.

Nothing is stopping the neckbeards (I can say that because I have one) from creating their own event truer to the original spirit of barcamps. I hardly think anyone involved with BCN would mind — on the contrary, I think they’d encourage it. The people involved with Barcamp might be “marketroids” or self-proclaimed social media mavens, but they understand the immense value of Nashville’s nerdier contingent. Frankly, I think it’s time for the Barcamp-haters to realize the converse is also true: marketing, social media and entrepreneurship might not be your bag, but that doesn’t mean they’re worthless. We need them. For every Steve Wozniak, there’s a Steve Jobs. These are the people that will keep interest piqued, businesses running, and investment capital flowing into the city while we nerds are too busy riding out 72-hour mountain-dew fueled hackfests. The ferocity of the criticism and eye-rolling directed at Barcamp and many other Nashville technology-related institutions (NTC, JSF, etc.) is undeserved, and it’s not cool. It’s easy to complain every year — what’s harder is to contribute or create a viable alternative. Don’t like Barcamp Nashville? Last time I checked, the organizational meetings are open to the public.

Don’t hate. Participate.

Thoughts?

There’s a new tool on the block called BlackSheep. It’s pretty clever — it’s a modified version of the firesheep codebase that makes fake HTTP requests of the sort that Firesheep would normally intercept and hijack, and detects the subsequent hijacking attempts. It won’t do anything to prevent the attack, but it will help verify your suspicions of the nerdy guy with the thinkgeek t-shirt in the back of the coffee shop who hasn’t ordered anything in a while.

First, a quick primer on HTTP session hijacking — the fundamental attack that firesheep uses:

Any website that requires authentication typically does so with a simple username and password. Most social networking sites (i.e. facebook, which for some reason has garnered most of the attention regarding this flaw) are no exception, and they configure their site so that your username and password is submitted over an encrypted (HTTPS) connection. In this way, an intruder snooping on the network would not be able to intercept the username and password. However, once you are authenticated to the website, many sites (facebook included), continue to use unencrypted HTTP, which means that anyone snooping can intercept that data — including something called session “cookies”. Cookies are little snippets of information that the web browser allows the webserver to store locally and resend with subsequent connections. In this way, facebook, for example, can store a cookie saying “this browser is logged in as chris wage”. This is convenient, because it allows you to use facebook without providing your username and password with every subsequent connection and/or reload of the page.

Unfortunately, however, if these “cookies” are sent in the clear (unencrypted), they can be intercepted — and that is precisely the case with facebook and many other websites. Once someone intercepts the authentication cookie, they can resend it to the targetted website and appear to be logged in as that user, with full access to everything.

HTTP Session Hijacking by itself is nothing new — you’ve been able to do it with a modicum of simple tools like tcpdump and wget for years. What firesheep has done is package the attack into a startlingly convenient extension for firefox. Simple clicking the “start” button on a sidebar and you start building a nice list of people/accounts and websites. Click the entry in the list and you get a tab with that site, logged in as them. It’s pretty insidious — and by making the exploit available to the lay-person, it has brought the insecurity of plaintext HTTP servers back to the limelight where it belongs.

Of course, if you, dear reader, are anything like us here at Centresource, you probably read about firesheep, thought, “whoa.. I wonder if it works?” and immediately downloaded it to try it. Many people are downplaying the significance of the threat because they downloaded firesheep, tried it, and failed to get any results other than their own web sessions. The reason for this, without getting too technical, boils down to the fact that out of the box, firesheep will only work on open, unencrypted wireless networks. The saving grace of wired networks is that they are (usually) switched — meaning that your workstation doesn’t see traffic with other workstations unless it’s required — and it’s certainly not for someone else logging in to facebook. So, the firesheep extension can sniff all it wants, but it won’t see anything but your own traffic.

If you think this means you’re safe from it, though, think again. I won’t get into the details here, but there are very simple (frighteningly simple) attacks against any switched network that overcomes this obstacle and makes anyone on the entire network vulnerable to this attack — provided they aren’t using SSL.

Which brings me to the last question you probably have: how do I protect myself from this? If you’re of the nerdy persuasion, you can opt to route all of your browser’s traffic through an encrypted proxy. There are plenty of HOWTOs on various ways to accomplish this: SSH + Squid, for example. This isn’t a perfect solution, because your traffic still leaves the endpoint of your tunnel and heads to facebook.com unencrypted — but you can probably trust the network at a datacenter more than your local coffee shop.

For the rest of us, we’re left with relying on our various websites to provide HTTPS. What’s relatively unknown is that facebook (and many other sites) actually do provide HTTPS — they simple don’t redirect you to it by default. If you open your browser and go to https://www.facebook.com/, your connection will be encrypted. The downside is that it’s somewhat slower, and also that it’s hard to remember. Fortunately, for the last inconvenience, there are remedies in the form of browser extensions/plugins that auto-detect and force HTTPS communication if it exists: ForceTLS for Firefox and KB SSL Enforcer for Chrome, among others.

So go forth, install, and browse — safe in the confidence that your pokes and farmville … farmings .. are safe and secure, hidden away from the prying eyes of your nosey coworkers.

1. The first step is to consider the nature of your website. Is it merely a “static” HTML/CSS website with no dynamic data? If not, does it use a database of any kind? How/where does that database store its data? Lastly, does your website include the storage of a large number of files (multimedia, documents, etc)? And if so, are these crucial to the operation of your website?
2. Next, how much data can you afford to lose? Of course, no one likes to think or talk about losing data, but any solution is a compromise between resources you’re willing to pay for and the amount of data that’s guaranteed protection. This “amount” is typically/best measured in time. Any backup solution will capture “snapshots” of your website — copies of the data at a certain point in time — that will vary anywhere from “realtime” to hours or days since the last snapshot was made. If you cannot afford to lose any data, then a realtime backup solution is the only option. These solutions will do their best to use a program running non-stop in the background making a backup (somewhere) of your data. If the answer is “some” — then you just make a determination of what your acceptable loss is. Is it a day? A week? 6 hours?
3. The second “time” factor is the time-to-restore. How much time is acceptable to go from a catastrophic failure to a complete restore of your website functionality? This time is determined by many things in addition to the actual type of backup you have, including the type of webhosting you have, and the form of the failure. If you host your website on a VPS (“Virtual Private Server”), for example — and this VPS was backed-up in its entirity on a regular basis — then restoring is as simple as copying over the VPS backup and restarting it to that point. If your website has a more complicated architecture (multiple frontend webservers and multiple databases, for example), then the restore process is likely to be a more lengthy ordeal of reconfiguring the servers ahead of time before any data is put back in place.
4. Where is the data backed up? It should be obvious that the best backup is to a location that is physically separate from where your website is hosted. This is not an absolute (in fact it’s quite relative — is another server rack in the same datacenter “off-site”? What about a different datacenter owned by the same company and on the same network?), but in general you should attempt to distance your data from the potential points of failure: fires, tornadoes, hosting company incompetence, nuclear strikes — you name it.
5. Can you test the restore functionality regularly? This is a factor of cost, to some extent, and also practicality. If you opted for any backup solution other than “realtime”, you’ve decided on an acceptable level of data (time) loss — so testing a full restore right to production is not advisable, since you’d lose data. But working with your hosting/backup vendor to verify the backup, and maybe test a restore to another location are things you should pursue, if possible.

These are all important questions that you need to ask of yourself, and also of your web hosting vendor and your web developer. Together, you can decide on a backup solution that makes sense for your website. That said, what kind of solutions are there? This is not a comprehensive list, but it’s a general summary of the types of ways you can guarantee continuity of your website. Interestingly, computing/server technology has come a long way in the last couple of years. Virtualization and cloud competing have blurred the lines between continuity, load-balancing, and backup. After all, if your website is running on a virtual server in the cloud, data loss is impossible, right? Well, no, not really.

• VPS (Virtual Private Servers) – as mentioned above, a VPS can be a good way of assuring a very quick time from hardware failure to restoration. A VPS is a “server” that is entirely self-contained and run in a “virtual” environment. This means that your server itself is essentially a file or files that can be copied (regularly, or realtime) to any other backup medium for later restoration. Typically, most hosting providers that offer hosting on a VPS will also offer a backup service that makes restoration of your VPS a cinch. The above questions, however, are still quite pertinent and should be asked: How often do they backup the VPS? How quickly can they restore? Some hosting vendors host their VPS on a “cloud” server architecture which makes hardware failure a nearly negligible factor. However, you should still consider how much you trust the competence of your hosting vendor and explore the option of backing up your data somewhere else independent of the VPS itself. Many other things can go wrong — virtualization software failure, data corruption, etc. These things can all leave your virtual server inoperable or unsalvageable.
• Backup “daemons” – “daemon” is computer nerd slang for any software that runs non-stop in the background. There are many common backup solutions that use a daemon running in the background to persistently backup your website and its data. Typically, this sort of solution is what’s required for a “realtime” backup of your data. Most hosting vendors that host larger dedicated managed server environments will have some offering in this regard.
• Scheduled backup jobs – there are a wide variety of ways this can be implemented, but loosely speaking, this is any sort of backup script/program that is merely scheduled on a regular basis (say, nightly) and involves a manual copy of the website data (files and database dump) to an arbitrary off-site location.

It’s very likely that any hosting vendor you choose will have one or more of the above options — and possibly others. But it’s not enough to simply sign-off on an arbitrary product without understanding what it does and what it guarantees. Hopefully, armed with the above questions and information, you can make the right decision about how to backup your website. In an age where a website is increasingly the face of your business, you can’t afford to leave it to chance.

It didn’t always get it 100% right, but even if it didn’t, it was still a convenient shortcut and it would get a majority of the info right. It was also amazingly good at picking “events” out of e-mail — either casual e-mail from a person or automated/invite type messages.

Then one day, it disappeared. Poof. Gone. Has anyone else noticed this? Does it still exist for anyone else out there? Is it unique to traditional gmail versus google apps?

Google, please bring this back, or I swear on the Golden Kraut I’ll have my revenge!

Recently, this customer began reporting that she was unable to receive e-mail from certain people. Some investigation yielded this information from Swirbo, while attempting to deliver a legit e-mail message from someone on aol.com:

Jun 14 10:50:10 mta1 postfix/smtp[5285]: 90D0D834038: to=, relay=redacteddomain.com[1.2.3.4], delay=3, status=bounced (host redacteddomain.com[1.2.3.4] said: 550 SITEGROUND Faked AOL, so you must be spam. (in reply to RCPT TO command))

This error message was not exactly clear, so I was forced to contact Siteground to try to determine why they were blocking this mail. Unfortunately, Siteground has no support number to call, or e-mail address to e-mail. So, I attempted to file a ticket, however even this requires 30 minutes of trudging/fighting through obscure attempts to deflect actual contact with support via their “knowledge base” and a barely-functionining java applet that tries to detect obvious problems first. Once I made it through this and obtained the customer’s password (which is required for some reason to even file a ticket), I managed to file a ticket asking them for an explanation.

This was their response, after which they immediately closed the ticket (which I can’t re-open to respond):

I carefully investigated your case and I noticed that your domain is using an mail exchange server as it’s MX records point to

redacteddomain.com. 14400 IN MX 10 mta1.swirbo.net.
redacteddomain.com. 14400 IN MX 10 mta2.swirbo.net.

Due to this fact the email messages are coming to our server from email address redacteduser@aol.com and hostname mta2.swirbo.net. due to the fact that mta2.swirbo.net. is relaying them to us.

Our spam prevention filters are checking if the email address domain and the host which the emails come from are the same and if they find a mismatch the email is rejected with the message that it must be fake mail.

My advice is to contact your mail exchanger support team and present them with this information in order to solve the issue more effectively.

Please do not hesitate to contact us if you have any further questions or comments.

Best Regards,

Andrey
Support Team
SiteGround.com

So, aside from their hideous customer service and refusal to give me a chance to respond to a support inquiry, we have what appears to be a very braindead attempt at spam-filtering by Siteground. What they seem to be saying is that when they receive an SMTP connection, they check the reverse-DNS for the IP, and if the domain name in the resulting A record doesn’t match the sender’s domain, they reject it. This makes no sense, and contravenes any number of standard practices involving SMTP relays. I’m assuming they only do this for certain larger domains (AOL, Yahoo, etc.), otherwise it’s hard to imagine how their customers get any mail at all. The potential here for false-positives is huge.

Similar problems as this emerge with SPF, but this is above and beyond even that.

Unfortunately, due to Siteground’s complete lack of support, broken e-mail, and inability to address this problem, we’ll probably be migrating what few accounts we have with them somewhere else.

Thursday 6/07

4PM — Our comcast cable goes down hard. We reboot the modem and lights sync up fine, but we have no access to the internet. One of our employees called Comcast and they dispatched for the following morning (Friday 6/08)

Friday morning 6/08

10AM — Comcast tech arrives on-site. He checks the signal and it’s fine, and then proceeds to replace the old SMC modem with a Netgear “business IP gateway”. Unfortunately this didn’t get us back up.

11AM — After an hour or so of phone calls, he managed to get a laptop online via NAT from the router. This is good because it means whatever originally went wrong has been fixed, but he had no explanation for what originally caused it.

11:05AM — Comcast tech looks satisfied and starts hinting that we’re good to go and I ask about the static IP configuration and remind him that we’re down until we get the static IP configuration. He clearly has no idea what this even means, so I have to convince him that he’s not done, since he was arguing at first that “the static IP is something y’all have to do, we don’t do that”.

11:30AM — After I convince him that we need the static IP (i nearly whiteboarded a network diagram to try to explain it to him, but he had no TCP/IP/routing/networking knowledge at all whatsoever), he called 2 different Comcast numbers, eventually getting someone. He wouldn’t let me just talk to them, so I had to relay word-for-word what I needed, which he relayed to them: “They.. they’re saying they .. they need a static.. static? right, static IP? a static IP configuration?”

11:40AM — After finally relaying this to the woman he was talking to, she informs him that “We don’t provide the static IP — that’s something the client has to provide..” As politely as I could, I let them know that I don’t even know what that means. We pay for a static IP. They give us the IP, route it, and we assign it to our firewall. Pretty simple. We’ve had it that way for 2 years.

12:30PM — Finally, they get the static IP config built and pushed to the new modem. At long last, we’re back up and running. I pack up and head home.

2:30PM — I get alerts from Nagios that our Comcast has gone down. Again. I venture into the office to check on it, and verify via troubleshooting that it appears our new modem has simply “lost” the static IP configuration again. (I know this because I can get online via NAT through the modem itself). I call comcast support, and an hour later, I finally get to a tech that knows what I am talking about, and they get the static IP re-pushed. I head home again.

3:30PM — I get alerts from Nagios that our Comcast has gone down again. I head back into the office. Same thing. I call Comcast, same song and dance. This time, I again ask what could be causing this that the modem would simply “lose” its config so often. He doesn’t seem to know, and asks if anyone “hard reset” the modem. No progress on that front, but he got us back up and running again.

4:45PM — I get alerts from Nagios that our connection has gone down again. This time, however, just as I arrive at the office, it comes back up. It appears that this perhaps was just a “normal” outage. (The fact that I consider outages like these to be “normal” is an indication of the quality of Comcast’s service.)

Saturday and Sunday (6/09 and 6/10)

During these two weekend days, our cable went up and down around 3 or 4 times, for 15-20 minutes at a time. Though it concerned me, it never does any good to call Comcast support to “let them know” that it went down, so if it comes back up at all, we consider ourselves lucky.

Monday (6/10)

Thinking that everything seems peachy keen, our employees arrived at the office fresh and ready for a productive day’s work for a change. Unfortunately, looks can be deceiving. Not long into the day, we started to realize that something is wrong — the re-emergence of a problem we had around 6 months ago that subsequently mysteriously disappeared.

The problem was that TCP sessions would timeout during inactivity. I knew this wasn’t an issue with our firewall, because of the following reasons:

• Our firewall config hasn’t changed or been touched substantially in over 2 years.
• The problem came and went (usually corresponding with Comcast needing to re-push the static IP config) while no changes were made to our firewall
• While the problem is occurring, I could still see a connection in the pf state table (our firewall is OpenBSD running on a Soekris box).
• I was able to reproduce the problem by setting up a laptop with the static IP and plugging it straight into the modem.

The problem was tolerable in the past, but this time it was much worse — connections were timing out after around 5 minutes of inactivity. This doesn’t sound like a big deal, but when you consider the amount of traffic we have that requires a persistent TCP connection, and how poorly some software deals with timeouts (*cough*Outlook*cough*), you can imagine how rough things were for us. Instant Messaging (AOL, MSN or Jabber) would stop working. E-mail (IMAP) would stop working. FTP control connections would stop working, etc.

Making matters worse, from a technical perspective, was that these connections didn’t merely “time out” — there was no RST or FIN packets sent to terminate the connection. It just disappeared. Traffic goes out, nothing comes back.

I will spare you the blow-by-blow of the troubleshooting for this problem, but you can probably imagine that a company with tech support that can barely understand and support the basic services they provide (i.e. our static IP configuration problems), getting them to understand a slightly esoteric (but devestating) TCP problem was a nightmare. Attempts were made to deflect the issue back to us were made at every turn. Every call was a new call — no one was able to stay familiar with the problem, and no one every seemed to be working on the issue. I eventually managed to get is escalated to “tier two”, who then said he was forwarding it on to their networking team, but I never heard anything from them ever again.

The only resolution came, a week later when I finally got a tier 1 tech to dispatch someone to try replacing the Netgear modem with an SMC modem similar to the one we originally had. This was scheduled for Thursday (6/13) between 10-12, unfortunately no one ever showed up. I got a phone call at 5PM from a dispatch tech saying he “had the wrong modem, could we reschedule?” So, Friday comes, and finally, around noon, he shows up with the modem and replaces it. After the usual phone-call song-and-dance he gets a tech that can push the config, and we’re back up and runing. Problem solved — no more timeouts. I don’t know if it’s a configuration issue, or if it’s an issue with Netgear, or what.

I do know that Comcast surely doesn’t know either — but what’s worse, is they don’t care. Our entire company was forced to work at home for a grand total of 7 business days due to these issues. To this date, no one has ever followed up on the original ticket(s) we filed about either issue to ask if it was resolved. I’m afraid to call, because as often as Comcast tech support fixes something, they break something else.

The important thing to realize about this story is that it’s not unique. It’s not a one-off experience. We’ve faced these difficulties every step of the way with Comcast, in multiple locations. They advertise their “business cable” as a business-grade Internet connection. It’s not. You get more bandwidth, but that’s it. No SLAs. No accountability. No customer service. Nothing.

Further, what I’ve learned about Comcast is that they are truly too big for their own good. They have so many different departments that even their employees don’t seem to understand it all. Dispatched techs often make 2-3 phone calls before they even get the right person. Tech support never knows whether or not someone is actually being dispatched. If you’ve ever heard the expression “the left hand isn’t talking to the right”, that perfectly describes Comcast, except Comcast is a multi-armed Shiva, and none of the hands are talking to eachother. At one point, a tier-1 tech actually transferred me to Gateway. No joke. I wish I could make this up. He said that “he thinks the problem is with our internet gateway, so he needs to transfer me to Gateway @home support” (or something like that). Next thing you know, I was on hold with Gateway computers. Naturally, I had to hang up and try again.

I know that after I post this, I’m sure I’ll get the standard “well, we have Comcast and it works fine!” comments. That’s great. I’m not saying there’s anything wrong with cable technology. It’s wonderful. The problem is with Comcast’s business practices and size. If you have no problems, generally, you’re good to go. But good luck if you ever have any problems. We’re moving to a new building this fall, and I can assure you, I won’t even remotely consider Comcast as an option. The ramifications of this is that our ISP cost will easily double or triple. Painful, right? Not really. Not when you factor in the cost of the downtime we’ve suffered at the hands of Comcast. It’s not a business-grade ISP. Period. It’s a no-brainer for us.

Technical details of permanent failure:
TEMP_FAILURE: SMTP Error (state 13): 450 : Recipient
address rejected: Greylisted for 5 minutes

Anyone see a problem? The error we returned was 450, yet Google seems to think it was a permanent failure. Here’s a bit from the SMTP RFC (2821):

4yz Transient Negative Completion reply
The command was not accepted, and the requested action did not
occur. However, the error condition is temporary and the action
may be requested again. The sender should return to the beginning
of the command sequence (if any). It is difficult to assign a
meaning to “transient” when two different sites (receiver- and
sender-SMTP agents) must agree on the interpretation. Each reply
in this category might have a different time value, but the SMTP
client is encouraged to try again. A rule of thumb to determine
whether a reply fits into the 4yz or the 5yz category (see below)
is that replies are 4yz if they can be successful if repeated
without any change in command form or in properties of the sender
or receiver (that is, the command is repeated identically and the
receiver does not put up a new implementation.)

SMTP 4xx errors are temporary and typically represent transient errors. An SMTP client, upon receiving an error like this, should (i.e. is encouraged to) try again. Greylisting is a spam-filtering technique that takes advantage of this fact by issuing temporary rejections to new clients, in effect filtering out mail sent by software not smart enough to retry (as any proper mailserver would). Google should be queueing and retrying these messages. Anyone else running into this?