1. The first step is to consider the nature of your website. Is it merely a “static” HTML/CSS website with no dynamic data? If not, does it use a database of any kind? How/where does that database store its data? Lastly, does your website include the storage of a large number of files (multimedia, documents, etc)? And if so, are these crucial to the operation of your website?
2. Next, how much data can you afford to lose? Of course, no one likes to think or talk about losing data, but any solution is a compromise between resources you’re willing to pay for and the amount of data that’s guaranteed protection. This “amount” is typically/best measured in time. Any backup solution will capture “snapshots” of your website — copies of the data at a certain point in time — that will vary anywhere from “realtime” to hours or days since the last snapshot was made. If you cannot afford to lose any data, then a realtime backup solution is the only option. These solutions will do their best to use a program running non-stop in the background making a backup (somewhere) of your data. If the answer is “some” — then you just make a determination of what your acceptable loss is. Is it a day? A week? 6 hours?
3. The second “time” factor is the time-to-restore. How much time is acceptable to go from a catastrophic failure to a complete restore of your website functionality? This time is determined by many things in addition to the actual type of backup you have, including the type of webhosting you have, and the form of the failure. If you host your website on a VPS (”Virtual Private Server”), for example — and this VPS was backed-up in its entirity on a regular basis — then restoring is as simple as copying over the VPS backup and restarting it to that point. If your website has a more complicated architecture (multiple frontend webservers and multiple databases, for example), then the restore process is likely to be a more lengthy ordeal of reconfiguring the servers ahead of time before any data is put back in place.
4. Where is the data backed up? It should be obvious that the best backup is to a location that is physically separate from where your website is hosted. This is not an absolute (in fact it’s quite relative — is another server rack in the same datacenter “off-site”? What about a different datacenter owned by the same company and on the same network?), but in general you should attempt to distance your data from the potential points of failure: fires, tornadoes, hosting company incompetence, nuclear strikes — you name it.
5. Can you test the restore functionality regularly? This is a factor of cost, to some extent, and also practicality. If you opted for any backup solution other than “realtime”, you’ve decided on an acceptable level of data (time) loss — so testing a full restore right to production is not advisable, since you’d lose data. But working with your hosting/backup vendor to verify the backup, and maybe test a restore to another location are things you should pursue, if possible.

These are all important questions that you need to ask of yourself, and also of your web hosting vendor and your web developer. Together, you can decide on a backup solution that makes sense for your website. That said, what kind of solutions are there? This is not a comprehensive list, but it’s a general summary of the types of ways you can guarantee continuity of your website. Interestingly, computing/server technology has come a long way in the last couple of years. Virtualization and cloud competing have blurred the lines between continuity, load-balancing, and backup. After all, if your website is running on a virtual server in the cloud, data loss is impossible, right? Well, no, not really.

• VPS (Virtual Private Servers) – as mentioned above, a VPS can be a good way of assuring a very quick time from hardware failure to restoration. A VPS is a “server” that is entirely self-contained and run in a “virtual” environment. This means that your server itself is essentially a file or files that can be copied (regularly, or realtime) to any other backup medium for later restoration. Typically, most hosting providers that offer hosting on a VPS will also offer a backup service that makes restoration of your VPS a cinch. The above questions, however, are still quite pertinent and should be asked: How often do they backup the VPS? How quickly can they restore? Some hosting vendors host their VPS on a “cloud” server architecture which makes hardware failure a nearly negligible factor. However, you should still consider how much you trust the competence of your hosting vendor and explore the option of backing up your data somewhere else independent of the VPS itself. Many other things can go wrong — virtualization software failure, data corruption, etc. These things can all leave your virtual server inoperable or unsalvageable.
• Backup “daemons” – “daemon” is computer nerd slang for any software that runs non-stop in the background. There are many common backup solutions that use a daemon running in the background to persistently backup your website and its data. Typically, this sort of solution is what’s required for a “realtime” backup of your data. Most hosting vendors that host larger dedicated managed server environments will have some offering in this regard.
• Scheduled backup jobs – there are a wide variety of ways this can be implemented, but loosely speaking, this is any sort of backup script/program that is merely scheduled on a regular basis (say, nightly) and involves a manual copy of the website data (files and database dump) to an arbitrary off-site location.

It’s very likely that any hosting vendor you choose will have one or more of the above options — and possibly others. But it’s not enough to simply sign-off on an arbitrary product without understanding what it does and what it guarantees. Hopefully, armed with the above questions and information, you can make the right decision about how to backup your website. In an age where a website is increasingly the face of your business, you can’t afford to leave it to chance.

It didn’t always get it 100% right, but even if it didn’t, it was still a convenient shortcut and it would get a majority of the info right. It was also amazingly good at picking “events” out of e-mail — either casual e-mail from a person or automated/invite type messages.

Then one day, it disappeared. Poof. Gone. Has anyone else noticed this? Does it still exist for anyone else out there? Is it unique to traditional gmail versus google apps?

Google, please bring this back, or I swear on the Golden Kraut I’ll have my revenge!

Recently, this customer began reporting that she was unable to receive e-mail from certain people. Some investigation yielded this information from Swirbo, while attempting to deliver a legit e-mail message from someone on aol.com:

Jun 14 10:50:10 mta1 postfix/smtp[5285]: 90D0D834038: to=, relay=redacteddomain.com[1.2.3.4], delay=3, status=bounced (host redacteddomain.com[1.2.3.4] said: 550 SITEGROUND Faked AOL, so you must be spam. (in reply to RCPT TO command))

This error message was not exactly clear, so I was forced to contact Siteground to try to determine why they were blocking this mail. Unfortunately, Siteground has no support number to call, or e-mail address to e-mail. So, I attempted to file a ticket, however even this requires 30 minutes of trudging/fighting through obscure attempts to deflect actual contact with support via their “knowledge base” and a barely-functionining java applet that tries to detect obvious problems first. Once I made it through this and obtained the customer’s password (which is required for some reason to even file a ticket), I managed to file a ticket asking them for an explanation.

This was their response, after which they immediately closed the ticket (which I can’t re-open to respond):

I carefully investigated your case and I noticed that your domain is using an mail exchange server as it’s MX records point to

redacteddomain.com. 14400 IN MX 10 mta1.swirbo.net.
redacteddomain.com. 14400 IN MX 10 mta2.swirbo.net.

Due to this fact the email messages are coming to our server from email address redacteduser@aol.com and hostname mta2.swirbo.net. due to the fact that mta2.swirbo.net. is relaying them to us.

Our spam prevention filters are checking if the email address domain and the host which the emails come from are the same and if they find a mismatch the email is rejected with the message that it must be fake mail.

My advice is to contact your mail exchanger support team and present them with this information in order to solve the issue more effectively.

Please do not hesitate to contact us if you have any further questions or comments.

Best Regards,

Andrey
Support Team
SiteGround.com

So, aside from their hideous customer service and refusal to give me a chance to respond to a support inquiry, we have what appears to be a very braindead attempt at spam-filtering by Siteground. What they seem to be saying is that when they receive an SMTP connection, they check the reverse-DNS for the IP, and if the domain name in the resulting A record doesn’t match the sender’s domain, they reject it. This makes no sense, and contravenes any number of standard practices involving SMTP relays. I’m assuming they only do this for certain larger domains (AOL, Yahoo, etc.), otherwise it’s hard to imagine how their customers get any mail at all. The potential here for false-positives is huge.

Similar problems as this emerge with SPF, but this is above and beyond even that.

Unfortunately, due to Siteground’s complete lack of support, broken e-mail, and inability to address this problem, we’ll probably be migrating what few accounts we have with them somewhere else.

Thursday 6/07

4PM — Our comcast cable goes down hard. We reboot the modem and lights sync up fine, but we have no access to the internet. One of our employees called Comcast and they dispatched for the following morning (Friday 6/08)

Friday morning 6/08

10AM — Comcast tech arrives on-site. He checks the signal and it’s fine, and then proceeds to replace the old SMC modem with a Netgear “business IP gateway”. Unfortunately this didn’t get us back up.

11AM — After an hour or so of phone calls, he managed to get a laptop online via NAT from the router. This is good because it means whatever originally went wrong has been fixed, but he had no explanation for what originally caused it.

11:05AM — Comcast tech looks satisfied and starts hinting that we’re good to go and I ask about the static IP configuration and remind him that we’re down until we get the static IP configuration. He clearly has no idea what this even means, so I have to convince him that he’s not done, since he was arguing at first that “the static IP is something y’all have to do, we don’t do that”.

11:30AM — After I convince him that we need the static IP (i nearly whiteboarded a network diagram to try to explain it to him, but he had no TCP/IP/routing/networking knowledge at all whatsoever), he called 2 different Comcast numbers, eventually getting someone. He wouldn’t let me just talk to them, so I had to relay word-for-word what I needed, which he relayed to them: “They.. they’re saying they .. they need a static.. static? right, static IP? a static IP configuration?”

11:40AM — After finally relaying this to the woman he was talking to, she informs him that “We don’t provide the static IP — that’s something the client has to provide..” As politely as I could, I let them know that I don’t even know what that means. We pay for a static IP. They give us the IP, route it, and we assign it to our firewall. Pretty simple. We’ve had it that way for 2 years.

12:30PM — Finally, they get the static IP config built and pushed to the new modem. At long last, we’re back up and running. I pack up and head home.

2:30PM — I get alerts from Nagios that our Comcast has gone down. Again. I venture into the office to check on it, and verify via troubleshooting that it appears our new modem has simply “lost” the static IP configuration again. (I know this because I can get online via NAT through the modem itself). I call comcast support, and an hour later, I finally get to a tech that knows what I am talking about, and they get the static IP re-pushed. I head home again.

3:30PM — I get alerts from Nagios that our Comcast has gone down again. I head back into the office. Same thing. I call Comcast, same song and dance. This time, I again ask what could be causing this that the modem would simply “lose” its config so often. He doesn’t seem to know, and asks if anyone “hard reset” the modem. No progress on that front, but he got us back up and running again.

4:45PM — I get alerts from Nagios that our connection has gone down again. This time, however, just as I arrive at the office, it comes back up. It appears that this perhaps was just a “normal” outage. (The fact that I consider outages like these to be “normal” is an indication of the quality of Comcast’s service.)

Saturday and Sunday (6/09 and 6/10)

During these two weekend days, our cable went up and down around 3 or 4 times, for 15-20 minutes at a time. Though it concerned me, it never does any good to call Comcast support to “let them know” that it went down, so if it comes back up at all, we consider ourselves lucky.

Monday (6/10)

Thinking that everything seems peachy keen, our employees arrived at the office fresh and ready for a productive day’s work for a change. Unfortunately, looks can be deceiving. Not long into the day, we started to realize that something is wrong — the re-emergence of a problem we had around 6 months ago that subsequently mysteriously disappeared.

The problem was that TCP sessions would timeout during inactivity. I knew this wasn’t an issue with our firewall, because of the following reasons:

• Our firewall config hasn’t changed or been touched substantially in over 2 years.
• The problem came and went (usually corresponding with Comcast needing to re-push the static IP config) while no changes were made to our firewall
• While the problem is occurring, I could still see a connection in the pf state table (our firewall is OpenBSD running on a Soekris box).
• I was able to reproduce the problem by setting up a laptop with the static IP and plugging it straight into the modem.

The problem was tolerable in the past, but this time it was much worse — connections were timing out after around 5 minutes of inactivity. This doesn’t sound like a big deal, but when you consider the amount of traffic we have that requires a persistent TCP connection, and how poorly some software deals with timeouts (*cough*Outlook*cough*), you can imagine how rough things were for us. Instant Messaging (AOL, MSN or Jabber) would stop working. E-mail (IMAP) would stop working. FTP control connections would stop working, etc.

Making matters worse, from a technical perspective, was that these connections didn’t merely “time out” — there was no RST or FIN packets sent to terminate the connection. It just disappeared. Traffic goes out, nothing comes back.

I will spare you the blow-by-blow of the troubleshooting for this problem, but you can probably imagine that a company with tech support that can barely understand and support the basic services they provide (i.e. our static IP configuration problems), getting them to understand a slightly esoteric (but devestating) TCP problem was a nightmare. Attempts were made to deflect the issue back to us were made at every turn. Every call was a new call — no one was able to stay familiar with the problem, and no one every seemed to be working on the issue. I eventually managed to get is escalated to “tier two”, who then said he was forwarding it on to their networking team, but I never heard anything from them ever again.

The only resolution came, a week later when I finally got a tier 1 tech to dispatch someone to try replacing the Netgear modem with an SMC modem similar to the one we originally had. This was scheduled for Thursday (6/13) between 10-12, unfortunately no one ever showed up. I got a phone call at 5PM from a dispatch tech saying he “had the wrong modem, could we reschedule?” So, Friday comes, and finally, around noon, he shows up with the modem and replaces it. After the usual phone-call song-and-dance he gets a tech that can push the config, and we’re back up and runing. Problem solved — no more timeouts. I don’t know if it’s a configuration issue, or if it’s an issue with Netgear, or what.

I do know that Comcast surely doesn’t know either — but what’s worse, is they don’t care. Our entire company was forced to work at home for a grand total of 7 business days due to these issues. To this date, no one has ever followed up on the original ticket(s) we filed about either issue to ask if it was resolved. I’m afraid to call, because as often as Comcast tech support fixes something, they break something else.

The important thing to realize about this story is that it’s not unique. It’s not a one-off experience. We’ve faced these difficulties every step of the way with Comcast, in multiple locations. They advertise their “business cable” as a business-grade Internet connection. It’s not. You get more bandwidth, but that’s it. No SLAs. No accountability. No customer service. Nothing.

Further, what I’ve learned about Comcast is that they are truly too big for their own good. They have so many different departments that even their employees don’t seem to understand it all. Dispatched techs often make 2-3 phone calls before they even get the right person. Tech support never knows whether or not someone is actually being dispatched. If you’ve ever heard the expression “the left hand isn’t talking to the right”, that perfectly describes Comcast, except Comcast is a multi-armed Shiva, and none of the hands are talking to eachother. At one point, a tier-1 tech actually transferred me to Gateway. No joke. I wish I could make this up. He said that “he thinks the problem is with our internet gateway, so he needs to transfer me to Gateway @home support” (or something like that). Next thing you know, I was on hold with Gateway computers. Naturally, I had to hang up and try again.

I know that after I post this, I’m sure I’ll get the standard “well, we have Comcast and it works fine!” comments. That’s great. I’m not saying there’s anything wrong with cable technology. It’s wonderful. The problem is with Comcast’s business practices and size. If you have no problems, generally, you’re good to go. But good luck if you ever have any problems. We’re moving to a new building this fall, and I can assure you, I won’t even remotely consider Comcast as an option. The ramifications of this is that our ISP cost will easily double or triple. Painful, right? Not really. Not when you factor in the cost of the downtime we’ve suffered at the hands of Comcast. It’s not a business-grade ISP. Period. It’s a no-brainer for us.

Technical details of permanent failure:
TEMP_FAILURE: SMTP Error (state 13): 450 : Recipient
address rejected: Greylisted for 5 minutes

Anyone see a problem? The error we returned was 450, yet Google seems to think it was a permanent failure. Here’s a bit from the SMTP RFC (2821):

4yz Transient Negative Completion reply
The command was not accepted, and the requested action did not
occur. However, the error condition is temporary and the action
may be requested again. The sender should return to the beginning
of the command sequence (if any). It is difficult to assign a
meaning to “transient” when two different sites (receiver- and
sender-SMTP agents) must agree on the interpretation. Each reply
in this category might have a different time value, but the SMTP
client is encouraged to try again. A rule of thumb to determine
whether a reply fits into the 4yz or the 5yz category (see below)
is that replies are 4yz if they can be successful if repeated
without any change in command form or in properties of the sender
or receiver (that is, the command is repeated identically and the
receiver does not put up a new implementation.)

SMTP 4xx errors are temporary and typically represent transient errors. An SMTP client, upon receiving an error like this, should (i.e. is encouraged to) try again. Greylisting is a spam-filtering technique that takes advantage of this fact by issuing temporary rejections to new clients, in effect filtering out mail sent by software not smart enough to retry (as any proper mailserver would). Google should be queueing and retrying these messages. Anyone else running into this?

The first thing that stands out in the new Firefox is the more modern, snappier look and feel. Everything is more shinny, more playful and more clickable.

Can anyone tell me what this means? This is followed by:

Tabbed browsing was a major browser innovation that Firefox popularized

No. Opera popularized tabs. But that’s a forgiveable mistake. It looks like Firefox 2.0 will of course be a decent browser and a vastly superior choice to even IE 7, but it doesn’t look substantially different from Firefox 1.5. This appears to be a “hey let’s make this a major number release for some reason” release — i.e. marketing.

I’ll continue to stick with Opera.

This week, their director of marketing contacted me asking to try to clear up the situation and convey their side of the story. I told him I wouldn’t amend the original post (barring for any inaccuracy), but that he was welcome to e-mail me an explanation. In the interest of fairness, here it is:

When you originally called ACS you spoke with an employee, V. Patel. Of
course we prefer that our employees would transfer all phone calls of
these matters to a principal, but that is in the past and what’s done is
done.

ACS is a company that strives to go above and beyond industry ethics. We
have never endorsed spam of any kind and we never will.

Our side of the story….

In this particular case one of our employees was assigned with the task
of link building for a client. The client had recently come to us and
brought with them a checkered past, which included lots of email and
comment spam. I’m assuming that because of this our employee thought it
would be acceptable for him to continue with these practices. It wasn’t.
There was really no excuse for him to do this, he knew our ethics and
policies.

Jabber/XMPP has gotten another big shot in the arm this week, with the announcement that Livejournal has launched a Jabber server for its users, complete with the friends-list pre-loaded as the Jabber roster. They will be incorporating s2s communication, along with lots of other features.

This comes on the heels of Google Talk, which last year launched its XMPP-based Google Talk service, opened it to XMPP s2s on the Internet. They also incorporated an (optional) user-friendly AJAX-based IM client into the gmail web client — look for Livejournal to do the same, as well as integrating the blogging/comment-notification functions into instant messaging as well.

With two big players like Livejournal and Google backing the open XMPP standard, things are looking more grim than ever for proprietary, closed, and centralized instant messaging services like AOL, Yahoo, et al. Here’s hoping the trend continues!

Now I don’t have to switch to Firefox after all.

The mystery started when sessions started mysteriously expiring prematurely on two of our most popular web applications: DekkoTime, and our internal CRM/groupware application. It started about two weeks ago, with no discernable changes to our configuration that could be responsible.

So to understand what was necessary to track down this problem, we have to explore a little bit about how PHP session data storage and expiration works:

When PHP creates a session with session_start(), it dumps a file in a particular path. This is governed by the session.save_path parameter in php.ini — /tmp by default. But naturally, as sessions go idle or are abandoned, they need to be cleaned up, so that our save_path isn’t overwhelmed with old session data before it has a chanced to be cleaned (usually on reboot, in the case of /tmp). Enter garbage collection.

Garbage collection in PHP is, from what I gather, piggybacked on invocations of the PHP interpreter itself. When (if) it runs, it deletes any session files in the save_path that haven’t been accessed in a certain amount of time, governed by another php.ini setting: session.gc_maxlifetime. There are other parameters that dictate the probability/frequency with which the garbage collection routine runs, but they are irrelevant to this discussion.

So, naturally, the first thing I checked to see why our sessions were expiring was session.gc_maxlifetime in our php.ini:

session.gc_maxlifetime = 72000

72000 seconds — that’s 20 hours. So, no problem there. It appeared from our experience that sessions were expiring between 45 minutes to an hour — far less than 20 hours. I roughly verified the time that sessions were disappearing by initiating a new session and doing this:

date; while true;
do
if [ ! -f sess_235f09d44d5288554cf7a55fdfbc6df7 ];
then echo "session has disappeared" | mail cwage@centresource.com;
break;
fi;
sleep 1;
done

That way, I’d get mailed when the session disappeared. Pretty sick, I know. This verified that sessions were disappearing after around 45 minutes of idle time. I could not find an explanation for this: session.gc_maxlifetime was set to 72000 in our php.ini. Maybe it was being overridden in that particular php environment? “print_r(ini_get(”session.gc_maxlifetime”))” bore the same result: 72000. No problem there.

Here I took a slight detour in wondering if there was something else diligently cleaning up an admittedly messy and full /tmp directory (~500 days of uptime will do that). So I started looking for some sort of utility that would let me monitor a file and see what process was responsible for unlinking it (the session file, that is). Sadly, there’s no utility that can accomplish this with a stock kernel in Linux: fwatch appears to accomplish this, but I wasn’t about to install a kernel module labelled as an alpha release just to track this down. Eventually I convinced myself, anyway, that the likelihood of some rogue process cleaning up /tmp was pretty unlikely, even for Linux.

So, I resorted to just googling my little heart out. Here’s where things get interesting.

Naturally, any application can override session.gc_maxlifetime to whatever pleases it — in fact, most OSS PHP applications do just this, in order to enforce its own particular idea of a sensible session expiration time. But here’s where things get sticky. If you override session.gc_maxlifetime in one particular environment, how does it know which sessions are its own, as opposed to others that should be adhering to the global setting?

Well, apparently, it doesn’t. When the PHP garbage collection routine runs, as far as I can tell, it blindly removes sessions from session.save_path that haven’t been accessed in longer than session.gc_maxlifetime — period. So, as it happens, what changed two weeks ago? We started playing with a number of PHP applications: Joomla! and Zen-Cart, both of whom (among others), take it upon themselves to override session.gc_maxlifetime to a smaller value, which appears to have been, drumroll please: around 45 minutes. So, every time the PHP interpreter was invoked in this environment, it obliterated sessions for all our other applications if they had been idle for 45 minutes or more. Harsh.

I am not sure what the preferred solution to this is supposed to be, and I’m also surprised that this isn’t a more common problem — overriding session.gc_maxlifetime is a fairly common thing for PHP applications to do these days. I am surprised these unexpected results would go unnoticed. In any event, my solution was just to create a hierarchy of per-application directories inside /tmp/php (owned by www-data, so Apache can write to them), and then adding a line to my Apache virtualhost config for each one, for example:

php_admin_value session.save_path /tmp/php4/dekko

In this way, the save_path is isolated for each application, so an overridden session.gc_maxlifetime for another codebase won’t affect it.

Phew.