VFEmail.net

We're currently running on the backup server. This server is woefully underpowered and you will experience slowness.

The primary server is being restored right now. Once complete you will 'fall back' to a mailbox snapshot from May 13th. We will then copy all mail delivered in the last 2 weeks to the main server.

Delivery delays - There are about 60,000 messages that are in limbo, waiting to be delivered. This is due to extremely slow deliveries on the backup server. The incomnig mail servers were VMs, and I've cannibalized the VM host to be the primary server. Once the VM's for incoming mails are brought back up on new hardware (after the restore is complete), those emails will be delivered.

I deeply regret the delays in service restoration and lack of communication. Unfortunately due to other circumstances, I have not been available to resolve these issues in a more timely manner.

Rick

_________________
VFEmail.net Admin

Thanks for the update Rick

good to know we will be up and running in time

I appreciate what you are doing to restore both service and old email.
Having said that, can you estimate when 1) service performance will
be back to normal? and 2) when the restore of old email might be complete?

I understand if that is more than hours, but are we talking 1-2 days? 1-2 weeks? A month?

A best guess or a statement of "It is going slowly, but I will post an update with
a progress report but XXX date/time" would be greatly appreciated.

How is the primary server restoration going?

Server restoration is slow. To get the data off of the backup server takes about 3 days. It's slow.
Hopefully we'll be up and running as of "May 13" by Saturday.

Here's the problem - Once that is done, I will need to manually copy all 'new' email from the August snapshot that we're currently running on to the new server. I really have no idea how long that will take, because every user directory will need to be traversed looking for mail newer than < 30 days ago, and then copying that to the correct location on the new server. Technically it's a one liner command, but with hundreds of thousands of users who have their own folder structure, on a slow backup server, I expect it will take a while.

That's what you guys see. I still will need to re-establish SQL replication, memcache replication, put the VMHost back together, etc, which should be pretty transparent to you, but is a lot of busy work and making sure things are all lined up right. For example - memcache replication is why the web interface was down for so long after the main crash - it wasn't working quite right when I shut down the crashed server.

_________________
VFEmail.net Admin

Ouch. But thanks for the update.

Having run my own ISP many years ago, I can imagine only 1/10 of what Rick must be going through

We are very appreciative of what he is doing for us (and have been doing so, for the past 11 years) to keep things running.

I have been keeping regular monthly backups of all my emails, so I wasn't affected much by the system crash.

I will state I have configured a temporary email account somewhere else for email to be delivered during the rebuild (as I have my own domain name) but I fully intend to move everything back to VFEmail once the rebuild is completed.

Have any of you *properly* read the fine print of Hotmail / GMail / Yahoo / etc. terms and conditions ?

I feel much more comfortable with Rick having my emails than those grubby corporates ...

Thanks again Rick for your hard work and dedication

I'm still seeing two day old stuff roll in, and some mail sent to me did get returned (got a letter from Progressive - luckily I didn't miss that bill!).... any updates? You mentioned Thursday in another thread as being the day for a new server.

I was able to bring up the new system - the old server had some burnt components, and the backup server is consumer grade, so I went with new core hardware and existing disk array - but there must be a bad something in the disk array as there are still issues under load.

It's running in depth diagnostics, but at 50% completion I don't see any issues being displayed. I've ordered some more drives and a new controller to try and have whatever needs to be replaced on hand as soon as possible. I think it's likely a drive issue, because only the data volume is showing issues.

Rick

_________________
VFEmail.net Admin

Rick,

I had to go out of town yesterday and had hoped that I'd have email upon arriving home about 30 min. ago.

I haven't received any email for my main acct (bp) since June 26th so that's 10 days of no email thus far.

Doesn't make sense that my father is receiving email and I think that Mom is even though they are running a few hours or a day late.

I even sent 2 different emails to you within the past 10 days but can only assume that you never received them.

Appreciate the status update but wondering how many more days I will have to wait to receive email and/or of most of it bounced back to the sender.

_________________
Outside the sound and the fury of living is the silence of being alive. We only need to learn how to listen. ~ Author Unknown

I also got some returned mail. I'm glad I got the returned message though - they are work related and I'd hate to have to call to ask if they went through.

Should I wait until this blows over to be sending critical emails?

Congratulations!!!! Your stamina is to be applauded!!! My mails are starting to come through and I now have the pleasure of wading through the backlog. >

I used to manage a team of System Adminstrators for an Investment Bank and suffered the slings and arrows hurled by disgruntled users, not a pleasant experience. I hope you emerge from this wiser and unbruised.

How did the diagnostics run turn out?

With all due respect Rick, when I/we can't send email and incoming mail either doesn't show up or shows up late, we'd like some updates more than once or twice a week. I think I speak for many of us when we say that the years of positive experience and faith in your skills and commitment make us want to stick through this until you get it fixed. Obviously we're all still here because we like you and appreciate everything you've done, otherwise we'd all have gone to gmail long ago. I have a long list of reasons why I prefer dealing with small businesses over larger ones. I also don't think anyone doubts that you're busting your hump right now trying to get this going again. However, when we don't hear from you, it leaves us wondering if the commitment is still there or if we should be jumping ship. I/we need to at the very least know what is going on so we can plan accordingly; I've had a queue of emails sitting and waiting to be sent because I don't want to enter into iffy correspondences for important business matters (important to me, anyway).

Also, please let me know whether you received my payment or not. Since paypal works via email, and the email is wonky right now... well, you get the idea. I'd like to know that the funds made it where they were supposed to.

I have to agree with jeffreywills. I'm hanging on, but the lack of "timely" updates is disconcerting and increasingly has me searching out other options, something I'd rather not be doing.

Thanks.