Downtime with unet responding to pings

This morning, unet was down (thanks to @ymasur for reporting this). The machine was still running, responding to pings, but we couldn't log in or even access the machine locally. A reboot resolved this, which unfortunately the reason for this glitch is unknown.

In addition, we connected again the KVM, which at some point appeared not to be working. This was resolved by moving the machine meknes from port 4 to 3, as port 3 was unoccupied. It is possible the port 4 of the KVM is dysfunctional, or that the KVM doesn't expect to have an unused port (3) in between two used ports (2 and 4).

Could it be the UPS?

It looks like our aging UPS is failing, and when doing so, sometimes cuts the power to all the devices is serves. Indecently, this exactly is what UPS is supposed to prevent, but for its defense one is not supposed to run a UPS without replacing its rechargeable battery for... 10 years. We decommissioned this aging UPS, which might have been the cause for the recent periods downtime.

Machine off

For the second time in the last couple of months, the server has been down, and when looking at the problem we found that the machine was just turned off. It wasn't manually turned off by mistake, and we couldn't find anything useful in the logs. Also the temperature in the rack is not exceptionally high, which unfortunately may point to failing hardware.

Faster connection

Starting today, we switched Internet provider, from Speakeasy to Comcast Business, and as a result have a faster connection in the office and to the unet server. We used to have (in MBits/s) 0.75 up / 3 down, and now have 6 up / 33 down of actual bandwidth. This is 8 times more bandwidth for the uplink, and more than 10 times more downlink.

Because of this, the IP address of our DNS server changed, so you might be unable to access the server until the DNS configuration propagates. If you're still having some trouble accessing the server 24 hours after the date of this post, please let us know by adding a comment here.

Server Crash November 14-15

We had a server crash this weekend, probably on Saturday 14 (not Friday 13!). The server was restarted on Sunday morning (Pacific time), and all the service are now working again as expected. As far as we know, no data has been lost. The reason for the crash is unknown at this point, and we are still investigating.

Issues with DSL router

We're having persistent problems with our new DSL router, and are working with our provider to have this fixed. In the meantime we have put back our old router; it won't use the new DSL line to its full potential, but we hope that it will be more stable. We'll make sure to keep you updated on this.

Downtime Jul 3rd, 2009

Exactly one year of smooth sailing since your last post, I have to report that the server was down this morning. The issue was with our new DSL router which required a little push in the form of a reboot.

Downtime on Jul 3rd

I should have posted about this earlier, but for completeness I'd like to mention that we experimented some downtime on the server last week Thursday for a few hours in the afternoon/evening.

Mail pilling up this morning

Mails have been piling up this morning for a few hours as the server has been running out of disk space and wasn't accepting new messages. The problem is now resolved and no mail has been lost in the process. Thanks Julien!

New DSL Modem Installed

During the last few months we had a number of issues with our DSL modem. Every time time the symptom was the same: the DSL model would drop the connection and would need to be rebooted. Of course this was quite annoying as every time this happened, the connection would be down until we go a chance to send someone to reset the modem. Hopefully with the new DSL modem we won't have to deal with those problems anymore :).

Server is Up

The server is up again, as of Saturday January 5, in the evening (Pacific time). As we suspected, it was a matter of rebooting the router. All the servers behind the router are alive and kicking; unet has uptime of 262 days.

Thank you Thomas for having taken care of this, and apologies to all for the inconvenience.

Downtime during the holidays

The server is not accessible right now and most likely won't be accessible until this weekend (January 6, 2008).

We have 4 servers in a rack behind this connection and none of them is accessible right now. So most likely this is "just" a problem with the DSL router that needs a kick in the but (i.e. reboot). This wouldn't be a big deal if someone was on site, but unfortunately the 4 persons that have access to the server are in Switzerland right now. (That's what you get for trusting a bunch of Swiss guys with your infrastructure!)

The first of us will be back on Saturday evening. So if everything goes well, we'll have our servers back this weekend. I will be flying back to San Francisco over the big pond on Sunday, and will post an update here early next week.

Power outage Thursday night

A power outage on Thursday April 19th during the night latest too long for our UPS. When the power came back, the server unfortunately didn't reboot properly. We fixed the problem on Friday morning, after a downtime of about 10 hours.

Both DNS server down this morning

Both our DNS servers were down this morning and possibly part of the weekend. The main server is now up and the second server will be brought up later today.

Downtime on Wednesday December 27th, 2006

The server was down for a few hours this morning. Only connectivity was lost for a few hours, and restarting the DSL modem this morning did the trick. This may or may not have been related to a nice storm we had in the area yesterday night.

Downtime on Sunday October 22nd, 2006 for hardware upgrade

One of the disks of our RAID array started failing last week. The server can still run with one disk down, but this put us at the risk of loosing data if one more disk failed. So replacing that disk became a priority. Instead of just replacing the faulty disk, we decided to replace all the drives. This improves reliability overall and since we bought bigger disks, as a side effect it also doubles our storage capacity.

And this is not the end of the story: instead of installing the disks on the old server, Julien installed them a slightly faster spare machine. This should give use slightly better performance, but more importantly the new 2 U rack-mounted server now fits much more nicely in our rack.

The server was down Sunday and during part of the night from Sunday to Monday (Pacific Time) to allow for this upgrade.

Downtime for a few hours on Tuesday Jun 14th

Our DSL line was down for a few hours yesterday. The connection came back up as we were trying to figure out what exactly the problem was.

Delay problem with FTP, SMTP, and Others Solved

Julien solved a delay problem that impacted a number of services, in particular FTP and SMTP. The effect was a long delay to connect to the FTP server or send an email through SMTP. The cause was wrong DNS configuration on the server itself; that is for the DNS servers used by services running on the server, not with the unet public DNS servers. I think we are all glad to see this problem finally fixed. Thank you Julien!

Update on "Server Down For the Day"

The problem mentioned in the last entry was resolved the same day, and was apparently due to an issue with the network card. A reboot of the server resolved the problem.

Server Down For the Day

The server will be down today. The machine shut down after a disk failure on hdi1. As I brought it up this morning, everything starts properly, except there seems to be a problem with the network. Julien will stop by later today to figure this out, so the server will be down today at least until this evening (PST).