Emergency Control Panel & Server Maintenance

Emergency Control Panel Maintenance

Thursday 26th July 08:00 UTC

We have been noticing some control panel timeouts the past few days which cause occasional short unavailability of the control panel and other miscellaneous errors. Some users have also reported issues accessing some reports and statistics information.

We need to perform some server hardware upgrades and some database software optimisations in order to fix the problem and improve performance. These optimisations may take up to a couple of hours to perform during which the control panel will be unavailable. Your server will continue to function though, you just won't be able to administer it.

We have identified Thursday 26th July 8:00 AM (UTC) to be a good and quiet time for this. This upgrade will occur within a 1 hour window and should take at most 3 hours.

Emergency Server Reboots

Friday 27th July 08:00 UTC

We completed our control panel upgrades and database optimisations this morning which appear to have fixed the issues of several days ago. We hope this emergency maintenance wasn't too disruptive to everyone.

Unfortunately though, as the optimisations took server hours, Centova Cast has since "lost" communication with some Shoutcast and Icecast instances. This means that although those servers are still up and broadcasting the Centova Cast control panel is unable to control them (i.e. bring them up and down or make changes).

The simplest way to fix this is to reboot each server so unless we find another solution in the meantime (we're working on it!) we'll be doing this at tomorrow morning (Friday 27th July 08:00 UTC). We're trying to minimise disruption by having the reboots occur at off peak times.

Your server should be brought back online automatically but we suggest you double check this and contact us if you have any further problems. We expect downtime of no more than 30 minutes. We will also apply a few other updates at the same time which we have been holding on to as they also require a reboot. This will save future downtime occurances.

We thank you for your understanding and if you need to make changes in the control panel in the meantime reply to this email and we'll restart you server manually now. Bear in mind all servers will be rebooted tomorrow regardless of wether we restart your server now so if you can wait please do.

We're just preparing for the reboots now. Everything is scheduled to start on time in a few moments at 08:00 UTC.

All servers are now offline and being updated. We'll post again in about 20 minutes when they are back online.

All servers have now been brought back online. Radio servers should being coming back online too if they were online before the reboot. We recommend you double check anyway to be sure. Thanks for your patience and if you notice any issues please contact support@internet-radio.com.

We're having some issues bringing uk1, uk3, and us1 back online. We're looking into it now and will report back shortly.

We've escalated the issue to the data centre and hope to hear back shortly.

No word from the DC yet so we're performing a migration on us1 to see if it fixes the issue. Should finish in about 20 minutes and if it's a success we'll do the same for uk1 and uk3. Thanks for bearing with us!

The migration didn't fix the issue. We'll just have to wait for the DC to respond. As soon as we have any updates we'll post here.

uk1 and uk3 and back online us1 to follow shortly.

us1 is back and we're now completely online. Phew!

The problem was that us1, uk1 and uk3 lost their networking stack after the upgrade. For some unknown reason they were set to not auto update their networking on boot like the other servers. We're investigating the the DC why this was the case. Apologies for the extended downtime on those servers. Things should be running smoothly now.