London Outage / Nov 5th – 6th Incident

First off. We’re sorry.

 

We know that for most people, the worst part of the outage was the lack of communication (it was for us too), so we’re going to answer some questions in-depth so you can be as informed as possible. The lack of communication from a provider in a situation like this can be excruciatingly frustrating, and we get that, so we hope that by providing you with this statement you are able to gain an understanding of the whole situation.

 

Were there two outages?

Yes.

The first occurred between 17:33 and 18:47 GMT (5th Nov 18) and then the second between 00:06 and 14:34 GMT (6th Nov 18).

 

Has the problem been 100% fixed?

It will be. While we don’t anticipate any outages again in the near future, as you will note further into our question set, we’re going to be performing some pretty massive changes to gain control over the environment we operate from to prevent this from happening again.

 

What type of outage were these incidents?

These were network outages. There was no power or cooling loss through the process that allowed servers to remain powered on at all times resulting in no data loss. The servers were never turned off.

 

What caused the outages?

The network feeds that supply our core infrastructure within London were suspended due to an administrative error that occurred between our colocation provider and the datacenter. The suspension was performed manually by the datacenter that resulted in all our systems being completely disconnected from the network.

 

Why were we unable to provide more information during the outage?

We fully understand that more than likely, the worst part out about the whole incident was the lack of information. We know. Unfortunately, we were in the same situation as you due to communication failings with our colocation provider and the datacenter to definitively relay the cause of the problems. The communication breakdown resulted in us being passed incorrect information and assumptions that were initially identified as being an outage caused by the datacenter. After direct communication with datacenter technicians, we decided not to post an announcement with such information until it could be verified.

 

Why was the initial outage corrected and the second wasn’t?

Our colocation provider followed standard procedures to make contact with the datacenter to inform them of the incident, to which they were able to bring the services back online while they looked into what caused it. The second outage was a manually performed suspension after the initial situation had been reviewed.

 

How did EL get notified of the outage and what was the response time?

We make use of status monitors on all core systems that we operate; these monitors let us know about the outage within 30 seconds of it starting on both occasions. Unfortunately, unlike other scenarios, having a short notification and fast response time of fewer than 2 minutes didn’t provide any benefit due to the communication failure.

 

What compensation are we providing customers that were affected?

While the outage was only in London, it also knocked off all web packages including our control panels that resulted in the vast majority of our users being negatively impacted in some form. All services located in London will be issued with a two-week extension, and all game servers within New York City and Chicago will receive a one-week extension on the chance that the customer experienced problems accessing the control interface. If you would like to receive this compensation, please let us know by contacting us via the ticket system with the request made to ‘Administration’.

 

What are we doing to prevent this from happening again?

Administration issues and mistakes do happen in business. With the uncertainty about the situations root cause, lack of current admin safeguards and the improvements still being made at our colocation provider, we have decided to make some significant changes within our London location. We will be discontinuing business with our colocation provider and are switching to colocation directly with the datacenter within the next few weeks, allowing us to have much greater control over all aspects of the service that we provide. All IP addresses will remain the same, but we will be scheduling some emergency maintenance within the next few days where the equipment will be re-racked within the facility. We will be posting a status update with the scheduled time window once we’ve completed contract signing and rack provisioning.

 

I have questions that aren’t answered here, what do I do?

If we haven’t answered a question that you have, please feel free to reach out to us directly at [email protected] (applicable to both ELHostingServices and PrimeNodes customers), and we will be more than happy to have a conversation with you to get your questions answered.

 

We are incredibly sorry for any inconvenience this situation has caused; we hope that the compensation and steps that we’re currently performing to ensure that this doesn’t happen again are enough to show you that we genuinely care about every one of our customers. Thank you for taking the time to read this statement and we hope that you have found the information you needed.

 

Once again, sorry.

 

William Wilson & William Phillips

Encrypted Laser Management


Most Notable Advancement To Date

Massive Encrypted Laser Limited Developments

Encrypted Laser Limited is about to go through the most significant change in its history within the next 1-2 weeks; completely changing almost every aspect of our business to continue improving and developing the services that we can provide to our incredible customers. It has been two years since ELHostingServices came into existence, and we’re rapidly approaching the two year anniversary of when Encrypted Laser Limited became a legally registered company. With the exclusion of our significant redesign performed on the 1st of June 2017, the business strategy we’ve been following has remained very passive. Times are, however, changing. We’re now VAT registered which has also prompted multiple changes in how we price our services & operate. On the 1st of Jan 2018, PrimeNodes launched, and the way we conduct business altered dramatically.

We’re performing massive changes, and we wished to let you know what we’re doing. If you have any questions about any aspect of it, please get in touch!

 

Infrastructure Improvements

• The hardware that is powering game servers & virtual private servers is being upgraded from rented i7 6700K (4.0GHz) dedicated servers to 100% self-owned i7 8700K (4.8GHz) rack servers.
• Bristol services are being migrated from Bristol to our new London based datacentre operated by Bandwidth Tech (Zare).
• New York City services are being migrated from ReliableSite & Choopa to our new NYC based datacentre operated by CoreSite (USDedicated).

 

DDoS Protection Upgrades

• Our UK DDoS protection is being upgraded from 100Gbps Corero DDoS mitigation to 140Gbps Corero DDoS mitigation.
• Our New York City DDoS protection is being upgraded from a custom 20Gbps solution by Choopa/ReliableSite to 40Gbps Corero DDoS mitigation.

 

Web Hosting Upgrades

• The base of our web servers, Apache, is being replaced with LiteSpeed offering faster request speeds, built-in anti-DDoS measures and many more improvements to provide a more stable & responsive experience.
• ELHS and PrimeNodes will now feature four web hosting package tiers, a change from our existing two. These tiers will include a range of different features and resource limits to allow customers to select a plan better tailored to them.
• We’re introducing a variety of off-site backup options for web hosting customers, ranging from monthly to hourly depending on the chosen package to offer peace of mind for data security.

 

Game Hosting Upgrades

• After consistent compatibility problems, a decision has been made to discontinue Rust game servers. Existing servers will be left operational, but no new servers can be provisioned for this game.
• We’re putting Arma 3 servers back in stock for deployment in all our locations.

 

VPS Upgrades

• Virtual private servers are getting upgraded to the same hardware that powers game servers, i7 8700K processors overclocked to 4.8Ghz.
• We’re increasing the amount of storage of our VPS plans significantly, ranging from a 20GB upgrade for our lowest plan to a 520GB upgrade for the more extensive packages.

 

Dedicated Server Upgrades

• We’re discontinuing the sale of new i7 6700K and i7 7700K dedicated servers in each of our locations, these were a mix of rented and colocated hardware. These product lines are being replaced with i7 8700K dedicated servers available in London and New York City only.

 

Price Changes

• We’re increasing the price of our GMod & CSGO servers from £0.38 to £0.40 GBP, but are introducing block packages allowing people to order thirty-two (32) & sixty-four (64) slot servers at a substantial discount.
• Our one-time payment web hosting plan, our ‘Basic Hosting’, is being replaced with a monthly recurring £0.99 GBP fee plan named ‘Dev Hosting’.

 

Reseller Plans & Partnerships

• PrimeNodes is introducing a set of reseller services enabling customers interested in getting into the hosting market to operate and sell services powered by our powerful & reliable servers with ease, all backed by our friendly & knowledgeable customer support. Customers have the option to bundle together multiple plans, allowing for the resale of both web servers and game servers from a white-label cPanel and game interfaces.


2018 Company Changes

2018. New Year. Lots of Changes.

We would like to begin by wishing everyone a Happy New Year! 2017 was a fantastic year for us, with the introduction of a new location, upgraded panel and better hardware. 2018 is going to be a little different for us, and you should see the benefits quite quickly. Since our launch almost 2 years ago, we’ve never gone flat out to get bigger or to expand; ELHS has operated in the slow lane content with offering a small select range of high-quality services. That business concept, however, is being left in 2017 and we’re powering on to 2018 with full steam ahead. Major changes are coming to ELHostingServices in the very near future with some changes already being put in place.

 

Discontinuation of TeamSpeak Servers

As of the 1st of January, ELHostingServices locked the sales of new TeamSpeak servers. With the rise in popularity of Discord and ELHostingServices’s adoption to offer support through this platform, TeamSpeak has become less and less popular over the past seven months. As a hosting provider that strives to innovate and provide the latest and greatest services, offering TeamSpeak servers doesn’t allow us to do that and makes us feel that we’re holding onto the past. We truly understand that some still love TeamSpeak. We pledge to every customer who has an existing server with us that we will continue to run it until the day that you choose to cancel the service. If you never cancel, we’ll never stop providing the services to you.

 

Performance Instances Revamped

Performance VPS Instances, introduced as a test type service in September have now proven their worth and are getting several boosts. These improvements include a better user control interface, unlocked SSD I/O speeds, upgraded network adapters, Virtio virtualisation for better performance as well as live DDoS statistics. Currently, VPS servers can be purchased in Bristol (UK), New York City (USA) and Chicago (USA), our aim for 2018 is to begin offering these services in Dallas (USA). We’re taking care to select a datacentre that offers superb DDoS protection and phenomenal networking to ensure that it operates that the same high standard as all of our other locations before it gets launched. ELHS have discontinued the sale of Game Cloud servers due to their incredibly time consuming and high maintenance nature. Existing customers are unaffected by these changes and can continue to upgrade and downgrade on request, new servers, however, are unable to be purchased.

 

These are just a two of the many changes we’re working on, as we make updates we’ll be sure to release it here and on our Twitter account. Stay tuned!


ELHostingSerivces News & Press Releases

ELHS are releasing… news?

We’ve created a section of our site dedicated to important ELHS news, press releases and major announcements. Previously, the knowledge of our changes and upgrades have only released to users in our Discord server and via email. It is now time to have a centralised location for all of our news. No more missing emails dismissed Discord notifications or wasted time searching for missed messages. If you wish to keep up to date with what we are doing to upgrade our services you can now do so by visiting ELHS News.