It was a great day for a meeting today. Spring was in the air, and the ‘IED Platform Ecosystem and Education Grid status and review’ meeting was scheduled poolside at the Splenda Island on the Main Grid of Second Life. I got there early to get a good seat by the pool. The flowers were in bloom and a few other people had arrived early. I wore the wrong outfit, and was not exactly producing any Vitamin D in my lounge chair. All was good in the virtual world.
Aaron Walsh was standing on the diving board over the pool at around 12:30pm PDT when I logged in. He and two other people were talking about the location for this meeting, and agreed that the pool would be a nice change of pace. Aaron said that he had to log out for a few minutes, and that he would be right back.
After twenty minutes it became obvious that something was wrong. Not only was Aaron not back, no other people had rezzed on the island for the meeting. As 1:00pm clicked closer, one of the others said that Aaron could not get back in, and that a notice had just been posted that many services on the grid were down (again).
Here’s a copy of the message posted to the SLED Listserv about the problems ten minutes before the start of the meeting:
[12:50 PM Pacific] In addition to the reduction of services outlined below, Operations will be immediately disabling the following services for at least 30 minutes:
- In-world transactions,
- Portions of Website functionality, including access to account pages.
We’ll keep you updated every 30 minutes at most on this, and announce a return to service ASAP.
In order to increase overall stability of the grid today during peak usage hours, our operations team has disabled a set of in world functions to reduce overall database load and create a more reliable experience for everyone. As a side effect of these temporary changes, some group and avatar profile services will not be available.
- Avatar profile information will not be transmitted to the viewer. This affects both floating and embedded profile windows.
- General group information (name, charter, etc.) will not display
- in floating or group embedded group info windows.
- Groups will not show their member lists.
- Group owners and officers will not be able to eject group members.
- Group proposals will open the UI, but will fail to create.
- About Land will show 0 for traffic. (Note: This is temporary, and will not cause the loss of traffic captured throughout the time the total is unavailable to display.)
Please note that all of these effects are temporary and will return to normal behavior when we re-enable these services later today. At that time you should either relog or run Client -> Clear Group Cache (a Debug option) in order to refresh group behavior.
We will update this blog post to indicate when these services have actually been disabled, and again when they are again re-enabled.
As previously blogged, these steps are part of a temporary mitigation strategy until additional hardware can be installed, along with other scaling refinements planned for the very near future.
Thanks for your patience.
I had set my laptop up in our conference room with a projector, and invited others to sit in on the presentation. I guess I picked a bad day to bring in visitors who are interested in using Second Life in education. First comment upon the news of rescheduling the meeting was how this would not be good if it were a class, and you were a teacher. This might be excusable if it had not been happening all week long. Last weekend the grid experienced an 18 hour offline stretch, which also impacted planned in-world events. Something has been very wrong this week with the grid.
This is yet another example of the on-going stability issues that seem to be a permanent fixture lately in Second Life. I once had a block of Netcom stock (a popular ISP back about 10 years ago). I was holding onto it, waiting for it to hit $60/share before selling. When it was within striking distance Netcom experienced a similar outage to Second Life’s last weekend, and the stock plunged to a fifth of its value in one day. An 18 hour outage, especially on a weekend, is a very big deal and can only serve to reinforce the image of instability of the Second Life grid.
I find it ironic that in the same week that Windlight was pushed out in the official viewer we’ve been plagued by these random issues and outages. It’s too bad more effort was not directed at these stability issues over the past year before adding enhancements to the environment of Second Life. In the Windlight podcast, the developers rave about the new sunsets and shiny sparkly water. Is it really a beautiful sunset if nobody is online to see it?
If you want to see just how bad it’s been, jump to the Second Life blog and scroll back through this past week. A quick look at the blog shows that there are scheduled downtimes from Sunday through Wednesday of next week. Let’s hope things settle down soon.