Master Server Down?

[[TWB]Pand]

Multiplayer Server list won't populate.   Once you're out of a server you can't get back in anywhere.

 

Numbers dropping drastically fast from 52 to now 18 online on WOL via online.il2forever.com.

 

Posted in Technical Issues and Bug Reports -> http://forum.il2sturmovik.com/topic/23733-master-server-down/

 

Edited July 5, 2016 by [TWB]Pand

[[TWB]Pand]

Multiplayer Server list won't populate.

 

[-TBC-AeroAce]

YEAH NUMS are down... this genre cant help itself

[[TWB]Pand]

YEAH NUMS are down... this genre cant help itself

I think you're confused what I'm reporting here.

[WWDubya]

Yup, and it's off-hours over there. Will probably be resolved first thing they arrive at the office. Guess it gonna be an RoF night... their master is up as I type this.

[[TWB]Pand]

Seems like at a minimum there should be monitoring in place, and ideally have this apparent single point of failure addressed by load balancing/clustering multiple systems in case one goes belly up.

[JG13_opcode]

Ugh.

 

That's just unprofessional.  Why am I having flashbacks to CLOD and the amateur way it was handled?

Edited July 6, 2016 by 13GIAP_opcode

[II/JG17_HerrMurf]

Looks like it. DED Normal has been down since last night as well. DED often goes down just after updates when they tinker with their maps, though. Maybe another update or the STEAM release?

[FTC_SkyVi]

I have the same issue, can't log see any server to get in but i was playing multiplayer later in the afternoon

[NO_SQDeriku777]

We don't know what happened but I would really respect the Devs if they were to give a clear explanation of what happened and what measures, if any, could be put into place to keep it from happening again.

[JG13_opcode]

We don't know what happened but I would really respect the Devs if they were to give a clear explanation of what happened and what measures, if any, could be put into place to keep it from happening again.

 

They won't.

 

This is 1C we're talking about.  Remember CLOD?  Getting anything out of them was like pulling teeth.

[SCG_CorZero]

Same problem with me.  Worked ok yesterday.

[NO_SQDeriku777]

They won't.

 

This is 1C we're talking about.  Remember CLOD?  Getting anything out of them was like pulling teeth.

Well, All I can do is monitor the forum and see if they provide some information. I have been impressed with how detailed the change logs have been for the game. That gives me hope that they know how to communicate effectively. I like the game so far but am taking a wait and see attitude towards buying BOM until how I see the multiplayer numbers keep shaking out for North America market. The manner in which this specific issue is handled should be a good indicator of how seriously they want to keep expanding that market.

[JG13_opcode]

 

 

The manner in which this specific issue is handled should be a good indicator of how seriously they want to keep expanding that market.

 

 

Quoted for truth.

[II/JG17_HerrMurf]

Wow, you guys sure put a lot of weight and speculation into a completely unknown situation. Fly off the handle much? Let's wait a bit and see what's up first.

[NO_SQDeriku777]

Wow, you guys sure put a lot of weight and speculation into a completely unknown situation. Fly off the handle much? Let's wait a bit and see what's up first.

Where did I fly off the handle? Where did I speculate? I expressed the hope that an honest post-Mortem analysis of why the Master Server was down and action plan to prevent this from happening again will be provided soon.

[[TWB]Pand]

Wow, you guys sure put a lot of weight and speculation into a completely unknown situation. Fly off the handle much? Let's wait a bit and see what's up first.

I really don't think anyone is flying off the handle; however, I do think Erik is on par asking for a root cause analysis, which is SOP for any business operating an environment like this.

 

It is very concerning that there has been no acknowledgement or communication from 1CGS or 777 indicating that they are aware and working the issue.  

 

The fact that the entire US player base is potentially just waiting for someone to "wake up" in Moscow to realize there is a problem, should be an immediate red flag that they need appropriate monitoring, redundant infrastructure, and 24/7 support for a global application that people paid real money for.  

[JG13_opcode]

Even a "direct connect by IP address" option would go a long way.

 

At least then we could bypass their useless master server.

[LLv24_Zami]

Wow, you guys really wan`t to play the game, don`t you?

 

Well that`s positive but the world is not going to end now. Relax. I don`t recall this ever happening before. Things happen sometimes even if they shouldn`t.

 

Carry on    

[71st_AH_Mastiff]

Wow, you guys really wan`t to play the game, don`t you?

 

Well that`s positive but the world is not going to end now. Relax. I don`t recall this ever happening before. Things happen sometimes even if they shouldn`t.

 

Carry on    

its happened before and for a whole 4 day holiday weekend at that!!!!

[LLv24_Zami]

its happened before and for a whole 4 day holiday weekend at that!!!!

Sorry, my bad. Still world is not going to end right now.

[71st_AH_Mastiff]

back up; for how long?

[216th_Lucas_From_Hell]

This is the highest level of unnecessary drama I've ever seen, nice one

 

The server crashed for a reason or another. It's summer so there is less staff around to fix it. Whenever someone's shift starts, they come in and fix the server. Problem solved, no need for a press release or to define the future of flight simulations through that.

[wtornado]

They will not post a tech to be onsite for 15-20 players EST when the server crashes

I doubt that very much.

 

Not worth it

Even if there were 50 it is not like 5000 players were online in the game

[1PL-Husar-1Esk]

If master server is down, we should have option to manualy connect to know servers ,save they ip and name would be helpfull to. But it is minor issue.

[Dakpilot]

In several years there have been two issues....get  grip people, consider where they operate from, their circumstances may not be the same as California 

 

It must be very demoralising to be compared to the chaos/silence of the later CLoD development days (different team)..this is a very erroneous comparison

 

Cheers Dakpilot

[SYN_Mike77]

The connection between this team and the CloD team is slight in the extreme.  I think they hired a guy or two from that team and those were widely praised here as good hires.  Other than that, there is a legal connection as 1C held the rights to the name IL2.  That's it.

 

And by the way, servers are back up.

Edited July 6, 2016 by SYN_Mike77

[Jade_Monkey]

Drama much?

[wtornado]

It is solar flares bringing the server down!

 

Their Commodore 64 server ran out of spare parts they are working on it 

 

Have you ever seen a game server not crash once in awhile?

 

They fix it put it back up end of story.

[[TWB]Pand]

The issue isn't around the crash.  Things break, they always do... It's how it is handled that matters.

1. Issue:  No redundant master server?  
   Why:   No one can play multiplayer globally while it is inoperable.
   Solution: Set up another master server (or whatever failed) and load balance/cluster them. 
   Cost: $20-$30/month @ aws and maybe some coding to make the master server software multi node aware.
    
2. Issue:  Had to wait until someone in Moscow "wakes up" to see and fix the issue.  
   Why:   No appropriate monitoring, alerting, staff in place to address issues.  Customers are aware of outages before the company is.
   Solution: Write a monitoring script or use one of many free open source monitoring packages like zabbix to detect, alert, and automatically page the support team out when an issue exists.  Support does not need to be onsite.  It is 2016 and people have computers outside of the office.
   Cost:  Use existing infrastructure, or spin up a new vm @ aws, nano at <$10/month, develop oncall rotation and configure in zabbix to rotate who from existing support team gets called.
    
3. Issue:  Communication
   Why:  Customers lose confidence because they see and experience outages that the company doesn't know about.  
   Cost:  Time to implement process for support team.  Support team acknowledges the issue and posts on this forum they are working it, and if possible, a potential estimated recovery time.  After the fact, communicate what is being done to prevent this from happening again. 

These are just merely suggestions on how to address these issues, and may not fit their business plan; however, I am confident they will come up with solutions that will work going forward. 

 

I find it interesting that many of the people saying "no big deal" haven't played or played very little multiplayer in months, and likely know very little about IT infrastructure.  Just because you don't care or understand, doesn't mean that the service provider shouldn't.

[LLv24_Zami]

I find it interesting that you guys made such a drama out of it. Few hours and it was back. Lol

 

It has nothing to do how much one play MP or understand so spare us from that BS. If I can't connect, I'll do something else in the mean time and try again later. Of course it can be better but it's not end of life.

[Jade_Monkey]

Read a magazine, watch TV, go for a walk, polish your sword...

[LeRocket]

some people get cranky when they don't get their toys.

[[TWB]Pand]

As I said... people who don't like baseball, don't care if the game gets rained out. 

[Jade_Monkey]

People who like baseball understand when the game is rained out once a year.

[LLv24_Zami]

The game server has been down several times and I haven't been able to play my campaign when I wanted.

 

Still, I did not think to commit suicide because of it. I did something else and tried again.

[LeRocket]

pwcg can be used in offline mode i believe.

[LLv24_Zami]

pwcg can be used in offline mode i

believe.

True. Also other custom missions.

[Dakpilot]

The issue isn't around the crash.  Things break, they always do... It's how it is handled that matters.

1. Issue:  No redundant master server?  

   Why:   No one can play multiplayer globally while it is inoperable.

   Solution: Set up another master server (or whatever failed) and load balance/cluster them. 

   Cost: $20-$30/month @ aws and maybe some coding to make the master server software multi node aware.

    

2. Issue:  Had to wait until someone in Moscow "wakes up" to see and fix the issue.  

   Why:   No appropriate monitoring, alerting, staff in place to address issues.  Customers are aware of outages before the company is.

   Solution: Write a monitoring script or use one of many free open source monitoring packages like zabbix to detect, alert, and automatically page the support team out when an issue exists.  Support does not need to be onsite.  It is 2016 and people have computers outside of the office.

   Cost:  Use existing infrastructure, or spin up a new vm @ aws, nano at <$10/month, develop oncall rotation and configure in zabbix to rotate who from existing support team gets called.

    

3. Issue:  Communication

   Why:  Customers lose confidence because they see and experience outages that the company doesn't know about.  

   Cost:  Time to implement process for support team.  Support team acknowledges the issue and posts on this forum they are working it, and if possible, a potential estimated recovery time.  After the fact, communicate what is being done to prevent this from happening again. 

These are just merely suggestions on how to address these issues, and may not fit their business plan; however, I am confident they will come up with solutions that will work going forward. 

 

I find it interesting that many of the people saying "no big deal" haven't played or played very little multiplayer in months, and likely know very little about IT infrastructure.  Just because you don't care or understand, doesn't mean that the service provider shouldn't.

 

Your first post was sensible and to the point, should have stopped there... this one not so much

 

by the way if you are going to quote me in your sig, please do it right

 

I have signed off with Cheers Dakpilot for many years and in many forums, but I don't believe I have ever done it all in CAPS

 

Cheers Dakpilot

[Guest deleted@50488]

When the server is down, I can still play at will - in offline mode :-)