Server freezes (Fixed!)

  1. ‹ Older
  2. 9 years ago

    My first pretty quick testing shows the same:

    I used the same mission as before and flew the same pattern. Above that, since it did not crash, I generated a heavy MCC mission. Still working good. My RAM usage fluctuate between 1965-1970MB.

    Lets watch it a bit more (will let this mission run over the night)

  3. Just tried ALiVE with our full mod pack and it crashed after about 30 minutes of game play. Not going to bother uploading the RPT considering we were trying with our full mod pack.. but I do have it if requested.

    There's still some kind of issue somewhere so we're sitting tight.

  4. Tupolov

    19 Feb 2016 Administrator
    Edited 9 years ago by Tupolov

    Rog thanks for the report :(

    Did the server crash or was it the freeze issue?

  5. Edited 9 years ago by uk_apollo

    First I'm going to say what a wonderful mod @alive is, and how it has been enjoyed by all of us at 3 Commando Brigade on every single one of our busy public server missions for well over a year. Awesome job, thanks!

    Unfortunately we have also been suffering with the "freezes" in this winter weather. Identical to all the reports you've previously had here. We do run a large modset so I won't bore you with rpt's as I know you need minimal mod installations to help find the solution. But I thought it worth reporting the issue to add to the data-set, especially as we have built up a lot of experience running Alive on 24/7 dedi servers catering for 20-40 players on most days of the year.

    Symptoms

    • players see others freeze in place or walking on the spot
    • vehicles rubber band
    • players will claim to be at the same place but not see each other
    • players may not be able to join in progress
    • red chains sometimes, but not always
    • desync = 0 in map/player info
    • suddenly occurs for all players, seemingly randomly from 5 minutes to 3 hours into a mission
    • server rpt huge, spammed with specific error messages (example below)
    • server reports very low on virtual memory (memory leak)

    Facts

    • it started happening sometime after the A3 v1.54 update and/or updates to @alive and @ace3 at that time. The same modset and same missions prior to that were stable.
    • the problem goes away if you remove @alive from a larger modset
    • it's still freezing with the Eden update
    • not all of our missions freeze: we have a number which use @alive/@ace3/@rhs_ amongst other mods, which appear to be 100% stable (10 hours+ up-time)

    Our conclusions so far

    • it's not related to spawn density (happens with minimal ai numbers spawned, even with all players still at base)
    • it's not player number related (has happened with just 2)
    • it's not related specifically to being in combat (has happened with all players in a blacklisted base)
    • it's probably not a specific @ace3 module setting, as all of our missions have identical parameters (ballistics etc); some work, others don't
    • it's probably not the 3rd party Opfor type selected in the @alive modules - server freezes with BIS AAF enemy too (although to be fair, RHS/Leights is still loaded in the modset, but not being used)
    • it doesn't appear to be related to a particular @alive module (such as C2Istar); I knocked it back to the bare minimum last night and still saw rpt spam; asymmetric or conventional warfare alike
    • it probably is a conflict between @alive and either @ace3 or @rhs_ (possibly @leights_opfor) apparent only since an update from about a month ago, in agreement with the consensus opinion here

    Server RPT sample
    17:39:30 WARNING: SuppressCache::Trace called with huge input. Function exited to prevent program stall inside. tBeg[-0.000000,165.000000,-0.818642]; tEnd[0.000000,0.574304,12276.468750]; ammoHit(1.745155); distance(0.000000)
    17:39:30 In last 500 miliseconds was lost another 2 these messages.
    17:39:49 Virtual memory total 4095 MB (4294836224 B)
    17:39:49 Virtual memory free 57 MB (59785216 B)
    17:39:49 Physical memory free 25890 MB (27147853824 B)
    17:39:49 Page file free 56668 MB (59421413376 B)
    17:39:49 Process working set 3530 MB (3701755904 B)
    17:39:49 Process page file used 3762 MB (3944747008 B)
    17:39:49 Longest free VM region: 3604480 B
    17:39:49 VM busy 4233289728 B (reserved 150560768 B, committed 4082728960 B, mapped 74436608 B), free 61546496 B
    17:39:49 Small mapped regions: 24, size 106496 B
    17:41:40 NetServer::finishDestroyPlayer(1970388062): DESTROY immediately after CREATE, both cancelled

    The WARNING message is spammed, thousands of times.

    www.3commandobrigade.com

  6. My server is after nearly 14h still running! Dunno if its fixed or we just have very much luck.

    A more in reasonable test will be done tomorrow. (Meaning we will actually play then! ^^)

  7. ditto to UK apollo. Ran my British Steel mission with another friend for over 2 hours with no issues. Thought we were in the clear. Called for CAS and a Helo Transport. As soon as we are about to land on the helo....desync.

    Checked the RPT...same out of memory crash that apollo mentioned above.

    Immediately restarted the server and launched the helicopters again, told them to land...landed fine this time.

    So....in conclusion....still getting crashes....not happening "consistently" in any one instance.

    My British Steel mission uses ALiVE + ACE...no RHS. 3CB and CAF Aggressors are the unit packs.

    I will say...there is a definite difference though in performance. I think we're close.

  8. Edited 9 years ago by SavageCDN

    Anyone still experiencing a server freeze - can you post what version of CBA_A3 you are using?

  9. @Tupolov Rog thanks for the report :(

    Did the server crash or was it the freeze issue?

    It froze. We were using Alive 1.0 and CBA 2.3. When I saw mention of no crashes with ACE and Alive using the updated CBA, I thought maybe it was CBA all along and decided to try it out. Looks like something still doesn't like ALiVE.

  10. I got the freeze w the latest version of everything, CBA alive ace, etc

  11. Tupolov

    19 Feb 2016 Administrator

    are you guys running CQB? What modules are you putting down?

  12. SOAR tested ALiVE/ACE/CBA/RHS/Leights last night with 1.56. We had CQB module placed down independent of another module. Along with the required 4 ALiVE modules, 1 opcommander module, and 1 civ objective module, as well as civilian placement/population modules. Pretty much a full mission.

  13. @Tupolov are you guys running CQB? What modules are you putting down?

    The entire Alive suite. Cqb, logistics, etc

  14. Edited 9 years ago by uk_apollo

    A3 1.56
    @cba_a3 v2.3.0
    Yes to cqb module.

    One of our public server Alive missions repeatedly fails with freezes so we stopped running it for now.

    With the same modset (incl. @alive, @ace3, @rhs_afrf3) another Alive mission has worked fine all day and was up for 28 hours.

    https://www.gametracker.com/server_info/144.76.162.200:2362/
    Around 25-30 players this evening.

    I don't envy you folks trying to sort this problem out!

  15. Tupolov

    20 Feb 2016 Administrator

    @uk_apollo A3 1.56
    @cba_a3 v2.3.0
    Yes to cqb module.

    One of our public server Alive missions repeatedly fails with freezes so we stopped running it for now.

    With the same modset (incl. @alive, @ace3, @rhs_afrf3) another Alive mission has worked fine all day and was up for 28 hours.

    https://www.gametracker.com/server_info/144.76.162.200:2362/
    Around 25-30 players this evening.

    I don't envy you folks trying to sort this problem out!

    can u share both missions?

  16. bump - Apollo any way we can look at those missions?

  17. Are you guys sure you're using CBA 2.3.0 and not 2.3.1? I had zero freezes with 2.3.0 when before the server froze in 10 minutes. After updating to 2.3.1 the server froze after 20 min. I will try reverting back to 2.3.0 and see what happens.

  18. @SavageCDN - private message sent, sorry for delay
    @JerkinMerkin - yes, we are running cba 2.3.0 and have had freezes on some missions

  19. Edited 9 years ago by SOAR_Jooce

    Performance Binary Thread

    I know the callExtension issue was brought up once and you responded to it @Tupolov , but could something have changed in 1.54 that doesn't like how you're managing the return values?

    I'm very ignorant of mod development and .sqf but I can't help but be curious since it's been brought up a couple times now in that thread.

  20. I guess i can piggy-back on this thread regarding the server hangs and crashes.

    My friend and I tried to make an Asymetric mission the other week and started to have these errors, mostly the server hangs and crashes after its memory has been filled up.

    So we dug into the problem and started a dev with @ALiVE (1.0.0.1602091) and @CBA (2.3.1) only (using the stable branch). The mission is made on Altis and in the "Eden" editor the mission.sqm binarization is turned off. Here are my impressions:

    -It seems to come from the Installation placement in the MIlitary AI commander, when activated, in local server i get the following error "Error undefined variable in expression: _state" "x\alive\addons\fnc_analysis\fnc_liveanalysis.sqf, line 514" when you start running trough the map towards the enemy "bastion" once the modules loaded up (until the Intel appears).

    -Removing the Assymetric mode or just simply deactivating the Installations stops the error. Other modes like Invasion and Occupation doesnt generate the error AFAIK.

    When loaded up on the dedicated, you can start the mission, load it up at 100%, leave it run for as many hours/days as you want it wont crash, its only when you log in and start walking towards the enemy that the error occurs. It freezes the server and chugs up all the RAM until the dedicated server crashes. You won't notice this if you play alone, only that the server on "#monitor" doesn't post anymore after a while.

    I did a "portfolio" with the mission file, and the two RPT's (one from my local machine testing in local host, and the other from our dedicated (*HUGE RPT 24mb*)) and also a screencapture of the error i got in local host running this mission. The mission inside is quite light (couple of kbs) but its made to stress test our machine also, we didnt want to overlook certain aspects, so if im correct this mission takes a bit to load up and has 86 objectives triggering the "80" limit warning. Also, if the mission is badly "structured" with the custom objectives let me know, for all i know maybe im trying to do something impossible with ALiVE and just plainly breaking it because im overwhelming the Military Commander with multiple Custom Objectives (is there a limit to the number you can synch to it?)

    Ive read that it might be the error that HighHead is working on but i might be mistaken. Hoping for the best.

    FTP download

    Hope this helps

  21. Edited 9 years ago by SpyderBlack723

    @SOAR_Jooce Performance Binary Thread

    I know the callExtension issue was brought up once and you responded to it @Tupolov , but could something have changed in 1.54 that doesn't like how you're managing the return values?

    I'm very ignorant of mod development and .sqf but I can't help but be curious since it's been brought up a couple times now in that thread.

    Crashes occur without the plugin doing anything so it's unlikely it's causing any problem.

  22. 9 years ago

    Tupolov

    26 Feb 2016 Administrator

    @SOAR_Jooce I know the callExtension issue was brought up once and you responded to it @Tupolov , but could something have changed in 1.54 that doesn't like how you're managing the return values?

    In relation to freezes, its not applicable as we have had freezes without using the extension (which is only required for persistence on server)

  23. Does someone found what causing this hangs up?

  24. Edited 9 years ago by SpyderBlack723

    It is still under investigation. If you are able to supply BI with memory dumps from a crashed/frozen session they can take a deeper look.

  25. Maybe it would be wise for the ALiVE team to start up a thread on the BI forums so we can all post our crash logs in one place. Though I don't think Bohemia is going to really offer to help fix anything that is not part of the vanilla game.

  26. I got a 98MB Rpt file from oure last game if you want it??

  27. Thanks but not needed. Whatever is happening does not get written to the RPT file. I'm not going to say much more other than the problem is still being worked on and that the devs are getting closer. Thanks everyone for your patience.

  28. At this point, that's all that needs to be said. I'm glad to hear it. If at any point the ALiVE team would like the updates tested prior to an official release, feel free to contact SOAR.

  29. Will do.. and thanks to the SOAR team for providing valuable reports

  30. I just thought I'd mention that the crashes occur even when playing the mission without any ACE-modules on the map (but with ACE activated).

    I hope the issue get's resolved! My group just started a homemade campaign using alive, ace and RHS and we haven't been able to play it due to the server crashes either. While this is frustrating to us I can imagine that it's even more frustrating for the devs, so I wish you the best of luck and just wanted to say that all of us in our group appreciates your mod and the work you do, thank you!

  31. Tupolov

    2 Mar 2016 Administrator

    So we are getting closer to understanding the issue. We now have a solid repro (without ACE or RHS).

    We believe its related to some of the remote AI functions/commands BIS provide that we leverage when distributing AI between clients and HC. We can repro when using an HC or at least putting down the HC in the mission. The mission needs at least 2 people connected.

    Just an FYI. We are continuing to chase this bug

  32. Awesome to hear, keep up the good work. Hopefully the chase is narrowing down now! Best of luck.

  33. So does the player and that same player flying around in Zues count as 2 players? Its frozen on me when doing that.

  34. Tupolov

    4 Mar 2016 Administrator

    We found 2 bugs that are causing a freeze in certain scenarios. One was related to a BIS function we were using. The other was related to how we were using remote AI commands.

    We have a build planned for this weekend and hope to release soon. The build will include a bunch of fixes for CBA, RHS, 1.56 and much more.

    We can't guarantee that this build will fix all freeze issues, we will need you guys to test to confirm.

  35. Ok, good to hear. With the current build, I had the impression - as already mentioned in the previous thread by numerous people - that no single mod alone causes the freezes, but that each additional mod on top of the (by the functions you identified as per above posting) already strained server can make him more and more unstable up until the point where he crashes.

    As a temporary fix I tested to play either with RHS OR with ACE, but not with both. This increased my server stabilty to a level where playing for 2-5 hours was possible. But mostly I was alone on my server and only once played for about 3 hours with a buddy in a No-ACE-COIN Mission.

    Happy to hear that a fix is on the way and will gladly test the new build with the same modset I use now.

  36. @Tupolov We found 2 bugs that are causing a freeze in certain scenarios. One was related to a BIS function we were using. The other was related to how we were using remote AI commands.

    We have a build planned for this weekend and hope to release soon. The build will include a bunch of fixes for CBA, RHS, 1.56 and much more.

    We can't guarantee that this build will fix all freeze issues, we will need you guys to test to confirm.

    Thanks!

  37. Tupolov

    4 Mar 2016 Administrator

    Thanks guys. We can repro the freeze when we hammer the server (lots of profiles, HC, persistence etc) with 2 clients connected. Like I said, I suspect there might be a couple of scenarios where a freeze can happen but we believe we have fixed 2 issues that likely contribute to the most common one.

    Hoping to get a test build for you guys very soon!

  38. Thanks Tupolov!

  39. Schweeet! Excited for the new build!

  40. @Tupolov We found 2 bugs that are causing a freeze in certain scenarios. One was related to a BIS function we were using. The other was related to how we were using remote AI commands.

    We have a build planned for this weekend and hope to release soon. The build will include a bunch of fixes for CBA, RHS, 1.56 and much more.

    We can't guarantee that this build will fix all freeze issues, we will need you guys to test to confirm.

    This makes me happy in my face region.

  41. Serious thanks for all the hard work on tracking this down, @Tupolov and the rest of the Alive team!

  42. Newer ›
 

or Sign Up to reply!