Server Shutdown error

[lishawj]

If you want to receive support we need this info to help you properly.
» Find Revision
L2J Revision Number:3225
L2JDP Revision Number:6316

Does anyone know how I may resolve this shut down error? The server just hangs there unless an admin manually closes the dos window. When no admin or server admin is around, my server is pretty much dead. I have a weekly schedule for the server to auto restart so this is becoming a problem for me. Any help identifying the issue is much appreciated.

Someone suggestioned using the option -Djava.awt.headless=true but this did not work for me.

[lishawj]

No one running L2J server on Windows Server 2003 R2 is having this issue? I had replaced all 4 GB of system RAM, re-install the OS, re-install Java runtime 6 U14, and I still can not shutdown the server normally. I had even gone as far as running my server on a completely different server and still coming up with the same error upon shutdown.

I started having this problem within the last week or so. I can no longer do automatic restart because the server now hangs upon shutdown/restart calls. The Java.exe thread go skyrocket high and uses 100% of the CPU time when the shutdown is initiated. I am getting many hits on Google when searching for "java AWT-Windows exception" but none that would help me to fix my issue.

I am running my server on the lastest nightly build (last night), Windows Server 2003 R2 (application server role), 4 GB RAM, 150GB HD, Java runtime 6 u14. I need some help here guys, please.

[_DS_]

I use 2003R2 standard, 6u3 and does not have such issue.

[janiii]

The Java.exe thread go skyrocket high and uses 100% of the CPU time when the shutdown is initiated.

this is from the shop item fix. the saving of shops could take 6 minutes on shutdown. just let it save all, wait that 6-10 minutes. like DS said, next time it should be already fine.

[lishawj]

The Java.exe thread go skyrocket high and uses 100% of the CPU time when the shutdown is initiated.

this is from the shop item fix. the saving of shops could take 6 minutes on shutdown. just let it save all, wait that 6-10 minutes. like DS said, next time it should be already fine.

Okay, now that I know it is because of the shop fix, should I expect corrupt databases etc if I am forcefully killing the Gameserver console? On the subsequent shutdown, the server will shutdown faster or will it now take 6-10 minutes to shutdown going forward? The reason I am asking is because I am doing maintenance on the server on a regular weekly schedule. What side-effects, if any, are going to be present? I haven't noticed any side effects for premature termination of the game server console but please let me know if there are.

Thank you.

[janiii]

hi,

could you try with clean core and dp? and default startGameServer.bat file? maybe some changes or custom mods are creating that error.

[MELERIX]

Log with clean Core/Dp

Code: Select all

2009.06.29 19:40:28,796 INFO    14  net.sf.l2j.gameserver.ThreadPoolManager All ThreadPools are now stoped2009.06.29 19:40:29,156 INFO    14  net.sf.l2j.gameserver.Shutdown  SIGTERM received. Shutting down NOW!2009.06.29 19:41:06,046 INFO    14  net.sf.l2j.gameserver.Shutdown  RaidBossSpawnManager: All raidboss info saved!!2009.06.29 19:41:07,562 INFO    14  net.sf.l2j.gameserver.Shutdown  GrandBossManager: All Grand Boss info saved!!2009.06.29 19:54:21,171 INFO    14  net.sf.l2j.gameserver.Shutdown  TradeController: All count Item Saved2009.06.29 19:54:22,078 INFO    14  net.sf.l2j.gameserver.Shutdown  Olympiad System: Data saved!!2009.06.29 19:54:22,437 INFO    14  net.sf.l2j.gameserver.Shutdown  Data saved. All players disconnected, shutting down.

13 minutes to shutdown!!!!

according to _DS_ this Changeset is the trouble: http://www.l2jserver.com/trac/changeset/3206

[janiii]

L2TradeList.L2TradeItem.restoreInitialCount() is making problems.
after debugging it, these numbers came out:
- called when looping through lists and items from getCurrentCount() method of a L2TradeItem
- L2TradeList.L2TradeItem.restoreInitialCount(): the while loop on line 267 loops each time 346205 times!
- mostly the RejectedExecutionException is thrown
- L2TradeList.L2TradeItem.saveDataTimer() updated for each time it was called (for each item in each list) 1488 items
-

the list has 3 items, 2 of them have count 5:

2009.06.30 07:47:58,489 TradeController: in the for-loop _lists.values(): npcId: 35383; listId: 135383
2009.06.30 07:47:58,489 TradeController: list has limited stock item
2009.06.30 07:47:58,489 TradeController: in the for-loop list.getItems()
2009.06.30 07:47:58,489 TradeController: in the for-loop list.getItems()
2009.06.30 07:47:58,598 L2TradeList.L2TradeItem.restoreInitialCount the while looped 346205 times.
2009.06.30 07:47:58,598 L2TradeList.L2TradeItem RejectedExecutionException!
2009.06.30 07:47:58,847 L2TradeList.L2TradeItem saveDataTimer updated 1488 items
2009.06.30 07:47:58,847 TradeController: in the for-loop list.getItems()
2009.06.30 07:47:58,879 L2TradeList.L2TradeItem.restoreInitialCount the while looped 346205 times.
2009.06.30 07:47:58,925 L2TradeList.L2TradeItem RejectedExecutionException!
2009.06.30 07:47:59,175 L2TradeList.L2TradeItem saveDataTimer updated 1488 items

[_DS_]

Yeah, i know about it and if today will have sufficient time - will try to get into the tradelists and optimise save.

[janiii]

it is just a try, just an idea, maybe DS can look at it though..

- calling the getCurrentCount method only once for the update
- changed the while loop to an if statement
- updating the savetimer only for the given item (not for all items with same restore delay)

http://www.pastebin.cz/20512

Code: Select all

Index: java/net/sf/l2j/gameserver/TradeController.java===================================================================--- java/net/sf/l2j/gameserver/TradeController.java (revision 3244)+++ java/net/sf/l2j/gameserver/TradeController.java (working copy)@@ -321,10 +321,11 @@                                        for (L2TradeItem item : list.getItems())                    {-                       if (item.hasLimitedStock() && item.getCurrentCount() < item.getMaxCount()) //needed?+                       long currentCount;+                       if (item.hasLimitedStock() && (currentCount = item.getCurrentCount()) < item.getMaxCount())                        {                            statement = con.prepareStatement("UPDATE merchant_buylists SET currentCount=? WHERE item_id=? AND shop_id=?");-                           statement.setLong(1, item.getCurrentCount());+                           statement.setLong(1, currentCount);                            statement.setInt(2, item.getItemId());                            statement.setInt(3, listId);                            statement.executeUpdate();Index: java/net/sf/l2j/gameserver/model/L2TradeList.java===================================================================--- java/net/sf/l2j/gameserver/model/L2TradeList.java   (revision 3222)+++ java/net/sf/l2j/gameserver/model/L2TradeList.java   (working copy)@@ -264,9 +264,9 @@             _nextRestoreTime = _nextRestoreTime + this.getRestoreDelay();                          // consume until next update is on future-            while (this.isPendingStockUpdate() && this.getRestoreDelay() > 0)+            if (this.isPendingStockUpdate() && this.getRestoreDelay() > 0)             {-                _nextRestoreTime = _nextRestoreTime + this.getRestoreDelay();+                _nextRestoreTime = System.currentTimeMillis() + this.getRestoreDelay();             }                          // exec asynchronously@@ -352,9 +352,10 @@             try             {                 con = L2DatabaseFactory.getInstance().getConnection();-                PreparedStatement statement = con.prepareStatement("UPDATE merchant_buylists SET savetimer =? WHERE time =?");+                PreparedStatement statement = con.prepareStatement("UPDATE merchant_buylists SET savetimer =? WHERE time =? and item_id =?");                 statement.setLong(1, _nextRestoreTime);                 statement.setInt(2, (int) (this.getRestoreDelay()/60/60/1000));+                statement.setInt(3, this.getItemId());                 statement.executeUpdate();                 statement.close();             } 

[lishawj]

I create two new multisell NPC around the time the shutdown error appeared. I did try a clean core and dp but I also imported the new custom NPC over. Today, after work I will try clean core and dp without the new custom NPC. Thank you guys!

[MELERIX]

it is just a try, just an idea, maybe DS can look at it though..

- calling the getCurrentCount method only once for the update
- changed the while loop to an if statement
- updating the savetimer only for the given item (not for all items with same restore delay)

http://www.pastebin.cz/20512

Code: Select all

Index: java/net/sf/l2j/gameserver/TradeController.java===================================================================--- java/net/sf/l2j/gameserver/TradeController.java (revision 3244)+++ java/net/sf/l2j/gameserver/TradeController.java (working copy)@@ -321,10 +321,11 @@                                        for (L2TradeItem item : list.getItems())                    {-                       if (item.hasLimitedStock() && item.getCurrentCount() < item.getMaxCount()) //needed?+                       long currentCount;+                       if (item.hasLimitedStock() && (currentCount = item.getCurrentCount()) < item.getMaxCount())                        {                            statement = con.prepareStatement("UPDATE merchant_buylists SET currentCount=? WHERE item_id=? AND shop_id=?");-                           statement.setLong(1, item.getCurrentCount());+                           statement.setLong(1, currentCount);                            statement.setInt(2, item.getItemId());                            statement.setInt(3, listId);                            statement.executeUpdate();Index: java/net/sf/l2j/gameserver/model/L2TradeList.java===================================================================--- java/net/sf/l2j/gameserver/model/L2TradeList.java   (revision 3222)+++ java/net/sf/l2j/gameserver/model/L2TradeList.java   (working copy)@@ -264,9 +264,9 @@             _nextRestoreTime = _nextRestoreTime + this.getRestoreDelay();                          // consume until next update is on future-            while (this.isPendingStockUpdate() && this.getRestoreDelay() > 0)+            if (this.isPendingStockUpdate() && this.getRestoreDelay() > 0)             {-                _nextRestoreTime = _nextRestoreTime + this.getRestoreDelay();+                _nextRestoreTime = System.currentTimeMillis() + this.getRestoreDelay();             }                          // exec asynchronously@@ -352,9 +352,10 @@             try             {                 con = L2DatabaseFactory.getInstance().getConnection();-                PreparedStatement statement = con.prepareStatement("UPDATE merchant_buylists SET savetimer =? WHERE time =?");+                PreparedStatement statement = con.prepareStatement("UPDATE merchant_buylists SET savetimer =? WHERE time =? and item_id =?");                 statement.setLong(1, _nextRestoreTime);                 statement.setInt(2, (int) (this.getRestoreDelay()/60/60/1000));+                statement.setInt(3, this.getItemId());                 statement.executeUpdate();                 statement.close();             } 

Tested.

Now shutdown is stucked in SIGTERM received. Shutting down NOW! & java.exe 100% cpu usage XD

[lishawj]

Just applied the patch and now my server does not even reach the AWT-Windows exception error. It's now 25 minutes and counting...

[janiii]

Tested.

Now shutdown is stucked in SIGTERM received. Shutting down NOW! & java.exe 100% cpu usage XD

i tested it too. this only changes the trade list save. shouldnt change anything else. for me it decreased the saving time of the tradelist.

Just applied the patch and now my server does not even reach the AWT-Windows exception error. It's now 25 minutes and counting...

the patch should only decrease the time of the high cpu usage on shutdown. the awt-error probably comes from some custom code, but not sure. would be good if someone could reproduce the awt-error too.

[lishawj]

Tested.

Now shutdown is stucked in SIGTERM received. Shutting down NOW! & java.exe 100% cpu usage XD

i tested it too. this only changes the trade list save. shouldnt change anything else. for me it decreased the saving time of the tradelist.

Just applied the patch and now my server does not even reach the AWT-Windows exception error. It's now 25 minutes and counting...

the patch should only decrease the time of the high cpu usage on shutdown. the awt-error probably comes from some custom code, but not sure. would be good if someone could reproduce the awt-error too.

Okay, I will start with clean core and dp again today without patching it with any of the custom codes I used. On the last test, I used clean core and dp but I patched it with a few custom codes without the custom NPC.

With clean core and dp using only above patch, I was able to shutdown. Now I will have to slowly add my custom item and see what happens.