epilogue server crash at every 9 hours!

[thorl2]

i just moved 2 day ago to l2j project, server is compiled and not nightlies. Server work superb whit more than 200+ online, but at every 9 hours server crash with an pool thread error and i have to restart it by force... Server is fresh so is not updated to be a reason to miss some files...
few of my setting are:

cpu: quad q6600, ram 8gb, hdd raptor 15000 rpm 73 gb
windos 2003 sp2 x64, java update 20, latest mysql

mysql:
max_connections=600
tmp_table_size=256M
innodb_log_buffer_size=16M
innodb_buffer_pool_size=1024M
innodb_thread_concurrency=60

"C:\Program Files\Java\jre6\bin\java" -Djava.util.logging.manager=com.l2jserver.util.L2LogManager -Xms1024m -Xmx4096m -cp ./../libs/*;l2jserver.jar com.l2jserver.gameserver.GameServer

ThreadPoolSizeEffects = 50
ThreadPoolSizeGeneral = 65
#Default 2
UrgentPacketThreadCoreSize = 10
#Default 4
GeneralPacketThreadCoreSize = 20
#Default 4
GeneralThreadCoreSize = 20
AiMaxThread = 20
MaximumDbConnections = 600

server crash even on default server/mysql configs

» Find Revision
L2J Revision Number: 7208
L2JDP Revision Number: 4089M

the error:

25.04.2010 02:13:12 com.l2jserver.gameserver.model.actor.L2Character$HitTask run
SEVERE: Failed executing HitTask.
java.lang.NullPointerException
25.04.2010 02:41:09 com.mchange.v2.resourcepool.BasicResourcePool prelimCheckoutResource
WARNING: com.mchange.v2.resourcepool.BasicResourcePool@24148662 -- an attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have either interrupted the Thread attempting checkout!
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at com.l2jserver.L2DatabaseFactory.getConnection(L2DatabaseFactory.java:221)
at com.l2jserver.gameserver.model.L2ItemInstance.updateInDb(L2ItemInstance.java:1521)
at com.l2jserver.gameserver.model.L2ItemInstance.updateDatabase(L2ItemInstance.java:1338)
at com.l2jserver.gameserver.model.L2ItemInstance.updateDatabase(L2ItemInstance.java:1309)
at com.l2jserver.gameserver.model.itemcontainer.ItemContainer.addItem(ItemContainer.java:266)
at com.l2jserver.gameserver.model.itemcontainer.PcInventory.addItem(PcInventory.java:424)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.addItem(L2PcInstance.java:3604)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.doAutoLoot(L2PcInstance.java:4678)
at com.l2jserver.gameserver.model.actor.L2Attackable.doItemDrop(L2Attackable.java:1501)
at com.l2jserver.gameserver.model.actor.L2Attackable.doItemDrop(L2Attackable.java:1412)
at com.l2jserver.gameserver.model.actor.L2Attackable.calculateRewards(L2Attackable.java:578)
at com.l2jserver.gameserver.model.actor.L2Character.doDie(L2Character.java:2103)
at com.l2jserver.gameserver.model.actor.L2Npc.doDie(L2Npc.java:1358)
at com.l2jserver.gameserver.model.actor.L2Attackable.doDie(L2Attackable.java:506)
at com.l2jserver.gameserver.model.actor.instance.L2MonsterInstance.doDie(L2MonsterInstance.java:206)
at com.l2jserver.gameserver.model.actor.status.CharStatus.reduceHp(CharStatus.java:197)
at com.l2jserver.gameserver.model.actor.status.NpcStatus.reduceHp(NpcStatus.java:51)
at com.l2jserver.gameserver.model.actor.status.AttackableStatus.reduceHp(AttackableStatus.java:49)
at com.l2jserver.gameserver.model.actor.L2Character.reduceCurrentHp(L2Character.java:6810)
at com.l2jserver.gameserver.model.actor.L2Attackable.reduceCurrentHp(L2Attackable.java:477)
at com.l2jserver.gameserver.model.actor.L2Attackable.reduceCurrentHp(L2Attackable.java:413)
at com.l2jserver.gameserver.skills.l2skills.L2SkillDrain.useSkill(L2SkillDrain.java:183)
at com.l2jserver.gameserver.model.actor.L2Character.callSkill(L2Character.java:6560)
at com.l2jserver.gameserver.model.actor.L2Character.onMagicHitTimer(L2Character.java:6159)
at com.l2jserver.gameserver.model.actor.L2Character$MagicUseTask.run(L2Character.java:2552)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
25.04.2010 02:42:13 com.mchange.v2.resourcepool.BasicResourcePool prelimCheckoutResource
WARNING: com.mchange.v2.resourcepool.BasicResourcePool@24148662 -- an attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have either interrupted the Thread attempting checkout!
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at com.l2jserver.L2DatabaseFactory.getConnection(L2DatabaseFactory.java:221)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.storeCharBase(L2PcInstance.java:7638)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.store(L2PcInstance.java:7609)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.store(L2PcInstance.java:7621)
at com.l2jserver.gameserver.network.L2GameClient.saveCharToDisk(L2GameClient.java:311)
at com.l2jserver.gameserver.network.L2GameClient$AutoSaveTask.run(L2GameClient.java:736)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
25.04.2010 02:42:22 com.mchange.v2.resourcepool.BasicResourcePool prelimCheckoutResource
WARNING: com.mchange.v2.resourcepool.BasicResourcePool@24148662 -- an attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have either interrupted the Thread attempting checkout!
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at com.l2jserver.L2DatabaseFactory.getConnection(L2DatabaseFactory.java:221)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.storeCharBase(L2PcInstance.java:7638)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.store(L2PcInstance.java:7609)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.store(L2PcInstance.java:7621)
at com.l2jserver.gameserver.network.L2GameClient.saveCharToDisk(L2GameClient.java:311)
at com.l2jserver.gameserver.network.L2GameClient$AutoSaveTask.run(L2GameClient.java:736)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
25.04.2010 02:42:30 com.mchange.v2.resourcepool.BasicResourcePool prelimCheckoutResource
WARNING: com.mchange.v2.resourcepool.BasicResourcePool@24148662 -- an attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have either interrupted the Thread attempting checkout!
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at com.l2jserver.L2DatabaseFactory.getConnection(L2DatabaseFactory.java:221)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.storeCharBase(L2PcInstance.java:7638)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.store(L2PcInstance.java:7609)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.store(L2PcInstance.java:7621)
at com.l2jserver.gameserver.network.L2GameClient.saveCharToDisk(L2GameClient.java:311)
at com.l2jserver.gameserver.network.L2GameClient$AutoSaveTask.run(L2GameClient.java:736)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
25.04.2010 02:42:33 com.mchange.v2.resourcepool.BasicResourcePool prelimCheckoutResource
WARNING: com.mchange.v2.resourcepool.BasicResourcePool@24148662 -- an attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have either interrupted the Thread attempting checkout!
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at com.l2jserver.L2DatabaseFactory.getConnection(L2DatabaseFactory.java:221)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.storeCharBase(L2PcInstance.java:7638)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.store(L2PcInstance.java:7609)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.store(L2PcInstance.java:7621)
at com.l2jserver.gameserver.network.L2GameClient.saveCharToDisk(L2GameClient.java:311)
at com.l2jserver.gameserver.network.L2GameClient$AutoSaveTask.run(L2GameClient.java:736)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

i really apreciate some help since i'm new for this project... thx!

[Onepamopa]

From what I understand, you maxed thread settings and your computer cant handle ALL of them.
try lowering the values and check if the machine runs out of recources.

Try to return thread settings to their default values and see if the problem occurs.
Also check your starting script, and make sure it has enough ram allocated for the server.
(and DONT set it too high!)

[JIV]

with 200 players use default thread settings. About mysql link error can be unclosed connection, try enable debug.

[thorl2]

default setting still the same but more time up 15 hours, and this time no more errors cleam console

[Onepamopa]

well, your server runs out of resources...
We use core 2 quad 4ghz with 8 gb ddr2 and 1180 users - no such problems, so... this is either caused by machine resources or wrong configuration.

[Szponiasty]

Looks that u have glith that doesnt properly close server connections. To confirm, add this to your server starting line:

Code: Select all

 # add this to set forcing connection close:#  -Dc3p0.unreturnedConnectionTimeout=1500 -Dc3p0.debugUnreturnedConnectionStackTraces=truejava -Dc3p0.unreturnedConnectionTimeout=1500 -Dc3p0.debugUnreturnedConnectionStackTraces=true -Djava.util.logging.manager=com.l2jserver.util.L2LogManager ........... 

You should now get stacktrace on console when problems appear. You may also configure to make forceclosing mysql connections that are idle for certain time:

Code: Select all

 java -Dc3p0.unreturnedConnectionTimeout=1500 -Dc3p0.debugUnreturnedConnectionStackTraces=true -Dc3p0.maxStatements=150 -Dc3p0.maxIdleTime=300 ...... 

If by adding that one above your server stop crashing, look for places in scripts and core where mysql connection is opened and verify all are correctly closed when not needed anymore.

PS. Maybe L2J should add forcibly closing mysql connection to trunk? As far as I know no connection is needed for longer than 5 minutes, and that simple addon helps a lot debugging not to mention it prevents server from crashing in case that there is some mysql connection glith...

[thorl2]

thank you! i just added the code in the gameserver.bat and i will restart server in few hours and see if dyes again.

[JIV]

Maybe L2J should add forcibly closing mysql connection to trunk? As far as I know no connection is needed for longer than 5 minutes, and that simple addon helps a lot debugging not to mention it prevents server from crashing in case that there is some mysql connection glith...

there is already implemented unclosed connection checker, but its enabled by debug only because there is no reason to have it running for each connection request. Still its not L2J problem if someone use crappy code, clean core close everything.

[Szponiasty]

Maybe L2J should add forcibly closing mysql connection to trunk? As far as I know no connection is needed for longer than 5 minutes, and that simple addon helps a lot debugging not to mention it prevents server from crashing in case that there is some mysql connection glith...

there is already implemented unclosed connection checker, but its enabled by debug only because there is no reason to have it running for each connection request. Still its not L2J problem if someone use crappy code, clean core close everything.

I've firstly used this cause of crashes due to unknown problem. Then found problem source - it was L2J's crappy UI config routines

[JIV]

im talking about current revision.

[thorl2]

server is working for a few hour not yet 15 hours to see if it crash again but whit szponiasty code i receice a debug error maybe this can be usefull:


26.04.2010 02:43:52 com.mchange.v2.resourcepool.BasicResourcePool removeResource
INFO: Logging the stack trace by which the overdue resource was checked-out.
java.lang.Exception: DEBUG ONLY: Overdue resource check-out stack trace.
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:506)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at com.l2jserver.L2DatabaseFactory.getConnection(L2DatabaseFactory.java:221)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.__call__(Unknown Source)
at org.python.core.PyMethod.__call__(Unknown Source)
at org.python.core.PyObject.__call__(Unknown Source)
at org.python.core.PyInstance.invoke(Unknown Source)
at org.python.pycode.serializable._pyx1272231202828.onAdvEvent$14(__init__.py:756)
at org.python.pycode.serializable._pyx1272231202828.call_function(__init__.py)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyFunction.__call__(Unknown Source)
at org.python.core.PyMethod.__call__(Unknown Source)
at org.python.core.PyObject.__call__(Unknown Source)
at org.python.core.PyObject._jcallexc(Unknown Source)
at org.python.core.PyObject._jcall(Unknown Source)
at org.python.proxies.main$Quest$415.onAdvEvent(Unknown Source)
at com.l2jserver.gameserver.model.quest.Quest.notifyEvent(Quest.java:406)
at com.l2jserver.gameserver.model.actor.instance.L2PcInstance.processQuestEvent(L2PcInstance.java:1727)
at com.l2jserver.gameserver.network.clientpackets.RequestBypassToServer.runImpl(RequestBypassToServer.java:218)
at com.l2jserver.gameserver.network.clientpackets.L2GameClientPacket.run(L2GameClientPacket.java:92)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

[BiggBoss]

as JIV said, you have a jython crappy script, which custom jython scripts are you using?

[thorl2]

i have only 2 custom in my core:

1. NPC-PC fake
2. Biohazard event

that is all....

[BiggBoss]

post both codes

[janiii]

do you have any php web scripts that access your database? (account controll panel, online player list, etc.) ?