L2J Server

Good morning ^^ here Migi asking for help

I'm wondering if my machines are enough or not, so they are optimized like on Nitrozik's guide, but.. when there's 200-300 ppl at the same place, it lags like hell (in events, yes).. but when playing normal, lvling or in party the connection/performance is fine.
I can add also that during an event, while the event people is lagged, if you aren't at this event you can walk fine. The lag is only for the event participants/expectators.

Now let's see my machines and what is hosted in each one:


SERVER 1:

Core: Intel Xeon Quad 4x 2.66+ GHz 12 MB L2 - FSB 1333 MHz 45nm
RAM Memory: 8 GB DDR2
HD: 2x 750 GB
Connection: 1Gbps

Here I host the main db (4 GB dedicated to it's cache), the web server and the mid-rate server (200 ppl tops and no lag)

SERVER 2:

Core: Intel Core2Duo 2x 3.00+ GHz 6 MB L2 - FSB 1333 MHz 45nm
RAM Memory: 4 GB DDR2
HD: 2x 1000 GB
Connection: 100 Mbps

Here we can find the low-rate server, remotely connected to the 1st server for the database. With 300 ppl in top moments, it almost never lags (well, it is growing and soon it will lag.. I know).

SERVER 3:

Core: Intel Xeon Quad 4x 2.66+ GHz 12 MB L2 - FSB 1333 MHz 45nm
RAM Memory: 8 GB DDR2
HD: 2x 750 GB
Connection: 1Gbps

Finally the problem: our high rates server, which lastly reached 750 ppl tops and in events most of players get freezed by lag. It connects to the main DB remotely, as the 2nd server.


Maybe I'm stupid and I'm using all this rly bad, with errors such as having only one database server (I need to have all the DBs in the same server, for the webpage queries..), but if you think that it is solvable looking for a better machine or changing something.. let me know please ^^

Ah, I use windows 2003 server, and it shows me only 1 hard disk on every server ;S

Thank you!! And sorry for this newbie english..

It is all probably just client lag.

- Client lag is when your frames stop and your screen freezes entirely.
- Server side lag or connection lag is when you stop, you cannot do anything but your character is still doing the little standby or combat mode animation...you know...when the character balaces.

The lag on your events is due to the number of players in one place. That is not just a server lag but more a client lag which is caused by the flood of packets sent by the server to every client.

Let's say there are 300 players in one place, everyone in the knowlist of the other...so they all can see eachother.
If they are all moving we can say there is an average of 2 packets per second send to every other client.
That would be 600 packets per second JUST FROM THE PEOPLE AROUND YOU (and they are just moving, not chatting, not casting any skill or changing their status in any way).

Now, do you really think the client can manage that much? I doubt.

If there is no lag to the players which are not in the events then the problem is the connection/client packet flooding, not at all the server.

A way to minimize this would be to make your players use a computer which matches the requrements of the game (I know, the client runs with alot less, but in these situations you need a powerful computer to manage all the packets the client receives.).
Also you could ask your players to use Alt+P which activates the minimum frame mode with the worse details.

I cannot think you could add a packet flood protector because that would mess up everything. The clients will not be updated as they should and it will be all weird.


Oh..and by the way...wait for L3, they will use their own unreal engine so no more UE2

Yesterday I've seen one of these events and I couldn't walk.. there were 5-10 seconds retards ;(
And of course, the graphic lags.. but this is client-side ^^

I'll solve it programming instantiable events for now. But tell me, it is an error to have all the DBs hosted in 1 machine? Or.. Should I change some machine?

I am afraid I cannot give a very accurate answer to that but there should be no problem.

Your HardDisks, at what rpm are they?

If you got anything below 15k rpm, then that could cause some serious delay, and by the sound of it if you got 500GBs and 1TB worth of space, they ain't 15k rpms.

Also if your accessing dbs remotely from 1 machine to another, that just completely sucks up your I/O Usage which is another lag-helper.

Other than those 2, you got some very solid machines.

Once last thing, are u using 32-bit OS, or a 64-bit OS?

I can't find the disk rpms anywhere ;S
And of course, the OS is 64-bit!! xD

I think that it is I/O problem, but not for the HD, it's the connection..

My provider is www.ovh.com

Thank you ^^

Saso wrote:Your HardDisks, at what rpm are they?
Also if your accessing dbs remotely from 1 machine to another, that just completely sucks up your I/O Usage which is another lag-helper.

I just planning to setup a server but this is disappointed me a bit. I have dedicated mysql box with 2 xeon quad cpus and 24 gig ram onboard, 6 sas drives (3 mirrors of 2 drives each). This box is not to busy, 10 gig of ram free, 95% idle on each of 8 cores and one disk free that i'm planning to use solely for l2j database. Also this box have 4 intel gigabit nics, one of this nics i'm planning to use for direct game server connection - will be in same rack with sql box. Game server will be dual-core athlon with 4 gig of ram, 2 nics - one for sql server connection and another for players, both boxes ubuntu x64 server. Maybe it is an little overkil for around 500 players online, but I belive it will give me less lags that single box setup. I have no idea how l2j server works under load, now i have server based on c4, that forbidden to be named here.

migi-chan wrote:I can't find the disk rpms anywhere ;S
And of course, the OS is 64-bit!! xD

I think that it is I/O problem, but not for the HD, it's the connection..

My provider is http://www.ovh.com

Thank you ^^

Write here the model name of your hard-disk ( control panel -> system -> device manager -> Disk drives )

I want to buy a dedicated server from them but i don`t think that they have hard-disk with more than 7200rpm ?
If somebody have server there and know better ... ?

Your hardware is more than enough. It doesnt seem to me a DB lag (which is not expected for such amount of players even using remote LAN mysql connections) I would bet for other sources:
- Custom stuff
- Web (apache) slowering mysql server
- Really bad ThreadPools config
- bandwidth. OVH has many options (burstable, standart...), check if your bandwidth is shared by all servers from your network (read SLA) or is independent
- windows update, firewalls and other shits

I would put each database to its own localhost server, just to check if you made something wrong in your config or slow LAN; actually you can keep your web services up just even if database are not in localhost. Also let us know your OS architecture and java version.

I have attached a modificated mmocore lib that you can try (I'm doing some changes on it internally), maybe you get better throughput using it (dont forget to backup the other just in case it doesnt work as expected)

Hey,

I am also using an OVH server and these are my stats:

2x 4x 2.66+ GHz Intel i7 920 processor
64 bit Windows Server 2003 R2
6GB DDR3 RAM
250 GB Harddrive
1 GB/s and 100Mbit SLA ....

What would be a good thread configuration ? I got this on my live server 550+ online:
(maybe someone is well experienced with this)

#============================================================#
# Thread Configuration #
#============================================================#
# Extreme caution should be here, set to defaults if you do
# not know what you are doing.
# These could possibly hurt your servers performance, or improve
# it drastically.

ThreadPoolSizeEffects = 22
ThreadPoolSizeGeneral = 28

#Default 2
UrgentPacketThreadCoreSize = 5
#Default 4
GeneralPacketThreadCoreSize = 10
#Default 4
GeneralThreadCoreSize = 10

AiMaxThread = 14

#Packet LifeTime in milliseconds, 0 - disabled feature
PacketLifeTime = 0

#Dead Lock Detector (a separate thread for detecting deadlocks)
#For improved crash logs and automatic restart in deadlock case if enabled
#Check interval is in seconds
DeadLockDetector = False
DeadLockCheckInterval = 20
RestartOnDeadlock = False

DrHouse wrote:- Custom stuff
Yes, this custom stuff sometimes force hundreds of people to join an small place. Now I've fixed it, I only have to update the live server.
- Web (apache) slowering mysql server
How? Can I change something to avoid this?
- Really bad ThreadPools config
Bigger than in nitrozik's guide :S
- bandwidth. OVH has many options (burstable, standart...), check if your bandwidth is shared by all servers from your network (read SLA) or is independent
I use the traffic series
- windows update, firewalls and other shits
I'm sure that this option not xD

I would put each database to its own localhost server, just to check if you made something wrong in your config or slow LAN; actually you can keep your web services up just even if database are not in localhost. Also let us know your OS architecture and java version.

I have attached a modificated mmocore lib that you can try (I'm doing some changes on it internally), maybe you get better throughput using it (dont forget to backup the other just in case it doesnt work as expected)

I can add that the server goes FINE, but when it brings a few hours online it begin lagging.. But when the server lags the thing is: 20-60 seconds fine, 5-10 seconds freezed
And all the time like this until restart -_-

My servers have a lot of people, used to a 24/7 online server.. I can't experiment with changing database hosts..

Thank you very much!!!!!!

Maybe threadpool config is too hight, just set it as ~3x default one. Also, what is your starting script? looks like you have GC problems... Disconnect apache if possible, it can use more ressources than you know (easy to flood)

I think Ligttpd ( http://www.lighttpd.net/ ) is better than Apache ( if u use it for a domain ) For windows : http://en.wlmp-project.net/
Ligttpd runs faster than Apache ( search on google for statics and graphs ) the only problem is that u dont have .htaccess support ( mod_rewrite is avaible) .

If you want help about server peformance you need post your mysql initialization parameters.

I've read a lot about mysql, mysql is 75% responsible for your server's performance. Btw i read the thread created by nitrozik in the old-forum, i didnt understand why he setups the MySQL to InnoDB mode, following what i've did read until now, it's not seems so smart because innodb is the slower engine storage in MySQL.

I really didnt understand yet why he choose InnoDB, the parameters seems ok, but the storage method seems wrong.

MyISAM x InnoDB ?

I believe the developers more experienced in MySQL could give us some lessons creating a wiki teaching how properly configure the MySQL to get best results based on machine specifications and servers population.

We could work to create 2 my.cnf files: 1 for Huge servers and 1 for medium servers, the configuration to small servers seems not necessary due to the mysql capacity with default parameters is ok.

I'm still sure that could be very nice create a forum section exclusive to performance discussion, including mysql, java, hardware and other discussions, there people interested on those and there people who needs those. The community must have official documentation about performance

I use all "static" (e.g. weapons, armors) tables in MyISAM and several large tables (characters, items) in InnoDB, due to several problems.