There are some options, in config files (.properties), which are "helping" memory cache, to grow, and eventually extend, its limit.
If you are running your game server, with full geodata, then it's possible, even on a machine with 4 GB of RAM, to face memory issues.
Maybe a restart on your pc, would solve the problem, but probably you would face the same problem in the future
Here is a link, that could help you, with finding the resources which are overloading your system.
http://blogs.opcodesolutions.com/roller ... error_java.
You can find a lot of similar references, on the net, for your case.