It might not be a problem - it may depend what's running on your system. I'm not an expert server admin but I believe unix systems, and common programs (php, mysql, etc) "reserve" a lot of RAM for their own use. If it were a windows server I'd be more concerned though hehe. I dunno about other people, but you couldn't pay me to host my sites on windows
As was suggested, from a shell prompt (SSH) run the command "top" or even just "w" and you'll see the load on the server. If it's anywhere under 1.0 it's probably ok. According to the output of "top", my server running absolutecross.com which has 2GB of RAM currently has about 512mb "free" out of that, but the system is running great. Since there's almost nothing else running on the server right now and that site is mostly static html, even with a few thousand visitors there's no way it is actually using that much RAM.
From SSH try typing free -m which tells you "the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel." The -m makes the numbers in MB. Using this command on my server, it shows that even though top showed there to be only around 512 or so MB free, there are actually almost 1.7GB free, just most is reserved. Type man free to learn more about the command and other usage options. Though I'm stepping out of my certain knowledge -- I'm pretty sure that those buffers give way to actual needs of the server as they arise.
Hope this helps 
|