What should I consider when doing capacity planning for memory requirements and processing time on UNIX?
Primer to Capacity Planning on UNIX
Some basic assumptions to understand if you want to extrapolate this information:
- My experience is with Sun Solaris Machines (Sunfire and e-Class)
- Most all process tuning I have done involved Java Virtual Machines.
- The last version I personally tested/tuned was Java 1.4
There are really four components to think about when planning and tuning:
- Memory Requirements
- Processing Performance
- File I/O
For this piece, I wanted to focus on the first two.
On windows there are lots of practices to follow, like the 50% rule of thumb and less than 1.8GB heap. These are primarily based on the 32-bit machine architecture and the OS requirements. With UNIX you can forget these numbers. There is a practical no-limit on Java heap sizes (albiet physical memory) and I have run just fine peaking at 80-90% memory utilization. (Actual performance varies based on paging). My point is I would worry far less about heap sizes and memory management at the OS once you move to UNIX from 32-bit. But for those who like numbers, here is what I use:
Shoot for your total memory (including JVMs) to be less than 80%. (Java = max heap)
On UNIX I like to keep min/max heap settings at the same value, unless the application tends to grow significantly over time. This allows good server-side memory management from startup (esp. with large singletons/caching). GC will always be more efficient when there is a difference in min vs max values. However, GC becomes a bigger issue with 32-bit and memory restrictions, and if you have good practices that have been proven to work about setting min vs max, use them. Just do some testing to prove your apps still behaves as expected with those rules on UNIX. When in doubt, set the min to equal your max.
Remember the max heap setting does not mean total max consumption. There is still JVM overhead like stack allocations and thread management. So 80% is a good number since everything else on the box will probably not grow as fast as your JVMs, the remaining 20% should keep you in a comfortable range (accounting for memory spikes).
JVM Processes are sized by both their memory consumption and their CPU utilization. Larger JVM are typically characterized by their memory usage rather than CPU due to Java’s history with an emphasis on memory and GC. So large JVMs are those bigger than 1.8GB where medium are between that 512kb, with the rest being small. Now the next question you need to ask is how process intensive is your application? Those that are not (e.g. less than 2-5%) are what I like to call 'agents'. These JVMs are used by off the shelf vendors to monitor, schedule or otherwise provide a service that is on-demand. With JVM agents I hardly worry about where it is the priority stack or how cycles are being allocated to it.
I have found concern for two types of JVM landscapes. The first is: many of JVMs (20+) with maybe only a handful being really intensive (e.g. >20% utilization). The second is: a few very large JVMs that are all equally important but maybe only between (2-10%) utilization. In these situations under load you can be faced with thrashing that limits your overall process throughput. One solution I like is process groups. It is a really nice practice. It means you can assign a process or group of processes a single "group" that gets priority. The process group can then be dedicated to a single processor on the machine and both limit memory thrashing as well as multi-tasking to allow that group to perform at is best.
So in the first scenario you can bind that one process to a single CPU and ensure it gets both priority (dispatching tables) and dedicated cycles without agents interfering. Now UNIX is so advanced that this can be setup so any unused cycles can be distributed if that process does not require the entire CPU, without degrading optimum perform of that process. In the second scenario you can offset the multi-tasking overhead and thrashing by putting each large process on its own group and manage it like its own Virtualized OS, complete with memory allocations and CPU time.
BTW, this concept is one of the reasons what makes UNIX so cool (safe and efficient) because you can do this exact thing with the UNIX kernel, hence the term KERNAL CAGE. Dedicating a portion of the processing time to a CPU to ensure the job gets done without interrupt and still run giants like ORACLE with all its I/O and shared memory to peak performance.