I use my Sirui T-005X tripod with a Sirui C-10X head. When I place my Nikon 300mm f4 AF lens on it, I slide it on and tighten both the plate lock and the position on the ball head. After letting go, I notice that the camera and lens move up a bit up (a few mm but at most 1cm) and finally stops moving. Afterwards I use a remote cable release to take a shot.
I assume when I hold it I'm pushing the lens and body a bit down? I'm use to compensating for this movement but I never bothered to ask until now.
I do not notice it with any lighter lens, although I don't have any other lens with a collar to test it.
Is this a sign that I need a better ball head (or other tripod head) or is this how ball heads work with heavy telephoto lens?
The Sirui C-10X ballhead has a capacity of 28.7 lb, about 13 kg (source : http://mob.sirui-photo.com/productseries.php?productclassid=28).
Your Nikon 300mm f4 AF weights about 1.3 kg and a regular camera body is about 1 kg (1.2 kg top for Nikon D3x : https://en.wikipedia.org/wiki/Comparison_of_Nikon_DSLR_cameras) so the total weight your ball has to carry is far under its rated capacity : 2.5 kg vs 13 kg.
A little movement is very common after fixing your body on your tripod. Whether its magnitude is bothering or not is up to you. Unless you headball encountered a very bad situation (you should see it just by looking at it), those are made to last. Mine (Manfrotto) is more than 15 years old and works like a charm.
Be careful to use a tripod collar if you are using a heavy lens, your Nikon 300mm f/4 should have one : http://www.dpreview.com/forums/post/50913294.
If you are not using it, the center of gravity of the [body+lens] isn't above your tripod ball, thus creating a huge torque that your head isn't designed to withstand.
It depends how the head is made. Most ball-heads move a little which is called drift. A millimeter or two is common but should not be more for anything reasonable. Otherwise it makes framing rather hard.
Certain ball-heads are designed specifically to prevent this and are called non-drift heads. Manfrotto for example has a Hydrostatic series like that. The one I personally use if the 468MGRC4 which took me too long to buy because I considered it too expensive and bought poorer ones instead.
Should you want no drift at all, what you need instead is a geared-head. Those are costly models that constantly hold the camera in an exact orientation and use gears to move the camera while its weight is constantly supported.
In my opinion, that ballhead is too lightweight for uses beyond holding smaller DSLRs with shorter/lighter lenses. Many tripod and ballhead manufacturers overstate their "rated" weight.
For instance, the Sirui C-10X says it has a capacity of 28.7 lbs. But what is that, in photographic equipment terms? A Nikon D810 weighs 1.98 lb (990 g). With a 600 mm ƒ/4G ED coming in at 11.16 lb (5.1 kg), the total camera + lens combination weighs 13.14 lb (6 kg), which is just half the rated capacity of the Sirui C-10X. I would not put a 600 ƒ/4 + D810 anywhere near that ballhead. Sure, it will probably clamp and hold the lens when everything is balanced. But if the system were tilted to provide an unbalanced torque on the ballhead, would you be confident the ball wouldn't slip? Not me.
So where is the "real" or practical limit of the ballhead? I have no idea. I don't believe in any of the "specs" at all, because I haven't seen a ballhead manufacturer publish testable, measurable numbers (i.e., something like "max holding torque"). When it comes to minimizing vibration, deflection, or drift, in general, more mass is better. Mass dampens vibrations. Assuming like metals, more mass means larger geometry (such as larger balls in the clamp) which provides longer moment arms to apply force (i.e., more resistant to torques applied by heavy loads).
If your use of the 300mm ƒ/4 (which is a pretty short and light lens for its reach and aperture) indicates you will be using more lenses of that general size or larger, then yes, you will need a better ballhead. You don't have to spend as much as a Really Right Stuff BH-55, but when it comes to stability of mount, that's the approximate size you should be looking at.