Using ensembles of two, three and four spheres immersed in a fermionic background we evaluate the (integrated) density of states and the Casimir energy. We thus infer that for sufficiently smooth objects, whose various geometric characteristic lengths are larger then the Fermi wave length one can use the simplest semiclassical approximation (the contribution due shortest periodic orbits only) to evaluate the Casimir energy. We also show that the Casimir energy for several objects can be represented fairly accurately as a sum of pairwise Casimir interactions between pairs of objects.