The paper presents a numerical implementation of the gravitational N-body problem with contact interactions between non-spherically shaped bodies. The work builds up on a previous implementation of the code and extends its capabilities. The number of bodies handled is significantly increased through the use of a CUDA/GPU-parallel octree structure. The implementation of the code is discussed and its performance are compared against direct N$^2$ integration. The code features both smooth (force-based) and non-smooth (impulse-based) methods, as well as a visco-elastic non-smooth method, to handle contact interaction between bodies. The numerical problem of simulating "rubble-pile" asteroid gravitational aggregation processes is addressed. We discuss the features of the problem and derive criteria to set up the numerical simulation from the dynamical constraints of the combined gravitational-collisional problem. Examples of asteroid aggregation scenarios that could benefit from such implementation are finally presented.