Massive stars in binaries can give rise to extreme phenomena such as X-ray binaries and gravitational wave sources after one or both stars end their lives as core-collapse supernovae. Stars in close orbit around a stellar or compact companion are expected to explode as "stripped-envelope supernovae", showing no (Type Ib/c) or little (Type IIb) signs of hydrogen in the spectra, because hydrogen-rich progenitors are too large to fit. The physical processes responsible for the stripping process and the fate of the companion are still very poorly understood. Aiming to find new clues, we investigate Cas~A, which is a very young ($\sim$340 \,yr) and near ($\sim$3.4\,kpc) remnant of a core collapse supernova. Cas~A has been subject to several searches for possible companions, all unsuccessfully. We present new measurements of the proper motions and photometry of stars in the vicinity based on deep HST ACS/WFC and WFC3-IR data. We identify stellar sources that are close enough in projection, but using their proper motions we show that none are compatible with being at the location of center at the time of explosion, in agreement with earlier findings. Our photometric measurements allow us to place much deeper (order of magnitude) upper limits on the brightness of possible undetected companions. We systematically compare them with model predictions for a wide variety of scenarios. We can confidently rule out the presence of any stellar companion of any reasonable mass and age (main sequence, pre main sequence or stripped). Although this finding is not in direct conflict with model predictions, it does rule out what many considered to be the most likely evolutionary scenario for type IIb supernova. (abstract abbreviated)