Photonic computing has the potential of harnessing the full degrees of freedom (DOFs) of the light field, including wavelength, spatial mode, spatial location, phase quadrature, and polarization, to achieve higher level of computation parallelization and scalability than digital electronic processors. While multiplexing using wavelength and other DOFs can be readily integrated on silicon photonics platforms with compact footprints, conventional mode-division multiplexed (MDM) photonic designs occupy areas exceeding tens to hundreds of microns for a few spatial modes, significantly limiting their scalability. Here we utilize inverse design to demonstrate an ultracompact photonic computing core that calculates vector dot-products based on MDM coherent mixing within a nominal footprint of 5 um x 3 um. Our dot-product core integrates the functionalities of 2 mode multiplexers and 1 multi-mode coherent mixers, all within the footprint, and could be applied to various computation and computer vision tasks, with high computing throughput density. We experimentally demonstrate computing examples on the fabricated core, including complex number multiplication and motion estimation using optical flow.