The task-specific optimization of robotic systems has long been divided into the optimization of the robot and the optimization of the environment. In this letter, we argue that these two problems are interdependent and should be treated as such. To this end, we present a unified problem formulation that enables for the simultaneous optimization of both the robot kinematics and the environment. We demonstrate the effectiveness of our approach by jointly optimizing a robotic milling system. To compare our approach to the state of the art we also optimize the robot kinematics and environment separately. The results show that our approach outperforms the state of the art and that simultaneous optimization leads to a much better solution.