While communication systems that employ millimeter wave (mmWave) frequency bands can support extremely high data rates, they must use large antenna arrays in order to overcome the severe propagation loss of mmWave signals. As the cost of traditional fully-digital beamforming at baseband increases rapidly with the number of antennas, hybrid beamforming that requires only a small number of radio frequency (RF) chains has been considered as a key enabling technology for mmWave communications. Intelligent reflecting surface (IRS) is another innovative technology that has been proposed as an integral element of future communication systems, establishing the favorable propagation environment in a timely manner through the use of low-cost passive reflecting elements. In this paper, we study IRS-aided mmWave multiple-input multiple-output (MIMO) systems with hybrid beamforming architectures. We first propose the joint design of IRS reflection pattern and hybrid beamformer for narrowband MIMO systems. Then, by exploiting the sparsity of frequency-selective mmWave channels in the angular domain, we generalize the proposed joint design to broadband MIMO systems with orthogonal frequency division multiplexing (OFDM) modulation. Simulation results demonstrate that the proposed joint designs can significantly enhance the spectral efficiency of the systems of interest and achieve superior performance over the existing designs.