Millimeter wave (mmWave) technology can achieve high-speed communication due to the large available spectrum. Furthermore, the use of directional beams in mmWave system provides a natural defense against physical layer security attacks. In practice, however, the beams are imperfect due to mmWave hardware limitations such as the low-resolution of the phase shifters. These imperfections in the beam pattern introduce an energy leakage that can be exploited by an eavesdropper. To defend against such eavesdropping attacks, we propose a directional modulation-based defense technique where the transmitter applies random circulant shifts of a beamformer. We show that the use of random circulant shifts together with appropriate phase adjustment induces artificial phase noise (APN) in the directions different from that of the target receiver. Our method corrupts the phase at the eavesdropper without affecting the communication link of the target receiver. We also experimentally verify the APN induced due to circulant shifts, using channel measurements from a 2-bit mmWave phased array testbed. Using simulations, we study the performance of the proposed defense technique against a greedy eavesdropping strategy in a vehicle-to-infrastructure scenario. The proposed technique achieves better defense than the antenna subset modulation, without compromising on the communication link with the target receiver.