In this paper, we address a physical layer security (PLS) framework for the integrated sensing and semantic communication (ISASC) system, where a multi-antenna dual-function semantic base station serves multiple single-antenna semantic communication users (SCUs) and monitors a malicious sensing target (MST), in the presence of a single-antenna eavesdropper (EVE), with both the MST and EVE aiming to wiretap information from the SCUs' signals. To enhance PLS, we employ joint artificial noise (AN) and dedicated sensing signal (DSS) in addition to wiretap coding. To evaluate the sensing accuracy, we derive the Cramer-Rao bound (CRB) as a function of the communication, sensing, and AN beamforming (BF) vectors. Subsequently, to assess the PLS level of the ISASC system, we determine a closed-form expression for the semantic secrecy rate (SSR). To achieve an optimal trade-off region between these two competing objectives, we formulate a multi-objective optimization problem for the joint design of the BF vectors. We apply semi-definite programming, Gaussian randomization method, and golden-section search techniques to address this problem. Simulation results demonstrate that the proposed scheme outperforms baseline schemes, achieving a superior trade-off between SSR and CRB.