With the advent of Fifth Generation (5G) and Sixth Generation (6G) communication technologies, as well as the Internet of Things (IoT), semantic communication is gaining attention among researchers as current communication technologies are approaching Shannon's limit. On the other hand, Large Language Models (LLMs) can understand and generate human-like text, based on extensive training on diverse datasets with billions of parameters. Considering the recent near-source computational technologies like Edge, in this article, we give an overview of a framework along with its modules, where LLMs can be used under the umbrella of semantic communication at the network edge for efficient communication in IoT networks. Finally, we discuss a few applications and analyze the challenges and opportunities to develop such systems.