In this work, we present a system that enables delivery drones to autonomously navigate and deliver packages at various locations around the house. The objective of this to reach a specific location in the house where the recipient wants the package to be delivered by the drone without the use of any external markers as currently used. This work is motivated by the recent advancements in semantic segmentation using deep learning that can potentially replace the specialized marker used by the current delivery drone for identifying the place where it needs to deliver the package. The proposed system is more natural in the sense that it takes an instruction input on where to deliver the package similar to the instructions provided to the human couriers. We propose a semantic segmentation-based lowering location estimator that enables the drone to find a safe spot around the house to lower from higher altitudes. Following this, we propose a strategy for visually routing the drone from the location where it lowered to a specific location like the front door of the house where it needs to deliver the package. We extensively evaluate the proposed approach in a simulated environment that demonstrates that the delivery drone can deliver the package to the front door and also to other specified locations around the house.