Multi-robot visual simultaneous localization and mapping (SLAM) system is normally consisted of multiple mobile robots equipped with camera and/or other visual sensors. The networked robots work independently or cooperatively in an unknown scene in order to solve autonomous localization and mapping problem. One of the most critical issues in Multi-robot visual SLAM is the intensive computation that is normally required yet overwhelming for inexpensive mobile robots with limited on-board resources. To address this problem, a novel task offloading strategy and dense point cloud map construction method is proposed in this paper. First, we develop a novel strategy to remotely offload computation-intensive tasks to cloud center, so that the tasks that could not originally be achieved locally on the resource-limited robot systems become possible. Second, a modified iterative closest point algorithm (ICP), named fitness score hierarchical ICP algorithm (FS-HICP), is developed to accelerate point cloud registration. The correctness, efficiency, and scalability of the proposed strategy are evaluated with both theoretical analysis and experimental simulations. The results show that the proposed method can effectively reduce the energy consumption while increase the computation capability and speed of the multi-robot visual SLAM system, especially in indoor environment.