Pixel-level crack segmentation is widely studied due to its high impact on building and road inspections. Recent studies have made significant improvements in accuracy, but overlooked the annotation cost bottleneck. To resolve this issue, we reformulate the crack segmentation problem as a weakly-supervised problem, and propose a two-branched inference framework and an annotation refinement module that requires no additional data, in order to counteract the loss in annotation quality. Experimental results confirm the effectiveness of the proposed method in crack segmentation as well as other target domains.