Fine robotic assembly, in which the parts to be assembled are small and fragile and lie in an unstructured environment, is still out of reach of today's industrial robots. The main difficulties arise in the precise localization of the parts in an unstructured environment and the control of contact interactions. Our contribution in this paper is twofold. First, we propose a taxonomy of the manipulation primitives that are specifically involved in fine assembly. Such a taxonomy is crucial for designing a scalable robotic system (both hardware and software) given the complexity of real-world assembly tasks. Second, we present a hardware and software architecture where we have addressed, in an integrated way, a number of issues arising in fine assembly, such as workspace optimization, external wrench compensation, position-based force control, etc. Finally, we show the above taxonomy and architecture in action on a highly dexterous task -- bimanual pin insertion -- which is one of the key steps in our long term project, the autonomous assembly of an IKEA chair.