In current production environments, industry 4.0 or smart production plays an essential role. The trend towards more and more individualization and an increasing number of product variants place high demands on people working in manufacturing. Assembly workers must achieve increasingly shorter cycle times and constantly learn how to assemble new products. The support from assistive systems (e.g. digital work instructions, pick-by-light, automatic component recognition for quality control) is necessary in these environments. But many systems and prototypes presented in this field work perfectly in lab environments, however, they may fall short when deployed in a real production environment. We developed a comprehensive and flexible assistance system for manufacturing and identified design and technical challenges, which may only appear when working at the shop floor and. In this paper, we present our experiences and discuss our key challenges and lessons learned.