Augmented Reality enhances user perception by overlaying real world information with virtual computer-generated information. The aims of the 6th Sense project are to improve real-time interaction between the real environment and the virtual world and to maximize the user experience in mobile Augmented Reality. To achieve these objectives a generic framework constituted of two main layers is proposed. The End-to-End Adaptation Layer adapts in real-time the parameters of the Augmented Reality system to provide the user with the best possible experience despite the varying operating conditions such as the transmission link and user head motion. The Generic Augmented Reality Layer encompasses solutions to the problem of overlaying adequate information in the real scene and manages multimodal interaction with the virtual environment.