Tourism is one of the fastest growing industries in the world. In Nigeria, it contributes significantly to job creation, income generation, foreign exchange earnings and cultural promotion However, if not planned and managed well; it exerts negative socio-cultural, economic and environmental impacts, especially on the host communities. To ensure sustainability, responsible tourism is advocated. Responsible tourism is an approach to tourism development which adheres to the principles of sustainable development, including economic and social benefits for destination communities and improved quality of life for hosts and their environment. This paper examines the adverse impacts of tourism and how responsible tourism helps to ameliorate these negative influences for the betterment of all tourism stakeholders. Economic, social, political, psychological and environmental empowerments are recommended as a means of achieving responsible and sustainable tourism development. Communities that are empowered politically and psychologically will have a voice in tourism development efforts which guarantee bottom-up development initiatives.