Distance Education is a very important instructional delivery for higher education in the 21st century. Along with the rapid development of computer related modern technology such as Internet, videoconferencing, Web conferencing etc., distance education is playing an increasingly important role in schools, colleges and other educational institutes. The advancement of information and communication technology (ICT) has brought revolutionary changes to the mode of distance education. Distance education in the form of E-learning can play a crucial role in broadening access to education for the whole society. The framework of this research focuses on the trends and issues of existing Distance Education in Bangladesh and explores the opportunities as well as the challenges of E-learning to develop a modern distance education system for the disadvantaged people of the country. Considering all issues and challenges this research proposes a model of E-learning assisted distance education system for Bangladesh. This paper also intends to suggest some recommendations for facing the challenges and strengthening infrastructures to implement the system for quality education.