The study assessed agricultural sustainability in South Asia (i.e., Bangladesh, Pakistan, India and Nepal) by computing multi-lateral multi-temporal Total Factor Productivity (TFP) indices and their six finer components (technical change, technical-, scale- and mix-efficiency changes, residual scale and residual mix-efficiency changes) and examined the role of capital in driving TFP growth covering a 34-year period (1980–2013). Results revealed that all countries sustained agricultural productivity growth at variable rates with Bangladesh experiencing highest rate estimated @1.05% p.a. followed by India (0.52%), Pakistan (0.38%) and Nepal (0.06% p.a.). There were little or no variation in technical and scale efficiency changes among the countries. However, residual scale efficiency increased @0.44% p.a. in Bangladesh, 0.12% p.a. in Pakistan, remained unchanged in India and declined −0.39% p.a. in Nepal. Similarly, mix efficiency increased @0.44% in Bangladesh, remained unchanged in India and declined @−0.12% p.a.in Pakistan and −0.39% p.a. in Nepal. The major drivers of agricultural TFP growth were the levels of natural, human and technology capital endowments whereas financial capital and crop diversification had opposite effects. Policy implications include land and tenurial reforms aimed at consolidating farm operation size and smooth operation of the land rental market to improve natural capital, investments in education to improve human capital and agricultural R&D to enhance technology capital in order to boost agricultural productivity growth in South Asia.