Herbarium collections shape our understanding of Earth’s flora and are crucial for addressing global change issues. Their formation, however, is not free from sociopolitical issues of immediate relevance. Despite increasing efforts addressing issues of representation and colonialism in natural history collections, herbaria have received comparatively less attention. While it has been noted that the majority of plant specimens are housed in the Global North, the extent and magnitude of this disparity have not been quantified. Here we examine the colonial legacy of botanical collections, analysing 85,621,930 specimen records and assessing survey responses from 92 herbarium collections across 39 countries. We find an inverse relationship between where plant diversity exists in nature and where it is housed in herbaria. Such disparities persist across physical and digital realms despite overt colonialism ending over half a century ago. We emphasize the need for acknowledging the colonial history of herbarium collections and implementing a more equitable global paradigm for their collection, curation and use.