Developing an understanding of the public discourse on COVID-19 vaccination on social media is important not only for addressing the current COVID-19 pandemic, but also for future pathogen outbreaks. We examine a Twitter dataset containing 75 million English tweets discussing COVID-19 vaccination from March 2020 to March 2021. We train a stance detection algorithm using natural language processing (NLP) techniques to classify tweets as `anti-vax' or `pro-vax', and examine the main topics of discourse using topic modelling techniques. While pro-vax tweets (37 million) far outnumbered anti-vax tweets (10 million), a majority of tweets from both stances (63% anti-vax and 53% pro-vax tweets) came from dual-stance users who posted both pro- and anti-vax tweets during the observation period. Pro-vax tweets focused mostly on vaccine development, while anti-vax tweets covered a wide range of topics, some of which included genuine concerns, though there was a large dose of falsehoods. A number of topics were common to both stances, though pro- and anti-vax tweets discussed them from opposite viewpoints. Memes and jokes were amongst the most retweeted messages. Whereas concerns about polarisation and online prevalence of anti-vax discourse are unfounded, targeted countering of falsehoods is important.