Described as a deeply personal book for the author, the novel was inspired by the close and enduring bonds formed in his youth and the soundtrack that accompanied them.
It explores a friendship that starts in the summer of 1986 and what happens when it is put to the test after the phone rings with unexpected news 30 years later.
The Waterstones Scottish Book of the Year specifically champions books by authors based in Scotland, or that have a strong Scottish setting.
O'Hagan said: "I am truly delighted that Mayflies has been selected by Waterstones as its Scottish Book of the Year.
"For an author, one of the great joys is to see a book being taken up by individual shops and individual booksellers, making every effort to put it into the hands of customers.
"This novel is so personal to me and the reaction of readers has been overwhelming. I feel a huge sense of gratitude to all Waterstones booksellers in Scotland."
Previous recipients of the award include Shuggie Bain by Douglas Stuart (2020), His Bloody Project by Graeme MacRae Burnet (2016) and Sunset Song by Lewis Grassic Gibbon (2015).
Angie Crawford, Waterstones Scottish buying manager, said: "Akin to a football game, Mayflies is a novel of two parts. The first half is a rollick through the 80sn and explores the friendships we cement in our teenage years.
"The second half takes these friendships and tests them to the limit. Funny and deeply moving, this novel is a masterpiece - a love letter to music, growing up in 80s Scotland and being real. A truly unforgettable read."
O'Hagan, who was born in Glasgow in 1968 and grew up in Ayrshire, has been nominated for the Booker Prize three times and has won a number of awards including the EM Forster Prize from the American Academy of Arts and Letters
O'Hagan is editor-at-large of the London Review of Books and a fellow of the Royal Society of Literature.