The building's distinctive appearance is due to the ornate design which uses red sandstone from Locharbriggs for the exterior masonry, and Westmorland green slate for the main roof.

The building was designed by Englishmen John Simpson (1858-1933) and E J Milner Allen (1859-1912). They won a competition for the design of the building set in 1891 by the Association for the Promotion of Art and Music. The competition received 62 entries, from which six were selected for further investigation. Simpson and Miller's joint design was adjudged the winner by Sir Alfred Waterhouse, the architect who designed the Science Museum in London.

There is a popular local myth that the building was built back to front, with the main entrance facing Kelvingrove Park instead of Argyle Street. The myth also has it that the architect was so distraught on realising his mistake that he jumped to his death from one of the building's high towers. This is most definitely not true, as the Museum was designed as a feature of Kelvingrove Park to complement the Glasgow University buildings, with the main entrance within the grounds of the park.