The Palazzo Brera owes its name to the Germanic braida, indicating a grassy opening in the city structure: compare the Bra of Verona. The convent on the site passed to the Jesuits (1572), then underwent a radical rebuilding by Francesco Maria Richini (1627–28). When the Jesuits were disbanded in 1773, the palazzo remained the seat of the astronomical Observatory and the Braidense National Library founded by the Jesuits. In 1774 were added the herbarium of the new botanical garden. The buildings were extended to designs by Giuseppe Piermarini, who was appointed professor in the Academy when it was formally founded in 1776, with Giuseppe Parini as dean. Piermarini taught at the Academy for 20 years, while he was controller of the city's urbanistic projects, like the public gardens (1787–1788) and piazza Fontana, (1780—1782).
For the better teaching of architecture, sculpture and the other arts, the Academy initiated by Parini was provided with a collection of casts after the Antique, an essential for inculcating a refined Neoclassicism in the students. Under Parini's successors, the abate Carlo Bianconi (1778–1802) and artist Giuseppe Bossi (1802–1807), the Academy acquired the first paintings of its pinacoteca during the reassignment of works of Italian art that characterized the Napoleonic era. Raphael's Sposalizio (the Marriage of the Virgin) was the key painting of the early collection, and the Academy increased its cultural scope by taking on associates across the First French Empire: David, Pietro Benvenuti, Vincenzo Camuccini, Canova, Thorvaldsen and the archaeologist Ennio Quirino Visconti. In 1805, under Bossi's direction, the series of annual exhibitions was initiated with a system of prizes, a counterpart of the Paris Salons, which served to identify Milan as the cultural capital for contemporary painting in Italy through the 19th century. The Academy's artistic committee, the Commissione di Ornato exercised a controlling influence on public monuments, a precursor of today's Sopraintendenze delle Belle Arti.
The Romantic era witnessed the triumph of academic history painting, guided at the Academy by Francesco Hayez, and the introduction of the landscape as an acceptable academic genre, inspired by Massimo D'Azeglio and Giuseppe Bisi, while the Academy moved towards becoming an institution for teaching the history of art. Thus in 1882 the Paintings Gallery was separated from the Academy.
From 1891 the exhibitions were reduced to triennial events, and architectural projects developed their autonomous course. During the period of the avant-garde when Modernism was becoming established, the director of the Academy Camillo Boito had as pupil Luca Beltrami, and Cesare Tallone taught Carlo Carrà and Achille Funi.
The Brera Observatory hosted the astronomer Giovanni Schiaparelli for four decades, and the Orto Botanico di Brera is a historic botanical garden located behind the Pinacoteca.