The castle's origins go back to the Sengoku period. At that time Shimadachi Sadanaga of the Ogasawara clan built a fort on this site in 1504 which was originally called Fukashi Castle. It is believed the castle was completed by 1593–94. During the Edo period, the Tokugawa shogunate established the Matsumoto Domain, of which the Matsudaira, Mizuno and others were the daimyo. For the next 280 years until the abolition of the feudal system in the Meiji Restoration, the castle was ruled by the 23 lords of Matsumoto representing six different daimyo families. In this period the stronghold was also known as Crow Castle (Karasu-jo) because its black walls and roofs looked like spreading wings.
In 1952 the keep, Inui-ko-tenshu (small northern tower), Watari-yagura (roofed passage), Tatsumi-tsuke-yagura (southern wing), and Tsukimi-yagura (moon-viewing room) were designated as national treasures.
The second floor of the main keep features a gun museum, Teppo Gura, with a collection of guns, armor and other weapons.