After Toyotomi Hideyoshi’s successful conquest of Kyūshū, local warlord Matsura Shigenobu was granted Hirado County and the Iki Island to be his domain. In 1599, Matsura Shigenobu erected a castle called Hinotake-jō on the site of the present-day Hirado Castle. However, he burned the castle down himself in 1613, as a gesture of loyalty towards Shōgun Tokugawa Ieyasu, having served in the losing Toyotomi side during the Battle of Sekigahara. In return, he was allowed to retain his position as daimyō of Hirado Domain under the Tokugawa bakufu.
The present Hirado Castle was constructed in 1704 by order of the 4th daimyō of Hirado domain, Matsura Takashi with the assistance of the Tokugawa shogunate. It was intended to be the keystone in coastal defense in the East China Sea region, as the government had by then implemented a policy of national seclusion against Western traders and missionaries. The design was partly influenced by the theories of the military strategist Yamaga Soko. The new construction was completed in 1718, and the castle remained home to the Matsura daimyō until the Meiji Restoration of 1868.
In 1871, with the abolition of the han system, all structures of Hirado Castle were dismantled, with the exception of the northern gate, a yagura and the moat, and the grounds turned into Kameoka Park, with a Shinto shrine dedicated to the spirits of the successive generations of the Matsura daimyō. The former residence of the final daimyō, Matsura Akira was turned into a local history museum.
In 1962, four yagura, the ramparts, and the keep were reconstructed. The modern keep is a five-story steel-reinforced concrete structure, and contains a museum with artifacts of the Matsura clan. One of these artifacts is a 93-cm long Japanese sword (tachi) dating from the Asuka period, and is locally purported to have been carried by a general during the time of the legendary Empress Jingu’s invasion of Korea. An heirloom of the Matsura clan, it is now owned by Kameoka Shrine and is a designated as a National Important Cultural Property (ICP).
In 2006, Hirado Castle was listed as one of the 100 Fine Castles of Japan by the Japan Castle Foundation.