Originally a Pictish Dun or Fort and one of a chain running along the coast within signalling distance of one another. Held by the Norse from around 800 AD it became known as Dun Dhaibhidh (Fort David).

Following the defeat of the Norse at Largs in 1263 the castle was abandoned and little is known of the period covering the next 300 years.

Next, Master of the Isles, Donald Gorm, is recorded as making Duntulm his home and initiating a great deal of building work and improvement. In 1540 King James V visited the area and was impressed by the castle and its situation.

Ownership of the castle and its domain was hotly disputed and changed hands several times, alternating between MacLeods and the MacDonalds until the Battle of Trotternish (C 16th). The MacDonalds were victorious and held Duntulm until they moved their base to Monkstadt in 1730. It is said that the move was as the result of a family tragedy. The young son and heir of Donald Gorm Og fell to his death from a window of the castle. It is more likely however, that the castle fell into a state of disrepair in the long periods in which it was unused and eventually became uninhabitable.