The site is forever associated with the princes of Deheubarth, the kingdom in south-west Wales. A lack of available excavation data means uncertainty remains as to the shape, form and history of any earlier fortifications, which may underlie the medieval castle. Despite this, present evidence suggests very strongly that the history of Dryslwyn Castle is entwined with the rule of the Lord Rhys (d. 1197).
Over time the castle changed hands between the princes of Deheubarth and gradually evolved into formidable fortresses. It eventually fell to the English Crown from 1287, serving as centres of royal administration and authority. By the end of the Middle Ages the castle had become ivy-clad ruins.