The Palace was built in several stages, and got its final baroque image at the time of Andrija Zmajevic, as witnessed by the inscription on the front stating the year 1664. It is built on a rock and of local stone. The Palace presents a rare architectural conception. Symmetric disposition, with protruding central part above the cave in the rock, with the wings on strong bearing walls and outer staircase, leaves impression of building in the same breath. However, the building was performed in several stages: from the original defensive tower, which now presents the central part of the Palace, through extension of the tower towards the hill, to the building of the northern and southern wing. Decoration of the Palace with frescoes was done by the well-known baroque painter Tripo Kokolja, whose maecenas was Andrija Zmajevic.
The Zmajevics are one of the most significant Perast families, originating from village Vrba of Njegui. They belonged to the kazada (extended family) Perojevic. The heraldic insignia of the family is a winged dragon and a star, above two crossed feathers that are the symbol of Perojevic kazada. The Zmajevics gave some of the most distinguished persons of Perast.
Today the Palace is in the final phase of entire and thorough reconstruction. The basic architectural concept from the period of family Zmajevic domination has been fully preserved. Certain structural changes have been made in the inner Palace, but the logics of its original construction has never been neglected, neither materials that could spoil the authenticity of the object dominating Perast have been ever used.