Perched atop the highest point of Pairikia is the Frankish castle, built in the 13th century. It was constructed in 1260 by Marcus Sanudus, who had conquered, along with Crusaders of Venetian origin, the Cyclades as well as other parts of Greece, and established the Duchy of the Aegean Sea, based in Naxos.

Nowadays, only one of the castle walls remains. The material used for its construction was taken from ancient sites that existed nearby, such as the temple of goddess Athena, the protector of the ancient town of Paros.