Le château trouve ses origines sur les bases d’une ancienne forteresse du Moyen Age érigée vers 1260 . Le fief de Combas , siège d’un prieuré au XIVe siècle, appartenait en l’an 1694 à Charles de Montagut, seigneur de Combas; il est resté la propriété de la famille de Montagut (Charles, Alphonse de Montagut, 1704 et Catherine de Levat sa veuve, en 1742), jusqu’au 11 juin 1749. Date à laquelle, il fût acheté par Jean-François Daydé. Vendu à Antoine Hyacinthe Roussel, il passa ensuite dans la famille de M. de Barrès Servian, le 24 juillet 1813. Il n’y resta pas longtemps puisque Etienne Balme et Etienne Laurès de Murviel en firent l’acquisition le 2 mars 1826.
Finally, on December 22, they decided to sell the Château to the Marquis of Thézan-Saint Géniès. The marquis of Thézan-Saint Géniès and his marquise, née de Bourdeille de Matha, carried out important extension projects of the 75-hectare Hermitage de Combas domain.
Thanks to him, we have some information about the restoration undertaken: "the already existing walls were used by moving the farm buildings outside in order to free the enclosure whose solid foundations served as a framework for a new construction. The towers were raised, a large staircase was built, as well as the chapel. On the second floor, large rooms were planned to receive the paintings, furniture and family heirlooms taken from the castles of St Génies and Corneilhan”.
He also referred to the framework created for the new castle: "The Marquise de St Géniès, endowed with an enlightened taste, traced and planted the formal garden, extended to the park’s north and encircled with trees; now old of a hundred years of age but which still procure a majestuous ornamental effect. A dam was built to stop the waters of St Michel and Combas river’s stream, its light was widened when it was close to the garden, forming afterwards a kind of lake; thereby the land coming out of it was used for many purposes ”.
It is therefore an interesting example of the early restoration of an ancient castle which did not completely satisfy the later wine-growing generation. Indeed, the last Marquis de Thézan-Saint Géniès added a bell tower and a pinnacle, nostalgically acknowledged: "space was needed, so the exterior was too often sacrificed to interior fittings". In fact, his ancestors knew that the major interest of their home would be the furniture, portraits and archives of Thézan-Saint Géniès which made L’Hermitage a place of memory as to the history of our country since the Middle Age
During the 18th century, the Saint Géniès couple had only one daughter (who became the wife of sir Baron de Maussac, member of the Lords of Corneilhan’s family), Louis XV erected in 1750 for the Baron’s benefit, the Saint Géniès’ marquisate including in particular the lands of this name and those of the Lords of Corneilhan. This second house died out with our precious informant (who married a Mandelvillle in 1896), his father then giving him this 52ha estate valued at 200,000F. He died in 1943 after having instituted as heir his cousin, the Duke of Lévis - Mirepoix, of the French Academy.