The Holy Rosary Seminary was first established as a Casa de Clerigos for the episcopal see of Nueva Caceres sometime in the early part of the 18th century. Later, on March 7, 1793, then Caceres bishop Antonio Orbigo, OFM, transformed it into a Seminario Conciliar de Caceres. Four Years thereafter, it was canonically erected. The seminary was located by the bank of the Bikol River, now Calle Caceres, in downtown Naga. Under Bishop Bernardo de la Concepcion, OFM, (1816-1829).
It was transferred close to its present site. The new building was badly damaged by a typhoon in 1855 and again by fire in 1860. Thus, Bishop Francisco Gainza, OP (1862-1879), caused the construction of another building, the present moorish-style structure; he also transferred its administration to the Vincentian fathers. Then in 1925, the seminary came to be known as Seminario del Santissimo Rosario.
The seminary has produced 21 bishops, including the first Filipino bishop, Jorge Barlin, and the first Filipino Cardinal to work in the Roman Curia, Jose Cardinal Sanchez. It has richly contributed as well to the national heritage through Jose Ma. Panganiban and Tomas Arejola and 7 of the 15 Bikol Martyrs. On January 29, 1988, the National Historical Institute declared the Holy Rosary Seminary as a National Historical Landmark.