In the 16th century, the castle became the residence of the royal family. During the period from 1536 to 1664, Dragsholm Castle was also used as a prison for noble prisoners. In the large tower at the northeast corner of the castle, prison cells were made. Some of the most well-known prisoners at Dragsholm Castle include the last catholic Bishop in Roskilde, James Hepburn, 4th Earl of Bothwell, third husband of Mary, Queen of Scots, and the mad squire, Ejler Brockenhuus. The Earl of Bothwell was subjected to particularly horrible treatment in the tower and was tied to a pillar and left to rot. He was given just what was necessary to keep him alive and he was said to have gone mad.
The Earl of Bothwell is one of the most famous ghosts of Dragsholm. The Earl has been seen riding into the courtyard of the castle with his horse and carriage. Many visitors claim to have heard the horses hoof beats upon the cobbled yard.
The castle is also said to be haunted by two other ghosts. There is a white lady who is said to be a daughter of one of the bovles who owned the castle. She fell in love with a commoner and when her father found out he was so angry that he imprisoned her in the dreaded tower. She was more than imprisoned, legend says that he walled her up in her prison cell and left her to die. It is said that every night she returns to the castle and walks around the corridors. There have been numerous sightings of her. There is also factual evidence to support this story. In the 1930's, when the old walls of the castle were torn down, workers found a hole in the wall and a skeleton with a white dress in it.
The last ghost to haunt Dragsholm Castle is a gray lady. She is the rarest of the three ghost and is seldom seen or heard. She is believed to be the ghost of a young serving girl who died of a tooth ache. She lingers in the shadows, hiding from sight, looking for a light in the darkness.