Garendon Hall was primarily associated with the Phillipps (or Phillips) family and, later, the De Lisle family.
Phillipps Family: Sir Ambrose Phillipps purchased the estate in 1684, and his family held it for a significant period. The estate was intimately connected with the Phillipps family during the 17th, 18th, and 19th centuries (source).
De Lisle Family: The De Lisle family of Leicestershire were originally the Phillipps from London, and they maintained ownership into the later years. The family used Garendon Hall primarily and had close ties to the estate (source).
For more information, you can read the Wikipedia entry on Garendon Hall and further details on The DiCamillo Companion.
Here are some images of Garendon Hall: