She was born in 1920, the daughter of Major Jack Spencer-Churchill (1880–1947), the younger brother of Winston Churchill, by his marriage to Lady Gwendoline ("Goonie") Bertie (1885–1941), a daughter of the 7th Earl of Abingdon, who had been married in 1908. She is thus a niece of Winston Churchill, who was Prime Minister during the Second World War, and a granddaughter of Lord Randolph Churchill, Chancellor of the Exchequer 1886–87, and his wife the American society beauty Jennie Jerome. Her paternal great-grandfather was the 7th Duke of Marlborough and her maternal great-great-grandfather, the 3rd Marquess of Londonderry, half-brother of the 2nd Marquess, who, as Viscount Castlereagh was Foreign Secretary during the Congress of Vienna of 1815 that followed the Napoleonic Wars.
Jack Churchill, born in 1880, became an army officer and served with distinction in the Boer War, after which he returned to civilian life, having been found a position as a stockbroker by the financier Sir Ernest Cassel. At the time this was considered an unsuitable career for a gentleman, and in 1907 his proposed marriage to the "vivacious" Lady Gwendoline had to be postponed because her mother thought him too poor. Though self-effacing and inoffensive, a good deal of unfounded rumour attached to him as a young man (as it did to much of the Churchill clan, although in some cases for better reason): among other things, it was suggested that his natural father was the fifth Earl of Roden (or, less plausibly, the Austro-Hungarian Ambassador to Britain, Count Karl Kinsky) and that he had murdered Lord Percy, heir to the Duke of Northumberland, who had died in mysterious circumstances in 1909 and was whispered to have been the lover of Clementine Hozier, whom Winston Churchill married in 1908. It appears also that Winston had proposed marriage to Lady Gwendoline, who had turned him down in favour of his brother. In the 1920s, the mere fact that his brother was a stockbroker caused some awkwardness when Winston was Chancellor of the Exchequer. Jack Churchill fought again in the First World War and was awarded both the Croix de Guerre and the Légion d'Honneur.
Clarissa Spencer-Churchill's elder brothers were Johnnie (1909–1992), an artist, and Henry Winston (known as Peregrine) (1913–2002).