Sargent is a descendant of Epes Sargent, a colonial military leader and jurist. Before John Singer Sargent's birth, his father, FitzWilliam (b. 1820 Gloucester, Massachusetts), was an eye surgeon at the Wills Eye Hospital in Philadelphia 1844–1854. After John's older sister died at the age of two, his mother, Mary Newbold Singer (née Singer, 1826–1906), suffered a breakdown, and the couple decided to go abroad to recover. They remained nomadic expatriates for the rest of their lives. Although based in Paris, Sargent's parents moved regularly with the seasons to the sea and the mountain resorts in France, Germany, Italy, and Switzerland. While Mary was pregnant, they stopped in Florence, Tuscany, because of a cholera epidemic. Sargent was born there in 1856. A year later, his sister Mary was born. After her birth, FitzWilliam reluctantly resigned his post in Philadelphia and accepted his wife's request to remain abroad. They lived modestly on a small inheritance and savings, living a quiet life with their children. They generally avoided society and other Americans except for friends in the art world. Four more children were born abroad, of whom only two lived past childhood.
Although his father was a patient teacher of basic subjects, young Sargent was a rambunctious child, more interested in outdoor activities than his studies. As his father wrote home, "He is quite a close observer of animated nature." His mother was convinced that traveling around Europe, and visiting museums and churches, would give young Sargent a satisfactory education. Several attempts to have him formally schooled failed, owing mostly to their itinerant life. His mother was a capable amateur artist and his father was a skilled medical illustrator. Early on, she gave him sketchbooks and encouraged drawing excursions. Sargent worked on his drawings, and he enthusiastically copied images from The Illustrated London News of ships and made detailed sketches of landscapes. FitzWilliam had hoped that his son's interest in ships and the sea might lead him toward a naval career.
At thirteen, his mother reported that John "sketches quite nicely, & has a remarkably quick and correct eye. If we could afford to give him really good lessons, he would soon be quite a little artist." At the age of thirteen, he received some watercolor lessons from Carl Welsch, a German landscape painter. Although his education was far from complete, Sargent grew up to be a highly literate and cosmopolitan young man, accomplished in art, music, and literature. He was fluent in English, French, Italian, and German. At seventeen, Sargent was described as "willful, curious, determined and strong" (after his mother) yet shy, generous, and modest (after his father). He was well-acquainted with many of the great masters from first hand observation, as he wrote in 1874, "I have learned in Venice to admire Tintoretto immensely and to consider him perhaps second only to Michelangelo and Titian."