Seth Phelps was named after his grandfather who served in the
American Revolutionary War and at times with
George Washington and was present at
Valley Forge. The senior Seth was later promoted to Captain and became an aide to General Washington.
 Seth Ledyard's father's name was Alfred Phelps, who served in the
War of 1812 under Winfield Scott in the
Battle of Queenston Heights in Ontario. After the war Alfred returned home, started a law practice, and then met and married Ann B. Towsley on July 1, 1820. Shortly thereafter Seth was born on January 13, 1824 in
Parkman, Ohio, the eldest of five siblings. His two younger brothers Alfred and Edwin soon followed. The Phelps family moved to
Chardon, Ohio and bought a farm just east of
Cleveland, a short distance from
Lake Erie. Later in life, Seth's father became active in Republican politics in Ohio.
 Seth grew up near the lake and listened along with his brothers to the stories of his father about his seafaring adventures, especially those of Oliver Hazard Perry. These stories are largely what inspired Seth to pursue a career in the Navy.
 He married Elizabeth Maynadier (born July 21, 1833, died May 27, 1897),
[c] on July 1, 1853, whom he would affectionately refer to as "Lizzie". She was the daughter of Captain Maynadier, of the Ordnance Department, Washington D. C. During his naval service Phelps frequently wrote to her of his life in the military.