Benedict Freedman was born in New York City in 1919 and was the son and grandson of writers. From a young age, Freedman showed great academic promise and was enrolled in accelerated courses for gifted boys during high school. He excelled in mathematics and received a medal for his achievements in the subject. After high school, Freedman attended Columbia University as a premed student, but had to drop out at the age of sixteen due to his father's sudden death.
Despite having to leave college, Freedman continued to pursue his interest in mathematics through private study. His main focus was on games and recreational mathematics, and he even assisted in writing a textbook on the subject. In addition to this, Freedman also worked on actuarial problems as a clerk to a consulting actuary. Freedman's passion for mathematics and his natural talent for the subject allowed him to make a significant contribution to the field, despite not completing a formal education in the subject.