In March 1854 a college was born for young people. The college was just a girl's college. The girl's college was across from the little white Congregation church in Fox Lake, Wisconsin. The village of Beaver Dam and Fox Lake both offered a site and donated money for a site. Beaver Dam got a college for men. Fox Lake got the college for young women. A man named Thomas G. Eggleston held the contract to build at a cost of $6,000 on a 5 acre lot. The lot was on the west side of College Avenue between Cherry and Green Street. The building was three stories high made of stones. The college opened in October of 1856. 58 students for the three terms per year paid $14.50 to $25.00. Room and board was $80.00. This included fuel and a stove in each room. As the college and staff grew so did the scholarships. Jason Downer helped pay for renovations to the college. He gave $9,500 of the $12,000 needed for a new building.The college was then dedicated to Jason Downer on May 5th ,1870. The college became known and continued to grow, but Fox Lake did not so Downer College moved to Milwaukee in 1895. After it moved, Downer College turned into a hotel, and it was in the same building. They never took the building down. It came down by a fire.