The first reference to the Fugger family in the Bavarian Free City of Augsburg is the arrival of Hans Fugger recorded in the tax register of 1357. He married Klara Widolf and became an Augsburg citizen. After Klara's death, he married Elizabeth Gfattermann. He joined the Weaver's Guild, and by 1396 he was ranked high in the list of tax payers.

His eldest son, Andreas Fugger, was a merchant in the weaving trade, and was nicknamed 'the rich Fugger', buying land and other properties.

Andreas' son, Lucas Fugger, was granted arms by the Emperor Frederick III, a golden deer on a blue background, and he was soon nicknamed 'the Fugger of the Deer'. However, he was too ambitious, and went bankrupt.

Hans Fugger's younger son, Jakob the Elder, founded another branch of the family, This branch progressed more steadily and they became known as the 'Fuggers of the Lily' after their chosen arms of a flowering lily on a gold and blue background. Jakob was a master weaver, a merchant and an alderman, and married Barbara Basinger, the daughter of a goldsmith. His fortune progressed, and by 1461, he was the 12th richest man in Augsburg. He died in 1469.

Jakob's eldest son, Ulrich, took over the business on his father's death, and in 1473 he provided new suits of clothes to Frederick, his son Maximilian and his suite on their journey to Trier to meet Charles the Bold of Burgundy and the betrothal of the young prince to Charles's daughter Maria. This started the very profitable relationship between the Fugger family and the Habsburgs. Ulrich died in 1510.

Ulrich's youngest son, Jakob the Younger, was born in 1459, and was to become the most famous member of the dynasty. He married Sibylla Artzt in 1498, but they had no children. He was elevated to the nobility of the Holy Roman Empire in May 1511. Jakob died in 1525. He is considered to be one of the richest persons of all time.

Jakob's successor was his nephew Anton Fugger, son of his elder brother Georg. Anton was born in 1493, married Anna Rehlinger, and died in 1560.

In 1511 Jakob deposited the large sum of 15000 florins as an endowment for some almshouses. In 1514 he bought up part of Augsburg and in 1516 he came to agreement with the city that he would build and provide a number of almshouses for needy citizens. By 1523, 52 houses had been built, and the Fuggerei had become into existence. It is still used today.

See also German colonization of the Americas