The term mathematical practice arose in the philosophy of mathematics to distinguish actual practices of working mathematicians (choices of theorems to prove, informal notations to persuade themselves and others that various steps in the final proof are formalizable, refereeing and publication) from the final result: proven and published theorems.
This distinction is considered especially important by adherents of quasi-empiricism in mathematics, a school in the philosophy of mathematics that denies the possibility of foundations of mathematics and attempts to refocus attention on the ways mathematical statements are arrived at.
The modern mathematical practices are what distinguish modern professional mathematicians from older ideas of folk mathematics. Those 'folk' practices may well include useful formulae or algorithms, but without the accompanying proof discipline.
The evolution of mathematical practice was slow, and some contributors to modern mathematics did not follow even the practice of their time, e.g. Pierre de Fermat who was infamous for withholding his proofs, but nonetheless had a vast reputation for correct assertions of results. Likewise there is contrast between the practices of Pythagoras and Euclid. While Euclid was the originator of what we now understand as the published geometric proof, Pythagoras created a closed community and suppressed results; he is even said to have drowned a student in a barrel for revealing the existence of irrational numbers. Modern mathematicians admire Euclid's practices, and usually frown on those of both Fermat and Pythagoras. Nonetheless, all three are considered important contributors to mathematics, despite the variance in method.
One motivation to study mathematical practice is that despite much work in the 20th century, the foundations of mathematics remain unclear and ambiguous. This shifted focus to some degree on 'what is meant by a proof' and other such questions of method.