We consider a new method to improve the quality of training in gradient boosting as well as to increase its generalization performance based on the use of modified loss functions. In computational experiments, the possible applicability of this method to improve the quality of gradient boosting when solving various classification and regression problems on real data is shown.