K-polystability of a polarized variety is an algebro-geometric notion conjecturally equivalent to the existence of a constant scalar curvature Kähler metric. When a variety is K-unstable, it is expected to admit a 'most destabilizing' degeneration. In this note we show that if such a degeneration exists, then the limiting scheme is itself relatively K-semistable.