The ball is the only thing that can break a rock, and breaking one puts two faster ones back on the court.
Rocks drift through the middle of the court and never reach your paddle.
The ball bounces off them — and breaks the big ones into two smaller, faster ones.
A rock never changes how fast the ball is going, only where it ends up.