A worm gear is the gear, which looks like a screw, mounted on the output shaft of the motor.
As the shaft spins, the power is transmitted from the screw to a spur gear engaged with it. This serves the purpose of speed reduction and torque multiplication. Additional gears transmit the power to the axles, allowing the locomotive to move down the track.
It is very important that the gears do not bind or jam, as this could damage the motor and decoder. The gear train should be examined and lubricated at regular intervals. Prior to conversion to DCC, check the operation of the locomotive and make any adjustments needed, as well as lubrication.
New locomotives may have too much, or not enough lubrication. The gears should be cleaned and lubed to allow them to wear in. If the locomotive was sitting for a long period of time, the grease used may have hardened on the gears and require removal prior to the locomotive being placed on the layout.
Older locomotives may have worn or broken gears. A noisy locomotive, or one that jumps in operation, is suspect. Some brands had issues with gears splitting, even when purchased brand new. Replacement gears and gearboxes are available, and it may even be possible to change the ratio to improve operation.
See the animation here (wikipedia): Worm Gear