Pins should be cleaned from rust. Dremel wire brush wheel works nicely.
As for the inside of the track "tube" where the pins go ... tricky to get to, but if it's rusty in there, it'll certainly adversely affect track conductivity, and slow down your train away from transformer power hookup. "Reckers" here on the forum offered a nice tip of using a "welder's tip cleaning tool" to poke into the track tubes ... sort of like small, round files.
Inserting/removing/insterting the pins themselves will help to dislodge any surface rust and improved metal-to-metal contact.