If you are going to keep the locomotive(s) away from the dirt, say up on ballasted rails, you are not likely to have any problems with properly affixed 'gravel' or 'dirt'. I sifted garden soil through pantyhose and got some nice fine stuff. I had tested the local beach sand that I used for ballast (on three layouts), but didn't think to test the fines that went through the pantyhose.
I mixed the fines with some plaster of Paris powder, shook it over the surface of the layout where I wanted it, rolled it flat (most of it was around the rails in my yard), and then sprayed it with alcohol and water. Once it was wet and setting, I sprayed it with dilute wood glue and some dish detergent to help it to penetrate. I never had any issues.