There is so much rolling stock on the used market that you should really be choosy or you'll end up with a mountain of stuff you'll never use. After two years in the hobby, I've all but stopped buying freight cars already. If I buy a piece, it's usually at a train show at a price I can't refuse.
Are you into postwar? RailKing? Scale size stuff? Also, do want to run freight or passenger? Both? Do you want to run prewar style wooden box cars or newer auto racks, ocean containers, etc.?
To give you some ideas, I decided I like things more scale sized and I want to concentrate on roadnames that might have been seen in my area in the 40's-70's. That required some brushing up on history. I'm still learning but to me that's fun. That is still a lot of road names though!
So for freight cars, I settled on a few road names that actually had their headquarters where I live (Missouri Pacific, MKT, Frisco, & Wabash). Within those road names, I usually only go for scale sized rolling stock and I only buy used, LNIB. That whittles it down a bit and keeps the budget down.
For passenger cars, I only go for road names that used St. Louis Union Station. Again, a pretty wide net. However, I don't consider any older passenger cars with silhouette windows, nothing under 18", and nothing with incandescent bulbs. I realize I could convert older cars to LED but the last thing I need is another PITA project and it adds cost. Again, there's such a glut of this stuff on the secondary market, if you have any patience at all, you'll find what you want (in spades!). I've pretty much settled on newer MTH passenger cars with LED lights. They hit my sweet spot for size/budget.
Ultimately, it's the old answer: it's up to you. Or the way it's usually put around here: it's your railroad...