You might be surprised to see how hot your fork oil gets, and how dirty it is when you drain it out. Go for a ride and drain it and see for yourself how hot and dirty it is.
This stuff went in pink and in 12K miles it turned black.
The teflon on the bushings wear and so do the sliders, depositing metal into the oil. The oil also breaks down. It begins to get thinner in the early stages, and if not changed will then start to thicken and turn to an almost mud like composition. The change in viscosity alone is enough to affect damping qualities (both compression and rebound).
But don't take my word for it, go ask any suspension expert that rebuilds forks and shock for a living. Go talk to the folks at Traxxion or Works Performance or Ohlins or any other suspension expert and see what they tell you about changing the fork oil.