Actually, the resin is considered the "matrix" and the fibers are really what carries the load.  The matrix resin distributes the load to the fibers thus having everything contributing to the structure.  Wood is an excellent composite being lignin reinforced cellulose.  And is probably the most fatigue resistant material around.  It will regularly go very close to ultimate starin and fully recover.  Not many materials can do that!

Your point is well taken however.  Generally though, carbon fiber fatigues at about 10 -20 % of the rate of aluminum or fiberglass.  So while aging doesn't do it any good, take heart in the fact that it's better than a lot of other materials.  Interestingly, that one reason why it's used on wind turbine blades.  It turns out that wood cored blades with carbon skins last a Hell of a long time with almost no deterioration.  Oddly, turbine blades are under very severe conditions and are prone to huge loads from gusts, torque, wobble (from passing it's wind shadow), sun, rain, high point loads at attachment points.  Aluminum and glass and almost all the other types had real problems.  Carbon and wood to the rescue!

Dan Newland