This is at best an oversimplification.  Flutter is a function of the amount of energy that the moving air can impart on the flexible parts of the aircraft.  It occurs when the energy going into the fluttering component exceeds the ability of the structure to absorb it.

To make a counter-oversimplification, if you take the amount of energy available from the relatively moving air, it is proportional to the square of the velocity, but it is also proportional to the mass of the air.  Velocity increases with higher TAS, but mass decreases with increased altitude, where IAS and TAS differ.  (This is an oversimplification because a lot more goes into the initiation of flutter than just the energy available in the air!)

Some aircraft are now built with Vne published at different values at different altitudes.  Others have a cruise speed (published in TAS) at high altitude which exceeds their Vne (published in IAS)!

Saying that "Vne is actually a function of TAS" is at least equally wrong as saying "it is simply a function of IAS."  The truth is somewhere in between, and varies from one aircraft design to another.

Flutter is a relatively rare occurrence, and in-flight breakups as a result rarer still.  I don't think a flutter-induced breakup has ever happened to a Rutan-designed aircraft, and if it's happened to RVs, it certainly hasn't happened often.  The published Vne numbers have a safety factor, and the designers should have taken flight typical altitudes into consideration when calculating them.  There are a lot of ways to get yourself killed in an experimental aircraft, and a few extra if you fly with an experimental engine.  The potential difference in Vne from altitude is not high on the list, and probably not on the list at all if your airplane is not substantially overpowered.