Hi Steve,
The torque curve for the rotary is fairly flat up into the 8000+ RPM range. So in general, HP will continue to go up with RPM until destruction of the bearings around 1100 is a number I have seen floated around for stock rotors. The prop will start becoming very inefficient as the tips go supersonic.
That means, you calculate your RPM for supersonic tip speed, and that will be best power/redline limit.
Peak efficiency is a different story.
Dave Leonard