See images, hope they explain things a little further. To get noise free results like a simple quick render with HDRI in environment slot I have to crank up the quality quite high, no matter if I use HDRI only or HDRI in the environment slot along with light portals (set to normal, simple work only for simple, small openings with no things in front of the window or overhangs and such). Tried it with various HDRIs and several setting, all the same. Other light work fine, same as the HDRI in a dome light. Same with camera: brightening up the image through ISO, FNumber or Shutterspeed gets kind of 'clamped' as well -> concearning the HDRI only. I ran into some strangeness now - rising the HDRI value does work up to a certain degree only - rising it higher does not bring brighter results! The light intensity of the HDRI is lower, true, but as I tend to stick to real light values and corresponding physical camera settings it wouldn't matter much to me tweaking the HDRI values - as those are kind of 'unsure' anyways.
Preview / LQ renders turn out pretty slick actually. Strange enough I was surprised by the speed and quality an HDRI, plugged into the environment slot, produced with an interior (I do exteriors mostly). I did not use VrayHDRIs in the environment slot before much because it always caused more grain / noise. Even though this thread is a little older I have to jump in here, a little confused.