Here is the calculation for figuring out minimum dot size. The minimum dot size control is found in the Calibration & Screening section of output process templates.

Print tint patches of the screen at the screen frequency you will use, concentrating the patches in even steps (for example, 0.5 percent or 1 percent) in the highlights and shadows, making sure to fully cover the printing range of your press.

The average distance in pixels between adjacent dot centroids is:

device_resolution / screen_frequency = average_dot_spacing

For example, at 150 lpi and 2400 dpi, the average dot spacing is 16 pixels.

This means that each dot is, on average, 16×16 = 256 pixels in size. The supercell screening in Prinergy will allow you to have more than 256 gray levels by distributing additional grays over a wider area, but this is a good number for calculation.
So, a 16-pixel dot cluster at 150 lpi corresponds to a tint of 16/256 = 6.25 percent. Likewise, a 4-pixel dot cluster is 4/256 = 1.56 percent. That is, if you see everything at 6 percent or lighter washing out at or not printing reliably at 150 lpi, you should set the minimum dot size to 16 pixels.

The full formula is:

Highlight minimum dot size = lowest_printing_tint × (device_resolution / screen_frequency^2

You can do a similar calculation for the other end of the tone scale, in the shadows. For example, if you can print as much as 85% at 150 lpi before totally plugging, then:
(100% - 85%) × 256 = 38 device pixels

So the formula is:

Shadow minimum dot size = (100% - highest printing tint) × (device_resolution / screen_frequency^2

The reason why the control specifies the minimum dot size in device pixels rather than a tint value is because, for a given printing process, the number of device pixels that can be reliably printed is usually constant, while the same number of device pixels corresponds to different actual tint values as you change the screen frequency. So, the idea is that you set the minimum/maximum dot value once for your press/plate combination, and leave this setting alone, even if you change to some other screen frequency.