Use the .size() built-in Deluge function to return the number of elements in the list. Refer sample script given below. You can add the script to the on user input block of the check box field. Here, Checkbox_1 is the name of the checkbox field n the form.
if (Checkbox_1.size() <3 )
alert "Please select at least three options";