It depends on your grow set up. I grow small plants in a small cabinet and use one gallon pots with no problems. There was a contest on here where they grew plants in 16 oz party cups.
Heres how to tell if you can downsize your pots, when you cut down whatever your growing in the 5 gallon buckets, dump them out and see how the roots look. if they are curling around the bucket or are rootbound, you need to keep the buckets, but if there is a lot of soil that the roots never got into, you can go with smaller pots.
@dizzle frost, your signiture is awesome