i try with drills all the time, with that last hole you forgot. and it is 10x harder..... it will work for this project but it will be a pain. aluminum should be like butter, even with old dull drill bits. stainless steel is the mother F#$@ER
honestly its a low speed, in intervals. with cutting lube as ONE person recommended (has no one done milling before) iv blasted through bit after bit. and finally used cutting oil and a spray bottle of water. and haven't broken a bit since, and all by bits last 5x longer
any bit will work for aluminum but carbon and titanium will stay sharper and be easier to cool....
start a pilot hole
lube up
low speed
EXTREME PRESSURE
and if you are struggling. stop and cool off the bit and the piece.
its excessive friction that heats and ruins bits. it binds and breaks them as well as lower rigidity and increase damage.
dont push dangerously hard to where your going to slide everywhere. but in short, you would never be able to apply the downward pressure a drill press can.... you dont think its a big difference but it is...... one thing i find easiest is to drill on a piece of scrap wood vs a vice or just over air.... it stops the indentation and bending of the piece and makes the bit eat a bit easier