Am I more likely to get an infection because I wear colored contacts?

According to a new study, yes.
The study examined clear and colored contacts (Acuvue 2 and Define) and exposed them both to Acanthamoeba. The authors found that Acanthamoeba adhered significantly more to colored contacts than clear contacts even when soaked in a multipurpose solution.

Why is that?
Because the surface of the colored lenses are more irregular and rough as compared to clear lenses which makes this a great place for Acanthamoeba to stick.