In software engineering we have this recurring pattern of rejecting "complex" things, and replacing them with "simple" ones, until it turns out the simple approach doesn't handle edge cases, has flaws, and after all the shortcomings are fixed, we end up with a complex solution again. There's nothing wrong with doing it this way (Gall's Law: "A complex system that works is invariably found to have evolved from a simple system that worked"), but you have to realize you're not getting rid of complexity with a silver bullet, you're merely starting from scratch.
Clear design is software architecture. If your design is uncelar because of "too much architecture" that's overengineering, YAGNI and just plain old poor design.
As your requirements grow, your simple architecture will either become a ball of mud, or you'll find yourself adding the patterns you dread. You can laugh at an AdapterFactory, but once you get a requirement to support several subsystems swappable at runtime, that's what you're going to write.
The benefit of having names for these patterns is clarity. Instead of winging it, you can plan it. You can communicate to new members how things are put together using a shared vocabulary.