It is often said that “premature optimization is the root of all evil”. The full quote from Donald Knuth is:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%

What this indicates is that worrying about optimization in non-critical parts of the code is not worth the effort.

During the early part of the development process, there should be focus on having a good clean design and all code should be written following the clean code principles. At the same time, we need to look for efficient code in every possible place, especially in critical sections.