When exposed to cold, your body prioritizes warmth distribution by pulling blood from the extremities to support the more important organ-housing torso. Before suiting up, take into account that your body will warm up significantly during strenuous activity by dressing as if the temperature is 10-20 degrees warmer than it is. Pamper yourself (unless your looking to do some “character building”), but dress in easily strippable layers so that you can remove them as the body heats up.
Ideally, you'll want to wear some type of moisture wicking material as your inner layer to keep the skin dry and prevent chaffing. Avoid cotton: polypropylene and wool are much better choices.
Second, add a fleece layer for warmth.
If it is nasty out, throw a weatherproof shell over it all.
Gloves and a beanie can do wonders for your motivation and are great at keeping body heat in.
You don't have to be a bank robber to don a balaclava. In other words, don't worry about looking silly. Idle onlookers will be too blinded by the shame of their own dormancy to notice even your most egregious fashion faux pas. The name of the game is warmth here, folks, do what it takes: dress for success.