Well... If you really care what other people, who in general probably don't even look at your watch, let alone the size of it then 34-36mm, if you find a watch that you like, because of the looks, the engineering, the style and the brand and it is 32 - 34mm and you like wearing it, then go for it!
This.
It depends on a few factors for me. Wrist size being a big factor. Overall body shape too. A lightly built bloke will get away with smaller compared to say a heavily built rugby player type.
I'm scrawny of wrist and frame so can get away with down to 30mm. Well I'd get no stares even from fellow WIS so...

Design of the watch another factor.
Here's a 30-31mm example.
1940's Alpina for the German navy in WW2. I reckon the pale dial works better than a dark. I had a near identical German Army "DH" with a black dial and it wore smaller.
Here's another 30mm on me.
1911 by the movement, but likely cased a little after that date.
Even for "god he's in need of a feed" me, some vintage are just
too small.
1950's 14kt Longines 28mm in width IIRC. I really like this watch, a great timekeeper with an amazing power reserve for the size and tim, but sadly it's just on the wrong side of small. It does look like a "womans" watch.
How fashions change though. Here's an ad from 1909 for an Elgin strap watch.
If that arm was any more masculine he'd be up a tree.
Ironically when women wore wristwatches in the late 19th early 20th century they often wore pocketwatches on leather straps and they would have run around 40mm. So it was the ladies who kicked off the big watch look.