The block number will always be correct by definition: It's number x in the chain, because it's chained on top of x-1.
However, as you say block.timestamp can be gamed a little bit - or with the cooperation of the economic majority of validating nodes a lot - which also means that the relationship of block.number to actual time can be gamed. So if you don't trust that a block.timestamp of 2017-01-01 will really be approximately 2017-01-01, you can't rely on counting the blocks that are supposed to be mined between now and 2017-01-01 either.
PS. People may be able to give you more helpful advice if you tell us what purpose you want to be secure for, and what you want to be secure against.
block.timestampspeculation is usually warrantless.block.timestampis very safe in real-life scenarios.block.timestampis safe for 99% use cases. If you are afraid of the frontrunning problem, then usingblock.numberdoes not really solve it. If you are afraid miners mess with your transaction, then you have an avenue of other attack vectors that usingblock.numberdoes not solve. – Mikko Ohtamaa Oct 12 '20 at 08:24