Let's say someone pays you, can't they figure out your balance since the ledger is exposed to the public?
Yes. Ethereum is fully transparent. With ZK-Snarks coming soon, that could very well change though.