There are all sorts of numbers but we can focus on natural numbers and real numbers. Natural numbers are intuitive concepts emerging from our experience of considering, exchanging, trading discreet quantities of things, such as for example, apples. The formal, mathematical version is essentially an elaboration on our intuitive notions.
Real numbers?
Fractions and therefore the idea of rational numbers also came through the bare necessities of exchanging goods, and sometimes, half or a quarter etc. of the usual quantity. The idea of negative numbers came intuitively out of the need to represent debt. So not only natural numbers, but also integers and rational numbers are intuitive conceptual constructs that emerge directly out of our everyday experience of the real world.
Real numbers? It is usually said that the first idea of real number was the idea of irrational number that mathematicians discovered when trying to find numerical solution to quadratic equations. However, the idea that the set of abscissa of points of an infinite straight line is the same as the set of real numbers demonstrates that we also have an intuitive notion of real numbers, essentially as distances in space and intervals in time.
The particularly interesting twist, though, is that we don't know how to construct all real numbers from rational numbers, even though we understand, again intuitively, that we could in principle find rational numbers infinitely close to any real number.
So the visual representation of numbers is perfectly valid as far as anyone can tell. It is valid probably because our idea of numbers come in large part from our experience of our visual perception of the world, being apples or distances.
Arithmetic is just the systematisation of our intuitive notions about numbers. As such, arithmetic essentially came an effort to represent formally our intuitive concepts. No intuition, no arithmetic.