You'd be removing one of the most important limits on spellcasters.
Consider a few different spellcasting classes up through level 10 (after which the balancing mechanisms are similar but less straightforward):
- Warlocks know about 1 spell per class level + 1, give or take (2-10 spells, and Patron spells are only added to the Warlock list, not necessarily known), and they can cast any of them 2x/short rest. They also get other stuff like Invocations and Pact benefits.
- Clerics know all their class spells (16, 17, 25, 11, 16, for about 9 spells per class level depending on domain and sources), of which they can choose around level+4 to prepare, and cast (2, 3, 6, 7, 9, 10, 11, 12, 14, 15) = about 1.5 per level. They also get other stuff like armor or even weapon proficiencies, depending on domain. Druids work similarly, with either kickass Shapeshifting or extra spells.
- Bards know just about level + 3 spells, and can cast one about 1.5x per level, like a Cleric. They also get a variety of bonus features, depending on subclass.
- Sorcerors know exactly level+1 spells, and can cast them with spell slots like a Cleric. They get Sorcery Points, to use for extra spell slots or metamagic shenanigans, and relatively minor bonuses from their subclass.
- Wizards know at least (2*level)+4 spells, probably more depending on the game. They can prepare about level+4 of them, and get nifty but modest benefits from their subclass.
Overall, the pattern is basically: every spellcasting class is limited in the amount of spell slots they have. They're also limited by either the spells they know, or the spells they can prepare. Technically, Wizards have both of these restrictions, but the number of spells they know is quite high, making the requirement to anticipate problems and prepare spells in advance the far more restrictive one on a daily basis.
And for all that, wizards don't get extra features as powerful as those of the Cleric, Druid, Bard, or Warlock. Why? Because wizard spells are deliberately more powerful. Wizards are the ones who bet everything on their magic, and the payoff is that they're overall better than other classes (except possibly the Sorceror) at doing damage, crowd control, defensive buffs, non-combat spells - basically everything except healing.
So if you let wizards cast from their spellbook, effectively removing the requirement to prepare spells, then you'd have an incredibly versatile caster who has the perfect spell for just about every occasion. Which could be a lot of fun, no doubt... but it'd probably make it a lot more difficult to spotlight the other players if the wizard can fix every problem on any given day. You'd have to really stress their spell slots before the other characters' usefulness became apparent, even more so than is already the case.