J.E. Lighter, Random House Historical Dictionary of American Slang (1994) observes that crackers was originally British English slang for crazy:
crackers adj. crazy; insane.—used predicatively. [First two cited instances:] 1925 in Fraser & Gibbons, Soldier & Sailor Words {ref to WWI}: To get the crackers, to go off one's head. Mad. 1928 in OEDS: I shall go "crackers," (meaning mad) if anything happens to Ted.
The popularization of the term seems intimately connected to World War I. A very early instance appears in James Milne, The War Stories of Private Thomas Atkins (1914) [combined snippets]:
In fact, this war is nothing but an artillery duel, and the country for miles is very wooded, which makes it harder for us, because we cannot see them till we are almost on top of them, and then they have first plonk at us. The Kaiser's crack regiment, the Prussian Guards, went crackers before we were out a fortnight. There was a pretty dust-up. We caught them coming across an open field. We let them come within 200 yards of us, and then we let go. We almost wiped them clean out. It was an awful sight when we finished.
The term "crackers" itself may owe something to firecrackers, but it also owes a considerable debt to cracked (in the sense of crazy)—a much older term. Again from Lighter:
cracked adj. 1. crazy; insane. Orig. S.E. [First three cited instances:] 1692 J. Locke, in OED. 1705 in OED: You are as studious as a cracked Chymist. 1775 S. Johnson, in OED: I never could see why Sir Roger is represented as a little cracked.
A number of early instances use the phrase "cracked in the head," suggesting either a disintegrating brain (on with cracks in it) or a person showing the effects of having been struck hard (cracked) on the head with a heavy object.
To be "driven crackers" simply means to be pushed toward insanity, in a figurative sense. The expression is usually used as a form of hyperbole, although it may sometimes refer to an event that was so horrific that it actually sent the victim over the edge into madness.