When I learned Swahili I found the fact that words are separated into classes and the number of classes that exist more difficult than the fact that it is an agglutinative language.
But if Alexander Galkin is right that might be due to the fact that German is my native language.
Sure, the two difficulties come together when you need to know which class adds what to the agglutinated word.
I do not remember much of my Swahili but one example: Back then we learned a word that means "to fell something". With it you could form a single word that expresses: "I fell it (now)(It is most likely some kind of tree because it belongs to the class of words mostly used for trees)". Now, where is the difficulty? Is it knowing how to agglutinate the word or is it to know which class uses which prefix or suffix?
Knowledge in Esperanto can help you with half the problem. But remembering all the classes, which class a word belongs to and which prefix it uses is something you just have to memorize.