For Germany, depending on your citizenship, you might even be able to apply after entering the country. Specifically, that's possible if you are from the US, Australia, Canada, Israel, Japan, New Zealand, Brazil, South Korea, a few tiny countries or, obviously, a member state of the European Union (in which case you don't need a visa or permit as such but might need to register your presence). Otherwise, you do need to apply for a student visa in advance.
As far as I know, you have to apply for that visa to the German consulate for your place of residence but the German authorities have their own notion of who is a resident and who isn't and you don't need to be a permanent resident to be able to apply. Thus, among the things you need to submit according to the German embassy in Washington, you will find:
- additional for non-US citizens: original of valid US alien registration card or valid US resident visa (type A, E, F, G, H, I, J, L, O, R) and two copies thereof
[…]
- your driver's license and/or utility bill in your name as proof of residence in the consular district where you plan to apply
which suggest that while a J visa isn't an “immigrant visa” under US law, it's still a “resident visa” as far as the German authorities are concerned and you could still apply there if you have a proof of address. On the other hand, a B-visa (temporary visit) wouldn't be sufficient.
For Canada, this question on the travel website suggests you could also apply from the US, if you are staying there for one year.