You can add DLL files as usual. Just run
Install-Package AjaxControlToolkit
to install the package from NuGet.
Then reference it as Anders Rask explains here: adding reference dll in package.
For resources you could just reference them in a Module (Elements.xml). As far as I know they will be unghosted because of file system restrictions in sandboxed solution. But this should put them at root:
<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
<Module Name="Assets">
<File Path="Assets\mystuff.js" Url="mystuff.js" />
</Module>
</Elements>
(Module called Assets).
Update:
To use a pure DLL you need to embed your resources within the DLL file and use a Feature Event Receiver to add them to SharePoint.
Here is one solution for embedding the resource: http://blogs.adatis.co.uk/blogs/martynbullerwell/archive/2007/11/30/embedding-images-into-dll-s.aspx .
But you could probably use any way you like, even generate some classes with Base64 strings (ugh).
Then you use any FileCollection method which support bytes, such as SPFileCollection.Add Method (String, Byte[], SPUser, SPUser, DateTime, DateTime)