I don't have experience of OSX, but at the least in Linux you can do this:
First, get some idea what system calls (syscall or int 80h in Linux) the library might be using, if any. You can use your favorite disassembler or code tracer for this part. If the library itself does not have any system calls, find out or deduce what system calls its child functions may be using. If you have no idea or this step gets too difficult due to the size and/or complexity of the library, you can skip this first part.
Then, start gdb with the program. If by now you already know which system calls to catch, then catch only those. Otherwise catch them all. The command is catch syscall. Then r to run the program.
At this point I usually do disp/10i $pc to see a little a bit of the code.
Once you're inside the system call, do backtrace to print the stack frames. Set breakpoints in all of them or only in the ones you're interested of. The instruction to use is break, for example break *0xdeadbeef. Then continue.
If you arrived in the desired library or function, great. If not, then delete the breakpoint you arrived in and then continue, and repeat until you arrive in the function you're interested of.
Hope this helps.