Well for starters you are going to want an in Kernel USB monitor like Linux has, a pass through USB analyzer (more expensive, but more flexible) or something like a logic analyzer.
Here is a video that goes into great detail of doing the latter far more than anyone will here https://www.youtube.com/watch?v=4FOkJLp_PUw Note from the video https://opentechlab.org.uk/videos:011:notes
To summarize you'll need a logic analyzer, potentially a way of getting the device to tell you it is about to send a packet in some manner (otherwise you will have better luck with a USB analyzer or in kernel USB monitor). Sigrok is software used there... but there are others.
Here is a little python code the guy in the video uses which will probably be of use to you. Which is just a basic example of using pyusb he was using it to talk to whatever micro controller board he was debugging you could use it to talk to your FPGA perhaps as you figure out what it's protocol is doing.
#! /usr/bin/env python3
import usb.core
import time
dev = usb.core.find(idVendor=0x0925, idProduct=0xD100)
if dev is None:
raise ValueError('Device not found')
res = dev.ctrl_transfer(
bmRequestType=0x40, # OUT, VENDOR, DEVICE request
bRequest=1, # Request #1
wValue=0xCAFE,
wIndex=0xD00D,
data_or_wLength=[0x01, 0x23, 0x45, 0x67])
The method you choose largely depends on what you need to know, if the device is functioning properly and you don't need to make changes to it, perhaps a software monitor is good enough. If it is indeed USB serial is it a dedicated chip? If so find it's datasheet... and investigate the circuit that may tell you some things. Common USB to serial chips are the the following which either use a standard USB CDC ACM protocol (check if chip in question is this then you can just grap the spec for it here ) or custom driver.
Note that the USB end of the protocol itself is probably baud rate independent but you may need to tell the USB serial convert what rate to run at... it depends alot on how it was designed. Probing the Uart output pints of the controller would allow you to figure out the correct baud rate if you can't figure it out from the circuit itself.
Another way to go about this would be to figure out at least one command that will give you a response... then exhaustively try different baud rates and configurations until you get the expected response.
Silicon Labs CP2102 and variants.
https://www.silabs.com/documents/public/data-sheets/CP2102-9.pdf
MicroChip MCP2200
http://www.microchip.com/wwwproducts/en/en546923
- FT232R http://www.ftdichip.com/Products/ICs/FT232R.htm