Hi,
Just as it’s not a good idea to parachute into the wilderness without testing your boots on a path first, don’t try debugging/ learning how to use both a new hardware toolchain and reverse engineer a device at once.
Break the task into parts - and get at least the first few to work first.
Attach the analyser to a known serial device and get clean data. USB serial adapter on a PC with a terminal emulator would be a start.
Oh, and don’t sit in front of a screen and STARE for hours. Go for a walk. I realised an odd wireline protocol was just Manchester encoding whilst on a bike.
Does the analyser have analogue inputs? If so, look at the actual voltages - mark/space might show you inversion, check 3V3 / 5V / 12V, and bad ringing. Again, start with a known good RS232 port and 'scope that so you understand timebase, one-shot triggers, etc.
If this helps, 
this post!