Good to know you found the issue. Depending on the system, even slight mismatches in the baud rate can cause issues. For example, when I design a UART receiver using the bit-bang method, I would design it such that it would read exactly at the middle of each bit so any minor fluctuations would not matter so long as it is within the bit rate. However if the receiver was designed to read towards the end of the bit, a slight mismatch in timing will create an error. Since there is no governing standard for UART, it isn't safe to be left to assumption.
You will experience issues if the wrong 'Collection Time' is set. The decoder requires sufficient data to be collected for it to display the correct result. e.g.
9600 baud = 104 us/bit
(start bit + 8 data bits): 104us x 9 = 936 us (0.936 ms)
Therefore a collection time near 1ms/div should be used.
I noticed your voltage threshold and hysteresis may have been set incorrectly. For a 5V signal I would suggest using a threshold of 3.7 V and a hysteresis of approximately 0.5 V. the software
Hope this information helps, if you experience any further problems, please feel free to let us know.
Technical Support Engineer