Here’s a universal suggestion for stream quality boost, no matter which encoder you use:
Play your game in the same resolution as your stream. So, no downscaling. OBS’ scaler favors speed, rather than quality. Streaming at the same res as your gameplay, looks a lot better.
Certain games lend themselves to this. Fighting games, for example. Those are absolutely fine in 720p. So, play and stream at 720p for better quality. Rather than downscaling from 1080p or whatever.