itch.io

So I did some reading and ran some diagnostics. What it comes down to is the Focusrite driver.

The problem is specifically with non-ASIO applications. Like most games, Discord, etc.
The Focusrite ASIO driver is great and has great latency. Focusrite's WDM driver however... is trash. 
I've measured the latency in a bunch of different applications and the latency can be as bad as 400+ ms. 

Now I'm not sure if this also affects older Gens but it is absolutely a problem on the 4th Gen Scarlett. 
Specifically combined with Windows 11 it seems.

I've tried a lot of things, including looking at LatencyMon, changing things in MS Interrupt Affinity Tool, messing with sample rate, buffer size and the list goes on.

The only solution I have found thus far is completely removing all Focusrite drivers, I also removed whatever left-over folders and "invisible" devices I could find in my sound settings. Also make sure to double check your Device Manager.
Once you've done that, restart your PC and plug your Scarlett back in. 

Using Window's native drivers for the interface, you'll have next to no latency. It is miles better than the Focusrite driver.
I was immediately able to play games I wasn't able to play before due to massive latency improvements and also noticed a huge improvement on Discord.

You'll miss out on Focusrite Control 2 but in turn you'll have no latency, besides most of the functions are buttons on the interface itself. You're not missing out on a whole lot, except for the loopback function and Focusrite USB ASIO (if you were using that in a DAW for example).

The next paragraph only applies to 2i2 and above due to the amount of channels, this is not an issue with the solo if you change it to 44100 kHz like another user mentioned. (Can't test this as I'm using a 2i2 with 4 channels).

Now doing this you will run into the issue a lot of other people are having trying to play The Choicer Voicer. 
Which is the game won't recognize your microphone signal at all. (Not to worry it does work in every other application I tested, seems to be isolated to the Godot engine).
There is a relatively easy workaround, though. I just route my microphone through Voicemeeter and select the Voicemeeter output in The Choicer Voicer. Sure this will add the tiniest bit of latency again but compared to what it was using the Focusrite driver, it's negligible. Took a while but I can finally play the game, and a lot of other games I wasn't able to play before.

I do hope Focusrite fixes this issue at some point, it's a seriously bad oversight. 

Quick edit: Once you're using the Windows drivers make sure to go into your sound settings and turn off any audio enhancements. Windows likes turning those on for some reason.