The downgrade patcher currently works only for the Steam version of Skyrim. GOG users cannot downgrade because there is no DRM-free version of 1.5.97 to apply.
There is frequent confusion between "Special Edition" (SE) and "Anniversary Edition" (AE): skyrim+skse+16640+2021