Hmm, well, it's virtually impossible to be sure what the issue is, at least from my PoV. Your project already has many many moving parts which could be the issue right now. Back when I was fiddling with different rendering solutions, I too ran into a lot of weird and buggy behavior. About all I can offer at this point is to tell you how I've been handling resolution on my own project and perhaps you can gleam something useful from that. As yet I've had 0 problems since I've started doing things this way:
I have a fairly simple 'Resolution' Class. It only really does two things. I set the desired target resolution for my game(the virtual resolution) on the Resolution class(it's a persistent service class), in my case 1280x720 at the moment. It then has a method for applying a new window resolution.
When the resolution is applied, it goes through a similar process as your 'SetScreen' method. First check to make sure the new desired height/width don't exceed the actual dimensions of the monitor. you can do this with GraphicsAdapter.DefaultAdapter.CurrentDisplayMode.Width/Height .
As long as the new desired size fits in that size, graphics manager sets the PreferredBackBufferWidth/Height to the new desired dimensions and applies changes. If setting fullscreen, check to see if the new fullscreen resolution is supported. GraphicsAdapter.DefaultAdapter.SupportedDisplayModes . Iterate through the modes and check to see if the new desired height/width match one of the supported modes. Set the back buffers as before and apply changes.
Next the viewport is set. This is where letterboxing and pillarboxing are calculated, if needed. I'll just post the code directly:
// The preferredWidth is the desired user window size. Dividing the width of the user window
// by the aspect ratio we want for our game screen, gives us the required height to maintain
// that aspect ratio. This is needed to ensure that the game screen isn't flattened if the
// user's window is shorter than needed.
int width = _preferredWidth;
int height = (int)(width / GetVirtualAspectRatio() + 0.5F);
if (height > _preferredHeight)
// If true, this means that the aspect ratio determined height of the game world screen
// is taller than the user's actual window size for the game. If true, the height of
// the user window becomes the limiting factor for the game screen, and pillarboxing
// will allow us to maintain the correct aspect ratio for the game screen.
height = _preferredHeight;
width = (int)(height * GetVirtualAspectRatio() + 0.5F);
Viewport view = new Viewport();
view.X = (_preferredWidth / 2) - (width / 2);
view.Y = (_preferredHeight / 2) - (height / 2);
view.Width = width;
view.Height = height;
view.MinDepth = 0;
view.MaxDepth = 1;
graphicsManager.GraphicsDevice.Viewport = view;`
In short, this will add colored bars(of whatever your graphics clear() color fill is) to the top and bottom, or left and right of the screen to fill up whatever space is unused due to maintaining the aspect ratio. Doing this is what prevents the stretch/warp of using window sizes with aspect ratios different than your target resolution.
The next thing to do is set some state data on the Resolution class which can be used by other classes. Just basic data about the current state of the resolution. The 2 important ones I'll list here:
GameScreenScale = new Vector2(graphicsManager.GraphicsDevice.Viewport.Width / VirtualResolution.X,
graphicsManager.GraphicsDevice.Viewport.Height / VirtualResolution.Y);
scalingFactor = new Vector3(GameScreenScale.X, GameScreenScale.Y, 1);
RenderingScale = Matrix.CreateScale(scalingFactor);
GameScreenScale is the similar scale to yours. However, note, it's based on the viewport and the virtual resolution, and not the backbuffer. This is in case letter/pillar boxing is happening due to our setup. RenderScale is a matrix, like the one I mentioned before, which uses the game screen scale we calculated. That's about it.
Moving on.. The last thing my Resolution class does is it fires off an 'OnResolutionChanged' event which other classes can subscribe to if needed, so they can deal with the new changes.
After that, it's fairly simple.
Each draw phase I create a new rendertarget, which is the same size as my target resolution. The render target does not get resized when the window changes sizes. So if you shrink your window to 200x100, the render target is still 1280x720. The only thing that actually changes when the window is resized is the viewport, and my state data(such as the current scale, and the scaling matrix for use by my final batcher). AFAIK, there is no need to change the size of your render target to anything but your intended target resolution. Maybe it works differently in 3D games? I haven't messed with 3D stuff yet. Anyway..
So my 1280x720 render target gets draw onto by all the various draw calls from my gui and whatnot. Also important to note here is that you do not scale your gui and other drawables, generally. That happens later. Right now you want to draw everything relative to your virtual resolution, which is the same as the render target. So if you have a GUI window you want to be at x:200 and y:200, with a width/height of 500x500 relative to your target resolution... then you draw that rectangle at exactly that position. No scaling or anything like that.
After all your batches .End() and everything is drawn onto the render target, you finally draw the actual render target as normal. The last piece of the puzzle here is that it's at this point and only this point that you apply scaling. So in your last batch here, include that Matrix scaler from earlier, from the Resolution class. This will scale up your render target... and by extension, everything you drew to the render target, is now scaled in one fell swoop at the very end.
So if my window is 640x480 (ratio of 0.75), then my viewport has been automatically resized to 640x360 (ratio of 0.5625) and centered on the screen, and letterbox bars will fill the rest. Everything gets drawn onto a 1280x720 render target.. and at the very end when I draw the target, I scale it. The scale is based on viewport and virtual res, which works out to a 0.5 scale in both dimensions. Scaling my 1280x720 render target by half gets me... exactly 640x360, which fits snugly into the viewport, along with all the things that were drawn onto the render target.
This is how my project does resolution handling.. so far, it's worked perfectly for me. All issues with aspect ratio and scaling are handled easily and with no fuss.. all your game code draws everything as though the resolution is 1280x720(or whatever you set), without having to be concerned about the user's window size, monitor size, aspect ratio, or anything. All the minutiae is handled by the Resolution class, and as long as you keep the Resolution class updated any time a window change happens, it just works.
The only thing you have to account for now is scaling the mouse position, which you already know how to do.
Anyway.. best of luck.