Sign In button doesn't redirect them to Google, then back to the site, as would have happened if they'd been using the basic OAuth 2.0 flows.
The HTML 5 web message specification standardised a solution to this problem, in the form of the window.postMessage() method. As you might guess, this lets you send a message from one window to another in a pretty straightforward way, even if they're from different origins. E.g.:
The second argument is the target origin - if the loaded window is actually from a different place, then the message will not be sent (so you don't accidentally send messages to a rogue window). Within the window, we can listen for a callback by adding an event listener.
var cw = document.getElementById("myiframe").contentWindow;
cw.postMessage("Hello World, "http://examplea.com");
Here we can check the source origin to make sure it came from where we expected. This handy little API is supported basically everywhere, including IE8+.
if(e.origin == "http://exampleb.com")
So what does this mean for sign-in? In the normal OAuth 2.0 (client side) flow
1. The application generated a URL, and redirects the user to the provider
2. The provider shows them a consent screen or similar to approve the application
3. Once the user has submitted the consent form, they are redirected back to the site with an access token that can be used to access resources
The server side flow is broadly similar, except a shorter-lived code is sent which the server can exchange for an access token. Using postMessage, we can make this experience a bit easier.
1. The sign in button can create a hidden iframe as a post message relay
2. This can pop up a window as needed for consent and sign in, with a redirect-uri of "postmessage" rather than another site
3. Once the user has submitted the consent form, the window sends back an access token via postMessage to the consuming code
This saves roundtrips for the user, and potentially makes the experience faster and smoother. Both sides can check the origin is what they expect (so registering the origin in console of the provider is still required). As it never puts an access token in the URL, its likely more secure as well.
It also allows some (potentially) useful functionality such as checking the session state without necessarily having to round-trip to a server.
So, wins! What's the downside?
2. Using this flow you can still request offline access which will return an access token for use in the client, and a short-lived code for the server*. However, as you're not generating the auth URL, you need to ensure you send a one-time code out to the client and back with the code, and check it matches. This is basically what you would have been using as the 'state' parameter in the old flow.
If you want to have a look at how this looks in practice, check out https://code.google.com/p/oauth2-postmessage-profile - this application actually uses the shindig project's gadgets.rpc for the inter-frame communication, but that uses postMessage underneath. This kind of functionality is also explicitly called out in the OpenID connect spec, specifically the bit on session management.
* The server can exchange this for both an access token and a longer lived refresh token, which it can exchange for access tokens as needed.