ADFS and SharePoint 2013: Re-authenticating every 4 minutes

We have our token to expire after 10 hours, but every 4-5 minutes SharePoint goes out and re-authenticates against the token. Browser activity doesn't matter. How do we keep SharePoint from re-authenticating against the token? I feel like this has something to do with the FedAuth cookie, but not sure.