Is it possible to restrict roles and resource_access in tokens?

The token I was able to obtain includes all roles and authorizations the user has. However the client application does not need all but specific ones. Depending on the number of applications existing in the organization, this increases the token unnecessarily.
Is there a means to include just the authorizations the specific client application needs?

Check out he docs for role mappings in the token: Server Administration Guide

There’s a lot of things you can do to the token using scopes and mappers.