on a modern site (team site) currently the recommended way to add JavaScript to the page would be using SPFx client app.
otherwise to modifiy a theme or create one and applying it to a modern site would require powershell.
more could be found here:
https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/modern-experience-customizations-customize-sites
a few things you cant do
Custom master pages; more extensive branding will be supported later
using alternative options.
Changing "modern" site to use "classic" seattle.master or oslo.master.
Custom page layouts; we are looking to have support for multiple
canvases in the future.
Enabling site or site collection scoped publishing features;
technically, features can be currently activated, but this is not a
supported configuration. User custom actions / custom JavaScript;
there will be a more controlled way to embed JavaScript on the pages
through the SharePoint Framework Extensions (currently in dev
preview).
"Modern" subsites; subsites created on "modern" team sites use the
"classic" experience, but you can change the user experience to be
similar to "modern" sites.
Ability to control available subsite template options. "Classic"
publishing features (WCM).
Activation of community feature or creation of community subsites
under "modern" team site.
so adding a ref in the masterpage is not possible.