If your rasters are likely to have irregular or rotated boundaries, you can load the rasters into postgres using raster2pgsql, then running the sql it generates, something like this
raster2pgsql -I -C -s <SRID> <PATH/TO/RASTER FILE> <SCHEMA>.<DBTABLE> | psql -d <DATABASE>
To do this, you'll need to make sure you create your database first, and enable to postgis extension.
e.g.
raster2pgsql -I -C -s 27700 "/path/to/xyz.tif" public.test | psql -d rastertest
Here, the database is 'rastertest' and the raster is written into its own table, 'test'.
You can then use ST_ConvexHull to generate a polygon geometry of the outline. ST_MinConvexHull does the same, but excludes NODATA pixels.
select ST_AsEWKT(ST_ConvexHull(rast)) from test;
I'm not sure how the speed would compare to your proposed numpy mask solution, but it might be worth trying. I found on my laptop it generated the convex hull in 142ms for a 2000x2000, 16Mb tiff.
As you'll be doing this on lots of images, you'll probably want to script it in python (the psycopg2 library is the best way to access postgres from python)