QGIS is using gdal_polygonoze https://www.gdal.org/gdal_polygonize.html for creating the polygons. It does not have on option for selecting a region of the input raster to be processed. It means that you must make a selection as a separate step. I would recommend to use gdal_translate https://www.gdal.org/gdal_translate.html and virtual raster output https://www.gdal.org/gdal_vrttut.html for that.
You seem to have an image with 40000 x 40000 pixels. If you want to have a try with a 10000 x 10000 image clipped from the top-left corner use this command:
gdal_translate -of VRT -srcwin 0 0 10000 10000 big_input.tif small_test.vrt
Once you have found the proper size for gdal_polygonize you will only need to slide the window with x and y offsets
-srcwin xoff yoff xsize ysize:
Selects a subwindow from the source image for copying based on pixel/line location.
VRT is just an instruction about how to read the original physical file. For such a big image it is also essential that the source image is tiled. Check the situation with gdalinfo and if the image is not tiled already, make a physical geotiff copy with gdal_translate first.
gdal_translate -of GTiff -co TILED=YES input.tif tiled_output.tif
You may also want to create a compressed output, see the options from https://www.gdal.org/frmt_gtiff.html.
rasteriooffers quite some possibilities in terms of windowing into rasters and processing them window by window. Check https://github.com/mapbox/rasterio/blob/master/docs/topics/windowed-rw.rst ... But this begs the question are you OK with using python? – user32882 Oct 27 '18 at 18:00