Im developing a deep texture processing engine. Right now I have a software renderer to generate the raster samples. Now, for bigger renders it would be nice to do this on hardware.
Is there a standard mechanism on how to capture the data of each shader evaluation in OpenGL? I could do some sort of depth peeling but it seems to me as a bit too brute forcelike approach. I don't mind the time cost just as long as it does not take ages to implement the infrastructure needed.
If you really want to obtain every fragment (shading sample) within the camera frustum, regardless of occlusion, one way to do it would be to disable depth testing, and then use a large SSBO together with an atomic counter to append data into the buffer from the fragment shader. Each fragment could then store a record of its screen position, depth, color, and any other desired information into the buffer to be read out later. The buffer has to be large enough to store every fragment rendered, as it will throw away any extra data once it's full.
Using one huge SSBO for the whole screen likely isn't ideal, as there will be huge contention on its counter with every fragment trying to increment it. A better way might be to slice up the screen in tiles with an SSBO+counter per tile. There is a limit of 16 bound SSBOs on all current GPUs, and on AMD GPUs, there is a further limit of 8 atomic counters (source: Sascha Willem's OpenGL Hardware Database). Those limits will determine how many tiles you can create. This will reduce contention and also ensure that even if you run out of memory in one tile, you'll still have valid data in the others.
