As a solution solution to the problem I explained here I decided to write an image renderer that will run along with my program, that will allow me to query it for a specific area of the screen.
The renderer itself works fine, but it still drops the frame rate quite a lot (but not as much as LibGDX taking a screenshot). However, since this is a separate system, I can create a Thread and then queue the stuff I want rendered using a LinkedBlockingQueue. I tried it and it almost works like a charm.
The reason why I say this works almost like a charm is because this is how the frame rate graph looks:
While the program runs at around 60 fps most of the time, every .2 seconds there is a lag spike that lasts for just a few frames, and it's just long enough to be noticeable. Since this is the result using multithreading for the image rendering, and from my experience I have found multiprocessing to give better results, I would like to know if this is a scenario where multiprocessing would be a good idea, since I also know that communication between processes tends to be more troublesome, and I want to avoid reworking my entire renderer to work with multiprocessing without being sure that it will be worth it.
TL;DR: I have neural nets on one side, real time image rendering on the other. What is a better option to distribute the load, multithreading or multiprocessing?
Edit: Normally I would not care much about these lag spikes, because they are barely noticeable, but I'm also using a physics engine, and these lag spikes are drastic enough to mess with it, which is why I want to get the smoothest FPS possible.
Edit 2: The way I render the image is as follows:
I have a WorldRenderer class that implements Runnable. In my "main" function (not really the main function, but it doesn't matter) I create a single Thread and start it right away. In the run method of the WorldRenderer class, I have a while loop that gets data from a LinkedBlockingQueue and passes it to another method for it to process that data. The data contains a Map<String, Object> where I put signals (to tell the renderer what to do) and RenderQuery instances, which are a class that just contain a bunch of different shapes. Then it calls the render method. This method checks every frame if it is the n-th frame, and if it is, it renders the image by iterating through an ArrayList<RenderQuery> instance, where each shape is drawn pixel by pixel. I implemented it so that it drew a fixed amount of shapes each frame to a backbuffer and when it had no more shapes left it would push the image to the frontbuffer, but that turned out to be laggier.
This is the WorldRenderer class:
package me.kolterdyx.artificiallife.graphics.renderer;
import com.badlogic.gdx.graphics.Color;
import me.kolterdyx.artificiallife.graphics.renderer.shapes.CircleShape;
import me.kolterdyx.artificiallife.graphics.renderer.shapes.LineShape;
import me.kolterdyx.artificiallife.graphics.renderer.shapes.RectShape;
import me.kolterdyx.artificiallife.graphics.renderer.shapes.Shape;
import me.kolterdyx.artificiallife.level.entities.Entity;
import javax.imageio.ImageIO;
import java.awt.RenderingHints;
import java.awt.Graphics2D;
import java.awt.image.BufferedImage;
import java.awt.image.WritableRaster;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
public class WorldRenderer implements Runnable {
private int[][][] backBuffer, frontBuffer;
private ArrayList<RenderQuery> renderQueries;
private final int width, height;
private ArrayList<Entity> entities;
private final BlockingQueue dataQueue;
private boolean exit = false;
private int counter;
private ArrayList<RenderQuery> extraQueries;
public WorldRenderer(BlockingQueue dataQueue, ArrayList<Entity> entities, int width, int height){
this.dataQueue = dataQueue;
frontBuffer = new int[width][height][3];
backBuffer = new int[width][height][3];
renderQueries = new ArrayList<>();
extraQueries = new ArrayList<>();
this.width = width;
this.height = height;
this.entities = entities;
}
private void setEntities(ArrayList<Entity> array){
// System.out.println("Received "+entities.size()+" entities");
entities = array;
}
private void getSignal(int signal){
System.out.println("Received signal: "+signal);
switch (signal){
case 0 -> {
saveImage();
}
case -1 -> {
exit = true;
}
}
}
public int[][][] getArea(int x, int y, int width, int height){
return null;
}
public int[][][] getBuffer() {
return frontBuffer;
}
private void setPixel(int x, int y, Color color){
int[] colorArray = new int[]{(int) (color.r*255), (int) (color.g*255), (int) (color.b*255)};
backBuffer[x+width/2][y+height/2] = colorArray;
}
public void renderShape(Shape shape){
switch (shape.getType()){
case CIRCLE -> {
CircleShape circle = (CircleShape) shape;
int radius = circle.getRadius();
int ox = (int) circle.getPos().x;
int oy = (int) circle.getPos().y;
if (circle.isFilled()){
for (int x = -radius; x < radius; x++){
int height = (int)Math.sqrt(radius*radius - x*x);
for (int y = -height; y < height; y++){
setPixel(x+ox, y+oy, circle.getColor());
}
}
} else {
for (float t=0;t<Math.PI*2; t+= Math.PI*2/20f){
int x = (int) (radius*Math.cos(t));
int y = (int) (radius*Math.sin(t));
setPixel(x+ox, y+oy, circle.getColor());
}
}
}
case RECT -> {
RectShape rect = (RectShape) shape;
int ox = (int) rect.getPos().x;
int oy = (int) rect.getPos().y;
if (rect.isFilled()) {
for (int x = ox; x < rect.getSize().x+ox; x++) {
for (int y = oy; y < rect.getSize().y+oy; y++) {
setPixel(x, y, rect.getColor());
}
}
} else {
for (int x=ox; x < rect.getSize().x+ox; x++){
for (int i=0; i<rect.getThickness(); i++){
setPixel(x, oy+i, rect.getColor());
setPixel(x, (int) (oy+rect.getSize().y)-i, rect.getColor());
}
}
for (int y=oy; y < rect.getSize().y+oy; y++){
for (int i=0; i<rect.getThickness(); i++) {
setPixel(ox+i, y, rect.getColor());
setPixel((int) (ox + rect.getSize().x)-i, y, rect.getColor());
}
}
}
}
case LINE -> {
LineShape line = (LineShape) shape;
}
}
}
public void addRenderQuery(RenderQuery query){
renderQueries.add(query);
}
public void render(){
counter++;
if (renderQueries.size() == 0){
for (Entity entity : entities){
if (entity.hasRenderQuery()){
addRenderQuery(entity.getRenderQuery());
}
}
for (RenderQuery query : extraQueries){
addRenderQuery(query);
}
frontBuffer = backBuffer.clone();
backBuffer = new int[width][height][3];
}
for (int i=0; i<1;i++){
if (renderQueries.size()==0)break;
renderQuery(renderQueries.get(0));
renderQueries.remove(0);
}
}
private void renderQuery(RenderQuery query) {
for (Shape shape : query.getShapes()){
try {
renderShape(shape);
} catch (ArrayIndexOutOfBoundsException e){
e.printStackTrace();
}
}
}
public void saveImage(int[][][] imagePixels){
int height = imagePixels.length;
int width = imagePixels[0].length;
int[][] flat = new int[width*height][3];
// Flatten the image into a 2D array.
int index=0;
for(int row=0; row<height; row++) {
for(int col=0; col<width; col++) {
for(int rgb=0; rgb<3; rgb++) {
flat[index][rgb]=imagePixels[row][col][rgb];
}
index++;
}
}
int[] outPixels = new int[flat.length*flat[0].length];
for(int i=0; i < flat.length; i++){
outPixels[i*3] = flat[i][0];
outPixels[i*3+1] = flat[i][1];
outPixels[i*3+2] = flat[i][2];
}
BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
WritableRaster raster = (WritableRaster) image.getData();
raster.setPixels(0, 0, width, height, outPixels);
image.setData(raster);
File outputFile = new File("image.png");
try {
ImageIO.write(rotateImage(image, 270), "png", outputFile);
} catch (IOException e) {
e.printStackTrace();
}
}
private BufferedImage rotateImage(BufferedImage buffImage, double angle) {
double radian = Math.toRadians(angle);
double sin = Math.abs(Math.sin(radian));
double cos = Math.abs(Math.cos(radian));
int width = buffImage.getWidth();
int height = buffImage.getHeight();
int nWidth = (int) Math.floor((double) width * cos + (double) height * sin);
int nHeight = (int) Math.floor((double) height * cos + (double) width * sin);
BufferedImage rotatedImage = new BufferedImage(
nWidth, nHeight, BufferedImage.TYPE_INT_ARGB);
Graphics2D graphics = rotatedImage.createGraphics();
graphics.setRenderingHint(
RenderingHints.KEY_INTERPOLATION,
RenderingHints.VALUE_INTERPOLATION_BICUBIC);
graphics.translate((nWidth - width) / 2, (nHeight - height) / 2);
// rotation around the center point
graphics.rotate(radian, (double) (width / 2), (double) (height / 2));
graphics.drawImage(buffImage, 0, 0, null);
graphics.dispose();
return rotatedImage;
}
public void saveImage(){
System.out.println("Saving image");
saveImage(frontBuffer);
}
@Override
public void run() {
try {
while (!exit){
getData(dataQueue.take());
dataQueue.clear();
render();
}
System.out.println("stopped");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void getData(Object object) {
if (object instanceof Map<?,?> data){
if (data.containsKey("signal")){
getSignal((int) data.get("signal"));
}
if (data.containsKey("entities")){
setEntities((ArrayList<Entity>) data.get("entities"));
}
if (data.containsKey("extraQueries")){
extraQueries = (ArrayList<RenderQuery>) data.get("extraQueries");
}
}
}
}
Edit 4: Removed Edit 3 because it didn't actually solve the problem.