0

As a solution solution to the problem I explained here I decided to write an image renderer that will run along with my program, that will allow me to query it for a specific area of the screen.

The renderer itself works fine, but it still drops the frame rate quite a lot (but not as much as LibGDX taking a screenshot). However, since this is a separate system, I can create a Thread and then queue the stuff I want rendered using a LinkedBlockingQueue. I tried it and it almost works like a charm.

The reason why I say this works almost like a charm is because this is how the frame rate graph looks:

1f/delta at each individual frame

While the program runs at around 60 fps most of the time, every .2 seconds there is a lag spike that lasts for just a few frames, and it's just long enough to be noticeable. Since this is the result using multithreading for the image rendering, and from my experience I have found multiprocessing to give better results, I would like to know if this is a scenario where multiprocessing would be a good idea, since I also know that communication between processes tends to be more troublesome, and I want to avoid reworking my entire renderer to work with multiprocessing without being sure that it will be worth it.

TL;DR: I have neural nets on one side, real time image rendering on the other. What is a better option to distribute the load, multithreading or multiprocessing?

Edit: Normally I would not care much about these lag spikes, because they are barely noticeable, but I'm also using a physics engine, and these lag spikes are drastic enough to mess with it, which is why I want to get the smoothest FPS possible.

Edit 2: The way I render the image is as follows: I have a WorldRenderer class that implements Runnable. In my "main" function (not really the main function, but it doesn't matter) I create a single Thread and start it right away. In the run method of the WorldRenderer class, I have a while loop that gets data from a LinkedBlockingQueue and passes it to another method for it to process that data. The data contains a Map<String, Object> where I put signals (to tell the renderer what to do) and RenderQuery instances, which are a class that just contain a bunch of different shapes. Then it calls the render method. This method checks every frame if it is the n-th frame, and if it is, it renders the image by iterating through an ArrayList<RenderQuery> instance, where each shape is drawn pixel by pixel. I implemented it so that it drew a fixed amount of shapes each frame to a backbuffer and when it had no more shapes left it would push the image to the frontbuffer, but that turned out to be laggier.

This is the WorldRenderer class:

package me.kolterdyx.artificiallife.graphics.renderer;

import com.badlogic.gdx.graphics.Color;
import me.kolterdyx.artificiallife.graphics.renderer.shapes.CircleShape;
import me.kolterdyx.artificiallife.graphics.renderer.shapes.LineShape;
import me.kolterdyx.artificiallife.graphics.renderer.shapes.RectShape;
import me.kolterdyx.artificiallife.graphics.renderer.shapes.Shape;
import me.kolterdyx.artificiallife.level.entities.Entity;

import javax.imageio.ImageIO;
import java.awt.RenderingHints;
import java.awt.Graphics2D;
import java.awt.image.BufferedImage;
import java.awt.image.WritableRaster;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Map;
import java.util.concurrent.BlockingQueue;


public class WorldRenderer implements Runnable {

    private int[][][] backBuffer, frontBuffer;

    private ArrayList<RenderQuery> renderQueries;

    private final int width, height;

    private ArrayList<Entity> entities;
    private final BlockingQueue dataQueue;
    private boolean exit = false;

    private int counter;
    private ArrayList<RenderQuery> extraQueries;

    public WorldRenderer(BlockingQueue dataQueue, ArrayList<Entity> entities, int width, int height){
        this.dataQueue = dataQueue;
        frontBuffer = new int[width][height][3];
        backBuffer = new int[width][height][3];
        renderQueries = new ArrayList<>();
        extraQueries = new ArrayList<>();
        this.width = width;
        this.height = height;
        this.entities = entities;
    }

    private void setEntities(ArrayList<Entity> array){
//        System.out.println("Received "+entities.size()+" entities");
        entities = array;
    }

    private void getSignal(int signal){
        System.out.println("Received signal: "+signal);
        switch (signal){
            case 0 -> {
                saveImage();
            }
            case -1 -> {
                exit = true;
            }
        }
    }

    public int[][][] getArea(int x, int y, int width, int height){
        return null;
    }

    public int[][][] getBuffer() {
        return frontBuffer;
    }

    private void setPixel(int x, int y, Color color){
        int[] colorArray = new int[]{(int) (color.r*255), (int) (color.g*255), (int) (color.b*255)};
        backBuffer[x+width/2][y+height/2] = colorArray;
    }

    public void renderShape(Shape shape){
        switch (shape.getType()){
            case CIRCLE -> {
                CircleShape circle = (CircleShape) shape;
                int radius = circle.getRadius();
                int ox = (int) circle.getPos().x;
                int oy = (int) circle.getPos().y;
                if (circle.isFilled()){
                    for (int x = -radius; x < radius; x++){
                        int height = (int)Math.sqrt(radius*radius - x*x);
                        for (int y = -height; y < height; y++){
                            setPixel(x+ox, y+oy, circle.getColor());
                        }
                    }
                } else {
                    for (float t=0;t<Math.PI*2; t+= Math.PI*2/20f){
                        int x = (int) (radius*Math.cos(t));
                        int y = (int) (radius*Math.sin(t));
                        setPixel(x+ox, y+oy, circle.getColor());
                    }
                }
            }
            case RECT -> {
                RectShape rect = (RectShape) shape;
                int ox = (int) rect.getPos().x;
                int oy = (int) rect.getPos().y;

                if (rect.isFilled()) {
                    for (int x = ox; x < rect.getSize().x+ox; x++) {
                        for (int y = oy; y < rect.getSize().y+oy; y++) {
                            setPixel(x, y, rect.getColor());
                        }
                    }
                } else {
                    for (int x=ox; x < rect.getSize().x+ox; x++){
                        for (int i=0; i<rect.getThickness(); i++){
                            setPixel(x, oy+i, rect.getColor());
                            setPixel(x, (int) (oy+rect.getSize().y)-i, rect.getColor());
                        }
                    }
                    for (int y=oy; y < rect.getSize().y+oy; y++){
                        for (int i=0; i<rect.getThickness(); i++) {
                            setPixel(ox+i, y, rect.getColor());
                            setPixel((int) (ox + rect.getSize().x)-i, y, rect.getColor());
                        }
                    }
                }
            }
            case LINE -> {
                LineShape line = (LineShape) shape;
            }
        }
    }

    public void addRenderQuery(RenderQuery query){
        renderQueries.add(query);
    }

    public void render(){
        counter++;
        if (renderQueries.size() == 0){
            for (Entity entity : entities){
                if (entity.hasRenderQuery()){
                    addRenderQuery(entity.getRenderQuery());
                }
            }
            for (RenderQuery query : extraQueries){
                addRenderQuery(query);
            }
            frontBuffer = backBuffer.clone();
            backBuffer = new int[width][height][3];
        }

        for (int i=0; i<1;i++){
            if (renderQueries.size()==0)break;
            renderQuery(renderQueries.get(0));
            renderQueries.remove(0);
        }

    }

    private void renderQuery(RenderQuery query) {
        for (Shape shape : query.getShapes()){
            try {
                renderShape(shape);
            } catch (ArrayIndexOutOfBoundsException e){
                e.printStackTrace();
            }
        }
    }

    public void saveImage(int[][][] imagePixels){
        int height = imagePixels.length;
        int width = imagePixels[0].length;
        int[][] flat = new int[width*height][3];


        // Flatten the image into a 2D array.
        int index=0;
        for(int row=0; row<height; row++) {
            for(int col=0; col<width; col++) {
                for(int rgb=0; rgb<3; rgb++) {
                    flat[index][rgb]=imagePixels[row][col][rgb];
                }
                index++;
            }
        }


        int[] outPixels = new int[flat.length*flat[0].length];
        for(int i=0; i < flat.length; i++){
            outPixels[i*3] = flat[i][0];
            outPixels[i*3+1] = flat[i][1];
            outPixels[i*3+2] = flat[i][2];
        }

        BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
        WritableRaster raster = (WritableRaster) image.getData();
        raster.setPixels(0, 0, width, height, outPixels);
        image.setData(raster);

        File outputFile = new File("image.png");
        try {
            ImageIO.write(rotateImage(image, 270), "png", outputFile);
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

    private BufferedImage rotateImage(BufferedImage buffImage, double angle) {
        double radian = Math.toRadians(angle);
        double sin = Math.abs(Math.sin(radian));
        double cos = Math.abs(Math.cos(radian));

        int width = buffImage.getWidth();
        int height = buffImage.getHeight();

        int nWidth = (int) Math.floor((double) width * cos + (double) height * sin);
        int nHeight = (int) Math.floor((double) height * cos + (double) width * sin);

        BufferedImage rotatedImage = new BufferedImage(
                nWidth, nHeight, BufferedImage.TYPE_INT_ARGB);

        Graphics2D graphics = rotatedImage.createGraphics();

        graphics.setRenderingHint(
                RenderingHints.KEY_INTERPOLATION,
                RenderingHints.VALUE_INTERPOLATION_BICUBIC);

        graphics.translate((nWidth - width) / 2, (nHeight - height) / 2);
        // rotation around the center point
        graphics.rotate(radian, (double) (width / 2), (double) (height / 2));
        graphics.drawImage(buffImage, 0, 0, null);
        graphics.dispose();

        return rotatedImage;
    }

    public void saveImage(){
        System.out.println("Saving image");
        saveImage(frontBuffer);
    }

    @Override
    public void run() {
        try {
            while (!exit){

                getData(dataQueue.take());
                dataQueue.clear();

                render();
            }
            System.out.println("stopped");
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    private void getData(Object object) {
        if (object instanceof Map<?,?> data){
            if (data.containsKey("signal")){
                getSignal((int) data.get("signal"));
            }
            if (data.containsKey("entities")){
                setEntities((ArrayList<Entity>) data.get("entities"));
            }
            if (data.containsKey("extraQueries")){
                extraQueries = (ArrayList<RenderQuery>) data.get("extraQueries");
            }
        }
    }
}

Edit 4: Removed Edit 3 because it didn't actually solve the problem.

Ciro García
  • 434
  • 2
  • 18
  • Before we can find a solution, we need to understand *why* the problem is happening so that our proposed solution isn't a shot in the dark. How are you breaking up the work to queue it? Are the spikes possibly the result of one worker spinning down and a new one spinning up? – John Glenn Feb 01 '22 at 02:48
  • @JohnGlenn I added information about the `WorldRenderer` class to my post. I only used one Thread since I don't really know how I could improve the rendering with multiple threads – Ciro García Feb 01 '22 at 03:06

0 Answers0