How to get world coordinates from clicking screen in Three.js using displacement maps?

Question

I'm trying to get world coordinates from clicking the screen, but using displacement maps. This makes it more complex as the floor is variable heights (hills). My approach so far is to calculate the point on the floor (assuming the floor was all flat at z=0), then ray trace from the camera to that point, and at each iteration see where the height of the ray becomes <= the real height of the floor at that point. Once its smaller or equal to, we know it hit the floor and have the (x,y,z). Points outside the map have z=0 too.

This is the code:

// https://stackoverflow.com/questions/13055214/mouse-canvas-x-y-to-three-js-world-x-y-z
public static Get3DPoints(camera:THREE.PerspectiveCamera, click_position:THREE.Vector2, targetZ:number) {
    var vec = new THREE.Vector3(); // create once and reuse
    var pos = new THREE.Vector3(); // create once and reuse
    vec.set(
        ( click_position.x / window.innerWidth ) * 2 - 1,
        - ( click_position.y / window.innerHeight ) * 2 + 1,
        0.5 );
    vec.unproject( camera );
    vec.sub( camera.position ).normalize();
    var distance = ( targetZ - camera.position.z ) / vec.z;
    pos.copy( camera.position ).add( vec.multiplyScalar( distance ) );
    return pos;
}

public static async Get3DCoordinatesFromClick(GAME_CONTEXT:Game_Context, camera:THREE.PerspectiveCamera, click_position:THREE.Vector2):Promise<THREE.Vector3> {
    return new Promise((resolve, reject) => {
        var start = camera.position.clone();
        var collision_point:THREE.Vector3;
        var point3d = this.Get3DPoints(camera, click_position, 0);
        //console.log(click_position, " fp --->", point3d);
        new TWEEN.Tween(start).to({
            x:point3d.x,
            y:point3d.y,
            z:point3d.z
        }, 60).onUpdate(vector => {
            if (vector.x < 0 || vector.y < 0 || vector.x > GAME_CONTEXT.width || vector.y > GAME_CONTEXT.height) {
                var z = 0;
            } else {
                var z = GAME_CONTEXT.heights_map.GetValue(vector.x/GAME_CONTEXT.width, vector.y/GAME_CONTEXT.width);
            }
            if (!collision_point && vector.z <= z) {
                collision_point = vector.clone();
            }
        }).onComplete(vector => {
            //console.log(click_position, "--->", collision_point || vector);
            resolve(collision_point || vector);
        }).start();
    });
}

This generally works and gives me fairly accurate measurements, perhaps off by 1-4 real world units. And maybe I could improve it by finding the point of the ray right when its z equals the floor height.

But the main problem is I'm getting inconsistent values each time I click. I'm clicking in the exact same spot on the screen but the (x,y,z) coordinate changes by a little each time. It's not giving me the same value.

I'm suspecting this has to do with Tweening and putting a fixed 60millisecond duration for the ray travel.

Anyone know how to fix this?