RealityKit – Visualizing Grid on detected planes

Question

I want to draw the mesh point on the detected plane as shown in the video link and I don't know how to achieve it.

Can you help me out to achieve this?

Thanks in advance.

Answer 1

In ARKit/RealityKit, in order to visualize the plane detection process similarly to the behavior you can see in ARCore, you should use the Scene Reconstruction feature – for iOS devices with a LiDAR scanner – however, this is a completely different story (Metal story), because you have to develop a procedural texture with soft mask for front edge.

In this post I want to show you how to use the "classic" visualization of plane detection process using ARKit/RealityKit. But in this case, I do not promise to show this process as visually pleasing as it's implemented in ARCore.

Here's a code:

import ARKit
import RealityKit

class Grid: Entity, HasModel, HasAnchoring {        
    var planeAnchor: ARPlaneAnchor
    var planeGeometry: MeshResource!
    
    init(planeAnchor: ARPlaneAnchor) {
        self.planeAnchor = planeAnchor
        super.init()
        self.didSetup()
    }
        
    fileprivate func didSetup() {            
        self.planeGeometry = .generatePlane(width: planeAnchor.extent.x,
                                            depth: planeAnchor.extent.z)
        var material = UnlitMaterial()
        material.color = .init(tint: .white.withAlphaComponent(0.999),
                            texture: .init(try! .load(named: "grid.png")))
        let model = ModelEntity(mesh: planeGeometry, materials: [material])
        model.position = [planeAnchor.center.x, 0, planeAnchor.center.z]
        self.addChild(model)
    }
    
    fileprivate func didUpdate(anchor: ARPlaneAnchor) {            
        self.planeGeometry = .generatePlane(width: anchor.extent.x,
                                            depth: anchor.extent.z)
        let pose: SIMD3<Float> = [anchor.center.x, 0, anchor.center.z]
        let model = self.children[0] as! ModelEntity
        model.position = pose
    }        
    required init() { fatalError("Hasn't been implemented yet") }
}

ViewController.swift

class ViewController: UIViewController {
    
    @IBOutlet var arView: ARView!
    var grids = [Grid]()

    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        arView.session.delegate = self
        
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal, .vertical]
        arView.session.run(config)
    }
}

ARSessionDelegate

extension ViewController: ARSessionDelegate {
    
    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        
        guard let planeAnchor = anchors.first as? ARPlaneAnchor else { return }
        let grid = Grid(planeAnchor: planeAnchor)
        grid.transform.matrix = planeAnchor.transform
        self.arView.scene.anchors.append(grid)
        self.grids.append(grid)
    }
    
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        
        guard let planeAnchor = anchors[0] as? ARPlaneAnchor else { return }
        let grid: Grid? = grids.filter { grd in
            grd.planeAnchor.identifier == planeAnchor.identifier }[0]
        guard let updatedGrid: Grid = grid else { return }
        updatedGrid.transform.matrix = planeAnchor.transform
        updatedGrid.didUpdate(anchor: planeAnchor)
    }
}

Only coplanar detected planes may be updated.