Goal

Find the dimensions (position and size) of each hole in a plate mounted in the vise of a milling machine.

Summary

Use a camera and machine vision to identity holes and drive LinuxCNC to probe each hole with a Haimer 3D Sensor. A second camera reads the mechanical gauge of the Haimer 3D Sensor, emulating an electronic probe. This setup could be used for reverse engineering a part or verifying machining operations. Software is written in Python, uses OpenCV, and is available on GitHub. A video demonstration is available.

Justification

It’s common to have a part without drawings that needs modifications, duplication, or to which a new part must be mounted. Measurements must be taken of the key features and the part recreated in CAD. Measurements can be done with hand tools, and symmetries in the part may make measurement quicker. Automation should make the process quicker and less error prone.

Method

Automation hole dimensions is implemented with two cameras and software driving probing operations of a CNC milling machine. To test the method, holes with known dimensions are added to a plate and compared to the measurements.

A camera attached to the milling machine and stares at the plate along the z-axis. After the upper-left corner and top of the plate are touched off, camera alignment can be corrected, including rotation and squareness. The plate is assumed to be rectangular, and corners of the plate are selected in the image. The selected points are used to warp the image and correct image distortions related to camera alignment. After warping, the dimensions (width and height) of the plate are entered. The software then estimates position and diameter each hole on the plate.

Warping of the image to correct camera alignment is implemented with an affine transformation is meant to improve estimations of a reasonably mounted camera. While a surprising amount of misalignment can be tolerated, it’s recommended to make a reasonable attempt at squaring the camera. If the camera could be made perfect, not requiring the warping, this step could be eliminated. However, having warping step allows the camera to be quickly mounted and removed, attached to the milling machine head using magnets. Related to the warping, by entering the plate dimensions, there’s no need to know the height of the camera. Entering the plate dimensions, with the plate extents known, is enough to convert pixel dimensions into millimeters. Once the corners of the plate are selected, the image of the plate is locked, and no longer updated. Locking the image improves system responsiveness by not processing the video stream. During probing much of the plate will be obscured.

A reference could be used to estimate distances. To improve the estimate, the opposite sides of the plate could be probed. These steps could eliminate the need to enter the plate dimensions, and possibly eliminate the need to select the plate corners. Simple references could be gauge blocks, a couple circles with known diameters and center distance, or a ruler of known length.

A list of approximate position and size of each hole is now known. A probe can measure each hole, improving the accuracy of the estimates. A Haimer 3D Sensor probe is used to measure the position and diameter of each hole. A second camera is attached to the probe so the software can read the mechanical face. How this is done is described in an earlier article. The probe is otherwise used normally, with the probe mounted in the spindle of the mill. LinuxCNC is the controller and software, the LinuxCNC driver, sends commands to LinuxCNC to move the probe. The total traveling distance of the probe is minimized by solving the travel-salesperson problem. The currently optimal path is shown and updated as the probe moves. Before automating the probing operation, the path is locked to improve responsiveness.

From LinuxCNC, driver software can read machine coordinates, active coordinate system, and offsets of each coordinate system. The driver sends MDI commands to LinuxCNC with G-code. It’s convenient in the driver software to send G54 positions, and otherwise operate in machine coordinates. Measurements come from the difference in positions and movements are mostly incremental, so coordinate system selection is only for convenience.

The upper-left corner of the plate is the G54 origin. The software expects to lower the probe tip into any hole without the probe reading changing. Z-axis origin is the plate’s top and the probe tip is 4mm in diameter. So, the software expects to lower the probe tip to -2mm (half the probe tip diameter) without the probe reading changing. Holes that are too small to fit the probe into the hole safely should be filtered. Minimal hole diameter is proportional to probe tip diameter and expected error in the image analysis estimates. If an obstruction is encountered while entering a hole, a warning should be recorded and the hole filtered. While probing, dimensions of each hole are written to a CSV file, including estimated position and size, probed position and size (if possible), and warnings.

Hole probing is similar to how one would touch off to find the center of a hole manually. The steps are:

  1. The probe lowers into hole at the estimated center.
  2. Probing of the right edge of the hole begins, and once found the position is recorded.
  3. Rapid move back to the estimated center.
  4. Probing of the right edge of the hole begins, and once found the position is recorded. The X-center is now known as the halfway point between the two probe edges.
  5. Rapid move back to the estimated Y- and known X-center.
  6. Probing of the bottom edge of the hole begins, and once found the position is recorded.
  7. Rapid move back to the estimated Y- and known X-center.
  8. Probing of the top edge of the hole begins, and once found the position is recorded. The Y-center is now known as the halfway point between the two probe edges.
  9. Rapid move back to the measured hole center, and raise the probe out of the hole.

The probing operations could optimized by not requiring the Haimer 3D Sensor to read 0. As edges of the hole are probed more information is known about the hole. Initially, while the probe tip is moving, little is known about the hole or the edge being probed, other than “the edge is still somewhere over there.” Only when an edge is located is useful information available. However, it’s not necessary for the software to drive the probe until the probe face reads 0. Once a measurement is shown on the face, the remaining distance is known, and the probing operation could be aborted. The software currently probes until the probe reads 0 to emulate what a person might do to aid demonstration.

Measured hole diameter comes from probing along the Y-axis. The initial probing along the X-axis will most likely be smaller than the diameter because the estimate center is unlikely to be the actual center. Probing a second time along the X-axis might be useful for verifying the diameter. Probing additional points around the hole could be used to verify the roundness of the hole, related to the ability of the machining operation to create the hole.

An inefficient way to probe the hole would use 3d probing following a grid pattern. The density of probing must be high enough to capture all the holes. Once a hole is found, with the probe able to descend below the plate top, an operation similar to the one listed above could find the dimensions. This method would require less assistance from the operator and no overall information, such as what the camera provides, but will be slow.

Next steps

  • Identification of slots (center of two holes, diameter or width of slow), rectangular holes (corner 1 and corner 2), and ellipse (center, minor axis, major axis). Initially, one axis would be aligned with a cardinal axis. Later, relax this and allow arbitrary major axes.
  • Combine the information from the approach described here with a ToF camera. The ToF camera would verify a hole has depth, while the web camera provides better estimation of position and diameter.
  • Measure hole depth. (Maybe the same video as the non-circular holes)
  • As approximate holes are measured, update the view of th e approximate holes with actual measurements, changing colors from green to red or something similar.
  • For the final image, capture a still instead of video frame. The Microsoft Cinema Lifecam delivers still with 2880 x 1620 resolution while video frames have 1280 x 720 resolution. Video frames are delivered faster and so reflects the most up to date view, but video is noisier. Averaging consecutive video frames would help with noise, but blur the image. Since the noise is mostly only noticeable when zooming, and the monitor resolution only accommodates the lower resolution, video is best for live views. The approximation of position and dimensions seem good enough already. Accommodating both video and still would require scaling of the blob detector parameters if the blob area filter is enabled.
  • Milling marks can cause false positive, and shadowing can cause false negatives during hole detection. The plate is likely to be a uniform color and equalizing the image across the region defined by the alignment points might reduce hole detection errors.
  • Probe only until a reading is shown on the face of the Haimer and use the measurement to calculate distance of the edge, instead of continuing to move until the probe face reads 0.

References