Blind Precise Assembly: Proprioceptive Sim-to-Real Peg-in-Hole for Diverse Parts

Blind precise assembly. Policies solve precise manipulation tasks using only proprioception—joint positions and previous actions—with no vision, no tactile feedback, no force or torque sensing, and no knowledge of part geometries or poses. Trained entirely in simulation, the peg-insertion policy transfers to real hardware and generalizes to unseen peg geometries; gear meshing and nut threading are shown in simulation. All three tasks execute in real time, demonstrating that rich sensory feedback isn't necessary for contact-rich assembly when policies learn to interpret their own motion.

Abstract

Every robot already carries high-resolution contact sensors: its own joints. Legged robots and humanoids exploit them to walk, run, and track whole-body motions without vision, but there the contacts are large and forceful. Is joint sensing also enough for precise, contact-rich assembly, where the robot must find a millimeter-scale opening, mesh gear teeth, or catch a thread through light contact? We study this question by training blind assembly policies that observe only a short history of joint positions, velocities, and their own actions, without visual, force, torque, or tactile sensing. The poses of both the fixture and the part are unknown to the policy, and the grasp is randomized substantially. The policy must therefore assemble the parts through contact, sensed exclusively through its own proprioception. Trained with reinforcement learning entirely in simulation over randomized parts, grasps, and contacts, such policies learn peg insertion, gear meshing, and nut threading. The peg insertion policy transfers zero-shot to the real world and inserts pegs of different shapes and sizes. Controlled comparisons with matched visual policies in simulation show that although blind policies are slower, they generalize far better to unseen part shapes and sizes. Training such policies needs thousands of different assemblies, each simulated with its clearance intact, so we also release an open-source generator that builds them procedurally from exact convex pieces for MuJoCo; it is more accurate and faster than convex decomposition or MuJoCo's signed-distance collision.

Precise assembly, felt through the joints

In all three tasks, a 7-DoF arm with a parallel-jaw gripper holds a part and must assemble it with a fixture that can sit almost anywhere in its workspace. The part's geometry, its pose in the grasp and the contact properties are randomized, but not observed by the policy. Episodes start with the part touching the fixture but misaligned, and succeed once it is fully assembled.

Peg insertion, gear meshing, and nut threading. (a–c) Blind policies run in your browser on a fresh random training start per episode. (d–f) Parts sampled from the three training distributions.

Probe the policies

During training, a decoder predicts task-relevant contact classes—whether the part is free, touching the opening, or inside—from the policy's latent representation. This supervision signal, labeled in simulation, helps the encoder learn to extract contact information from proprioception alone. Below, the decoder's predictions reveal what the policy infers from its history of joint positions and velocities. Each episode runs in your browser on MuJoCo (WebAssembly) at the training speeds: 1 kHz physics, 50 Hz policy.

Peg insertion, gear meshing and nut threading from proprioception alone. Each policy reads a 30-frame (0.6 s) history of joint positions and velocities, the gripper pose and twist computed from them, and its own previous actions, and moves the seven joint targets. The heatmap on the right is that input, frame by frame. A training-only decoder predicts the contact class from the policy's history latent, shown as colored dots on the part and in the ring diagram.

Simulating tight insertion at scale with mjfit

Training for this robustness needs thousands of different assemblies simulated in parallel, each with its clearance intact. MuJoCo collides meshes through their convex hulls, so a concave hole must be given a signed-distance field (SDF) or built from convex pieces, which simulation benchmarks assemble from boxes by hand.

mjfit automates the hand-built hole and generalizes it to any planar outline. The plate around the hole is split into convex polygons and extruded; the peg is the hole outline offset inward by the clearance, so every peg edge sits exactly the clearance from its hole edge. Curves are tessellated by a circumscribed polygon, so the wall never intrudes into the true hole; the only error is a little extra play at the corners. Companion generators do the same for threads and gears, and a sampler draws random outlines, islands and dimensions.

Assets generated by mjfit, simulated as emitted and drawn from their collision geometry, one color per convex piece. Hover a part for its exact values; click it to turn it in 3D and pull its pieces apart.

Does it fit?

The automatic alternatives, convex decomposition (CoACD, V-HACD) and MuJoCo's mesh SDF, tolerate errors far larger than an assembly's clearance. As a result, off-the-shelf alternatives jam at looser tolerances than mjfit.

mjfit vs. baseline collision approaches on peg insertion. A centered, aligned peg released 1 mm above each method's plate. With mjfit the peg falls through at every clearance. Convex decomposition leaves wall material inside the hole, so the peg stops on the rim until the clearance grows past that intrusion; the mesh SDF rounds the rim and catches the peg at the tightest fit.

Faster and more accurate

Against analytic ground truth, mjfit is more accurate than every baseline at every speed, on the CPU and on the GPU with MuJoCo Warp. Each assembly is only a few dozen convex pieces, so every one of our 12,288 parallel training worlds can hold a different one.

Speed against accuracy for a 16 mm round peg with 100 µm clearance, (a) on the CPU, one environment on one thread, and (b) on the GPU, per environment step across 256 MuJoCo Warp worlds, with the error also measured on Warp. Each line is one method's Pareto front over all its settings (faded: dominated settings); lower left is better, and hollow markers fill the clearance. Hover a point for its setting and to find the same setting on the other hardware.

Learning to feel the fit

At each step the actor observes its joint positions and velocities, the gripper pose and twist computed from them by forward kinematics, and its previous action. It never receives the hole pose, the part geometry, a force estimate or an image. A temporal convolution compresses the last 0.6 s of observations into a latent, and an MLP maps the latent and the current observation to joint-position increments, tracked by a joint impedance controller on an arm model identified from real-world measurements.

Policy architecture. A temporal convolution compresses the last 0.6 s of proprioception (joint positions and velocities, gripper pose and twist, previous action) into a latent z_t, from which an MLP sets joint targets for a joint impedance controller. In training, a contact decoder (amber) guesses from z_t alone which points on the peg's bottom edge are free, touching, or inside the opening; its loss against the simulator's labels trains the encoder, but the guess never reaches the policy and is dropped at deployment.

We train with PPO and a sparse reward. Two training-only signals speed up training without giving the actor anything privileged: an asymmetric critic that also sees the part's pose relative to the fixture, and the contact supervision above, which gives the encoder a dense signal about what the arm is touching.

Blind policies learn all three tasks

Proprioception alone is enough to learn all three tasks. Blind policies, which sense only their own joints, reach the same near-perfect training success as visual policies with a wrist camera.

Properties of visually blind behaviors

On peg insertion, we compare the blind policy with a visual policy trained on the same task from a wrist camera, and with a scripted spiral search as a geometric baseline.

1

Blind, but not open-loop

The blind policy succeeds at least as reliably as the visual one in distribution but takes longer, and it travels far less than a scripted spiral: proprioceptive feedback guides an efficient search.

Proprioception guides an efficient search without vision. From the same start, the blind policy slides across the surface until it finds the hole, so it seats later than the visual policy (a, b); yet over all matched starts it travels far less than an optimistic scripted spiral search (c). (a) replays each policy's recorded peg in 3D (its lighter band is the depth that goes in); Top looks down on the plate, where the path of each peg center shows through, and dragging any view turns all three. Press play, drag the time bar or hover over (b) to step through all three searches together; along the top of (c), this start's travel runs out against the full distributions. Another start replays 15 more matched starts drawn at random. The spiral is schematic, level and aligned from its start, and replayed at the blind policy's speed.

2

Visually blind, yet intelligent

The policy's history latent encodes whether the peg is lifted, searching, partly engaged or inside, in simulation and in the real world, and its actions depend on it: it acts on whether it has found the hole, not on where the hole is.

The policy's joint history encodes the insertion mode, in simulation and in the real world. A linear probe fitted only in simulation reads the mode from the policy's history latent. (a) It cleanly separates modes that call for different actions (lifted, moving, stuck), but not near from far search; hover a cell or a mode for details, or click a cell to find it in (b). (b) In the real world it decodes the same sequence of modes as in simulation: play or drag along the timeline, or pick any of the 30 real-world trials on peg I3 from the strip above. Camera frames are for illustration only; the policy never sees them.

3

Proprioception generalizes better

A camera sees a part's size and shape directly, so the visual policy learns what the training parts look like. Contact changes far less: the rim of a larger or narrower hole resists a push just as a training hole's does. On unseen sizes, proportions and cross-sections, the blind policy degrades gently while vision falls steeply.

Proprioceptive policies remain more robust to geometric variation. Success of the blind and visual policies (both with a 30-step history) on each task, in distribution (left of the dashed line) and on unseen parts that each widen one property beyond its training range while the others stay within theirs. Peg bars average two training seeds, whose range the error bars span; gear and nut bars are one seed. Hover a bar for its exact success and test range (on the peg, its whole widened axis).

Zero-shot transfer to the real world

We deploy the frozen blind peg insertion policy on a Flexiv Rizon 4S with a Robotiq 2F-85 at 50 Hz, with no real-world data or fine-tuning. It receives only encoder-derived joint state; no camera, force/torque sensor or object pose enters the policy. Each trial starts from a state drawn from the simulated reset distribution, with the peg grasped at a randomized position and tilt and touching the fixture near the hole.

Real-world vs. simulation results for the blind policy on the same peg. Error bars are 95% Wilson intervals. Hover over each peg for details; drag the pegs to turn them. Click any trial mark to watch it's outcome in real and simulation.

BibTeX

@misc{bronars2026blindpreciseassemblyproprioceptive,
      title={Blind Precise Assembly: Proprioceptive Sim-to-Real Peg-in-Hole for Diverse Parts},
      author={Antonia Bronars and Younghyo Park and Kushal Kedia and Pulkit Agrawal},
      year={2026},
      eprint={2610.23849},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2610.23849},
}

Acknowledgments

This research was financially supported by the Ministry of Trade, Industry, and Energy (MOTIE), Korea, under the "Global Industrial Technology Cooperation Center program" supervised by the Korea Institute for Advancement of Technology (KIAT) (Grant No. P0028435).