MotionGeneralizer uses procedural primitives and LLM-guided placement to create diverse, cluttered, and physically feasible workspaces at scale.
PerFACT overview. MotionGeneralizer creates diverse workspaces and planning data; MπNetsFusion learns an open-loop motion policy for simulation and real-world deployment.
Neural motion planners learn from experience, but small, manually designed datasets limit how well they generalize to everyday scenes. Monolithic architectures can also underrepresent sparse yet critical planning signals.
PerFACT addresses both challenges. MotionGeneralizer combines procedural generation with LLM-guided object selection and placement to collect 3.5M trajectories. MπNetsFusion uses bottleneck fusion and action chunking to plan in 0.22s on average while remaining competitive with sampling-based and neural baselines.
average simulated planning time — 4.5–18.6× faster than sampling-based and neural-sampler baselines.
training trajectories collected across ~1000 MotionGeneralizer workspaces via cuRobo.
average simulated success rate — competitive with AIT*, MPNets, and SIMPNet baselines.
successful real-world trials (69%) on a physical UR5e across four task categories, zero-shot.
MotionGeneralizer uses procedural primitives and LLM-guided placement to create diverse, cluttered, and physically feasible workspaces at scale.
Given a robot and a randomly chosen number of surrounding tables, MotionGeneralizer queries a fine-tuned LLM twice in a few-shot manner: once to pick how many primitives go on each table, and once to place their origin and orientation — before validating and de-duplicating the result.
Combine primitive pools from Neural MP and NVIDIA SceneSynthesizer to generate everyday articulated objects — tables, cabinets, dishwashers, microwaves, shelves — with randomized sizes and joint configurations.
GPT-4 is prompted with robot and table specifications to choose primitive counts per table, then prompted again with the generated primitives to set each object's pose relative to its table.
Collision-normal correction resolves any residual overlaps; a text-embedding similarity check against prior workspace descriptions rejects near-duplicate scenes to keep the dataset diverse.
MotionGeneralizer Framework. A fine-tuned LLM is prompted in a few-shot manner twice per workspace: first to determine the number of primitives per table, then — once those primitives are procedurally generated — to specify each primitive's origin and orientation. The process repeats to generate an arbitrary number (N) of diverse planning workspaces.
Generated Workspaces. MotionGeneralizer produces an arbitrary number of cluttered, diverse workspaces for a given manipulator (UR5e shown) drawn from a pool of everyday primitives.
Rather than rendering occluded, slow RGB-D point clouds in simulation, MotionGeneralizer synthesizes a privileged workspace point cloud by uniformly sampling points directly on obstacle and manipulator meshes at arbitrary configurations — enabling large-scale training before fine-tuning on real, partial camera reconstructions at deployment. It then samples collision-free poses within the generated primitives and the manipulator's reachable region to define robot-agnostic, scene-specific motion-planning problems that any off-the-shelf planner (OMPL, cuRobo) can solve for data collection.
Point Cloud Synthesis. Uniform surface sampling on workspace primitives (a) and on the manipulator at arbitrary configurations (b) gives dense, unoccluded training point clouds.
Planning-Problem Generation. Collision-free start/goal poses are sampled within the workspace and solved via robot-specific inverse kinematics for any embodiment (UR5e shown).
MotionGeneralizer vs. MotionBenchMaker. MotionBenchMaker (top) offers only eight fixed workspace types with a single major obstacle each. MotionGeneralizer (bottom) generates an arbitrary number of cluttered, multi-object workspaces — only 8 shown here for a fair visual comparison.
Against a random-placement baseline using identical primitives, MotionGeneralizer is evaluated on four motion-planning-relevant metrics: scene- and object-level collision rates, object-category entropy (type diversity), and table surface coverage (clutter density).
MπNetsFusion routes cross-modal information through learnable bottleneck tokens, keeping robot, workspace, and configuration signals balanced while predicting complete action chunks.
Current/goal configurations and robot/scene point clouds are tokenized separately, processed by unimodal transformer layers, then fused: each modality attends only to a shared pool of bottleneck tokens inside the fusion transformer, rather than attending directly to every other modality.
An MLP tokenizes current/goal configurations; PointNet++ set-abstraction layers tokenize robot and scene point clouds into compact embeddings.
Unimodal transformer layers refine each modality; a fusion transformer routes cross-modal information exclusively through learnable bottleneck tokens, preventing any one modality from dominating.
A transformer decoder attends over the fused memory with learnable action tokens to output a chunk of delta-joint actions, executed open-loop and recursively fed back as the next input.
MπNetsFusion Architecture. Current/goal configuration and robot/scene point-cloud embeddings pass through unimodal transformer layers (×Lf), fuse through bottleneck tokens inside the fusion transformer (×Lb), and are decoded — via learnable action tokens — into a chunk of delta-joint actions.
MπNetsFusion is trained end-to-end with an MSE loss on delta-joint actions, using the 3.5M-trajectory dataset collected via MotionGeneralizer and cuRobo across roughly 1000 generated workspaces for a UR5e manipulator.
Simulated Rollouts. MπNetsFusion planning across all held-out evaluation tasks plus an out-of-distribution scene generated by MotionGeneralizer (bottom row), each solved in roughly 0.2–0.3s. Frames show the path profile from start to goal.
We evaluate MπNetsFusion against AIT*, MPNets, and SIMPNet across six held-out environments using the same UR5e and collision-free success criteria.
In an unseen MotionGeneralizer scene, MπNetsFusion reaches every goal and achieves 28% collision-free success in 0.28s—comparable success at a fraction of baseline planning time.
How should robot, workspace, and configuration embeddings talk to each other? We compare MπNetsFusion against a vanilla pairwise-attention baseline (ACT), two variants that separate modalities further (All: 4 separate modalities; Config: current+goal configuration merged into one token stream), and ViTacFormer's cross-attention design. Bottleneck-token fusion — in any of its variants — consistently beats plain pairwise attention.
ACT Baseline. Standard pairwise attention over all planning modalities, with no bottleneck restriction on information flow.
ViTacFormer. Learns cross-modal interaction via explicit cross-attention between robot and workspace embeddings.
Neural MP is an open-loop planner with an RNN-based decoder; its GMM head adds multi-modal, sampling-based best-of-N behavior. MπNetsFusion matches or exceeds Neural MP's GMM head — most notably on Bins (84.5% vs. 58.5%) — using a single forward pass instead of recurrent, multi-modal decoding.
Swapping MπNetsFusion's action head for a Diffusion Policy head (same ACT-style encoder, 100 denoising steps, closed-loop) isolates the effect of the generative action head itself. MπNetsFusion matches or beats Diffusion Policy's success rate while avoiding its iterative denoising cost.
Because MπNetsFusion unrolls a fixed number of total steps, chunk size trades planning time for re-sampling frequency of the robot's body point cloud. Success stays flat, but planning time drops sharply as chunk size grows — the main model (chunk size 10) is roughly 6× faster than chunk size 1.
Sweeping the number of bottleneck tokens from 0 (ACT) to 8 shows that any nonzero bottleneck sharply improves success, and performance is stable across 1–8 tokens — 4 tokens is used as the main configuration for its favorable accuracy/size trade-off.
The PointNet++ set-abstraction radius controls how local each point-cloud token's receptive field is. Performance is nearly flat across radii 0.05–0.3m, since the downsampled tokens (16 robot / 128 scene) already provide sufficient spatial coverage regardless of radius.
Training on matched-size datasets (~430K trajectories, 150K steps) generated by MotionGeneralizer versus random primitive placement isolates the value of LLM-guided scene composition, independent of dataset size. MotionGeneralizer's workspaces improve success on every task.
Extending the shelf evaluation to six shelf variants (versus three in the main benchmark) isolates why shelf tasks are hard: MπNetsFusion always reaches a nearby pose, but open-loop rollout — with no iterative collision checking — is far more prone to scene-collisions in these narrow, cluttered spaces.
MπNetsFusion transfers zero-shot to a physical UR5e. Calibrated RGB-D cameras reconstruct the same point-cloud representation used during simulation training.
Experimental Setup. UR5e manipulator with a calibrated Intel RealSense D435i RGB-D camera.
Evaluation Categories. TableTop, Bins, Articulated (drawers/microwave), and Shelf tasks.
Real-world results. The planner reaches the goal in 100% of all 200 real-world trials; 138 (69%) are fully successful and collision-free. Shelf tasks remain hardest (36%), consistent with simulation — narrow passages and limited shelf-primitive diversity in training.
TableTop: 48/50 (96%) successful, all 50/50 goals reached.
Bins: 42/50 (84%) successful, all 50/50 goals reached.
Articulated: 30/50 (60%) successful, all 50/50 goals reached.
Shelf: 18/50 (36%) successful, all 50/50 goals reached.
PerFACT pairs MotionGeneralizer, an LLM-guided procedural workspace generator, with MπNetsFusion, a bottleneck-fusion action-chunking transformer, to train an end-to-end neural motion planner that matches classical and neural-sampler baselines in success rate while planning up to an order of magnitude faster — and transfers zero-shot to a physical UR5e.
Procedural primitives plus chain-of-thought LLM placement produce cluttered, physically feasible workspaces at a scale hand-designed benchmarks can't match.
Collision-free poses sampled within each workspace and the manipulator's reachable region yield planning problems solvable by any embodiment's inverse kinematics.
Restricting cross-modal attention through learnable bottleneck tokens keeps sparse configuration signals from being drowned out by dense workspace point clouds.
@article{soleymanzadeh2025perfact,
title = {PerFACT: Motion Policy with LLM-Powered Dataset Synthesis and Fusion Action-Chunking Transformers},
author = {Soleymanzadeh, Davood and Liang, Xiao and Zheng, Minghui},
journal = {arXiv preprint arXiv:2512.03444},
year = {2025}
}