Nippun Sabharwal

• Trained an action-conditioned latent flow-matching world model on 0.5B frames of human egocentric and simulated robot video to jointly predict future visual states and actions, learning a generative model of embodied dynamics rather than a deterministic next-frame predictor.
• Formulated representation learning through the evidence lower bound (ELBO), balancing reconstruction likelihood against KL divergence to a latent prior; learned conditional probability paths with flow matching so the model could represent multiple plausible futures in a compact latent space.
• Factorized long-horizon prediction into causal history modeling and DiT-style latent flow generation, then reparameterized transitions as latent deltas between consecutive frames so the model focused on state change and accumulated less drift during autoregressive rollouts.

Built an end-to-end autonomy stack on an embedded vehicle, closing the loop from local sensing and online map construction through goal-directed planning, visual recognition, and motor control while exposing live vehicle telemetry over wireless links.

• SLAM & Localization: Implemented SLAM by converting servo-scanned ultrasonic ranges into a continuously updated 2D occupancy grid and tracking vehicle pose through velocity-based dead reckoning, periodically rescanning as new regions became observable.
• Online Planning & Safety: Implemented A* search with obstacle inflation for vehicle clearance and repeated replanning toward coordinate goals, combining the global route with a reactive proximity override for newly observed and dynamic obstacles.
• Edge Visual Autonomy: Integrated an embedded camera with OpenCV and quantized TensorFlow Lite inference for traffic-sign and object recognition, coupling semantic detections to behaviors such as stopping for signs or pedestrians before resuming the planned route.
• Vehicle Telemetry & Control: Built bidirectional Bluetooth control and telemetry plus a Wi-Fi REST service and Electron interface for remote actuation, sensor monitoring, and vehicle-state visualization.

Built complementary geometric, learned, and hybrid manipulation systems in robosuite: an RGB-D cube-stacking pipeline, state-based and ResNet18 imitation policies, and a Simplex wiping controller combining reactive perception with behavioral-cloning fallback.

• Geometric Manipulation: Built an 8-joint URDF and PD waypoint controller with SLERP orientation, then implemented RGB-D segmentation, metric-depth deprojection, world-frame point clouds, adaptive clustering, clearance-aware grasp selection, and staged green–red–green stacking without simulator poses entering the control loop.
• Imitation Learning: Trained state-based MLP and ResNet18 visuomotor policies on 3,942 state transitions and 5,336 labeled image-action frames, reaching approximately 89% and 95% validation action accuracy, respectively; implemented interactive DAgger collection for correcting policy-induced failure states.
• Simplex Hybrid Control: Combined reactive dirt detection and boustrophedon planning with behavioral-cloning fallback, achieving 97.6% ± 2.0% coverage and 100% success across five trials versus 40% vision-only and 20% BC-only.

Contributors: Nippun Sabharwal, Shivansh Patel

Aligned a photorealistic 3D Gaussian Splat of a real ALOHA workspace with its MuJoCo twin, creating a shared coordinate frame for physically grounded rendering, simulation, and policy-facing multi-camera observations.

• 3DGS–Physics Alignment: ICP registration pipeline aligning a real-world Gaussian Splat capture (iPhone LiDAR + COLMAP) of an ALOHA bimanual setup with its MuJoCo twin, achieving 7.0 mm RMSE via point-to-plane ICP with Tukey robust loss and adaptive voxel downsampling.
• Real-Time Composite Rendering: MuJoCo robot foreground (segmentation mask) + custom NumPy Gaussian rasterizer background at 30–60 FPS across 7+ viewpoints including wrist cameras.
• Policy-Ready API: Observation interface matching the real ALOHA multi-camera topology, outputting dicts compatible with LeRobot and OpenPI policy servers for direct sim-to-real transfer.

Built a markerless 6-DoF plug-pose tracking pipeline from Intel RealSense RGB-D recordings and a CAD model, combining SAM2 segmentation, masked-depth point clouds, multiscale ICP, and calibrated pose-frame correction.

• Perception Pipeline: Separated SAM2-based 2D object tracking from metric 3D estimation using depth deprojection, STL registration, and multiscale ICP.
• Held-Out Evaluation: Calibrated the pose-frame correction on 60% of ArUco-visible frames and evaluated on the held-out 40%, reaching 4.40 mm mean translation error and 11.98° mean rotation error.
• Diagnostics: Produced per-frame overlays, pose trajectories, and failure analysis for partial geometry, mask contamination, symmetry, and tracking drift.

Thesis, Dept of ECE, UIUC · Advisor: Prof. Narendra Ahuja

At NCSA and CSL, developed Wave2Plane as part of a broader investigation into physics-informed recovery of scenes degraded by refractive turbulence. The framework combines simulator-grounded supervision, spatiotemporal geometry estimation, and multi-frame optimization to reconstruct planar underwater scenes without paired real-world ground truth.

• Physics-Grounded Simulation: Built a spectral wave simulator with surface-slope estimation, Snell-law ray tracing, and subpixel image warping, generating clean-texture, slope, and warp ground truth while establishing oracle performance ceilings and noise budgets.
• Modular Spatiotemporal Inference: Built multi-head 3D CNN (or ViT) models with NeRF-style positional encodings over local 31 × 31 × 9 video cuboids for joint per-pixel flatness, directional slope, and refractive-warp prediction, achieving under 0.5° mean absolute slope error. Trained and evaluated model variants on CUDA-accelerated NCSA clusters.
• Dense Scene Recovery: Evaluated every pixel across candidate frames and stitched the locally least-distorted observations into a reconstructed image, reaching up to 32 dB PSNR across synthetic benchmarks and validating recovery on real turbulent-water videos.
• Global Physics Consistency: Developed test-time restoration objectives based on splat-buffer variance and clean-image/coordinate reprojection, enforcing cross-frame geometric agreement without clean targets and improving reconstruction by an additional 2–3 dB PSNR.

Contributors: Nippun Sabharwal, Shreyanka Sinha

Engineered an intuitive VR teleoperation system that enables precise, real-time remote robot control by translating head movements into robot actions and providing immersive 3D visual feedback for enhanced depth perception.

• Real-Time Head-Driven Control: Captured 3-DOF head-pose at 60 Hz from an Oculus headset via ROS, translating movements into joint commands for a multi-DOF robotic neck. This facilitates smooth, natural robot motion through imitation learning.
• Immersive 3D Visual Feedback: Integrated a stereo RGBD camera, streaming dual-view video into the Oculus, completing a closed loop where the headset user sees the robot's environment and can repeatedly perform actions with immediate feedback, providing operators with crucial depth perception for fine manipulation tasks.
• Modular & Extensible Architecture: Designed a Unity–ROS–Dynamixel workflow with configurable DOF settings, ensuring adaptability to various robot platforms and paving the way for future full-body telemanipulation. Implemented ROS dampening filters and H.264/H.265 video compression, achieving 60 Hz command updates with 45–60 ms round-trip delay and sub-degree motion accuracy.
• Demonstrated Results: Successfully executed remote folding and assembly tasks over a 3,000-mile network. Trained imitation learning policies for long-horizon tasks (sorting, insertion, folding), cutting fine manipulation time by ~30%, establishing a robust foundation for scalable imitation-learning data collection.

STL credit: OpenTeleVision

Implemented shape-general 2D convolution from scratch in CUDA, progressing from direct and implicit-GEMM FP32 kernels to FP16/BF16 Tensor Core execution with FP32 accumulation. Built a scalar CPU oracle and regression suite for arbitrary strides and non-aligned tensor shapes, reaching 428 GFLOP/s on an RTX 3050.

• Kernel Design: Implemented direct NCHW convolution with constant-memory filter reuse and a 16 × 16 shared-memory implicit GEMM, then built four-warp WMMA kernels with genuine FP16/BF16 storage, shared weight tiles, tail-safe indexing, and FP32 accumulation.
• Benchmarking & Correctness: Built deterministic shape sweeps with CUDA-event, end-to-end, and five-trial median timing. Tensor Core kernels reached 428.4 GFLOP/s, 1.89× direct CUDA, and over 350× the single-thread CPU oracle; maximum absolute error remained below 1.2 × 10⁻³ for FP16 and 7.9 × 10⁻³ for BF16 across the regression suite.
• Optimization Study: Replaced ineffective in-kernel FP16 conversion with true 16-bit tensor storage and Tensor Core execution, compared FP32/FP16/BF16 accuracy–throughput tradeoffs, and kept mixed precision as an explicit dispatch choice rather than silently changing numerical behavior.

Research with Prof. Bin Hu · NeurIPS 2025

Co-developed EngDesign, a multi-domain benchmark for evaluating whether frontier LLMs can produce executable engineering designs rather than answer static technical questions. Contributed the digital-hardware evaluation pipeline, using simulation and testbench feedback to measure functional correctness and support iterative design refinement.

• Executable Benchmark Design: Helped formulate constraint-driven engineering tasks evaluated through domain-specific simulators, shifting assessment from answer matching to whether generated artifacts satisfy real design requirements.
• Hardware Evaluation: Built hierarchical SystemVerilog tasks and automated graders that compiled and simulated generated modules against testbenches, exposing functional errors that text-only evaluation could not detect.
• Closed-Loop Refinement: Integrated simulation and synthesis tools into an agentic verification loop that returned structured failure feedback for iterative correction and systematic comparison of frontier models.

Developed a full UNIX-style operating system kernel and robust journaling filesystem from scratch for the RISC-V architecture.

• Kernel: Implemented core OS functionalities including a bootloader, trap handling, Sv39 virtual memory with demand paging, and process abstraction. Supports essential user-mode syscalls (open, close, read, write, ioctl, exec, fork, wait, usleep, fscreate, fsdelete). Created cooperative and preemptive threading models using condition variables and timer (mtime) interrupts.
• Custom Filesystem + Block Cache: Engineered a block-based filesystem with a write-ahead journal for metadata consistency and crash recovery. Supports create, read, write, delete, flush, and multi-level indirection. Mountable via VirtIO block device or in-memory "memio" for rapid testing. Implemented a write-back cache with configurable associativity, reducing I/O latency by ~45% while ensuring data consistency through the journaling mechanism.
• Device Drivers & MMIO: Developed drivers for UART (polling & interrupt-driven), Real-Time Clock (RTC), Platform-Level Interrupt Controller (PLIC), VirtIO block & RNG devices (with custom ISR integration), and GPIO, SPI interfaces for embedded peripherals.
• Games! Built a unified I/O interface to load and execute ELF binaries (Star Trek, Doom, Rogue, Zork) on QEMU RISC-V, validated by automated tests for correct loading, execution, and system-call handling.

Designed and implemented a 16-bit CPU based on a reduced instruction set, x86-inspired ISA in SystemVerilog, end-to-end from ISA specification through FPGA verification.

• ISA & Pipeline: 16-bit processor with Program Counter, Instruction Register, general-purpose registers, ALU, and a 3-stage fetch–decode–execute pipeline supporting ADD, AND, NOT, BR, JMP, JSR, LDR, STR, and PAUSE. FSM control unit sequences memory access, ALU operations, and I/O interactions via a custom cpu_to_io bridge interfacing with on-board switches and hex displays.
• Memory-Mapped I/O & BRAM: Mapped UART, switches, and seven-segment displays into the CPU's address space using on-chip Spartan-7 Block RAM, handling read/write timing without an external "ready" signal.
• Graphics Controller: Developed an IP-core-based HDMI graphics controller for 80×30 character text rendering over AXI4 on Vivado IP Integrator. Implemented monochrome and color text output using VRAM and font ROM, supporting inverse text and palette-based coloring.
• FPGA Verified: Synthesized and achieved timing closure in Vivado, deploying and verifying stable operation at 50 MHz on a Xilinx Spartan-7 board.