Nippun Sabharwal

• Trained an action-conditioned latent flow-matching world model on 0.5B frames of human egocentric and simulated robot video to jointly predict future visual states and actions, learning a generative model of embodied dynamics rather than a deterministic next-frame predictor.
• Formulated representation learning through the evidence lower bound (ELBO), balancing reconstruction likelihood against KL divergence to a latent prior; learned conditional probability paths with flow matching so the model could represent multiple plausible futures in a compact latent space.
• Factorized long-horizon prediction into causal history modeling and DiT-style latent flow generation, then reparameterized transitions as latent deltas between consecutive frames so the model focused on state change and accumulated less drift during autoregressive rollouts.

Built an end-to-end autonomy stack on an embedded vehicle, closing the loop from local sensing and online map construction through goal-directed planning, visual recognition, and motor control while exposing live vehicle telemetry over wireless links.

• SLAM & Localization: Implemented SLAM by converting servo-scanned ultrasonic ranges into a continuously updated 2D occupancy grid and tracking vehicle pose through velocity-based dead reckoning, periodically rescanning as new regions became observable.
• Online Planning & Safety: Implemented A* search with obstacle inflation for vehicle clearance and repeated replanning toward coordinate goals, combining the global route with a reactive proximity override for newly observed and dynamic obstacles.
• Edge Visual Autonomy: Integrated an embedded camera with OpenCV and quantized TensorFlow Lite inference for traffic-sign and object recognition, coupling semantic detections to behaviors such as stopping for signs or pedestrians before resuming the planned route.
• Vehicle Telemetry & Control: Built bidirectional Bluetooth control and telemetry plus a Wi-Fi REST service and Electron interface for remote actuation, sensor monitoring, and vehicle-state visualization.

Deploying learned policies in the real world requires more than good imitation: it requires robustness to novel states, graceful degradation, and integration with classical perception and control. This project builds the full stack from first principles (URDFs, PD control, 3D vision) through modern imitation learning (DAgger, ResNet18 visuomotor policies), culminating in a Simplex safety architecture that achieves 100% task success where vision-only (40%) and learned-only (60%) each fail, the kind of hybrid system needed for reliable real-world robot deployment.

• 3D Perception: RGBD → point-cloud deprojection → hand-eye calibration → HSV segmentation → DBSCAN clustering → coarse-to-fine ICP for 6-DOF pose. Multi-view fusion from 5+ cameras enables autonomous pick-and-place with zero ground-truth state.
• Imitation Learning + DAgger: Trained MLP (state) and ResNet18 (vision) behavioral cloning policies on 2k+ teleoperated demos. DAgger closed the covariate shift gap → 80%+ success on randomized pick-and-place.
• Simplex Architecture (100% Success): Vision-based reactive controller (HSV dirt detection → boustrophedon paths) with automatic BC fallback. 97.6% ± 2.0% cleaning, 100% success vs. 40% vision-only and 60% BC-only on 150-marker contact-rich wiping.
• Classical Foundations: Custom 8-joint URDF, PD waypoint controllers with SLERP orientation, boustrophedon coverage planners.

Contributors: Nippun Sabharwal, Shivansh Patel

The visual sim-to-real gap is the biggest bottleneck in scaling robot learning: policies trained on flat-shaded MuJoCo scenes fail against real-world lighting and textures. This project fuses a photorealistic 3D Gaussian Splat of a real lab with MuJoCo physics so robot policies train in an environment that looks real and behaves real, tackling the same visual grounding problem that labs like Physical Intelligence, Google DeepMind, and Toyota Research Institute are racing to solve.

• 3DGS–Physics Alignment: ICP registration pipeline aligning a real-world Gaussian Splat capture (iPhone LiDAR + COLMAP) of an ALOHA bimanual setup with its MuJoCo twin, achieving 7.0 mm RMSE via point-to-plane ICP with Tukey robust loss and adaptive voxel downsampling.
• Real-Time Composite Rendering: MuJoCo robot foreground (segmentation mask) + custom NumPy Gaussian rasterizer background at 30–60 FPS across 7+ viewpoints including wrist cameras.
• Policy-Ready API: Observation interface matching the real ALOHA multi-camera topology, outputting dicts compatible with LeRobot and OpenPI policy servers for direct sim-to-real transfer.

Thesis, Dept of ECE, UIUC · Advisor: Prof. Narendra Ahuja

At NCSA and CSL, developed Wave2Plane as part of a broader investigation into physics-informed recovery of scenes degraded by refractive turbulence. The framework combines simulator-grounded supervision, spatiotemporal geometry estimation, and multi-frame optimization to reconstruct planar underwater scenes without paired real-world ground truth.

• Physics-Grounded Simulation: Built a spectral wave simulator with surface-slope estimation, Snell-law ray tracing, and subpixel image warping, generating clean-texture, slope, and warp ground truth while establishing oracle performance ceilings and noise budgets.
• Modular Spatiotemporal Inference: Designed a modular pipeline supporting ViT- and NeRF-based inference components in selected experiments, then established a multi-head 3D CNN over local 31 × 31 × 9 video cuboids for joint per-pixel flatness, directional slope, and refractive-warp prediction, achieving under 0.5° mean absolute slope error. Trained and evaluated model variants on CUDA-accelerated NCSA clusters.
• Dense Scene Recovery: Evaluated every pixel across candidate frames and stitched the locally least-distorted observations into a reconstructed image, reaching up to 32 dB PSNR across synthetic benchmarks and validating recovery on real turbulent-water videos.
• Global Physics Consistency: Developed test-time restoration objectives based on splat-buffer variance and clean-image/coordinate reprojection, enforcing cross-frame geometric agreement without clean targets and improving reconstruction by an additional 2–3 dB PSNR.

Contributors: Nippun Sabharwal, Shreyanka Sinha

Engineered an intuitive VR teleoperation system that enables precise, real-time remote robot control by translating head movements into robot actions and providing immersive 3D visual feedback for enhanced depth perception.

• Real-Time Head-Driven Control: Captured 3-DOF head-pose at 60 Hz from an Oculus headset via ROS, translating movements into joint commands for a multi-DOF robotic neck. This facilitates smooth, natural robot motion through imitation learning.
• Immersive 3D Visual Feedback: Integrated a stereo RGBD camera, streaming dual-view video into the Oculus, completing a closed loop where the headset user sees the robot's environment and can repeatedly perform actions with immediate feedback, providing operators with crucial depth perception for fine manipulation tasks.
• Modular & Extensible Architecture: Designed a Unity–ROS–Dynamixel workflow with configurable DOF settings, ensuring adaptability to various robot platforms and paving the way for future full-body telemanipulation. Implemented ROS dampening filters and H.264/H.265 video compression, achieving 60 Hz command updates with 45–60 ms round-trip delay and sub-degree motion accuracy.
• Demonstrated Results: Successfully executed remote folding and assembly tasks over a 3,000-mile network. Trained imitation learning policies for long-horizon tasks (sorting, insertion, folding), cutting fine manipulation time by ~30%, establishing a robust foundation for scalable imitation-learning data collection.

STL credit: OpenTeleVision

Developed and profiled six custom CUDA implementations of a CNN forward pass, systematically improving convolution performance through memory-hierarchy optimization, mixed-precision computation, tiled GEMM, and asynchronous execution.

• Memory Hierarchy: Weights in constant memory, I/O tiles in shared memory, register accumulators, achieving order-of-magnitude reduction in global memory traffic.
• FP16 Mixed Precision + Tree Reduction: GPU performance is memory-bandwidth bound: FP16 halves memory footprint, doubling effective bandwidth and allowing 2× more data in cache. Modern GPUs (Tensor Cores, FP16 ALUs) also deliver 2× compute throughput on half-precision ops. Implemented via CUDA intrinsics (__half, __hmul) with 3D thread-block tree reduction over input channels, eliminating serial loops and halving register pressure to enable higher occupancy, the same quantization strategy that makes large-scale training feasible.
• Tiled GEMM (im2col): Convolution re-cast as matrix multiply; 16×16 shared-memory tiled kernel with unrolled inner loops for near-peak occupancy.
• Streams & Layer-Adaptive Dispatch: Direct conv for shallow layers, GEMM for deep layers. 10 concurrent CUDA streams pipelining kernel execution with memory transfers.
• Nsight-Guided Profiling: Used NVIDIA Nsight Systems and Nsight Compute to profile warp occupancy, memory throughput, and kernel launch behavior, guiding loop unrolling, tile sizes, and stream placement to remove stalls and reach stable high-utilization runs.

Research with Prof. Bin Hu · NeurIPS 2025

Co-developed EngDesign, a multi-domain benchmark for evaluating whether frontier LLMs can produce executable engineering designs rather than answer static technical questions. Contributed the digital-hardware evaluation pipeline, using simulation and testbench feedback to measure functional correctness and support iterative design refinement.

• Executable Benchmark Design: Helped formulate constraint-driven engineering tasks evaluated through domain-specific simulators, shifting assessment from answer matching to whether generated artifacts satisfy real design requirements.
• Hardware Evaluation: Built hierarchical SystemVerilog tasks and automated graders that compiled and simulated generated modules against testbenches, exposing functional errors that text-only evaluation could not detect.
• Closed-Loop Refinement: Integrated simulation and synthesis tools into an agentic verification loop that returned structured failure feedback for iterative correction and systematic comparison of frontier models.

Developed a full UNIX-style operating system kernel and robust journaling filesystem from scratch for the RISC-V architecture.

• Kernel: Implemented core OS functionalities including a bootloader, trap handling, Sv39 virtual memory with demand paging, and process abstraction. Supports essential user-mode syscalls (open, close, read, write, ioctl, exec, fork, wait, usleep, fscreate, fsdelete). Created cooperative and preemptive threading models using condition variables and timer (mtime) interrupts.
• Custom Filesystem + Block Cache: Engineered a block-based filesystem with a write-ahead journal for metadata consistency and crash recovery. Supports create, read, write, delete, flush, and multi-level indirection. Mountable via VirtIO block device or in-memory "memio" for rapid testing. Implemented a write-back cache with configurable associativity, reducing I/O latency by ~45% while ensuring data consistency through the journaling mechanism.
• Device Drivers & MMIO: Developed drivers for UART (polling & interrupt-driven), Real-Time Clock (RTC), Platform-Level Interrupt Controller (PLIC), VirtIO block & RNG devices (with custom ISR integration), and GPIO, SPI interfaces for embedded peripherals.
• Games! Built a unified I/O interface to load and execute ELF binaries (Star Trek, Doom, Rogue, Zork) on QEMU RISC-V, validated by automated tests for correct loading, execution, and system-call handling.

Designed and implemented a 16-bit CPU based on a reduced instruction set, x86-inspired ISA in SystemVerilog, end-to-end from ISA specification through FPGA verification.

• ISA & Pipeline: 16-bit processor with Program Counter, Instruction Register, general-purpose registers, ALU, and a 3-stage fetch–decode–execute pipeline supporting ADD, AND, NOT, BR, JMP, JSR, LDR, STR, and PAUSE. FSM control unit sequences memory access, ALU operations, and I/O interactions via a custom cpu_to_io bridge interfacing with on-board switches and hex displays.
• Memory-Mapped I/O & BRAM: Mapped UART, switches, and seven-segment displays into the CPU's address space using on-chip Spartan-7 Block RAM, handling read/write timing without an external "ready" signal.
• Graphics Controller: Developed an IP-core-based HDMI graphics controller for 80×30 character text rendering over AXI4 on Vivado IP Integrator. Implemented monochrome and color text output using VRAM and font ROM, supporting inverse text and palette-based coloring.
• FPGA Verified: Synthesized and achieved timing closure in Vivado, deploying and verifying stable operation at 50 MHz on a Xilinx Spartan-7 board.

Contributors: Nippun Sabharwal, Vayun Gupta, Siddarth Natarajan

Developed an autonomous security system to modernize access control, replacing traditional key/card-based systems with sensor-triggered visual verification and remote actuation.

• Problem: Campus and home access relying on physical cards or keys creates delays, extra costs, and accessibility barriers.
• Solution: An ultrasonic sensor (HC-SR04) detects approaching individuals, triggering an Arduino UNO and OV7670 camera. The captured image streams via UART → PC → Telegram bot. The owner sends "door open" / "door close" over Telegram, and an ESP32 actuates SG90 servo motors to lock or unlock.
• Tech Stack: HC-SR04 ultrasonic sensor (50 cm trigger range, power-saving standby), OV7670 camera over UART, Arduino UNO for coordination, ESP32 for Wi-Fi and Telegram API, SG90 servos for actuation, Make.com for workflow automation.
• Features: Live video feed for verification, automated entry/exit logs, updatable face database, upgradable firmware. Real-world applications include keyless Airbnb access, ID-free campus entry, and improved accessibility.

My first project! Built and scaled a Harry Potter fan community to 12,500+ registered users and 4,000+ social media followers. Led and coordinated a team of 30 volunteers to develop quizzes, discussion forums, and engaging content, fostering a highly active online platform.