Research project case study · Summer 2026

HOPE —
Autonomous Humanoid Table Tennis

What I built, what failed, and the research questions that followed.

Jeremy Junming Wei

Mathematics–Computer Science
University of California, San Diego

I worked on HOPE as one of four student developers. My primary work covered OptiTrack calibration and rigid-body tracking, reflective-marker ball design, and three manually authored, predefined serve motions. I also contributed to physical-trial analysis and deployment; the learned return policy and end-to-end system were team efforts.

See My Contributions Team codebase HOPE project site Résumé Email

6: formal marker layouts
50+: marked balls fabricated
3: serve motions deployed on hardware
100: racket-orientation × contact-speed pairs evaluated

Three consecutive system returns Balls manually introduced by Jeremy · late-stage laboratory trial

01 / Project context

A full physical loop, not an isolated model.

HOPE is an open platform for autonomous humanoid table tennis. Its system-level structure follows a perceive–predict–plan–strike pipeline. Ball and robot observations were converted into state and trajectory information, used to predict future ball motion, select a desired contact, and command the humanoid.

The public HOPE team repository provides a usable baseline prototype spanning training, simulation, planning, motion-capture integration, and physical deployment on the AgiBot A3.

The project was demonstrated at WAIC 2026, and the team later competed at the 2nd World Humanoid Robot Games in Beijing.

Model-based planner + whole-body policy · Summer 2026 implementation

Select a stage to explore

01 Motion-capture sensing Ball and robot observations
02 State estimation / trajectory history Recent observations → position and velocity
03 Ball-trajectory prediction Estimated future ball path
04 Planning / control policy Desired contact and robot motion
05 Humanoid execution Commanded physical movement
06 Ball interaction Contact and resulting trajectory

01 / Method

Motion-capture sensing

Motion capture provides timestamped 3D ball positions and robot base poses. Joint encoders provide joint angles and velocities. During play, racket pose is calculated from the base pose and joint angles using forward kinematics.

Motion capture: 300–360 Hz
Racket pose: Forward kinematics

02 / Method

State estimation / trajectory history

A sliding window retains the 31 most recent ball observations by default. Independent least-squares quadratic fits to x(t), y(t), and z(t) estimate the ball’s position; their derivatives give its velocity. The history resets around detected bounces or sudden velocity changes.

Default window: 31 observations
Time span: ≈ 100 ms at 300 Hz

03 / Method

Ball-trajectory prediction

Starting from the fitted position and velocity, numerical integration propagates a gravity and quadratic air-drag model, including table-bounce restitution. It predicts the ball’s position, velocity, and arrival time at the selected hitting plane.

Integration step: 1 ms
Flight model: Gravity + quadratic drag

04 / Method

Planning / control policy

The planner solves for the outgoing ball velocity needed to reach the desired landing point, then uses a racket–ball collision model to determine racket velocity and face orientation. The predicted interception provides the contact position and timing. The whole-body policy combines its strike-target inputs with robot observations to produce a 31-dimensional action vector.

Policy rate: 50 Hz
Action vector: 31 dimensions

05 / Method

Humanoid execution

Policy actions are converted into joint-position targets. Low-level PD control tracks these targets using position and velocity feedback, coordinating the arms, waist, and legs.

Command: Joint-position targets
Feedback control: PD tracking

06 / Method

Ball interaction

Racket contact redirects the ball. Subsequent motion-capture observations update the estimated ball state and support the next prediction cycle.

Physical event: Racket–ball contact
Feedback: New ball observations

02 / Individual scope

My Contributions

My primary implementation work appears first; contributing and team-support roles are labeled separately.

Primary contribution · Perception infrastructure

Motion Capture and Tracking

Maintained the OptiTrack calibration and tracking environment and created or updated rigid-body models.
Reconfigured camera placement and height, then recalibrated and validated the system when detection quality degraded.
Diagnosed tracking and calibration failures during training, testing, and competition.

What the image shows

OptiTrack detects the ball’s ten reflective markers and identifies their relative positions within the rigid-body model.

On-site tracking validation

OptiTrack Motive interface showing a ten-point rigid-body model for a marked table-tennis ball — Tracking evidence **Ten-marker rigid-body model**

Primary contribution · Physical sensing design

Reflective Marker Design

Designed six formal marker-layout variants and fabricated more than 50 marked balls across formal and exploratory configurations.
Created an OptiTrack rigid body for each candidate and evaluated tracking dropouts during live rallies with teammates, including high balls, smashes, and spin.
Selected layouts based on continuity of position and orientation tracking; helped fabricate and prepare standardized eight-marker balls for distribution to participating university teams.

Design trade-off

The eight-marker ball was the standardized version prepared for participating teams; my later ten-marker design used smaller reflective patches.

I initially expected eight larger markers to improve detection; ten equally large patches left too little spacing for reliable identification. The larger patches wrinkled on the ball even after trimming, and I observed changes in contact and bounce during live rallies and ball-toss tests. I chose ten smaller markers to balance tracking reliability with bounce behavior.

Compare ball variants

Workbench with marked table-tennis balls, reflective material, cut markers, and fabrication tools — Fabrication evidenceMarker-layout workbench

Primary contribution · Motion development

Serve Motion Development

Developed three manually authored serve motions in MuJoCo and transferred them to the physical humanoid. The first supported qualification and later served as a backup; the other two were used primarily in competition.
For one motion, compared 100 racket-orientation and contact-speed pairs under limited perturbations, using target landing error as the selection criterion.
Adapted two human serve references into robot-feasible joint-space trajectories; all three motions were predefined rather than online-adaptive.

What testing showed

Hardware testing showed that nominally accurate settings still needed margin for net clearance and deviations in contact position and speed. PD-controlled execution did not always fully realize the intended racket motion.

Hardware trial & failure analysis

Simulation evidencePredefined serve motion

04–05

Contributing role · Return-system evaluation

Physical Trial and Failure Analysis

Reviewed prior work, contributed to discussions about return-policy design, and analyzed recorded robot trials for missed strikes, instability, and recovery failures to identify priorities for subsequent testing.

Post-contact stability analysis

Team support · Deployment operations

Physical System Integration

Helped standardize the testing environment; measured reference geometry and marker spacing; installed grippers and robot-mounted markers; supported networking and debugging workflows; and coordinated hardware and experimental-material logistics.

03 / Selected visual evidence

Field Testing and Public Demonstrations

From physical prototypes to venue testing, public showcases, and competition.

Five ball prototypes held in one hand with English annotations: a 15-marker ball, two 10-marker versions with different patch sizes, the standardized eight-marker ball, and a fully reflective ball — Design iterations **Five ball variants, side by side**
Five physical prototypes from a broader process of six formal marker layouts and more than 50 fabricated balls.

The photo shows a 15-marker layout, two 10-marker versions with different patch sizes, the standardized eight-marker ball, and a fully reflective variant.

Why I chose ten smaller markers

Technical communication Explaining marker distribution

Jeremy explains marker spacing and distribution in a Hunan TV field report. English subtitles available. View the original report .

HOPE team members and collaborators with the humanoid robot and table-tennis setup at the project’s WAIC 2026 showcase — Public showcase **HOPE at WAIC 2026**
Official showcase of the physical HOPE system before its later competition deployment in Beijing.

Jeremy rallying with a professor at the Beijing competition venue while an orange reflective-marker ball passes over the table inside the OptiTrack capture volume — On-site validation **Testing marked-ball tracking at the competition venue**
Jeremy and a professor rallied with the finalized reflective-marker ball to evaluate whether the venue OptiTrack setup could retain a usable track during live exchanges.

Official HITCH OPEN group photograph with participating teams, organizers, and the humanoid table-tennis robot at the Second World Humanoid Robot Games in Beijing — Official competition **HOPE at the Second World Humanoid Robot Games**
Official HITCH OPEN competition photograph at the Beijing venue · August 2026.

Additional evidence Earlier integration clip

Early integration Manually introduced ball test

A professor manually introduces the ball while the team evaluates the physical setup, documenting integration before the full serve mechanism was in place.

04 / Technical problems

What the physical system made visible.

These deployment failures motivate the research questions that follow.

Problem 01

Dynamic Interception

A plausible trajectory is not yet a successful contact event.

Serving and returning both depended on a narrow temporal and spatial contact window. For returning, recent motion-capture observations were used to predict the ball’s future trajectory and determine a desired hitting position and time. In practice, nominal trajectory accuracy alone did not guarantee physical interception.

Physical deployment Competition-venue serve execution

At the competition venue, the robot executed one of the predefined serve motions I developed but did not contact the ball. A communication or timing issue was suspected; no logs are available to confirm the cause. Successful physical serves had already been achieved during laboratory testing, but those trials were not recorded.

Measurement quality Timing & synchronization Planning & command latency Prediction / model error Physical feasibility Execution error

This experience led me toward a central question: how should state estimation and prediction support downstream robotic decisions when uncertainty can enter at multiple points in the loop?

Problem 02

Whole-Body Stability

Reaching the ball and remaining recoverable were inseparable objectives.

Whole-body balance remained a major challenge. The team experimented with whole-body imitation and later emphasized upper-body tracking while allowing greater freedom in the lower body. Lateral movement could still trigger increasingly aggressive corrective motions, instability, and eventual falls.

Physical failure case Post-contact loss of balance

After executing the return, the robot exhibits large trunk sway and repeated corrective steps, culminating in a safety-supported near-fall. The trial illustrates how task-level contact can still leave the whole-body state difficult to recover.

Jeremy contributed through physical-trial analysis and balance- and recovery-related debugging; the return controller was a team effort.

The failures exposed the coupling among task execution, balance, recovery, and whole-body control: an action can be locally useful for contact while making the next state harder to control.

05 / Emerging research direction

Questions I Want to Explore

HOPE began as a systems and deployment experience. Failures observed in physical trials raised a connected set of questions about how robots convert uncertain observations into timely, physically feasible actions.

01
How can a robot estimate a fast-moving object’s evolving state—and predict its future trajectory—from imperfect observations while representing uncertainty?
02
How should uncertainty in those estimates shape the selection and timing of a physically feasible interception action?
03
How can real-world trials distinguish errors in sensing and prediction from those introduced by planning and execution—and use that attribution to improve the full interaction loop?

Research direction

The challenges I encountered in HOPE shaped my current interests in state estimation, tracking and prediction, and probabilistic modeling, particularly how they inform planning and control in dynamic robotic systems.

These interests are a starting point, not a fixed boundary. I am also open to related projects in perception, robot learning, planning, and control, and to joining research groups approaching these problems from different perspectives.