wtf is taco?

TacO is a research codebase from UC San Diego, Carnegie Mellon University, and Seoul National University that benchmarks tactile sensors for robotic object manipulation. The project trains and deploys learning policies using four different types of tactile sensing: visual, acoustic, magnetic, and resistive. These policies are tested on three manipulation tasks: picking up objects and placing them down, reorienting objects, and inserting plugs. To collect training data, the team used a teleoperation system called GELLO, which lets a human operator guide the robot through tasks. These recorded demonstrations must be converted into a specific file format called HDF5 before training can begin. Each recording contains robot state information, action commands, camera images from two viewpoints, and tactile sensor readings from both sides of the robot gripper. The robot state and action data are padded to 128 dimensions, and the tactile data can be stored either as flattened arrays or as images, depending on the sensor type. Setting up the project requires Miniconda. Users create a Python 3.11 environment and run a setup script. A local configuration file must also be created from a provided template to specify machine-specific paths for saving and exporting model checkpoints. This local file is excluded from version control so individual setups stay private. Training and sensor behavior are controlled through configuration files. Dataset configs specify which recordings serve as training data and which serve as validation data. Sensor configs describe the type of tactile sensor being used, whether its data is treated as raw arrays or images, and the dimensions of the readings. Model configs define the neural network architecture, including layer counts, attention heads, and learning rates. The project uses a policy architecture called ACT, which processes camera images and tactile data together to predict robot actions. Training runs through a single Python command with arguments for batch size, learning rate, number of epochs, and configuration file names. The codebase is still a research artifact with a small number of stars, and the README is focused on helping other researchers reproduce the benchmark results.