Here's a video: https://www.youtube.com/watch?v=zCNmNm3lQGk.
Interfacing with existing excavators for enabling remote teleop (or autonomy) is hard. Unlike cars which use drive-by-wire technology, most of the millions of excavators are fully hydraulic machines. The joysticks are connected to a pilot hydraulic circuit, which proportionally moves the cylinders in the main hydraulic circuit which ultimately moves the excavator joints. This means excavators mostly do not have an electronic component to control the joints. We solve this by mechanically actuating the joysticks and pedals inside the excavators.
We do this with retrofits which work on any excavator model/make, enabling us to augment existing machines. By enabling remote teleoperation, we are able to increase site safety, productivity and also cost efficiency.
Teleoperation by the operators enables us to prepare training data for autonomy. In robotics, training data comprises observation and action. While images and videos are abundant on the internet, egocentric (PoV) observation and action data is extremely scarce, and it is this scarcity that is holding back scaling robot learning policies.
Flywheel solves this by preparing the training data coming from our remote teleop-enabled excavators which we have already deployed. And we do this with very minimal hardware setup and resources.
During our time in YC, we did 25-30 iterations of sensor stack and placement permutations/combinations, and model hyperparams variations. We called this “evolution of the physical form of our retrofit”. Eventually, we landed on our current evolution and have successfully been able to train some levels of autonomy with only a few hours of training data.
The big takeaway was how much more important data is than optimizing hyperparams of the model. So today, we’re open sourcing 100hrs of excavator dataset that we collected using Flywheel systems on real construction sites. This is in partnership with Frodobots.ai.
Dataset: https://huggingface.co/datasets/FlywheelAI/excavator-dataset
Machine/retrofit details:
Volvo EC380 (38 ton excavator)
4xcamera (25fps)
25 hz expert operator’s action data
We’re just getting started. We have good amounts of variations in daylight, weather, tasks, and would be adding more hours of data and also converting to lerobot format soon. We’re doing this so people like you and me can try out training models on real world data which is very, very hard to get.
So please checkout the dataset here and feel free to download and use however you like. We would love for people to do things with it! I’ll be around in the thread and look forward to comments and feedback from the community!
I've actually spent a decent amount of time running an excavator, as my Dad owns a construction / road building company. It was a great summer job!
An important note about the pilot hydraulics is that they _provide feedback to the operator_. I would encourage any system that moves these controls on behalf of a remote human operator or AI to add strain gauges or some other way to measure this force feedback so that this data isn't lost.
The handful of "drive by wire" pieces of equipment that my Dad or other skilled operators in my family have ran were universally panned, because the operators are isolated from this feedback and have a harder time telling when the machine is struggling or when their inputs are not sufficiently smooth. In the automotive world, skilled drivers have similar complaints about fully electronic steering or braking systems, as opposed to traditional vacuum or hydraulic boosting approaches where your foot still has a direct hydraulic connection to the brake pads.
I'd like to get a chance to talk to you and your Dad to get feedback. How do I reach you? My email is contact at useflywheel dot ai
I am curious if something like this is an opportunity for a whole new type of controls and feedback. Since the operator doesn’t have to be in the excavator physically they could take on any position: standing, sitting, lying down, etc. Instead of sending haptic feedback to the joystick it could be sent to a vibrating wrist band. You could hook up the equivalent of a Nintendo Power Glove to have the scoop operated by the operator simulating scooping action. Turning the excavator can be controlled by the operator turning their head and moving it around can be done by the operator walking on an infinite treadmill. Motor strain can be done via color of light or temperature rather than sound. You could have a VR helmet that can also show you a birds eye view from a companion drone, overlay blueprints, show power and water lines, measure depth, etc. I don’t know if it is possible but maybe you could even measure soil composition somehow to show large rocks, soft soil that is dangerous to drive over, inclination angles where the excavator is about to drive, etc.
I imagine skilled operators prefer familiar controls but perhaps there are interesting improvements unlocked by exploring alternatives. It might also fundamentally change how accessible it is for non-professionals to use these machines. I rented an excavator from Home Depot a few years ago to dig a foundation and the learning curve was not shallow. I wonder if a more “natural” interface would help keep people safer.
I cannot imagine this being useful to me unless the virtual operators cab closely mimicked an actual machine. It would have to have audio from the machine and be on a platform that tilted relative to the real thing. It would also need 270 degrees of monitors with a virtual mirror to see behind. On the front monitor, minimally, would need the to see vertically up and down too.
I also imagine all of this would be more useful to seasoned operators who can do most things on excavators in their sleep (definitely not me lol)
So if we are able to have really good autonomous safety layers to ensure safe movements, and dynamically resize remote teleop windows, you'd make the operator more efficient. So while we stream 360 degree view, we get creative in how we show it.
That's on the vision side. We also stream engine audio, and do haptic feedback.
Takeuchi are interesting! Rare ones to have blades even on bigger sizes - is that why you got one?
One big advantage would be cameras mounted on the boom and rear view cameras, as many machines have obstructed views.
I imagine an excavator, meant to touch and dig through things, and lift things, benefits from force feedback for the same reason VR would.
Have you played those VR sword games? BeatSaber works great because you're cutting through abstract blobs that offer no resistance. But the medieval sword-slashing games feel weird because your sword can't impact your opponent.
I saw a video recently of a quadcopter lifting heavy objects. When it's overloaded, it can't maneuver because all its spare power is spent generating lift to maintain altitude. If the controls had force feedback, the copter's computer could tell you "I'm overloaded, I can't move" by putting maximum resistance on the sticks.
So even if the electric system fails completely, you can still actuate the brakes.
The Looming Disaster Under America's Biggest Oil Field [video] - https://news.ycombinator.com/item?id=45361022 - September 2025
Texas has thousands of abandoned oil and gas wells. Who is responsible for cleaning them up? - https://www.texastribune.org/2025/05/08/texas-orphan-wells-e... - May 8th, 2025
The Rising Cost of the Oil Industry’s Slow Death - https://www.propublica.org/article/the-rising-cost-of-the-oi... - February 22nd, 2024
Well plugging SOP:
https://www.epa.gov/natural-gas-star-program/well-plugging
https://www.osha.gov/etools/oil-and-gas/abandoning-well
https://en.wikipedia.org/wiki/Relief_well
Sigh. No. Actual TAM is $0 which is why these wells are orphaned
For contrast, maritime shipping IMO (my current obsession) has set a $100-$400 carbon price. So now the efuel market a realistic $10B ZNZ pot to aim at.
There was a YouTube video recently of an AI-assisted digger making a wall out of the concrete rubble from a demolition.
I believe the applications for really smart excavators must be huge. Sounds like this might be a step on that voyage.
I have a Cat 289D skid steer I would happily contribute to the effort if you guys move into the compact equipment space (compact being a relative term, as my machine is only 6 tons compared to your 38 ton machine)
So in reality, CANbus control should be no different than hydraulic control to your system - just another style of I/O
But... I assume you're intending to run on extremely large sites such as highway construction, open pit mines and the likes primarily? Because my experience (if a bit dated cough) is running small 750kg baby excavators under sometimes extreme space constraints - digging trenches for telco in urban and rural areas, which often enough meant having to work with 5-10 cm distance to walls, lighting posts or other infrastructure, and directly next to workers shoving soil into the bucket. Will you add stuff like 360° camera vision, LIDAR etc. to make that safe and help a remote/AI operator, or are you planning on large sites with less danger potential only?
You might be interested in the work of Peter Corke also, he's automated horizontal mine shaft loaders and huge drag line shovels in his research:
https://www.youtube.com/watch?v=YUb9_Ysd2Hw
I think he used a different approach than you do, using visual servoing to get feedback and data from a camera. Maybe there's some value in combining both approaches, learn to control a machine from an operator, and also keep track of what is being moved with a camera to add another layer of control.
The range of skillset on these things is large, there are 2 dominant (fairly swappable - meaning most machines do both) different operating modes (excavator / backhoe) and I see that operators have a specific one they are best with.
Honestly, when I see a real pro using one of these machines I think this is one area AI is not going to win at soon - in the real world there's a whole support crew working with the excavator operator