Very cool. They've done what I've daydreamed about, and actually got it to work!
When I played Elite: Dangerous I used a "hand on throttle and stick" (HOTAS) setup, along with foot pedals. I couldn't help but think that there must be a better way to control a spaceship: your ship can pitch, yaw, and roll in addition to being able to fire thrusters in 6 directions.
I wanted a handheld ship model that I could move such that the ship in Elite would move in the same way. The linked project looks like it could do just that. Thrust would be controlled in a similar way, but with my other hand.
Strange or new input models like that are so amazing to me. Our imagination can really fly high with these sorts of capabilities.
One of my fondest memories is when I was 16 or so. I bought a 3D mouse (it’s like a mouse that you can control by waving it around) and taped it onto a hat that said Marines. Then I immediately went into CS: Source and got ready to annihilate everyone.
I think I played about the same, but it would’ve been funnier if I’d gotten creamed.
The theory was that my head is much more precise than my hands, so using a combination of both should improve my aim. It might sound like you’ll end up looking left and trying to see out of the corner of your eye, but in practice the two-mouse system ends up working out. I’ve always wanted to revive the idea for modern times, but you end up looking like a huge dork with a dildo glued to your head. I like that style though.
Modern consoles (and modern controllers I assume) can do this "double-input" for aiming, using a joystick and a gyroscope. I'm still bad at shooters, but it feels super good. Plus you don't look like a huge dork with a dildo glued to your head.
I have the slightly newer version, branded by HP as the SpacePilot, and use it daily in Fusion 360 CAD. It has a native USB connection, and 6 user-definable hotkeys below an unimpressive LCD.
It requires some old drivers that aren't officially supported (why would they remove support for perfectly good hardware from the newer drivers? To send good stuff to the landfill, of course!), but when Autodesk tried to move Fusion to the new driver model exclusively, user outcry persuaded them to leave the old drivers in as an option. Apparently there's quite a few of us using those SpacePilots, and the phrase "from my cold, dead fingers" comes up not infrequently.
That's really nice, before now I hadn't really understood how to use 2 sticks, but now it makes sense.
I used a CH Pro throttle, which has enough hats to sort of mimic that 2 stick action. Main throttle back forward for main engine and retros. A smaller thumb stick/hat for each of Y and Z axis.
I always imagined a sphere suspended / held by a minimum number of strings (or rods?) / attachments. By physically pushing, pulling, and twisting the sphere you could detect these movements via compression and tension in the attachments. You could motorize those strings/rods to give resistence and feedback to the piloting.
Yeah that's almost exactly what I was daydreaming of! Weaker gyroscopes might mean a ship can rotate in one direction slower than others, and feedback to a controller could model that for the user.
> I couldn't help but think that there must be a better way to control a spaceship: your ship can pitch, yaw, and roll ... in 6 directions.
15 years later and I'm still convinced that the SpaceOrb 360 is THE GREATEST GAME CONTROLER OF ALL TIME. See what I did with the all cap there? It's the G.O.A.T. and it's a tragedy that it lost the battle to "WASD" keyboard and mouse gaming and the "two thumbs" style of controller.
If you've ever tried a SpaceOrb, you probably already know this. If you didn't live through the SpaceOrb's hayday (around 1994-5 with the original Quake, and Descent, truly one of the greatest games ever) then this post will sound like I'm a crotchety old dude trying desperately to relive my the golden hour of my youth. Ok, so you got me. What of it? ;)
I had purchased a box of these when they were going away, and sponsored a custom driver. For a while there were other solutions available, like OrbShield:
Reading your comment with the linked project in mind, I started thinking about the idea of a “glove holding a sphere” sort of control system: the glove is the substrate for the ArUco codes, but the sphere could contain the accelerometers, but could also contain gyroscopes to provide inertial feedback? I don’t know anywhere enough about either technology but I imagine controlling a ship in 3d space would be a constant fight against inertia, and it would be awesome to feel that.
If you like the puzzle of firing thrusters manually, try "ΔV: Rings of Saturn". The game is 2D, but you get six keys to individually fire different thruster combos, and a separate key to fire the main engine.
Then try staying in control with an unbalanced ship, or after you inevitably smashed one of the thrusters into a rock. It's a pretty space-nerd game too, they simulate details like reaction wheel saturation.
The folks at Sublight Dynamics had a really neat product in this vein that I was excited about. Seems like the perfect flightsim controller. They had trouble getting fully to market. Website is still up though so I'm holding out hope!
Given that a spaceship isn't all that different from a fixed-wing aircraft, and the stick + pedals arrangement has become a standard for controlling the latter for over a century, I think it'd be hard to improve on.
The difference is that spaceships have thrusters, so they have the ability to accelerate up, down, left, or right, in addition to all the controls of a fixed-wing aircraft (throttle + stick + pedals).
Do fixed wing aircraft spend a lot of the time going backwards? Do they, for example, need to turn around and point their engine in the direction of travel in order to slow down? If a fixed wing accelerates to 1000mph and then rotates it’s nose by 90 degrees, how fast is it going and in what direction? What about a space craft?
Why do you think so?
When I was doing undergrad, the emphasis was on scientific work.
So novelty or applicability were not important for grades, but if your written communication about it was scientific. I didn't see a write-up of this project, so I couldn't judge it grade wise
I agree, and this is one of the things that are problematic with academia. Not that scientific rigor / work / dissemination shouldn't be rewarded--it should. But so should work like OP's.
Very cool. The use of a webcam really makes me wonder if there's a future where our regular single ~78° FOV webcams are going to be replaced by dual (stereo) fisheye webcams that can:
- Enable all sorts of new UX interactions (gestures with eye tracking)
- Enable all sorts of new peripheral interactions (stylus like this, but also things like a steering wheel for racing games)
- Enable 3D 180° filming for far more flexible webcam meetings, including VR presence, etc.
The idea of being able to use the entire 3D space in front of your computer display as an input method feels like it's coming, and using a webcam the way OP describes feels like it's a little step in that direction.
I thought this was around the corner years ago when Intel and partners had RealSense modules being built into laptops but it seems like all the players have shifted focus to more enterprise and industrial markets.
> An inertial measurement unit (IMU) is an electronic device that measures and reports a body's specific force, angular rate, and sometimes the orientation of the body, using a combination of accelerometers, gyroscopes, and sometimes magnetometers. When the magnetometer is included, IMUs are referred to as IMMUs.[1]
Moasure does displacement estimation with inertial measurement (in a mobile app w/ just accelerometer or also compass sensor data?) IIUC:
https://www.moasure.com/
Are there a limited set of possible-path-corresponding diffraction patterns that NIRS (Near-Infrared Spectroscopy) could sense and process to make e.g. a magic pencil with pressure sensitivity, too?
> Can low-cost lasers and Rdyberg atoms e.g. Rydberg Technology solve for [space-based] matter-wave interferometry? [...] Does a fishing lure bobber on the water produce gravitational waves as part of the n-body gravitational wave fluid field, and how separable are the source wave components with e.g. Quantum Fourier Transform/or and other methods?
I'm sadly bearish on this kind of stuff. The Mac touch-panel showed that even a context-aware, reactive and fully integrated input struggled to break the flow gained by keyboard+mouse.
Even today some people are still on the fence about the new fangled mouse!
I'm sure you've heard of TrackIR, which can do head tracking with some head-mounted reflectors. But without mounting anything to your head, the tobii eye tracker can do both head and eye tracking completely passively. It seems like adding hand/finger gesture controls, and peripherals like stylus, mouse, etc... would be an obvious next step, but it doesn't look like they or anyone else is doing that.
Some random-ish thoughts from exploring "a laptop keyboard... with hand pose, 3D stylus, and touch". Adding buttons yields a 3D mouse - but camera coverage can be a pain. Note the body is largely empty (and battery could be slimmed) - I could more-or-less type while holding a slim chopstick/toothbrush-like stylus (even with the markers, and a weirdly big tip). A big tip (sliced from a ping-pong ball sized xmas decoration) could slide fairly smoothly on a ThinkPad keyboard (and gave room for a less compact force sensor, and an extra tip marker). Thin stranded silicone ribbon cable can be string-like flexible - I just tethered the stylus to an arduino to get started.
Hmm... I wonder what the inertial sensor might make of something like a dimpled metal clicker, as a button (or three)?
The rolling shutter compensation is pretty cool and isn't something I would have thought of. Did you know that would be an issue from the start or notice it only after you built the rest of the system?
I knew it would have an effect (most of the literature for similar projects just uses global shutter cameras for this reason), but wasn't sure how significant it would be. It turned out to be small enough that it usually wasn't super noticeable, but in certain cases it really showed up (e.g., rotating the pen while keeping the tip in one place).
The thing I was most surprised by was how effective my solution was, given that it's a pretty gross approximation of reality. There are lots of much more sophisticated techniques for dealing with it, which I didn't end up needing.
One thing I would've liked to try out is using rolling-shutter-aware PnP [1], which can theoretically estimate pose and velocity simultaneously from one image, by exploiting the rolling shutter distortions.
I'm a big fan of all things 6DOF ! Nice work on the hardware and computer vision pose work but I am almost more impressed by the software surface you are drawing into and able to rotate. Thats interesting and could be used with any tangible user interface control like a finger slider for the same effect. Good project for problem solving skills looks like you nailed it bravo!
Btw the first 6DOF controller I had other (than a hacked WiiMote controller as a ir led Bluetooth receiver [1]) was the logitec mx air which was ahead of its day [2].
Since the move to full remote work at my company, I've been longing for whiteboard sessions. The best I could conjure up was to use an old iPad and load up a shared whiteboarding webapp in a browser on both iPad and my desktop, and then share screen from my desktop, using a cheapo stylus on the iPad to draw.
It's pretty good, but my iPad is super old and slow. Replacing it is not affordable. But a conventional web cam + a relatively low-tech stylus would be much better, and drop the need for an external device completely.
Readit News
When I played Elite: Dangerous I used a "hand on throttle and stick" (HOTAS) setup, along with foot pedals. I couldn't help but think that there must be a better way to control a spaceship: your ship can pitch, yaw, and roll in addition to being able to fire thrusters in 6 directions.
I wanted a handheld ship model that I could move such that the ship in Elite would move in the same way. The linked project looks like it could do just that. Thrust would be controlled in a similar way, but with my other hand.
Strange or new input models like that are so amazing to me. Our imagination can really fly high with these sorts of capabilities.
I think I played about the same, but it would’ve been funnier if I’d gotten creamed.
The theory was that my head is much more precise than my hands, so using a combination of both should improve my aim. It might sound like you’ll end up looking left and trying to see out of the corner of your eye, but in practice the two-mouse system ends up working out. I’ve always wanted to revive the idea for modern times, but you end up looking like a huge dork with a dildo glued to your head. I like that style though.
Your idea sounds way cool. You should make it!
https://en.m.wikipedia.org/wiki/SpaceOrb_360
https://www.tindie.com/products/vputz/orbotron-9001-version-...
It requires some old drivers that aren't officially supported (why would they remove support for perfectly good hardware from the newer drivers? To send good stuff to the landfill, of course!), but when Autodesk tried to move Fusion to the new driver model exclusively, user outcry persuaded them to leave the old drivers in as an option. Apparently there's quite a few of us using those SpacePilots, and the phrase "from my cold, dead fingers" comes up not infrequently.
Two sticks?
https://old.reddit.com/r/EliteDangerous/comments/16xi20a/dua...
https://youtu.be/T2-IHgNYaKA
I used a CH Pro throttle, which has enough hats to sort of mimic that 2 stick action. Main throttle back forward for main engine and retros. A smaller thumb stick/hat for each of Y and Z axis.
https://www.chproducts.com/Pro-Throttle-v13-d-719.html
Is it a track-ball, a rotary encoder, and a 3D stick?
15 years later and I'm still convinced that the SpaceOrb 360 is THE GREATEST GAME CONTROLER OF ALL TIME. See what I did with the all cap there? It's the G.O.A.T. and it's a tragedy that it lost the battle to "WASD" keyboard and mouse gaming and the "two thumbs" style of controller.
If you've ever tried a SpaceOrb, you probably already know this. If you didn't live through the SpaceOrb's hayday (around 1994-5 with the original Quake, and Descent, truly one of the greatest games ever) then this post will sound like I'm a crotchety old dude trying desperately to relive my the golden hour of my youth. Ok, so you got me. What of it? ;)
https://www.hanselman.com/blog/the-best-controller-for-fps-a...
Degrees of freedom: https://hanselmanblogcontent.azureedge.net/WindowsLiveWriter...
I had purchased a box of these when they were going away, and sponsored a custom driver. For a while there were other solutions available, like OrbShield:
// via SeedStudio: https://www.seeedstudio.com/Orbshield-v1-0-kit-p-671.html?qu...
// via Tindie: https://www.tindie.com/products/vputz/orbotron-9001-version-...
Then try staying in control with an unbalanced ship, or after you inevitably smashed one of the thrusters into a rock. It's a pretty space-nerd game too, they simulate details like reaction wheel saturation.
What would be the word for "To Kagi"? They use "Fetch" in the search field, but it hasn't really stuck with me.
Undergrad! If you didn't get top marks on this there is no justice
I do give a flying fuck about applicability and novelty, especially for an undergrad.
- Enable all sorts of new UX interactions (gestures with eye tracking)
- Enable all sorts of new peripheral interactions (stylus like this, but also things like a steering wheel for racing games)
- Enable 3D 180° filming for far more flexible webcam meetings, including VR presence, etc.
The idea of being able to use the entire 3D space in front of your computer display as an input method feels like it's coming, and using a webcam the way OP describes feels like it's a little step in that direction.
There are infrared depth cameras in various phones and laptop cameras now.
[VR] Motion controllers: https://en.wikipedia.org/wiki/Motion_controller#Gaming
Inertial navigation system: https://en.wikipedia.org/wiki/Inertial_navigation_system
Inertial measurement unit : https://en.wikipedia.org/wiki/Inertial_measurement_unit :
> An inertial measurement unit (IMU) is an electronic device that measures and reports a body's specific force, angular rate, and sometimes the orientation of the body, using a combination of accelerometers, gyroscopes, and sometimes magnetometers. When the magnetometer is included, IMUs are referred to as IMMUs.[1]
Moasure does displacement estimation with inertial measurement (in a mobile app w/ just accelerometer or also compass sensor data?) IIUC: https://www.moasure.com/
/? wireless gesture recognition RSSI: https://scholar.google.com/scholar?q=wireless+gesture+recogn...
/? wireless gesture recognition RSSI site:github.com : https://www.google.com/search?q=wireless+gesture+recognition...
Awesome-WiFi-CSI-Sensing > Indoor Localization: https://github.com/Marsrocky/Awesome-WiFi-CSI-Sensing#indoor...
3D Scanning > Technology, Applications: https://en.wikipedia.org/wiki/3D_scanning#Technology
Are there a limited set of possible-path-corresponding diffraction patterns that NIRS (Near-Infrared Spectroscopy) could sense and process to make e.g. a magic pencil with pressure sensitivity, too?
/q.hnlog "quantum navigation": https://news.ycombinator.com/item?id=36222625#36250019 :
> Quantum navigation maps such signal sources such that inexpensive sensors can achieve something like inertial navigation FWIU?
From https://news.ycombinator.com/context?id=36249897 :
> Can low-cost lasers and Rdyberg atoms e.g. Rydberg Technology solve for [space-based] matter-wave interferometry? [...] Does a fishing lure bobber on the water produce gravitational waves as part of the n-body gravitational wave fluid field, and how separable are the source wave components with e.g. Quantum Fourier Transform/or and other methods?
Because the digitizer
Even today some people are still on the fence about the new fangled mouse!
Some random-ish thoughts from exploring "a laptop keyboard... with hand pose, 3D stylus, and touch". Adding buttons yields a 3D mouse - but camera coverage can be a pain. Note the body is largely empty (and battery could be slimmed) - I could more-or-less type while holding a slim chopstick/toothbrush-like stylus (even with the markers, and a weirdly big tip). A big tip (sliced from a ping-pong ball sized xmas decoration) could slide fairly smoothly on a ThinkPad keyboard (and gave room for a less compact force sensor, and an extra tip marker). Thin stranded silicone ribbon cable can be string-like flexible - I just tethered the stylus to an arduino to get started.
Hmm... I wonder what the inertial sensor might make of something like a dimpled metal clicker, as a button (or three)?
The thing I was most surprised by was how effective my solution was, given that it's a pretty gross approximation of reality. There are lots of much more sophisticated techniques for dealing with it, which I didn't end up needing.
One thing I would've liked to try out is using rolling-shutter-aware PnP [1], which can theoretically estimate pose and velocity simultaneously from one image, by exploiting the rolling shutter distortions.
[1] https://www-sop.inria.fr/members/Philippe.Martinet/publis/20...
Btw the first 6DOF controller I had other (than a hacked WiiMote controller as a ir led Bluetooth receiver [1]) was the logitec mx air which was ahead of its day [2].
[1] https://web.cs.ucdavis.edu/~okreylos/ResDev/Wiimote/MainPage...
[2] https://www.cnet.com/reviews/logitech-mx-air-review/
2. Helpful documentation
3. Nice real world example for the use of a Kalman Filter!
Since the move to full remote work at my company, I've been longing for whiteboard sessions. The best I could conjure up was to use an old iPad and load up a shared whiteboarding webapp in a browser on both iPad and my desktop, and then share screen from my desktop, using a cheapo stylus on the iPad to draw.
It's pretty good, but my iPad is super old and slow. Replacing it is not affordable. But a conventional web cam + a relatively low-tech stylus would be much better, and drop the need for an external device completely.
[1] https://wacomstore.ca/product/one-by-wacom-small/