I'm begging you, please post a full-speed video that shows a seemingly random assortment of pieces sorted into boxes with identical color/size/shape pieces. I think it would be insanely satisfying for many people to watch. I would literally grab popcorn and watch it for hours (and I think many others would too).
Given your current setup, it might be easiest to just take one bin of identical pieces and then sort it by color into various bins. I'm salivating over what this would look like.
If you posted that video, I bet it would rack up millions of YouTube views.
Ok, will do but first I have to speed up that first feeder stage because I really don't want to stand there for an hour to drop pieces on the recognition belt.
Any good ideas for that first stage are very welcome. I've looked at all kinds of mechanism (belts, pickers, vibrating drums) and none so far have the required speed, regularity and are quiet enough to be used in a home.
A combination of pins and holes will get you roughly sized items.
A vibrating table with lots of W snapped channels underneath is going to help too.
A belt with "scoops" is a great way to feed out of your initial hopper, if you drop it onto a perpendicular belt (one with W grooves) your going to get a good initial separation more naturally.
Air knives and puffers are your friend as well. Since you already have a detection system running it twice might be of benefit to you, just puff off anything that you can't sort as a single item.
I don't think "quite" and "lego" really go together so no happy solution for you there other than sound isolation.
What about a big subwoofer for a car, and you can try driving it with random sound waves until you find one that works. Or go through a set of tones that can hit the resonant frequencies of various pieces?
Massive hopper (non-tapered if the tapering causes jams), feeding into slowly rotating (60 rpm) spiral-inner drum, falling onto very fast moving (10 feet per second) belt.
Parts get spread into a ~2 inch thick layer inside the drum, and then falling (at least a foot) onto the fast moving belt will then likely separate them further. For parts that end up entangled, end up on the belt too close together, or are unrecognisable in the orientation they fall, simply redirect them back into the top of the hopper to be resorted.
Even a 50% successful sort rate should be good enough.
Hmm. Tough problem. What about a couple of "tables" which vibrate back and forth at a really shallow angle, just enough to move pieces in one direction? I have gravel sorting tables with screens in mind, but the same principle with a solid table to just break down a heap into a flat layer may just work.
I think that a couple of tables, cones, and dividers (to create lanes of pieces) could do a reasonable job of turning a heap of pieces into a steady stream of individual pieces.
> The next morning I woke up to a rather large number of emails congratulating me on having won almost every bid (lesson 1: if you win almost all bids you are bidding too high)
That reminds me of the economist George Stigler's observation: "If you never miss a plane, you're spending too much time at the airport".
That's a strange observation. I would think an economist of all people would know that the negative utility of missing a plane is huge compared to that of spending a little unnecessary time at the airport. I'm rarely doing anything productive with the time prior to the airport arrival on "travel days" anyway.
It's an analogy for understanding the idea of fair value. If you want to spend a "fair" amount of time at the airport, it means that you need to balance the utility of missing one plane against the utility of always spend a lot of time at the airport in order to never miss a plane.
If you take 1000 planes in your life, and spend an extra 1 hour per plane in order to be sure that you will be on time, that's 1000 hours in your life. If, instead, you cut down that hour to 30 minutes, and that makes you miss 3 planes in your life, you'd still have "earned" 500 hours, while losing 3 planes. It's a better trade off overall.
But yeah, just an analogy, I bet the guy never missed a plane...
A few years ago a friend and I started work on a similar machine for sorting Magic: the Gathering cards. We weren't using any machine learning (this was beyond our capabilities) but our design was similar, except the bins were on a moving platform which slowed down as the weight of the cards in bins increased - due to the cheap parts/weakness of the motors we were using. Unfortunately it occupied an entire room in my house and was never completed and ultimately dismantled.
That's really interesting actually. In the distant past I spent more time sorting the cards than playing the game so there is probably a good trade off on automating it.
I went and bought a duel deck the other day to play with the wife so this is going to become another issue in the future again.
There are card identifier apps that connect to your webcam - you just flash a card under the camera and it will identify the card and set (used most often to give a quick market price or help you catalog your collection digitally) but you could definitely fit that into a larger automation project.
Yeah I remember seeing these for the first time not long ago. We were scraping magiccards.info without the owner's permission to use color coherence vector for comparison. As software engineers this was the really fun stuff for us - trying to figure out computer vision problems having absolutely no experience.
We were trying to it to sort by set at first - when ingesting a large collection of cards (my wife owned a brick and mortar card shop) they were often unsorted. Then with multiple passes we wanted to pull rarity, color and even alphabetize uncommons and rares [way, way too many passes]. The real goal was to be able to process a booster box quickly, so when a new set was released 40 cases in singles could be available very quickly - never even came close to that goal. It turns out that on release night customers are always willing to sort a booster box for you for a pack of cards...
A while back, I sorted my childhood LEGO pile before sending it to my brother. 73 pounds of LEGO sorted by hand. I'm sure it says things about how strangely wired my brain is, but this was an eminently enjoyable endeavor.
Sorting into sets would be a lot harder. Just grouping by similar pieces is a fairly fast, automatic process. I could just sort of zone out and run on pleasant zen autopilot. I didn't need to keep running counts, etc.
I found that by looking for pieces with certain shapes, my brain/eyes would lock in after a little while and it got really easy to spot them as I sifted through the pile.
Sorting things by hand is very relaxing and built in to us humans. I sort perls with my daughter when we make those perl-plates that you can melt with a hot iron. Or puzzle pieces, lego, whatever:)
An idea: I have a three-month-old daughter who I'll definitely be buying Lego for when she gets old enough. What I'll be looking for is whatever's the least valuable in your collection -- big bins of cheap mixed stuff to see what she invents. Maybe that would be a cool side output for your project? Cheap chaff packs for kids of hackers ...
Oh that's a great idea. Thank you! I have garbage bags full of 2x4, 2x3 and 2x2 bricks in common colors that are worth relatively little. I'll ask my panel of experts (5 and 7 years old) to see what they would do with such a treasure trove.
I second that- when I was a kid I had tons of common bricks (1- and 2-by-N's) and regularly built decent-sized houses with full-height walls and multiple rooms and such. My 5 year-old daughter recently graduated from Duplo to Lego and I don't see a good way of building up a decent foundation of just "normal" bricks from which she can build lots of things. I definitely feel like there's a lack of sets with good quantities of them. Even the large "Classic" sets are pretty light on them compared to other pieces, and what they do have is spread out over like 15 colors, so you can forget about trying to make anything match.
I've looked on eBay, but most of what I see in bulk there are lots of just one size or one color bricks, or just pounds of completely random pieces. I'd love it if I could find something like an assortment of 1,000 mixed 1-by-1/2/3/4/6/8 bricks in 4-5 colors, or the same with 2-by-N bricks.
Not a fanatic builder at all, been almost two decades since I built something with Lego, but really surprised to see that the categories are not color separated.
If you're building any esthetic Lego project would have color restrictions. Also I imagine some colors are worth more than others, just because they're used in more projects.
Assuming there's a practical limit to the number of sorting categories that rules out separation by color and function, I'd much rather have them separated by function as shown.
Is it easier to find the functional piece you need in a bin full of same-colored pieces, or to find the color you need in a bin full of same-function pieces? For me, the latter would be far faster/easier.
That said, I do agree that individual pieces labeled by function and color is probably ideal for sale to buyers looking for a single elusive piece (or groups of pieces).
I had mine separated out by type (function) as you describe, and unsorted by color. That worked, and would continue to do so until the collection reached a scale far greater than anything I ever contemplated.
You're right about the need for individual separation when selling parts. Bricklink does that, so that people in need of, say, exactly four orange cheese wedges, can go and buy them. That seems to be the preferred style for most builders, since it makes possible a mapping from a parts list to a purchase order; there are other styles in use, including bulk sale by mass or approximate quantity, but the individual price for a given part is much lower when sold in bulk than alone. So if it can be made feasible to automatically, or mostly automatically, sort down to the individual part level by type and color, that'll show the best return on investment per part at the point of sale.
(Update: based on jacquesm's comments elsethread, I don't suppose I can any longer recommend selling on Bricklink. But I'd imagine the pricing effects I describe are similar elsewhere.)
Why not go big? Write up a business plan that involves buying 100 tons of lego, making a bigger machine with 200 bins off the belt, or whatever, and making a new brickowl / bricklink site of your own. I'm sure you could find investors, you've got great publicity already. And you could propose it as a general sorting AI (or web sale of small item) firm in the long term, with the lego being your initial application.
You'd need to solve the inverse problem of how to efficiently go from bin to packing container, which doesn't seem too hard. Rotating carousels of bins with little robots or something. Set the item on a tray, and if isn't the right item, puff it back into the bin.
If it won't work small it won't work big. First learn to walk. If I can overcome all the challenges and there is a sufficiently large market for the end product and when all the costs (energy, for instance) have been accounted for there is still a business case left then this might be worth it. But there is a ton of work to be done before you could begin to answer those questions. For now it is a fun project and I'm trying hard to keep it that way. I've been approached by some of the larger bulk resellers that would love to get rid of the humans they employ, none of them made a good enough case why I should partner with them or even seemed to be nice people.
The general nature of neural nets for sorting and classification I think is well recognized in industry and there are many companies that are capitalizing on this (and have been for some time). I highly doubt I'm doing anything original in that sense.
Are there CAD models available for the various Lego components? I'm wondering if, given such models, you could use rendering software to generate labeled training images. That could allow you to get a large training set with many images for each kind of component in many different orientations, even for components you do not yet have.
If the CAD models are not available, how much work is it to make a CAD model given an instance of a particular component? I'd expect that for some components, such as those NxM plates, you'd only need to build one model by hand in CAD software, and then could algorithmically generate the models for other sizes.
I don't know CAD models, but it would probably make sense to scrape the lego website where you can select all their bricks.
Each named brick has 8 images (various orientations):
...This also makes me wonder whether it would make sense to set up multiple cameras, to get multiple orientations for every block. Use the image that has the lowest classification error, but use all orientations to train the network.
The images are not very useful to me because the images the machine takes are very different from those. But maybe it would somehow contribute. I'd be very ware of using other people's copyrighted content though.
Readit News
Given your current setup, it might be easiest to just take one bin of identical pieces and then sort it by color into various bins. I'm salivating over what this would look like.
If you posted that video, I bet it would rack up millions of YouTube views.
Any good ideas for that first stage are very welcome. I've looked at all kinds of mechanism (belts, pickers, vibrating drums) and none so far have the required speed, regularity and are quiet enough to be used in a home.
A combination of pins and holes will get you roughly sized items.
A vibrating table with lots of W snapped channels underneath is going to help too.
A belt with "scoops" is a great way to feed out of your initial hopper, if you drop it onto a perpendicular belt (one with W grooves) your going to get a good initial separation more naturally.
Air knives and puffers are your friend as well. Since you already have a detection system running it twice might be of benefit to you, just puff off anything that you can't sort as a single item.
I don't think "quite" and "lego" really go together so no happy solution for you there other than sound isolation.
Massive hopper (non-tapered if the tapering causes jams), feeding into slowly rotating (60 rpm) spiral-inner drum, falling onto very fast moving (10 feet per second) belt.
Parts get spread into a ~2 inch thick layer inside the drum, and then falling (at least a foot) onto the fast moving belt will then likely separate them further. For parts that end up entangled, end up on the belt too close together, or are unrecognisable in the orientation they fall, simply redirect them back into the top of the hopper to be resorted.
Even a 50% successful sort rate should be good enough.
I think that a couple of tables, cones, and dividers (to create lanes of pieces) could do a reasonable job of turning a heap of pieces into a steady stream of individual pieces.
That reminds me of the economist George Stigler's observation: "If you never miss a plane, you're spending too much time at the airport".
If you take 1000 planes in your life, and spend an extra 1 hour per plane in order to be sure that you will be on time, that's 1000 hours in your life. If, instead, you cut down that hour to 30 minutes, and that makes you miss 3 planes in your life, you'd still have "earned" 500 hours, while losing 3 planes. It's a better trade off overall.
But yeah, just an analogy, I bet the guy never missed a plane...
I went and bought a duel deck the other day to play with the wife so this is going to become another issue in the future again.
http://imgur.com/a/OJaq6
A while back, I sorted my childhood LEGO pile before sending it to my brother. 73 pounds of LEGO sorted by hand. I'm sure it says things about how strangely wired my brain is, but this was an eminently enjoyable endeavor.
I found that by looking for pieces with certain shapes, my brain/eyes would lock in after a little while and it got really easy to spot them as I sifted through the pile.
An idea: I have a three-month-old daughter who I'll definitely be buying Lego for when she gets old enough. What I'll be looking for is whatever's the least valuable in your collection -- big bins of cheap mixed stuff to see what she invents. Maybe that would be a cool side output for your project? Cheap chaff packs for kids of hackers ...
I've looked on eBay, but most of what I see in bulk there are lots of just one size or one color bricks, or just pounds of completely random pieces. I'd love it if I could find something like an assortment of 1,000 mixed 1-by-1/2/3/4/6/8 bricks in 4-5 colors, or the same with 2-by-N bricks.
If you're building any esthetic Lego project would have color restrictions. Also I imagine some colors are worth more than others, just because they're used in more projects.
Also, color separated Lego bins just look pretty.
Is it easier to find the functional piece you need in a bin full of same-colored pieces, or to find the color you need in a bin full of same-function pieces? For me, the latter would be far faster/easier.
That said, I do agree that individual pieces labeled by function and color is probably ideal for sale to buyers looking for a single elusive piece (or groups of pieces).
You're right about the need for individual separation when selling parts. Bricklink does that, so that people in need of, say, exactly four orange cheese wedges, can go and buy them. That seems to be the preferred style for most builders, since it makes possible a mapping from a parts list to a purchase order; there are other styles in use, including bulk sale by mass or approximate quantity, but the individual price for a given part is much lower when sold in bulk than alone. So if it can be made feasible to automatically, or mostly automatically, sort down to the individual part level by type and color, that'll show the best return on investment per part at the point of sale.
(Update: based on jacquesm's comments elsethread, I don't suppose I can any longer recommend selling on Bricklink. But I'd imagine the pricing effects I describe are similar elsewhere.)
If you have a bin of the same pieces in various color, you just pick the color you want.
You'd need to solve the inverse problem of how to efficiently go from bin to packing container, which doesn't seem too hard. Rotating carousels of bins with little robots or something. Set the item on a tray, and if isn't the right item, puff it back into the bin.
The general nature of neural nets for sorting and classification I think is well recognized in industry and there are many companies that are capitalizing on this (and have been for some time). I highly doubt I'm doing anything original in that sense.
What does anyone else think? Should Jacques open it up for investment?
If the CAD models are not available, how much work is it to make a CAD model given an instance of a particular component? I'd expect that for some components, such as those NxM plates, you'd only need to build one model by hand in CAD software, and then could algorithmically generate the models for other sizes.
https://shop.lego.com/Pick-a-Brick
I find it hard to believe that there are only 1421 bricks, including color combinations.
Example: "Apple with leaf", Color Family: Dark Green, Exact Color: Bright green, Category: Foodstuff Element ID: 4107050, Design ID: 33051, orientation 8: https://sh-s7-live-s.legocdn.com/is/image/LEGOPCS/4107050_s8
...This also makes me wonder whether it would make sense to set up multiple cameras, to get multiple orientations for every block. Use the image that has the lowest classification error, but use all orientations to train the network.
The images are not very useful to me because the images the machine takes are very different from those. But maybe it would somehow contribute. I'd be very ware of using other people's copyrighted content though.
I do use multiple orientations.
I believe he mentioned somewhere that he's using mirrors to achieve essentially the same effect with a single camera.