> But if I had any kind of point, it would probably be that spending the time to do things like write tools and explore the debugging space is pretty much always worth it.
> Strangely, I've found that there is a not-insignificant number of people are against this - especially in a professional environment. I think this has something to do with the JIRA-fication of development process, where all work is divided and sub-divided, and anything that does not directly check a box and produce a deliverable is wasted effort. It's disheartening that there is a whole group of people - developers and managers alike - who do not understand or see the value in exploration as part of the process.
So true, being able to create your own small tools is a superpower which is often at the heart of 10x programmers. It is sadly an art often practiced in the shadow.
I've run into this as well. There's a cog in the wheel developer and ... they have their place, and in many orgs they're the only thing that works.
But if you're not that person (and I like to think I'm not) and you are curious and stop to think about second order effects, future features that are a little obvious, and so on ... it's really painful working with those developers who do not.
I worked with one recently, not a bad person at all, but now I'm cleaning up his project that 100% did exactly what the tickets told him to do. And the project that he worked on that was designed to replace a piece of software that had issues ... has every single issue the old one had for all the same reasons (someone coded it in the easiest possible way + only what the ticket told them to).
So frustrating.
I will say this though, I find the use of the term "10x developer" a sort of buzzword smell for the kind of reckless developer with no oversight who "gets the job done" at great cost ... because they can do what they want and then everything is siloed and potentially a disaster when anyone has to touch it or they bolt when it becomes clear their tower of cards is ready to come down.
Not to say I disagree with your statement generally, just that term tends to make me worry.
I'd say that there are two kinds of 10x devs, as per who is doing the determination of '10x':
1. From the perspective of the money folks (managers & up): they get their jobs done very quickly, thus spending minimal time, meaning minimal money.
2. From the perspective of excellent systems design folks: they design an excellent system that addresses subtle problems and foresees and addresses system problems that will occur down the road.
Type #2 folks tend to not add to the org's technical debt; Type #1 folks don't GAF about such considerations -- they meet their quotas, get their perf reviews, and are on their way to their next promotion/position/project.
I'd say there is excellence for excellence's sake, and then there's money-making excellence. Sure, orgs need to make money (or at least break even) to survive, but there's a lot of overlap where being too much one way or the other will be an impediment to long-term viability.
I'm kind of in that boat as a "reckless developer". I come in and write little tools to help people workflows, automate something that was a manual process or a shell script to get the job done instead of doing it by hand. Some of these scripts can grow into big projects over time depending on if I'm the end user or someone else. No one asks me to make these things, I just see a issue to resolve and I resolve it. I like to call my self a hacker really since it makes sense in the old terminology of what I do with the many hats that i wear.
> if you're not that person and you are curious and stop to think about second order effects, future features that are a little obvious, and so on ... it's really painful working with those developers who do not.
Not only that, but it's also really painful working in a system which doesn't really value your artwork because you're not on the product team, and shouldn't be making product decisions.
I understand your concerns around 10x Dev. May I suggest a different term more specific to this discussion? "Tool builder", as in, one who builds tools for them self, and possibly shares with others. I have worked with programmers that were not outstanding in terms of pure computer science, but could build a tool or two to get leverage, especially around system transparency/debugging, etc.
> It's disheartening that there is a whole group of people - developers and managers alike - who do not understand or see the value in exploration as part of the process.
To steelman, because, as Frank is sending ethernet packets, we're stuck picking up the slack, doing things that are part of the job description, that are needed by the org as a whole. Why doesn't he just innovate in the actual problem we're working on, since there's a near infinity backlog?
I think, ideally, everyone is given exploration time, but that requires a manager that can strictly defend it, even when upper management is made aware of it and inevitably says "we need <project> done sooner". It's also a problem when other managers become aware of it with "You're allowing that team to hire more to compensate for the "lost time"? We need those people!". It really needs to be an org level thing, which even Google gave up on [1].
"Unethical" solution: Pad your development time to include some of it.
You may be on a team that has decided "developer burnout" is just an inevitable and acceptable cost of business.
> since there's a near infinity backlog?
Which is a problem in and of itself. In any case I'm only going to be able to give you like 4, real, solid hours of work a day on that log. The other 4 will be team coordination, managing that log, and stress management maybe while I hate eat my lunch.
> everyone is given exploration time
Call it "skill investment time." We're in a fast moving industry and keeping heads down for too long is destructive personally and organizationally. It's also the pathway to getting more than the basic level of engagement above.
10x developers (sweeping generalization here) are often not the ones that are beholden to a manager / product owner telling them what has the highest priority though. Whether that's because they won't be told what to do, are the manager themselves, or have no manager at all is not known to me.
That said, writing your own tools if applicable is useful, however I'd argue that a productive developer writes the least code and instead uses what's off the shelf. To use this article as an example, nobody should be writing their own TCP stack, there are perfectly fine existing implementations.
That said that said, writing your own stuff is probably the best way to get a deep understanding, and a deep understanding of things is a constituent part of the mythical 10x developer. I just wouldn't expect any employer to pay you for it.
A good manager recognizes that a good 10x developer needs minimal management. Not that they don't need managing, but that too much management throws a wrench in the mental gears and drags them down, burns them out, and forces them to quit. All they need is a light hand on the steering wheel to keep them pointed at the right problems.
But sibling comment is also right. The 10x developer has more "free" time outside of their Jira tickets. Some choose to focus on the next business problem, others drift and experiment.
Either way the business problem gets solved and you retain a very skilled employee. Their explorations may turn up something valuable, or just add to their own personal dragon hoard of skills, when then usually still benefits you in the end
The reason a “10x developer” can work on whatever they want is because it only takes 10% of their work hours to complete their job requirements- they are then free to experiment and play with a lot of their time.
"Wow that's really helpful. Your tool confirmed our model."
And
"Did you ask the PM first? You have to be careful with rabbit holes."
I usually just do it if it'll be under a day's worth of work. It's never gone wrong and never gone to waste. At worst I and up copy/pasting the code into something else later
Recently at work I developed a small suite tools (in a mixture of python and shell, running in WSL) that left my boss impressed when he saw me debugging a customers system (IOT).
Then he started asking me to make them accessible to non programmers, and suddenly those tools seemed a lot more than I bargained for.
I'm not sure it's Jira-fication. I think it's mapping effort back to something that has a direct result to the bottom line.
Example:
I used to work for a company that basically made slot machines. We needed to produce something called parsheets to give to submit to get our slots certified and to the casino. I was given the project but no manager wanted to give any developers to implement it. No one wanted to give the money to buy a computer with enough power to run simulations to produce the theoretical hold. But this was something we NEEDED to help get our games approved.
Since the title is rather vague, this is the start of a series about building a TCP/IP and Ethernet framing stack from scratch for a microcontroller. The author uses a chip (W5100) which can handle TCP/IP itself, but also supports handing it pre-built Ethernet frames, although the chip handles the preamble and CRC calculation. Most of the article is about trying to communicate with the chip itself, and sending a test packet (which I'm guessing is hardcoded, although it's not called out in the article).
(I was hoping it would be about bit-banging Ethernet on some improbable bit of hardware.)
The RP2040 can be convinced to bit-bang Ethernet (yes even 100Mb/s) with only a transceiver. Part of the trick is to modify the common PHY + MagJack breakout boards to let the RP2040 generate the clock signal so that the signals are in sync and you don't have to oversample the RMII.
Just 10Mb/s can be done even dirtier. I'm still waiting for someone to combine an RP2040 with DVI/HDMI video output and Ethernet into a "modern" glass terminal (just telnet, SSH is probably asking too much).
VGA is much easier to produce, and the RP2040 can do 1280x1024@60 no problem. The official examples use several clock cycles per pixel (2 IIRC, but it might be even more), but you don't have to.
I made half a terminal with that output, aiming for something that ran the original VT100 ROM. I never finished the emulator part, but I did write video output (including scan line emulation) for VT100 memory -> VGA. Adding SSH once the rest works should be perfectly possible. (Not bit banging ethernet at the same time, but using some external chip.)
I should probably put that online somewhere. Or finish it and then put it online.
I took a strange career jump recently into FPGA engineering with a focus on Ethernet. It's been a fun journey, which culminated in me finally designing my own Hard MAC IP and sending a packet over some custom PHY IP. I'd highly recommend it for those looking to try a "Hard Mode" version of this challenge. I feel like networking is very abstracted from users, so understanding how Ethernet cards, modems and switches put together and pull apart sections of packets, and how the PHY/PCS recovers signals over a link was really valuable.
A while ago (about a decade, oh no) I was involved in a project which did direct TCP from an FPGA. As part of debugging and verification, I made it so you could hook up the simulated version from Verilator to the Linux TUN/TAP device, enabling you to connect directly into it from the developer machine without tying up the physical hardware (which we only had one of). Fun project.
Sounds fun! Were you already familiar with networking/Ethernet when you started? If not, which resources - if any - did you use to get a broad overview of everything involved? My knowledge of networking ends at a very basic understanding of the OSI model, and I am very interested in taking a deep dive into the networking world.
I'll tell you how I learned, I got thrown head-first into a network switch design. I knew almost nothing about networking.
The most useful resource while I was learning was RFC1812 "Requirements for IPv4 Routers" [1]
Its an ancient document written the same year I was born detailing how future routers should be built on this relatively new thing called the internet. The language is highly approachable and detailed, often explaining WHY things are done. It is an awesome read.
To be honest you don't need to finish it. I only read the first few chapters, but I googled EVERYTHING I did not understand. The first few paragraphs took several hours. Talk to LLM's if you need a concept explained. Take notes. In a few days you'll have a very solid grasp.
Wireshark, plus a book about using Wireshark. I have "Wireshark Network Analysis", which looks to be shockingly expensive right now - to be fair, it's worth whatever price, but maybe not while you're speculativey getting your toes wet - and there are others out there. I am nowhere near an expert (I've only worked through about 1/3 of that book - just enough to solve the problem I had at the time!), but that's what's taught me what I know, and I know can take me further.
I wasn't at all, to be honest. I joined a multidisciplinary team and they were short an Ethernet SME. I said I'd be happy to just in feet first and learn, as I was desperate to get into anything and everything FPGAs at the time, having become bored by the CPU world I resided before. Mostly I learnt everything through building stuff using our User Guides. Everything else was just reading; the Ethernet spec, random Cisco stuff online, Wikipedia, etc. I just searched for any word or acronym I didn't understand, and read up on it (meaning of 64/66b encoding, PCS vs PHY vs MAC, PTP, OTN, FlexE, FEC, ANLT, and so on). Once you get passed the TLAs, it's actually pretty simple (except for PTP - that's the bane or my working life at the moment).
First time seeing the retcons for MOSI/MISO: main out/subordinate in instead of master out/slave in. I'll use that so I can still refer to the pins as MOSI/MISO. The COPI/CIPO alternative (controller out/peripheral in) never worked themselves into my brain stem properly.
I'm baffled as to why the author is using STM32F401 with a W5100 Ethernet shield when they could just as easily use an STM32F407 board, which includes a built-in Ethernet MAC, coupled with a cheap (2 for ~$12) Ethernet PHY board. Lots of example Ethernet projects available and just as easy to develop for as the STM32F401.
Also,
> Due to the complexity of the signalling involved with ethernet, a dedicated ASIC is generally used
In the context of microcontrollers, I think this is generally not true. In most cases Ethernet functionality is incorporated as a embedded peripheral within the microcontroller itself, as it is with the STM32F407 and the ESP32 (both of which happen to use the same Ethernet peripheral sourced from an external provider).
Author here - the reason is pretty underwhelming: These are the parts I had on hand when I decided to start on the project. Using a chip with a builtin Ethernet peripheral would definitely make more sense (though I'd be trading any complexity of configuring the W5100 for the complexity of configuring STs peripheral). The networking code already abstract the actual chip itself into a driver interface (think read/write/ioctl), so porting the code would be pretty straightforward.
I'll look into the STM32F407 for the main series. Thanks
> though I'd be trading any complexity of configuring the W5100 for the complexity of configuring STs peripheral
It's not too bad as far as such things go. The documentation on how the DMA system works leaves something to be desired, but it's not bad (and it's a heck of a lot faster than spitting packets over SPI).
> for some people the fun is just in learning, not necessarily doing the state of the art things.
Over a decade ago, when I was just learning Linux, I set out on a quest to turn a CentOS box into a router that could perform NAT between two different networks. I spent an entire weekend researching and following every suggestion I could find until it finally worked. I was so proud when my pings reached their respective destinations.
I took it apart the next day and never did anything more with it, but the journey and the reward was the fun of it.
Totally get and applaud that. Indeed, I built a high-performance Ethernet driver for the STM32F4 chips just because I wanted to say I've done it.
But his stated goal is to build a TCP/IP stack, not futz around with SPI and the particulars of an idiosyncratic network chip. There will be plenty of work (and learning) to do once he starts climbing up the network stack.
You must be new here. It's a law of nature around HN that whatever you post that you've done, there's some twit who will condescendingly lecture you on how you should have done it.
RFC 791 which defines IP talks about both packets and datagrams. IP sends datagrams. Each IP datagram is fragmented into one or more packets depending on the underlying L2 network.
But RFC 791 dates from when the underlying L2 network was most likely ARPAnet, which had 128 byte packets. Today the distinction is (or should be) moot - if you're relying on IP fragmentation to send your large IP datagram over multiple L2 packets, you're doing something wrong. IPv6 doesn't even support in-network fragmentation.
In practice, the distinction has largely disappeared over time. Everyone I know talks about packets and very rarely bothers to distinguish between frames, datagrams and packets, because in practice one Ethernet frame generally carries one IP datagram, and everyone calls it a packet.
Only on HN could someone implement networking from scratch, only to be dismissed by someone in the comments implying they've no idea what they're doing.
If you've got a hankering for wired ethernet on a microcontroller, several of the larger STM32 Nucleo boards [1] have 100Mbps ethernet built in - and at ~$25 they're pretty affordable.
Their 'STM32Cube' software gets mixed reviews, but it will spit out a working example of ethernet communication.
WT32-ETH01 about $7 on Aliexpress if you want something ESP32 based. As easy or easier to get started with than the STM parts in my opinion. Comes with MQTT, HTTP clients & servers etc.
When I last used Cube-MX, it was a very unpleasant experience throughout. I'd use stm32-hal or libopencm3 out of preference if I was using the parts again. The tool itself and the code it spat out had all sorts of nasty bugs and edge cases that cost days of debugging. Maybe it's improved since.
note that if you want to play with writing your own network stack on linux, you can use `socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))` and operate on not too dissimilar level of abstraction as this is.
SOCK_RAW packets are passed to and from the device driver without
any changes in the packet data. When receiving a packet, the
address is still parsed and passed in a standard sockaddr_ll
address structure. When transmitting a packet, the user-supplied
buffer should contain the physical-layer header. That packet is
then queued unmodified to the network driver of the interface
defined by the destination address.
And to do it from the other direction, you can open a tun (virtual IP) or tap (virtual Ethernet) interface just as easily. This adds a virtual interface to the network stack (e.g. a VPN interface), while a packet socket does the opposite and lets you communicate directly through with a real interface.
Readit News
> Strangely, I've found that there is a not-insignificant number of people are against this - especially in a professional environment. I think this has something to do with the JIRA-fication of development process, where all work is divided and sub-divided, and anything that does not directly check a box and produce a deliverable is wasted effort. It's disheartening that there is a whole group of people - developers and managers alike - who do not understand or see the value in exploration as part of the process.
So true, being able to create your own small tools is a superpower which is often at the heart of 10x programmers. It is sadly an art often practiced in the shadow.
But if you're not that person (and I like to think I'm not) and you are curious and stop to think about second order effects, future features that are a little obvious, and so on ... it's really painful working with those developers who do not.
I worked with one recently, not a bad person at all, but now I'm cleaning up his project that 100% did exactly what the tickets told him to do. And the project that he worked on that was designed to replace a piece of software that had issues ... has every single issue the old one had for all the same reasons (someone coded it in the easiest possible way + only what the ticket told them to).
So frustrating.
I will say this though, I find the use of the term "10x developer" a sort of buzzword smell for the kind of reckless developer with no oversight who "gets the job done" at great cost ... because they can do what they want and then everything is siloed and potentially a disaster when anyone has to touch it or they bolt when it becomes clear their tower of cards is ready to come down.
Not to say I disagree with your statement generally, just that term tends to make me worry.
1. From the perspective of the money folks (managers & up): they get their jobs done very quickly, thus spending minimal time, meaning minimal money.
2. From the perspective of excellent systems design folks: they design an excellent system that addresses subtle problems and foresees and addresses system problems that will occur down the road.
Type #2 folks tend to not add to the org's technical debt; Type #1 folks don't GAF about such considerations -- they meet their quotas, get their perf reviews, and are on their way to their next promotion/position/project.
I'd say there is excellence for excellence's sake, and then there's money-making excellence. Sure, orgs need to make money (or at least break even) to survive, but there's a lot of overlap where being too much one way or the other will be an impediment to long-term viability.
Not only that, but it's also really painful working in a system which doesn't really value your artwork because you're not on the product team, and shouldn't be making product decisions.
To steelman, because, as Frank is sending ethernet packets, we're stuck picking up the slack, doing things that are part of the job description, that are needed by the org as a whole. Why doesn't he just innovate in the actual problem we're working on, since there's a near infinity backlog?
I think, ideally, everyone is given exploration time, but that requires a manager that can strictly defend it, even when upper management is made aware of it and inevitably says "we need <project> done sooner". It's also a problem when other managers become aware of it with "You're allowing that team to hire more to compensate for the "lost time"? We need those people!". It really needs to be an org level thing, which even Google gave up on [1].
"Unethical" solution: Pad your development time to include some of it.
[1] https://hrzone.com/why-did-google-abandon-20-time-for-innova...
You may be on a team that has decided "developer burnout" is just an inevitable and acceptable cost of business.
> since there's a near infinity backlog?
Which is a problem in and of itself. In any case I'm only going to be able to give you like 4, real, solid hours of work a day on that log. The other 4 will be team coordination, managing that log, and stress management maybe while I hate eat my lunch.
> everyone is given exploration time
Call it "skill investment time." We're in a fast moving industry and keeping heads down for too long is destructive personally and organizationally. It's also the pathway to getting more than the basic level of engagement above.
That said, writing your own tools if applicable is useful, however I'd argue that a productive developer writes the least code and instead uses what's off the shelf. To use this article as an example, nobody should be writing their own TCP stack, there are perfectly fine existing implementations.
That said that said, writing your own stuff is probably the best way to get a deep understanding, and a deep understanding of things is a constituent part of the mythical 10x developer. I just wouldn't expect any employer to pay you for it.
But sibling comment is also right. The 10x developer has more "free" time outside of their Jira tickets. Some choose to focus on the next business problem, others drift and experiment.
Either way the business problem gets solved and you retain a very skilled employee. Their explorations may turn up something valuable, or just add to their own personal dragon hoard of skills, when then usually still benefits you in the end
"Wow that's really helpful. Your tool confirmed our model."
And
"Did you ask the PM first? You have to be careful with rabbit holes."
I usually just do it if it'll be under a day's worth of work. It's never gone wrong and never gone to waste. At worst I and up copy/pasting the code into something else later
Then he started asking me to make them accessible to non programmers, and suddenly those tools seemed a lot more than I bargained for.
Example: I used to work for a company that basically made slot machines. We needed to produce something called parsheets to give to submit to get our slots certified and to the casino. I was given the project but no manager wanted to give any developers to implement it. No one wanted to give the money to buy a computer with enough power to run simulations to produce the theoretical hold. But this was something we NEEDED to help get our games approved.
(I was hoping it would be about bit-banging Ethernet on some improbable bit of hardware.)
Just 10Mb/s can be done even dirtier. I'm still waiting for someone to combine an RP2040 with DVI/HDMI video output and Ethernet into a "modern" glass terminal (just telnet, SSH is probably asking too much).
I made half a terminal with that output, aiming for something that ran the original VT100 ROM. I never finished the emulator part, but I did write video output (including scan line emulation) for VT100 memory -> VGA. Adding SSH once the rest works should be perfectly possible. (Not bit banging ethernet at the same time, but using some external chip.)
I should probably put that online somewhere. Or finish it and then put it online.
https://hackaday.com/2014/08/29/bit-banging-ethernet-on-an-a...
The most useful resource while I was learning was RFC1812 "Requirements for IPv4 Routers" [1]
Its an ancient document written the same year I was born detailing how future routers should be built on this relatively new thing called the internet. The language is highly approachable and detailed, often explaining WHY things are done. It is an awesome read.
To be honest you don't need to finish it. I only read the first few chapters, but I googled EVERYTHING I did not understand. The first few paragraphs took several hours. Talk to LLM's if you need a concept explained. Take notes. In a few days you'll have a very solid grasp.
[1] https://datatracker.ietf.org/doc/html/rfc1812
Also,
> Due to the complexity of the signalling involved with ethernet, a dedicated ASIC is generally used
In the context of microcontrollers, I think this is generally not true. In most cases Ethernet functionality is incorporated as a embedded peripheral within the microcontroller itself, as it is with the STM32F407 and the ESP32 (both of which happen to use the same Ethernet peripheral sourced from an external provider).
I'll look into the STM32F407 for the main series. Thanks
It's not too bad as far as such things go. The documentation on how the DMA system works leaves something to be desired, but it's not bad (and it's a heck of a lot faster than spitting packets over SPI).
Over a decade ago, when I was just learning Linux, I set out on a quest to turn a CentOS box into a router that could perform NAT between two different networks. I spent an entire weekend researching and following every suggestion I could find until it finally worked. I was so proud when my pings reached their respective destinations.
I took it apart the next day and never did anything more with it, but the journey and the reward was the fun of it.
But his stated goal is to build a TCP/IP stack, not futz around with SPI and the particulars of an idiosyncratic network chip. There will be plenty of work (and learning) to do once he starts climbing up the network stack.
Packets are an IP concept :)
But RFC 791 dates from when the underlying L2 network was most likely ARPAnet, which had 128 byte packets. Today the distinction is (or should be) moot - if you're relying on IP fragmentation to send your large IP datagram over multiple L2 packets, you're doing something wrong. IPv6 doesn't even support in-network fragmentation.
In practice, the distinction has largely disappeared over time. Everyone I know talks about packets and very rarely bothers to distinguish between frames, datagrams and packets, because in practice one Ethernet frame generally carries one IP datagram, and everyone calls it a packet.
https://en.wikipedia.org/wiki/Ethernet_frame
Their 'STM32Cube' software gets mixed reviews, but it will spit out a working example of ethernet communication.
[1] https://www.st.com/en/evaluation-tools/nucleo-f439zi.html
When I last used Cube-MX, it was a very unpleasant experience throughout. I'd use stm32-hal or libopencm3 out of preference if I was using the parts again. The tool itself and the code it spat out had all sorts of nasty bugs and edge cases that cost days of debugging. Maybe it's improved since.
https://github.com/egnor/wt32-eth01
I recently discovered the ESP32 and my only complaint so far is that Ethernet (and with it PoE) is not a first class citizen of the platform.
https://liliputing.com/waveshare-esp32-p4-nano-is-a-tiny-ris...
Edit: I just saw there are way cheaper modules, although probably not as performant, like the WT32-ETH01
Deleted Comment
Sometimes. On some hardware it's broken.
Thankfully there's more than one CMake projects on github as an alternative.