I'm fortunate to work for a company with great PMs at the moment, but in the past, it's usually been the case that PMs blur the line between PMs and engineers by making us do their job for them. That they can use AI to appear productive is not a promising development.
Most of the big CEOs appear to be CEO of multiple companies (Musk and Bezos come to mind). If you can be the CEO of five different companies, it really doesn't seem like it can be that hard of a job. ChatGPT might be able to do that just fine.
> Most of the big CEOs appear to be CEO of multiple companies (Musk and Bezos come to mind).
Most big CEOs are definitely not in charge of multiple companies. You hear about Musk and Bezos because they’re all over the news, but they’re all over the news because they’re not normal CEOs.
It’s well known that Musk’s relationship to his companies is primarily one of ownership and delegation. Bezos hasn’t been in charge of Amazon for a long time but people conveniently forget that at every opportunity.
> it really doesn't seem like it can be that hard of a job.
It’s funny how often I hear this about AI replacing jobs, with one exception: Everyone who repeats it is highly confident that AI can replace other jobs but once you get to the work they do they’ll stop and explain why it’s actually much harder than you think.
And, at least for one of them, they also tweet all day while somehow being invaluable in providing leadership for like 5 companies. At the same time. I can't even successfully effectively juggle 4-5 _projects_ at a time.
At that level its more about persuasion and personality than the ability to do the job. High performing CEOs can persuade employees, customers, government, media, investors, suppliers.
LLMs still cannot persuade anybody right now but maybe soon.
Real-talk: using an LLM for CEO-work makes sense for when hard, rational, decisions need to be made (e.g. new divisions, layoffs, expansion, etc) - but CEOs also need to be a leader and inspire confidence in their subordinates - a non-human agent with non-human motivations simply can’t do that.
> using an LLM for CEO-work makes sense for when hard, rational, decisions need to be made (e.g. new divisions, layoffs, expansion, etc)
Hard disagree. Using an LLM for these decisions transforms it into a game of manipulating inputs.
If you thought it was bad when people were gaming metrics for performance reviews, imagine the nightmare of a company where everyone is trying to manipulate their work to appeal to the current preferences of the prompts HR is feeding to the LLM. Now imagine the HR people manipulating the prompts to get the answers they want, or leaking the prompts to their friends.
Without humans in the loop to recognize when incentives are being gamed it becomes a free for all.
I'd expect that AI making top-level decisions would be disastrous for employees, especially in tech. An AI is going to relentlessly push for efficiency, and human talent simply can't be reduced to a single input to measure against cost (salary, benefits, devtools, etc). AI will constantly suggest lower costs and thus fewer people, it has no perception of product quality or satisfaction, and it's always going to question the value of the employees' contributions to building software (or other stuff), especially if those employees are themselves relying on AI assistance.
A theme of pre-LLM computer-human hybrid systems, was one expensive/scarce expert might be replaced by a less-expensive computer-facilitated redundant group of lesser experts.
So perhaps imagine something like Kaggle competitions, but for Harvard Business School case studies. Open to LLMs, humans, and collaboratives.
A first step might be to create a leadership/managerial LLM test set. I wonder if HBS case studies are in training sets. And whether we can generate good case studies yet. Perhaps use military leadership training material?
In my vision the automated CEO is able to talk to all of the customers, all of the employees and all the investors simultaneously 24/7. It doesn't have to be perfect, if it is able to do that at all it would dramatically change the game. What is also hilarious is that it has a unit of attention so it can accurately divide it over the audience. A human CEO has few tokens that are extremely hard to divide.
> “No, no,” she said. “Engineers aren’t allowed to edit the prompts. It’s only the PMs and domain experts who do prompt engineering. They do it in a custom UI, and then the prompts are committed to the codebase.”
This feels like a desperate power grab.
Why can’t engineers be involved with the prompts? Why aren’t they allowed to do things like automated A/B testing or to implement ideas from papers they’ve read?
Banning engineers from prompts altogether feels extremely arbitrary. If this was a healthy relationship they’d at least say the engineers and PMs work together on prompts.
These company blog articles are usually for marketing. Humanloop develops software to assist with prompting during the development process, so the author’s conclusions reflect the company’s intentions more than an objective industry observation.
AI is transforming how we prototype and iterate, and products like v0 or Replit are scratching the surface. However, historically, low-code platforms lacked a good integration with complex development cycles. There were many attempts, but they either failed or shifted their focus: Microsoft Expression Blend had a brilliant concept of integrating early sketching and ideation with development, but the product ultimately died with Silverlight; Framer had an editor that allowed users to integrate React components with a design tool, but they repurposed their product into a CMS-oriented tool like Webflow; Builder.io is following a similar path. It seems that in today’s market, there is no clear fit for the RAD tools of the late 1990s. Maybe AI can change that and create the new equivalent to Visual Basic. The hardest part is the extra mile that goes from the prototype to something robust and complies with multiple quality attributes: scalability, performance, security, maintainability, and testability.
I’m SWE on a product and generally agree with this, at least for user-facing prompts. Once you’re walking an LLM thru OOXML it’s better done directly by ENG.
For tools, not clear how this works since as you adjust parameters and whatnot you’re also presumably changing the code downstream when you “execute the call”.
But probably both sides of this will be done by LLMs directly in the future. I rarely write or tune prompts by hand now.
Some of them are painful. Some are impressive. The projects are small. Sometimes pulling in powerful off-the-shelf modules. They are getting better fast.
As a greybeard software architect, my current annoyance is that I'm spending all day talking to people when I want to get some practice voice prompting code :P
Readit News
Most of the big CEOs appear to be CEO of multiple companies (Musk and Bezos come to mind). If you can be the CEO of five different companies, it really doesn't seem like it can be that hard of a job. ChatGPT might be able to do that just fine.
Most big CEOs are definitely not in charge of multiple companies. You hear about Musk and Bezos because they’re all over the news, but they’re all over the news because they’re not normal CEOs.
It’s well known that Musk’s relationship to his companies is primarily one of ownership and delegation. Bezos hasn’t been in charge of Amazon for a long time but people conveniently forget that at every opportunity.
> it really doesn't seem like it can be that hard of a job.
It’s funny how often I hear this about AI replacing jobs, with one exception: Everyone who repeats it is highly confident that AI can replace other jobs but once you get to the work they do they’ll stop and explain why it’s actually much harder than you think.
LLMs still cannot persuade anybody right now but maybe soon.
Hard disagree. Using an LLM for these decisions transforms it into a game of manipulating inputs.
If you thought it was bad when people were gaming metrics for performance reviews, imagine the nightmare of a company where everyone is trying to manipulate their work to appeal to the current preferences of the prompts HR is feeding to the LLM. Now imagine the HR people manipulating the prompts to get the answers they want, or leaking the prompts to their friends.
Without humans in the loop to recognize when incentives are being gamed it becomes a free for all.
So perhaps imagine something like Kaggle competitions, but for Harvard Business School case studies. Open to LLMs, humans, and collaboratives.
A first step might be to create a leadership/managerial LLM test set. I wonder if HBS case studies are in training sets. And whether we can generate good case studies yet. Perhaps use military leadership training material?
This feels like a desperate power grab.
Why can’t engineers be involved with the prompts? Why aren’t they allowed to do things like automated A/B testing or to implement ideas from papers they’ve read?
Banning engineers from prompts altogether feels extremely arbitrary. If this was a healthy relationship they’d at least say the engineers and PMs work together on prompts.
Banning them is just politics.
AI is transforming how we prototype and iterate, and products like v0 or Replit are scratching the surface. However, historically, low-code platforms lacked a good integration with complex development cycles. There were many attempts, but they either failed or shifted their focus: Microsoft Expression Blend had a brilliant concept of integrating early sketching and ideation with development, but the product ultimately died with Silverlight; Framer had an editor that allowed users to integrate React components with a design tool, but they repurposed their product into a CMS-oriented tool like Webflow; Builder.io is following a similar path. It seems that in today’s market, there is no clear fit for the RAD tools of the late 1990s. Maybe AI can change that and create the new equivalent to Visual Basic. The hardest part is the extra mile that goes from the prototype to something robust and complies with multiple quality attributes: scalability, performance, security, maintainability, and testability.
For tools, not clear how this works since as you adjust parameters and whatnot you’re also presumably changing the code downstream when you “execute the call”.
But probably both sides of this will be done by LLMs directly in the future. I rarely write or tune prompts by hand now.
Some of them are painful. Some are impressive. The projects are small. Sometimes pulling in powerful off-the-shelf modules. They are getting better fast.
As a greybeard software architect, my current annoyance is that I'm spending all day talking to people when I want to get some practice voice prompting code :P