negativeonehalf (u/negativeonehalf)

negativeonehalf commented on AI Alone Isn't Ready for Chip Design spectrum.ieee.org/chip-de... · Posted by u/pseudolus

Meanwhile Google has already used AI for this exact stage of chip design with AlphaChip:

https://deepmind.google/discover/blog/how-alphachip-transfor...

And they've responded to the absurd degree of skepticism that followed:

"That Chip Has Sailed: A Critique of Unfounded Skepticism Around AI for Chip Design" - https://arxiv.org/abs/2411.10053

negativeonehalf commented on That Chip Has Sailed: Critique of Unfounded Skepticism Around AI for Chip Design arxiv.org/abs/2411.10053... · Posted by u/foweltschmerz

AshamedCaptain · a year ago

If you actually look at the full quote

> [whistleblower] stated he did not have evidence to support his suspicion of fraud, that he needed to cross a much larger threshold to prove his suspicion

is exactly saying what I was saying.

But in addition, this is hearsay, "quoted" only by Google's rep. It was never actually mentioned by the whistleblower. It has exactly 0 value. Using this quote at face value is intentionally misleading no matter which way you put it. They're literally the defendants - they're basically quoting themselves.

> Seems like the "whistleblower" didn't even have that.

Before he was fired?

Also, I find it funny that for all the talk of the crisis of reproducibility, anyone would trust for a second the authors of the paper more than the attempts done by a 3rd party (and literally done by one of the most important names of the entire floorplanning academic community, to begin with). At least the EDA community has used some benchmarks that have been often used by other papers, allowing some resemblance of a comparison, and a criticism that "these ancient benchmarks do not reflect our holy ways or whatever" is a criticism that maybe I also share; but it's a hoop that everyone who has ever published any such paper (including all the big names) has had to pass in order to be published, unless apparently if you are called Google and publish in Nature.

Nature doesn't exactly have an stellar track record ensuring Google's results are verifiable ... https://retractionwatch.com/2024/05/14/nature-earns-ire-over...

Frankly, at this point I don't even know why would anyone bother with Google's paper. It feels as if they've managed to alienate the entire floorplanning academic community, and whenever I read one of Google's "responses" I see why.

negativeonehalf · a year ago

> Nature doesn't exactly have an stellar track record ensuring Google's results are verifiable ... https://retractionwatch.com/2024/05/14/nature-earns-ire-over...

They open-sourced AlphaFold-3 a week ago, so I'm not sure how you can say this is part of a pattern of being overly closed: https://www.nature.com/articles/d41586-024-03708-4

> Before he was fired?

I don't know how long someone should expect to remain employed when making baseless allegations of scientific misconduct against his colleagues instead of doing actual work. Again, he did not have evidence to support his suspicion of fraud, and he admitted this at the time.

I'm sorry that the "most important names of the entire floorplanning academic community" are struggling with ML basics, but it is what it is. The "Chip Has Sailed" paper makes this pretty clear. This pattern is unfortunately common when ML comes for a new field -- some researchers adapt and build, and others fail and complain (or worse, don't really even try).

> it's a hoop that everyone who has ever published any such paper (including all the big names) has had to pass in order to be published

If the hoop doesn't match what modern chip design needs, we shouldn't expect researchers to hop through it. No one is comparing Vision Transformers against AlexNet on MNIST. Meanwhile, AlphaChip is already used in production to make real layouts for real chips. TPU is a big deal!

I think the one thing we agree on is that this field desperately needs large public benchmarks that are representative of modern chip design.

negativeonehalf commented on That Chip Has Sailed: Critique of Unfounded Skepticism Around AI for Chip Design arxiv.org/abs/2411.10053... · Posted by u/foweltschmerz

AshamedCaptain · a year ago

This at the very least is a very misleading rebuke:

> However, this “whistleblower” admitted to a Google investigator that he had no reason to believe fraud occurred: “he stated that he suspected that the research being conducted by Goldie and Mirhoseini was fraudulent, but also stated that he did not have evidence to support his suspicion of fraud

Er... obviously he didn't catch them on the act, literally manufacturing the data. In any case like this, before discovery, the most damning evidence you could possibly have is that you cannot reproduce the results, and in no court of law this would considered evidence of fraud. Evidence of fraud needs to show knowledge and intent.

To point this as if the whistleblower had actually recanted his accusation is misleading and clearly in bad faith. Google settled with the whistleblower (the case was for wrongful dismissal, not "fraud"), so we will never know.

negativeonehalf · a year ago

The "whistleblower"'s admission that he "did not have evidence to support his suspicion of fraud" is pretty damning. He fails to meet a much, much lower bar than direct observation -- he admitted he had no evidence at all to even support his suspicion of fraud.

> the most damning evidence you could possibly have is that you cannot reproduce the results

Seems like the "whistleblower" didn't even have that. From the paper by the AlphaChip authors: "We provided the committee with one-line scripts that generated significantly better RL results than those reported in Markov et al., outperforming their “stronger” simulated annealing baseline. We still do not know how Markov and his collaborators produced the numbers in their paper."

negativeonehalf commented on That Chip Has Sailed: Critique of Unfounded Skepticism Around AI for Chip Design arxiv.org/abs/2411.10053... · Posted by u/foweltschmerz

negativeonehalf · a year ago

Update:

Synopsys disavowed Markov's paper: "Regarding the CACM article that Igor Markov's comments and writings do not represent Synopsys views or opinions in any way. Synopsys is also aligned with you on the potential of Reinforcement Learning AI for chip design" (https://x.com/JeffDean/status/1859431937640665474)

Jeff Dean's post on the overall situation: https://x.com/JeffDean/status/1858540085794451906

negativeonehalf commented on How AlphaChip transformed computer chip design deepmind.google/discover/... · Posted by u/isof4ult

dogleg77 · a year ago

Science is not done by quotes from VPs, and we don't know how MediaTek used these methods. Also, would you like to hear from VPs who wasted their company resources on Google RL and gave up?

The more marketing claims we see, the less compelling the Google story is.

Your perseverance is as admirable as it is suspicious. You are the lonely voice here defending the Google announcement.

negativeonehalf · a year ago

Unfortunately, there aren't publicly available benchmarks for modern technology node sizes, at least not that I'm aware of. Kahng compared on 45nm and 12nm chips, which are very different from a physical design perspective from the 4nm technology node size used by Dimensity 5G, or the sub-10nm technology node size of TPU. "MLContra" used a >100nm technology node size, which is just crazy.

Even if the AlphaChip authors redid Kahng's study properly, this still wouldn't give us useful information -- what matters is AlphaChip's ability to optimize chips in a real-life, production setting, for modern chips, where millions of dollars are on the line.

negativeonehalf commented on How AlphaChip transformed computer chip design deepmind.google/discover/... · Posted by u/isof4ult

negativeonehalf · a year ago

There's a lot of... passionate discussion in this thread, but we shouldn't lose sight of the big picture -- Google has used AlphaChip in multiple generations of TPU, their flagship AI accelerator. This is a multi-billion dollar project that is strategically critical for the success of the company. The idea that they're secretly making TPUs worse in order to prop up a research paper is just absurd. Google has even expanded their of AlphaChip use to other chips (e.g. Axion).

Meanwhile, MediaTek built on AlphaChip and is using it widely, and announced that it was used to help design Dimensity 5G (4nm technology node size).

I can understand that, when this open-source method first came out, there were some who were skeptical, but we are way beyond that now -- the evidence is just overwhelming.

I'm going to paste here the quotes from the bottom of the blog post, as it seems like a lot of people have missed them:

“AlphaChip’s groundbreaking AI approach revolutionizes a key phase of chip design. At MediaTek, we’ve been pioneering chip design’s floorplanning and macro placement by extending this technique in combination with the industry’s best practices. This paradigm shift not only enhances design efficiency, but also sets new benchmarks for effectiveness, propelling the industry towards future breakthroughs.” --SR Tsai, Senior Vice President of MediaTek

“AlphaChip has inspired an entirely new line of research on reinforcement learning for chip design, cutting across the design flow from logic synthesis to floor planning, timing optimization and beyond. While the details vary, key ideas in the paper including pretrained agents that help guide online search and graph network based circuit representations continue to influence the field, including my own work on RL for logic synthesis. If not already, this work is poised to be one of the landmark papers in machine learning for hardware design.” --Siddharth Garg, Professor of Electrical and Computer Engineering, NYU

"AlphaChip demonstrates the remarkable transformative potential of Reinforcement Learning (RL) in tackling one of the most complex hardware optimization challenges: chip floorplanning. This research not only extends the application of RL beyond its established success in game-playing scenarios to practical, high-impact industrial challenges, but also establishes a robust baseline environment for benchmarking future advancements at the intersection of AI and full-stack chip design. The work's long-term implications are far-reaching, illustrating how hard engineering tasks can be reframed as new avenues for AI-driven optimization in semiconductor technology." --Vijay Janapa Reddi, John L. Loeb Associate Professor of Engineering and Applied Sciences, Harvard University

“Reinforcement learning has profoundly influenced electronic design automation (EDA), particularly by addressing the challenge of data scarcity in AI-driven methods. Despite obstacles including delayed rewards and limited generalization, research has proven reinforcement learning's capability in complex electronic design automation tasks such as floorplanning. This seminal paper has become a cornerstone in reinforcement learning-electronic design automation research and is frequently cited, including in my own work that received the Best Paper Award at the 2023 ACM Design Automation Conference.” --Professor Sung-Kyu Lim, Georgia Institute of Technology

"There are two major forces that are playing a pivotal role in the modern era: semiconductor chip design and AI. This research charted a new path and demonstrated ideas that enabled the electronic design automation (EDA) community to see the power of AI and reinforcement learning for IC design. It has had a seminal impact in the field of AI for chip design and has been critical in influencing our thinking and efforts around establishing a major research conference like IEEE LLM-Aided Design (LAD) for discussion of such impactful ideas." --Ruchir Puri, Chief Scientist, IBM Research; IBM Fellow

negativeonehalf commented on How AlphaChip transformed computer chip design deepmind.google/discover/... · Posted by u/isof4ult

kayson · a year ago

I'm pretty sure Cadence and Synopsys have both released reinforcement-learning-based placing and floor planning tools. How do they compare...?

negativeonehalf · a year ago

Unfortunately, commercial EDA companies generally have restrictive licensing agreements that prohibit direct public comparison.

Still, the fact that Google uses it for TPU is pretty telling - this is a multi-billion dollar, mission-critical chip design effort, and there's no way they'd make TPU worse just to prop up a research paper. MediaTek's production use is also a good indicator.

negativeonehalf commented on How AlphaChip transformed computer chip design deepmind.google/discover/... · Posted by u/isof4ult

anna-gabriella · a year ago

[flagged]

negativeonehalf · a year ago

[flagged]

negativeonehalf commented on How AlphaChip transformed computer chip design deepmind.google/discover/... · Posted by u/isof4ult

wegfawefgawefg · a year ago

In reinforcement learning pre-training reduces peak performance. We can argue about this, but it is not a sufficiently strong point to stop reading from alone.

negativeonehalf · a year ago

See this ISPD 2022 paper where the AlphaChip authors dive more into the value of pre-training (Figure 7, Figure 8): https://dl.acm.org/doi/pdf/10.1145/3505170.3511478

negativeonehalf commented on How AlphaChip transformed computer chip design deepmind.google/discover/... · Posted by u/isof4ult

AshamedCaptain · a year ago

What is Google doing here? At best, the quality of their "computer chip design" work can be described as "controversial" https://spectrum.ieee.org/chip-design-controversy . What is there to gain by just making a PR now without doing anything new?

negativeonehalf · a year ago

In the blog post, they announce MediaTek's widespread usage, the deployment in multiple generations of TPU with increasing performance each generation, Axion, etc.

Chips designed with the help of AlphaChip are in datacenters and Samsung phones, right now. That's pretty neat!