duchenne (u/duchenne)

duchenne commented on OpenAI's new open-source model is basically Phi-5 seangoedecke.com/gpt-oss-... · Posted by u/emschwartz

lifis · 4 months ago

Does anyone know how synthetic data is commonly generated? Do they just sample the model randomly starting from an empty state, perhaps with some filtering? Or do they somehow automatically generate prompts and if how? Do they have some feedback mechanism, e.g. do they maybe test the model while training and somehow generate data related to poorly performing tests?

duchenne · 4 months ago

I have done that at meta/FAIR and it is published in the Llama 3 paper. You usually start from a seed. It can be a randomly picked piece of website/code/image/table of contents/user generated data, and you prompt the model to generate data related to that seed. After, you also need to pass the generated data through a series of verifiers to ensure quality.

duchenne commented on DumPy: NumPy except it's OK if you're dum dynomight.net/dumpy/... · Posted by u/RebelPotato

duchenne · 7 months ago

Looks awesome. Can we get the same thing for pytorch?

duchenne commented on Ask HN: How do you propose to rebuild industry in a post-apocalypse world? · Posted by u/hnthrowaway0315

duchenne · 9 months ago

There is a manga/anime about this: doctor stone.

For the knowledge preservation, I guess that a copy of deepseek has most of the required information. But, it would be hard to run it in a primitive world.

duchenne commented on Ask HN: How do you propose to rebuild industry in a post-apocalypse world? · Posted by u/hnthrowaway0315

cratermoon · 9 months ago

Tom Murphy over at Do The Math[1] argues that we would not be able to reboot modernity after an apocalypse. The short version of his argument is that we long ago mined/drilled/farmed/harvested the easy resources that can be gotten with pre-industrial technology. The raw materials we are extracting today require modern methods and materials. We can't bootstrap ourselves from human and animal muscle power back to industrial capacity because it is inadequate to the task.

We might have the theoretical knowledge to do it, but we'd be unable to build out the infrastructure.

And no, scavenging won't solve it because the energy and technology required to recover the raw materials again requires industrial processes and materials we won't have the ability to harness.

1 https://dothemath.ucsd.edu/

duchenne · 9 months ago

But we have landfills which are full of great raw materials. I would argue that it is easier to collect steel from a landfill than from a mine during the industrial revolution.

duchenne commented on MacBook Air M4 apple.com/macbook-air/... · Posted by u/tosh

woodson · 9 months ago

From what I’ve seen, they overheat a lot just from having a browser and VSCode running.

duchenne · 9 months ago

I had one for years. Never had overheating issues, except if I put it on my blanket for long.

duchenne commented on MacBook Air M4 apple.com/macbook-air/... · Posted by u/tosh

ruuda · 9 months ago

My 4-year old Dell XPS 15 is up for replacement, but somehow no manufacturer aside from Apple is making laptop with decent specs nowadays? I want 2TB storage, a 4k (or close) HiDPI display, good build quality, and not a bulky gaming laptop. The XPS 15 was perfect, it had those specs, except it only had 1TB storage which is now full. I was expecting that to not be an issue 4 years later ... But now Dell discontinued XPS, and their new Pro/Premium models have worse specs in almost all ways. The only non-Apple thing that I can find that even comes close, is a bulky 16" ThinkPad.

And then there is Apple who pack everything I want in a sleek 14" or 15" device, plus a very fast CPU and battery life that is years ahead of anything else ... Why is there no competition here? I'm willing to compromise on battery life, and I don't need the fastest CPU, just a good quality work laptop where I can run `cargo build` / `docker pull` without worrying about filling up the disk, and mostly just a browser aside from that. Why is the gap so large?

duchenne · 9 months ago

The asus zenbook pro is great. The 16inch version is not really bulky. It is 2.4kg, 2TB, 3.2k resolution, great design and build quality. $2200

The 14.5 inch version is 1.6kg, 2TB, 2.9k resolution, also great design and build quality. $1700

https://www.asus.com/laptops/for-creators/zenbook/zenbook-pr...

duchenne commented on DeepSeek open source DeepEP – library for MoE training and Inference github.com/deepseek-ai/De... · Posted by u/helloericsf

breadwinner · 10 months ago

Zuckerberg should stop claiming Meta is open sourcing AI (they are even running TV ads) when they are only releasing the weights, and not the code. Only DeepSeek is real OSS AI.

duchenne · 10 months ago

Come on... Meta has been refining pytorch for more than a decade. It basically contains all that you need to train LLMs, including the latest technologies. What more do you need? The part of the code that is specific to Meta infrastructure?

duchenne commented on OpenAI says it has evidence DeepSeek used its model to train competitor ft.com/content/a0dfedd1-5... · Posted by u/timsuchanek

duchenne · a year ago

The reasoning happens in the chain of thoughts. But OpenAI (aka ClosedAI) doesn't show this part when you use the o1 model, whether through the API or chat. They hide it to prevent distillation. Deepseek, though, has come up with something new.

duchenne commented on AMD Open-Source 1B OLMo Language Models amd.com/en/developer/reso... · Posted by u/ebalit

duchenne · a year ago

Training a 1B model on 1T tokens is cheaper than people might think. A H100 GPU can be rented for 2.5$ per hour and can train around 63k tokens per second for a 1B model. So you would need around 4,400 hours of GPU training costing only $11k And costs will keep going down.

duchenne commented on OpenAI to become for-profit company reuters.com/technology/ar... · Posted by u/jspann

Havoc · a year ago

That was the point musk was complaining about.

In practice it’s doable though. You can just create a new legal entity and move stuff and/or do future value creating activity in the new co. IF everyone is on board with the plan on both sides of the move then that’s totally doable with enough lawyers and accountants

duchenne · a year ago

But, if the non-profit gives all its assets to the new legal entity, shouldn't the new legal entity be taxed heavily? The gift tax rate goes up to 40% in the US. And 40% of the value of openAI is huge.