Fubwubs (u/Fubwubs) - Readit News

Fubwubs commented on BitNet b1.58 2B4T Technical Report arxiv.org/abs/2504.12285... · Posted by u/galeos

They give some description of how their weights are stored: they pack 4 weights into an int8, indicating that their storage format isn't optimal (2 bits per weight instead of the optimal ~1.58 bits). But I don't know enough about LLM internals to know how material this is.

Could anyone break down the steps further?

Fubwubs · 4 months ago

This model maps weights to ternary values {-1, 0, 1} (aka trits). One trit holds log(3)/log(2) ≈ 1.58 bits of information. To represent a single trit by itself would require 2 bits, but it is possible to pack 5 trits into 8 bits. This article explains it well: https://compilade.net/blog/ternary-packing

By using 4 ternary weights per 8 bits, the model is not quite as space-efficient as it could be in terms of information density. (4*1.58)/8 = 0.79 vs (5*1.58)/8 = 0.988 There is currently no hardware acceleration for doing operations on 5 trits packed into 8 bits, so the weights have to be packed and unpacked in software. Packing 5 weights into 8 bits requires slower, more complex packing/unpacking algorithms.

Fubwubs commented on MicroPython on Flipper Zero lab.flipper.net/apps/upyt... · Posted by u/psvisualdesign

0cf8612b2e1e · a year ago

What are people doing with the Flipper? It seems neat, but I fear I would get one and then forever leave it in a drawer having never done anything real with the device.

Fubwubs · a year ago

My neighborhood has a vehicle gate that is opened by an RF clicker, and a pedestrian gate opened by an RFID tag. I copied both of mine to my flipper. A couple months ago the coin cell battery in my clicker died, but I had my flipper! I also use the cloned RFID tag fairly regularly. There are two RFID tags in the house and more than two people using them, so I use my flipper for that too.

Fubwubs commented on Butterfly wing patterns emerge from ancient ‘junk’ DNA news.cornell.edu/stories/... · Posted by u/hhs

jakobnissen · 3 years ago

This is a pet peeve of mine - the common misconceptions that "junk DNA" is just DNA with an unknown function.

By far most junk DNA really is junk. We know this because we know its nonfunctional origin (transposons, pseudogenes and similar), because we can do studies of conservation between species, and because there is a huge range of junk DNA content in otherwise similar species.

The misconception come from media bias: You'll never hear a story about "junk DNA really is junk, researchers find"

Fubwubs · 3 years ago

Transposons and pseudogenes may be nonfunctional for an individual organism, but aren't they useful for evolution to create new genes? Wikipedia suggests some usefulness:

"While some TEs confer benefits on their hosts, most are regarded as selfish DNA parasites" (https://en.wikipedia.org/wiki/Transposable_element#Evolution)

"Pseudogene sequences may be transcribed into RNA at low levels, due to promoter elements inherited from the ancestral gene or arising by new mutations. Although most of these transcripts will have no more functional significance than chance transcripts from other parts of the genome, some have given rise to beneficial regulatory RNAs and new proteins." (https://en.wikipedia.org/wiki/Pseudogene)

I would define truly junk DNA as any DNA that if removed would be beneficial or not harmful to the majority of organisms in a species, and be beneficial or not harmful to the ability of a species to adapt and evolve its genome.