Readit Newsdspoka commented on Show HN: Wordllama – Things you can do with the token embeddings of an LLM github.com/dleemiller/Wor... · Posted by u/deepsquirrelnet
Looks cool! Any advantages to the mini-lm model - it seems better on most mteb tasks but wondering if maybe inference or something is better.
dspoka commented on The False Promise of Imitating Proprietary LLMs arxiv.org/abs/2305.15717... · Posted by u/lebek
Sensational title that misrepresents the message in paper.
However, when conducting more targeted automatic evaluations, we found that the imitation models close little to none of the large gap between LLaMA and ChatGPT. In particular, we demonstrate that imitation models improve on evaluation tasks that are heavily supported in the imitation training data. On the other hand, the models do not improve (or even decline in accuracy) on evaluation datasets for which there is little support. For example, training on 100k ChatGPT outputs from broad-coverage user inputs provides no benefits to Natural Questions accuracy (e.g., Figure 1, center), but training exclusively on ChatGPT responses for Natural-Questions-like queries drastically improves task accuracy.
Just because this might not be the way to replicate the performance of ChatGPT across all tasks, it seems to work quite well on whichever tasks are in the imitation learning. That is still a big win.
Later on this also works for factual correctness. (leaving aside the argument whether this is the right approach for factuality)
For example, training on 100k ChatGPT outputs from broad-coverage user inputs provides no benefits to Natural Questions accuracy (e.g., Figure 1, center), but training exclusively on ChatGPT responses for Natural-Questions-like queries drastically improves task accuracy.
dspoka commented on Show HN: GPT-4-powered web searches for developers phind.com... · Posted by u/rushingcreek
It couldn't find the answer to my question but the response contained enough supplementary information to show that I wasn't going to find it easily by googling either. That in-itself is a massive timesaver.
Q: What is the token window size of the Alpaca model?
It understood the question and knew what Alpaca was. So it passes the recent information test.
Alpaca finetuned a llama so:
https://news.ycombinator.com/item?id=35186185#:~:text=Tuned%....
dspoka commented on Carbon offsetting is just another form of greenwashing theartnewspaper.com/2022/... · Posted by u/boredemployee
I don't think the paper you posted disagrees with OP.
Tree planting can be a reasonable force of good for dealing with climate change however it needs to be done in such a way that the local ecosystem (both currently and in the targeted "restored state") is sustainable and able to survive with minimal human management in the long term.
And importantly, tree planting is only a viable solution of carbon offsetting as long as it isn't harming or displacing the local communities.
A good example of this is Tentree/Veritree's efforts. Restoring the heavily logged Mangrove forests on the coasts of Kenya, Madagascar, and in Indonesia recreates the local ecosystems for animals, fish, insects, and plants that live in marshes while also rebuilding the natural sea wall that protects inland areas from flooding due to weather. It's a good carbon sequestration project while more importantly serving to repair local ecosystems and reduce the impacts of further climate change on the local residents. Importantly these projects also focus on educating the locals on responsible forest management so that they can continue to harvest lumber for construction purposes without impacting the ecosystem or the sustainability of the recreated marshlands.
You can have good tree planting but it's more than just sticking saplings into the ground. Plenty of projects do really good work with the money they get towards forest restoration and most importantly these projects don't serve solely to offset environmental costs in western society but rather to repair ecosystems of disadvantaged regions and help protect these communities against the oncoming threat that is climate change.
TLDR: Tree planting as "more tree == less carbon" is obviously ineffective but in the bigger picture tree planting efforts can really make a difference as long as you put a modicum of research into what projects you are funding.
My main qualm with the parent comment was this in particular, "Number one, is that forests work as long term carbon storage and sinks." Even with the surrounding context it sounded like this strategy will just "work".
The example you give with mangroves is a great one which does in fact work. Pragmatically and historically most of the attempts however, have not due to mismanagement and other unseen complications.
Seeing the further comments I see the point the parent was making is more around first principles of Forrests as carbon sinks not about its implementations.