Wow, I'm currently reading the Scallop paper, so funny to see it posted here!
I really love the concept. This isn't just differentiable neurosymbolic declarative probabilistic programming; Scallop has the flexibility of letting you use various (18 included) or custom provenance semirings to e.g. track "proofs" why a relational fact holds, not just assign it a probability. Sounds cool but I'm still trying to figure out the practicality.
Also worth pointing out that it seems that a lot of serious engineering work has been done on Scallop. It has an interpreter and a JIT compiler down to Rust compiled and dynamically loaded as a Python module.
Because a Scallop program (can be) differentiable it can be used anywhere in an end-to-end learning system, it doesn't have to take input data from a NN and produce your final outputs, as in all the examples they give (as far as I can see). For example you probably could create a hybrid transformer which runs some Scallop code in an internal layer, reading/writing to the residual stream. A simpler/more realistic example is to compute features fed into a NN e.g. an agent's policy function.
The limitation of Scallop is that the programs themselves are human-coded, not learnt, although they can implement interpreters/evaluators (e.g. the example of evaluating expressions).
I'm wondering if this is a limitation though. If it can be learnt from training data, would it not be part of the neural network training data? I imagine we use Scallop to bridge the gap where we can't readily learn certain rules based on available data or perhaps we would prefer to enforce certain rules?
Ever since I learned about category theory and its relationship with symbolic reasoning I've suspected that AGI will come from elegantly combining symbolic reasoning and probabilistic reasoning. This is the first project I've seen that seems to be positioned that way. Very cool.
Neural networks are actually somewhere in between. They don't directly operate on symbolic expressions or explicit logical rules. And while they rely on probabilistic aspects for training (and sometimes for inference), they rely more on continuous-valued transformations in extremely high dimensional spaces. But if your goal is human-like intelligence, they are a pretty good bet, because we know the human brain also doesn't perform symbolic reasoning at its core and these things only emerge as high-level behaviour from a sufficiently complex system. But it also makes neural networks (and us too) prone to failure modes that you would not see in strictly symbolic reasoning processes.
Related: Graph&Category-Theory-based Neuro-Symbolic AI
> The work uses graphs developed using methods inspired by category theory as a central mechanism to teach the model to understand symbolic relationships in science.
How does Scallop scale on large knowledge bases (KBs) for doing probabilistic reasoning? I'm currently working on large KB with ~ 12M facts and trying to do probabilistic inference on it. So far I've been using [cplint](https://friguzzi.github.io/cplint/_build/html/index.html) which is based on SWI-Prolog. It works fine for toy examples, however, it doesn't finish running for the large KB - even after waiting for it for more than a week. Does know any Probabilistic Logic Programming (PLP) libraries that are fast and scale to large KBs? Preferably in Prolog ecosystem, but not a hard requirement.
Unfortunately it doesn't seem to be available yet. Scallop and Lobster are both from UPenn, and the Scallop website says "We are still in the process of open sourcing Scallop," so I assume it's a matter of time.
you seem to be more in the know than me :) Please could you just sketch out a few bullets and explain the relationship between Scallop and Lobster and what you think is going on?
I read the paper on Lobster a little bit. Scallop does its reasoning on the CPU - whereas Lobster is an attempt to move that reasoning logic to the GPU. That way the entire neurosymbolic pipeline stays on the GPU and the whole thing runs much faster.
The problem with scallop is that no one has ever shown a single program that wasn't easier to write without it. Their papers usually contain also no examples, and the ones that do are a heck of a lot more complicated than asking the model to do it directly. The programming languages world let's them get away with this because they're excited to be relevant in the LLM universe. But I wouldn't accept any of this work if I was their reviewer; they need to write real programs and compare against baselines. Many of them. And show that their language provides any value.
Just look at the examples on their website. All 3 are lame and far easier without their language.
It's like publishing that you have a new high performance systems language and never including any benchmark. They would be rejected for that. Things just haven't caught up in the ML+PL world.
I think you misunderstand what a neuro-symbolic programming language (like Scallop) is for.
It's not about performance, but safety.
Making safe decisions becomes exponentially more important as ML / agents evolve, to avoid "performant" but ultimately inefficient/dangerous/wasteful inferences.
Then show me programs that meaningfully improves safety. And compare them to baseline options to demonstrate this. None of these examples improve safety beyond a trivial check on the output which I can also do with a simple prompt.
I looked at Scallop a year ago and decided that it was not a replacement for Prolog - for me.
I may re-evaluate now, thinking of smoother LLM integration as well as differentiability.
Has anyone here used Scallop for a large application? I ask because in the 1980s I wrote a medium large application in Prolog and it was a nice developer experience.
Not Scallop related but did you try Mercury? It is prolog with types and flagging of deterministic functions; the Prolog we ported had a very large (... vague, I know) performance boost and that is a lot of code. Porting is gradual mostly.
Love to see this! I’m a huge fan of Neurosymbolic methods, but more advanced examples might be needed to help convince folks to adopt or try Scallop. The three on the page feel very toy. An example rooted in NLP or working with an LLM front and centered might help
Readit News
I really love the concept. This isn't just differentiable neurosymbolic declarative probabilistic programming; Scallop has the flexibility of letting you use various (18 included) or custom provenance semirings to e.g. track "proofs" why a relational fact holds, not just assign it a probability. Sounds cool but I'm still trying to figure out the practicality.
Also worth pointing out that it seems that a lot of serious engineering work has been done on Scallop. It has an interpreter and a JIT compiler down to Rust compiled and dynamically loaded as a Python module.
Because a Scallop program (can be) differentiable it can be used anywhere in an end-to-end learning system, it doesn't have to take input data from a NN and produce your final outputs, as in all the examples they give (as far as I can see). For example you probably could create a hybrid transformer which runs some Scallop code in an internal layer, reading/writing to the residual stream. A simpler/more realistic example is to compute features fed into a NN e.g. an agent's policy function.
The limitation of Scallop is that the programs themselves are human-coded, not learnt, although they can implement interpreters/evaluators (e.g. the example of evaluating expressions).
https://www.cis.upenn.edu/~mhnaik/papers/neurips21.pdf
https://dl.acm.org/doi/10.1145/3591280
There is a 135 page book on Scallop https://www.cis.upenn.edu/~mhnaik/papers/fntpl24.pdf
In the other hand, these problems are routinely analyzed and solved by differentiable algorithms running on neural net substrates (e.g. you).
> The work uses graphs developed using methods inspired by category theory as a central mechanism to teach the model to understand symbolic relationships in science.
https://news.mit.edu/2024/graph-based-ai-model-maps-future-i...
Category theory can be leveraged to make faster theorem provers (making complex symbolic reasoning practical at larger scales).
Don't ask me how, hopefully someone who studies it will chime in and correct me / expand.
https://liby99.github.io/res/papers/lobster.pdf
you seem to be more in the know than me :) Please could you just sketch out a few bullets and explain the relationship between Scallop and Lobster and what you think is going on?
Just look at the examples on their website. All 3 are lame and far easier without their language.
It's like publishing that you have a new high performance systems language and never including any benchmark. They would be rejected for that. Things just haven't caught up in the ML+PL world.
It's not about performance, but safety.
Making safe decisions becomes exponentially more important as ML / agents evolve, to avoid "performant" but ultimately inefficient/dangerous/wasteful inferences.
I may re-evaluate now, thinking of smoother LLM integration as well as differentiability.
Has anyone here used Scallop for a large application? I ask because in the 1980s I wrote a medium large application in Prolog and it was a nice developer experience.
Very pleasant branding though. Great work! :)