The book that includes the results from these slides has broader scope, and also can be downloaded for free from arXiv: https://www.maths.ed.ac.uk/~tl/ed/
"The starting point is the connection between diversity and entropy. We will discover:
• how Shannon entropy, originally defined for communications engineering, can also be understood through biological diversity (Chapter 2);
• how deformations of Shannon entropy express a spectrum of viewpoints on the meaning of biodiversity (Chapter 4);
• how these deformations provably provide the only reasonable abundance-based measures of diversity (Chapter 7);
• how to derive such results from characterization theorems for the power means, of which we prove several, some new (Chapters 5 and 9).
Complementing the classical techniques of these proofs is a large-scale categorical programme, which has produced both new mathematics and new measures of diversity now used in scientific applications. For example, we will find: [...]"
"The question of how to quantify diversity is far more mathematically profound than is generally appreciated. This book makes the case that the theory of diversity measurement is fertile soil for new mathematics, just as much as the neighbouring but far more thoroughly worked field of information theory"
Ah I was wondering about that. Their formula look suspiciously like the definition of Renyi entropy.
I'm not too sure where the category theoretical stuff enters though. They mention that metric spaces have a magnitude, but their end result looks more like a channel capacity (with the confusion matrix being the probability to confuse one species with another). Which, you know, makes sense, if you've got 'N' signals but they're so easily confused with one another that you can only send 'n' signals worth of data then your channels are not too diverse.
They do mention that this is equivalent to some modified version of the category theoretical heuristic, but is that really interesting? The link to Euler characteristic is intriguing, but from the way they end up at their final definition I'm not sure if metric spaces are really the natural context to talk about these things. It almost feels like they've stepped over an enriched category that would provide a more natural fit.
Metric spaces are enriched categories. They are enriched over the positive reals. The 'hom' between a pair of points is then simply a number: their distance.
This is precisely the question answered by the OP. The answer is, "because there is a whole spectrum of things you might mean by 'diversity', of which 'number of distinct species' is only one extremum".
(Less emphasis of the category theory, and more attention to the basic math behind entropy-like diversity measures)
(TLDR) There are two important aspects:
1. Generalize diversity measures to when the categories are not fully distinct (as assumed for the Shannon entropy calculation) but have similarities parametrized in a "Z"-matrix here.
2. A parameter "q" to represent whether you value a category highly (for diversity purposes) even when it has only a single distinct example highly (q --> 0) or whether you value it highly only when it has many many examples (q --> infinity)
* With Z = identity matrix (categories completely distinct) they show how different values of q reproduce different measures that have been considered before. (nicely summarized in a table)
* The generalization (parameterized by Z) when the categories can/do have some overlap is very elegant, and feels like an important step forward. (especially how it makes the measure robust to how we partition a bunch of examples into distinct categories, so long as we keep track of the similarity between the categories)
* The paper also summarizes a bunch of sensible properties that we would want any diversity measure to satisfy (like the one I just mentioned above).
This presentation is getting into the idea of "magnitude" the invariant from category theory and how it can be used in mathematical ecology, especially when we're trying to max out diversity. It's put forward as a possible answer to a head-scratcher in mathematical ecology: How do we find the maximum diversity if we're given a list of n species and a similarity matrix?
The presentation gives us a whole range of views on biodiversity, making it clear that we need to think about both common and rare species when we're figuring out diversity. It hints that this magnitude idea might give us a more detailed picture of diversity by considering how important different species are relative to each other. Then the presentation gets into the weeds of category theory, chatting about enriched categories and size-like invariants. It talks about stuff like monoidal categories, V-enriched categories, and linear categories. These ideas are brought up as tools to help us understand and calculate magnitude.
The presentation also explores how these category theory ideas relate to metric spaces. It suggests that getting a handle on this relationship could give us even more insight into how to max out diversity. Lastly, the presentation brings up the Euler characteristic, which is a concept from algebraic topology. It hints that the Euler characteristic and magnitude are pretty tight, and understanding this connection could give us even more insight into how to max out diversity.
So, to wrap it up, the presentation is a thorough look at how category theory ideas, especially magnitude, can be used to tackle problems in mathematical ecology. It suggests that these ideas can give us a more detailed understanding of diversity and might even help us figure out how to get the most diversity.
I am actually sympathetic to category theory, and the research at hand. That being said, what was presented might be interesting if left in terms of linear algebra or graph theory (both legitimate fields). Trying to put a category theory gloss on it just makes it look hollow.
It’s literally the result of a category theorist “following his nose” and solving a problem - mathematical ecologists are perfectly capable of doing some graph theory/linear algebra!
Amazing to see 'magnitude' on the front page of Hacker News! If you are interested in a direct application of this invariant beyond ecology, check out our recent pre-print in which we study the generalisation behaviour of neural networks: https://arxiv.org/abs/2305.05611
My personal approach to magnitude is not based on category theory but rather based on weightings of a metric space. If your metric satisfies certain properties, you can obtain a measure of the 'effective number of points' of a metric space. This is particularly relevant when looking at the metric space from different scales---zooming in gives you a lot of disconnected points, while zooming out gives you clusters. Magnitude then captures the changes in the number of points in a principled manner.
An interesting result, but saying that “the maximum diversity problem is completely solved by an invariant that comes from category theory” sounds like a parody. In the end it is just a mathematical result with a biological metaphor. But who knows, maybe in near future it will be used for solving diversity problems of boards of directors.
This is an interesting talk! I love reading about applications of math/computer science to ecology. The parent page has links to relevant papers that are worth reading, too. [0] The species similarity paper has some concrete examples on coral, butterflies, and gut microbiomes that I felt missing from the slides. [1]
Wow. The fact that there is an objective answer that is independent of any perspective on the importance of rare species is a rare gift, at least for this part of the problem.
Some questions and thoughts.
It seems that the result could vary based on how you construct the similarity matrix Z, e.g. is it purely taxonomic? or does it try to account for the ecological roles that a species is playing in the community, etc.
A seeming limitation is that the optimization works only for a fixed set of n species.
While it is useful for managing existing communities, it means that there is still a question of whether larger n is strictly better, and leaves open questions of how to deal with transient or migratory members (if the community is spatially bound).
The answer I think, is that it depends on how the similarity matrix is constructed. If every species is fully dissimilar then increasing n is always a good thing. If you use niche space to construct it and new species do not some add or enter new niches so they overlap with others, then they will be close to another species in the matrix and increasing n will not have much impact. On the other hand if you use a purely taxonomic approach then you wind up balancing the number of birds and mammals regardless of niche.
It is not clear to me whether it is possible to construct a similarity matrix that can account for the interaction between n, the carrying capacity of the ecosystem, and the number of available niches (or the ability of species to create new niches). By analogy if you have a stream (sunlight) powering water wheels, how many wheels and how many levels of gears (layers in the ecosystem) can be added, created, and/or sustained? At what point does adding an additional species mean that either two species are forced to be close together in the similarity matrix or both their populations must shrink in size because they must compete for the same energy sources?
Does the model sometimes produce impractical results, e.g. that it is good to have a single member of a sexually reproducing species (this is probably an orthogonal concern and you would want to scale to real population sizes such that the minimum corresponded to the smallest viable a self sustaining population)?
Is there evidence that maximizing diversity using this measure actually produces more robust and stable ecologies?
Readit News
"The starting point is the connection between diversity and entropy. We will discover:
• how Shannon entropy, originally defined for communications engineering, can also be understood through biological diversity (Chapter 2);
• how deformations of Shannon entropy express a spectrum of viewpoints on the meaning of biodiversity (Chapter 4);
• how these deformations provably provide the only reasonable abundance-based measures of diversity (Chapter 7);
• how to derive such results from characterization theorems for the power means, of which we prove several, some new (Chapters 5 and 9).
Complementing the classical techniques of these proofs is a large-scale categorical programme, which has produced both new mathematics and new measures of diversity now used in scientific applications. For example, we will find: [...]"
"The question of how to quantify diversity is far more mathematically profound than is generally appreciated. This book makes the case that the theory of diversity measurement is fertile soil for new mathematics, just as much as the neighbouring but far more thoroughly worked field of information theory"
I'm not too sure where the category theoretical stuff enters though. They mention that metric spaces have a magnitude, but their end result looks more like a channel capacity (with the confusion matrix being the probability to confuse one species with another). Which, you know, makes sense, if you've got 'N' signals but they're so easily confused with one another that you can only send 'n' signals worth of data then your channels are not too diverse.
They do mention that this is equivalent to some modified version of the category theoretical heuristic, but is that really interesting? The link to Euler characteristic is intriguing, but from the way they end up at their final definition I'm not sure if metric spaces are really the natural context to talk about these things. It almost feels like they've stepped over an enriched category that would provide a more natural fit.
Deleted Comment
Dead Comment
(Less emphasis of the category theory, and more attention to the basic math behind entropy-like diversity measures)
(TLDR) There are two important aspects:
1. Generalize diversity measures to when the categories are not fully distinct (as assumed for the Shannon entropy calculation) but have similarities parametrized in a "Z"-matrix here.
2. A parameter "q" to represent whether you value a category highly (for diversity purposes) even when it has only a single distinct example highly (q --> 0) or whether you value it highly only when it has many many examples (q --> infinity)
* With Z = identity matrix (categories completely distinct) they show how different values of q reproduce different measures that have been considered before. (nicely summarized in a table)
* The generalization (parameterized by Z) when the categories can/do have some overlap is very elegant, and feels like an important step forward. (especially how it makes the measure robust to how we partition a bunch of examples into distinct categories, so long as we keep track of the similarity between the categories)
* The paper also summarizes a bunch of sensible properties that we would want any diversity measure to satisfy (like the one I just mentioned above).
Fun stuff!
The presentation gives us a whole range of views on biodiversity, making it clear that we need to think about both common and rare species when we're figuring out diversity. It hints that this magnitude idea might give us a more detailed picture of diversity by considering how important different species are relative to each other. Then the presentation gets into the weeds of category theory, chatting about enriched categories and size-like invariants. It talks about stuff like monoidal categories, V-enriched categories, and linear categories. These ideas are brought up as tools to help us understand and calculate magnitude.
The presentation also explores how these category theory ideas relate to metric spaces. It suggests that getting a handle on this relationship could give us even more insight into how to max out diversity. Lastly, the presentation brings up the Euler characteristic, which is a concept from algebraic topology. It hints that the Euler characteristic and magnitude are pretty tight, and understanding this connection could give us even more insight into how to max out diversity.
So, to wrap it up, the presentation is a thorough look at how category theory ideas, especially magnitude, can be used to tackle problems in mathematical ecology. It suggests that these ideas can give us a more detailed understanding of diversity and might even help us figure out how to get the most diversity.
References:
https://arxiv.org/abs/2012.02113
https://arxiv.org/abs/1606.00095
My personal approach to magnitude is not based on category theory but rather based on weightings of a metric space. If your metric satisfies certain properties, you can obtain a measure of the 'effective number of points' of a metric space. This is particularly relevant when looking at the metric space from different scales---zooming in gives you a lot of disconnected points, while zooming out gives you clusters. Magnitude then captures the changes in the number of points in a principled manner.
[0] https://www.maths.ed.ac.uk/~tl/genova/
[1] https://www.maths.ed.ac.uk/~tl/mdiss.pdf
Some questions and thoughts.
It seems that the result could vary based on how you construct the similarity matrix Z, e.g. is it purely taxonomic? or does it try to account for the ecological roles that a species is playing in the community, etc.
A seeming limitation is that the optimization works only for a fixed set of n species. While it is useful for managing existing communities, it means that there is still a question of whether larger n is strictly better, and leaves open questions of how to deal with transient or migratory members (if the community is spatially bound).
The answer I think, is that it depends on how the similarity matrix is constructed. If every species is fully dissimilar then increasing n is always a good thing. If you use niche space to construct it and new species do not some add or enter new niches so they overlap with others, then they will be close to another species in the matrix and increasing n will not have much impact. On the other hand if you use a purely taxonomic approach then you wind up balancing the number of birds and mammals regardless of niche.
It is not clear to me whether it is possible to construct a similarity matrix that can account for the interaction between n, the carrying capacity of the ecosystem, and the number of available niches (or the ability of species to create new niches). By analogy if you have a stream (sunlight) powering water wheels, how many wheels and how many levels of gears (layers in the ecosystem) can be added, created, and/or sustained? At what point does adding an additional species mean that either two species are forced to be close together in the similarity matrix or both their populations must shrink in size because they must compete for the same energy sources?
Does the model sometimes produce impractical results, e.g. that it is good to have a single member of a sexually reproducing species (this is probably an orthogonal concern and you would want to scale to real population sizes such that the minimum corresponded to the smallest viable a self sustaining population)?
Is there evidence that maximizing diversity using this measure actually produces more robust and stable ecologies?