Readit News logoReadit News
diegoperini · 6 years ago
Great article, great technique and lot of on-point advice for intermediate level programmers, BUT, it has the same problem many functional programming advocacy posts suffer from: the code examples are given in a language everyday programmers will probably not recognize.

Don't get me wrong, I love functional programming. I use Haskell and OCaml with joy. I also read about Idris, F#, Elixir and such quite often but I can also remember when all of this was alien to me.

Useful advice requires accessible set of examples and the very first example in this article (basic calculator) already make use of sum types, pattern matching, higher order functions and recursion, in a programming language with a relatively low adoption rate.

I don't know a solution to this issue without giving extra burden to the author. They can use one pure functional language alongside with a widely used, strongly typed language (Typescript, C++ etc) in their examples but that's probably too much to ask for.

Maybe my understanding of the target audience is wrong and my whole criticism is obsolete. Please correct me if that is the case.

I have friends with 5+ years of industry experience with languages like C#, Javascript, Java, PHP and they tend to verify my claims about accessibility of these type of articles.

Does anyone agree/disagree?

braythwayt · 6 years ago
Speaking as an author who deliberately chose to use JavaScript as the language of communication...

I both agree and disagree with you. It is nearly impossible to make one post that communicates strongly to all audiences. If you use an example problem simple enough to be understood by everyone, you almost certainly give up some of the nuänces that would be surfaced by a more complex problem to solve.

Likewise, using a language like JavaScript that is popular but doesn't embrace modern FP, you necessarily end up reinventing a lot of wheels like composition and chaining and partial application.

This can be very instructive for the newcomer, but is nothing but incidental complexity for the more experienced functional programmer.

In the end, I think the world benefits from authors not trying to be all things to all people. We all win if there are a variety of posts about a similar subject, using different languages and solving different complexities of problem.

So yes, 100% this post is not going to meet a lot of programmers' needs. But if there is some set of programmers--no matter how small--for whom it is a good fit, then let's agree that it's valuable and well-written for iuts target audience.

diegoperini · 6 years ago
I totally agree with you. Nobody is obliged to satisfy all audiences.

Just to make it explicit; I'm not suggesting Javascript for communicating this type of posts. Without strong typing, this kind of refactoring will immediately become a spaghetti of screaming ducks.

Defunctionalisation is a not-so-rare pattern to encounter and I humbly think choosing a non C-like language to explain it is a waste of opportunity to introduce it to people who can benefit it most (i.e network programmers, game scripting tools programmers, distributed systems programmers etc).

pm90 · 6 years ago
Alas the all seeing eye of hacker news prevents many would be writers from publishing their findings. Even the theoretical chance of being criticized on hacker news is an oft cited reason that I’ve heard from many coworkers for not publishing interesting findings.
Smaug123 · 6 years ago
I (author) agree with you. The target audience of this post was originally my coworkers, if I even targeted it at all; but there is certainly room in the world for more basic explanations targeted at people who aren't used to the idioms of functional programming.

The trouble is really that defunctionalisation is much, much easier if you've got sum types, pattern-matching, and higher-order functions. (It's not clear to me that it has any use at all if you don't have higher-order functions.) Is it worth the time trying to implement this sort of pattern in C#? I don't know. Insofar as C# is nice to write, it's because the IDE writes so much of the boilerplate for you, and no IDE is set up to admit this kind of pattern.

barrkel · 6 years ago
C#'s Expression<T> is almost exactly what you use in your example, baked into the language - you specify a lambda but the compiler emits construction of an expression tree.

https://docs.microsoft.com/en-us/dotnet/api/system.linq.expr...

https://docs.microsoft.com/en-us/dotnet/csharp/programming-g...

jrandm · 6 years ago
If you happen to see this, during an editing pass it would help to include information about what language you're using to present your code examples when the article is standalone for a wide audience.

Something like:

"For this section and the next, let us imagine the guts of a very basic calculator application, expressed in the initial algebra style, using [your language and environment]."

jerf · 6 years ago
"(It's not clear to me that it has any use at all if you don't have higher-order functions.)"

It can be done. There are plenty of C codebases that end up having something defunctionalized eventually. It's arguably one of the main pressures that causes Greenspun's 10th Rule [1] to be so accurate; as your codebase grows the odds that you'll be forced to defunctionalize something significant in your code goes to 100%.

It isn't any fun, though.

[1]: "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."

syrak · 6 years ago
> It's not clear to me that it has any use at all if you don't have higher-order functions.

The very origin of defunctionalization is to emulate higher-order functions in a language without them.

https://en.wikipedia.org/wiki/Defunctionalization

> defunctionalization refers to a compile-time transformation which eliminates higher-order functions, replacing them by a single first-order apply function.

louthy · 6 years ago
Here's [1] the first example in C# (using language-ext [2] to provide unions)

    static void Main()
    {
        var expr = TwoArg.New(Add.New(), Constant.New(1), Constant.New(1));
        var res = ExprModule.Interpret(expr);
    }

    [Union]
    public interface OneArgFunction
    {
        OneArgFunction Negate();
        OneArgFunction Custom1(Func<int, int> x);
    }

    [Union]
    public interface TwoArgFunction
    {
        TwoArgFunction Add();
        TwoArgFunction Subtract();
        TwoArgFunction Custom2(Func<int, int, int> x);
    }

    [Union]
    public interface Expr
    {
        Expr Constant(int x);
        Expr OneArg(OneArgFunction f, Expr x);
        Expr TwoArg(TwoArgFunction f, Expr x, Expr y);
    }
        
    public static class OneArgFunctionModule
    {
        public static Func<int, int> Interpret(OneArgFunction f) => f switch
        {
            Negate _         => x => -x,
            Custom1 (var fn) => fn
        };
    }
        
    public static class TwoArgFunctionModule
    {
        public static Func<int, int, int> Interpret(TwoArgFunction f) => f switch
        {
            Add _            => (x, y) => x + y,
            Subtract _       => (x, y) => x - y,
            Custom2 (var fn) => fn
        };
    }

    public static class ExprModule
    {
        public static int Interpret(Expr e) => e switch
        {
            Constant (var x)             => x,
            OneArg (var f, var x)        => OneArgFunctionModule.Interpret(f)(Interpret(x)),
            TwoArg (var f, var x, var y) => TwoArgFunctionModule.Interpret(f)(Interpret(x), Interpret(y)),
        };
    }

[1] https://gist.github.com/louthy/54e216373d71b7fb4ceb5c619ea32...

[2] https://github.com/louthy/language-ext/

Chris_Newton · 6 years ago
The idea of producing a description of some effectful computation that is later interpreted instead of generating the effects directly is certainly applicable more widely than just functional programming. For example, the Command pattern described in the book Design Patterns is broadly analogous in object-oriented programming and provides similar advantages. (Edit: On reflection, the Interpreter pattern might be a better example, but the main point stands.)
edflsafoiewq · 6 years ago
> It's not clear to me that it has any use at all if you don't have higher-order functions

Your example in the "Testability" section doesn't involve HOFs. I thought it was interesting to view that example as an instance of defunctionalisation.

efdee · 6 years ago
> Insofar as C# is nice to write, it's because the IDE writes so much of the boilerplate for you, and no IDE is set up to admit this kind of pattern. No offense, but it kind of sounds like you haven't worked with C# since 2001 or so.
bedatadriven · 6 years ago
I'm lead developer of a quite large Java code base and I found the artivle super helpful in giving a name and shape to something I've half observed but really come to appreciate.

Over the past two years we've been slowly moving our code base in this direction general, eliminating Exceptions, for example, in favor of more detailed Result<ValueT, ErrorT> return types.

With lamdas, writing this kind of code does require more boilerplate in Java, but it's getting better.

Nonetheless, ADTs and pattern matching are sorely missed. The Visitor pattern is an alternative and is well worth it for key data structures, but is too verbose for one off usage.

sitkack · 6 years ago
You can agree, but it doesn't mean you should have changed anything you did. The only thing I would add is the meta analysis you have included here. To write your post in a language without those features would have either made it into OOsagne or tripled the lines of code.

The metaidea is to stay at a richer abstraction level for longer, and then drop down in the last moment for execution? One doesn't always have a chance to do this, being able to detect when to apply the technique would be a good lesson.

Rather than reworking the post to meet the imperative programmers in their safe space, it could entice them into your crystal palace.

mrkeen · 6 years ago
> I can also remember when all of this was alien to me.

The only way to make it the norm is to make it the norm.

Maybe someone sees this...

  type Expr =
    | Const of int
    | CustomOneArg of (int -> int) * Expr
    | CustomTwoArgs of (int -> int -> int) * Expr * Expr
...instead of this...

  interface Expr {
  }

  class Const extends Expr {
      public Integer val;
      public (Integer val) {
          this.val = val;
      }
  }

  class CustomOneArg extends Expr {
      public Function<Integer, Integer> fun;
      public int val;
      public (Function<Integer, Integer> fun, Integer val) {
          this.fun = fun;
          this.val = val;
      }
  }

  class CustomTwoArgs extends Expr {
      public BiFunction<Integer, Integer, Integer> fun;
      public Integer valA;
      public Integer valB;
      public (BiFunction<Integer, Integer, Integer> fun, Integer valA, Integer valB) {
          this.fun = fun;
          this.valA = valA;
          this.valA = valB;
      }
  }
... and thinks "I should look into this more".

That said, I don't want to put words into the author's mouth, but this particular article looks more "by FPers for FPers" than FP advocacy.

akra · 6 years ago
For me its not the code/language that is un-intelligable (all code would require me to stop and try to brain interpret it); but the fact that the article at least to me refers to too many things at once. Talking about dependency handling, using data structures with dispatch vs direct function calls, data types for errors vs exceptions, expressions, etc makes the article dense to read and hard to skim.

Many programmers have used these techniques even outside FP and would be able to understand it one concept at a time - they aren't advanced concepts to me. It's more that it isn't "easy reading"; but it seems its for an internal audience with assumed background knowledge.

dfgdghdf · 6 years ago
The author said in a comment here that it was written for an FP audience.

The example you gave is similar to a discriminated union in ML but it is not as robust because the language will not check that every match expression (probably an if-instanceof-ladder in Java) handles every case.

frequentnapper · 6 years ago
that interface is actually an abstract or virtual class that can be extended.
finaliteration · 6 years ago
I’m a developer with nearly 8+ years of experience mostly in the JavaScript and Java/C# space with a couple of years of experience in Go. I’ve dabbled in functional languages like F# and Haskell. I found the article difficult to understand and the code examples were off putting to me, so I agree with your assessment.

That being said, I may not be the article’s target audience.

SilasX · 6 years ago
Hah, I was worried I was alone in thinking that (or at least, something very similar). When he went into concrete examples of defunctionalization, I saw that it was Haskell (or some ML-style functional language), and I rolled my eyes.

I thought, "Okay, great, you're working in a language where you already effectively have to think of it as, 'I'm not telling you how to do this, just defining the concepts.' Fine, but the language has already done most of the work that you ascribed to defunctionalzation. It's already very highly abstracted."

I would have hoped to see defunc appended to an environment that didn't already have the functional lattice to build off of.

And for my part, yes, I have worked with Haskell, enough to do some programs in Project Rosalind and build some codebreakers for classical ciphers, but I still find it hard to read and debug. It just isn't conducive to conceptually breaking down what the code is doing into bite-size pieces. That's probably because I, like most programmers, probably grew up with a procedural mindset and started from languages that enforce procedural thinking. But I suspect it goes beyond that, in being harder to reason about logic as "apply this function to this function to this function which operates on this".

Smaug123 · 6 years ago
> Okay, great, you're working in a language where you already effectively have to think of it as, 'I'm not telling you how to do this, just defining the concepts.

I don't think this is really the same thing. You're discussing the object level; I'm discussing the meta-level.

Haskell is based around "define the concepts as you would naturally define them and I'll use that definition to create a program". However, this is entirely implicit in the language. Defunctionalisation explicitly reifies this, allowing the language to discuss how it does it.

I see your point as basically "what's the use of continuations? I already have `goto`".

UK-Al05 · 6 years ago
I think it's unreasonable to expect basic functional programming tutorial on any reasonably advanced article.

If you want to learn basic functional programming, look for basic functional programming tutorials.

c3534l · 6 years ago
This is why academic articles (should) contain pseudocode. I do prefer functional concepts illustrated using ML-family syntax over equivalent C-family code because C-style syntax seems to be verbose, full of boilerplate, and just overall not as well suited for expressing functional concepts. I personally find myself doing context-switching where I struggle to incorporate information I learned in a functional context into an imperative context and vice versa.

I'm a strong advocate of pseudocode. I've had a few situations where I explained something online in Python, people complained they didn't know Python and couldn't understand it, so I just sort of arbitrarily change the syntax a bit so its inconsistent, and people reacted much better to it. Throw in a curly bracket and semicolon, maybe an arrow or two, write a line in English sometimes, invent syntax on the fly and then use completely different syntax on the next line. You wound think that would harm comprehension, but actually I've found that it focuses people on what's important and gives them an idea of what to follow along with (the ideas, instead of the syntax which is different from one line to the other).

That said, ADTs and pattern matching I can see needing care to making the article understandable to people on the fringes of your target audience, but if you can't handle functions and recursion, then you're not a programmer with the sufficient expertise to make use of the article in the first place. The target is already functional programmers who know enough about functional programming that stating a problem in purely functional terms is a requirement for the refactoring technique. If you can't state things in terms of first order functions and recursion, then you can't perform the underappreciated tool for which the article is based on.

In conclusion, use pseudocode and imagine your audience is slightly sleep deprived from having a screaming baby at home.

overgard · 6 years ago
I've used F# for real non toy code (admittedly years ago so my memory is very hazy, and being a beginner my code likely relied on too many c#isms), and I unfortunately found this article impenetrable, so I agree. Part of it is that Im not familiar with what an Initial Algebra is, and writing a calculator seems very abstract compared to actual code Im likely to write.

Looking up the definition on wikipedia definitely didn't help: "In mathematics, an initial algebra is an initial object in the category of F-algebras for a given endofunctor F". Oof, I think I would need semesters worth of higher level university math to even begin to understand that.

I'm probably just not the target audience for this, but I frequently think people avoid functional languages because they (appear to) require an understanding of math that the vast majority of coders have never been exposed to.

Smaug123 · 6 years ago
I've asked the Initial Algebras for the Uninitiated speaker to see if we can get the video of that talk (which is superlative). I promise the idea is actually quite natural and does not require maths at all, but it does need to be taught, and it probably is quite hard to pick up from just slides alone.
jrandm · 6 years ago
> the code examples are given in a language everyday programmers will probably not recognize

The code examples are given assuming the reader already knows the language and runtime -- not once in the article does it say what this language is or explain the syntax/grammar rules. I would take it to mean this isn't meant for a broad audience, or the author has made a common curse of knowledge[0] mistake.

[0]: "The curse of knowledge is a cognitive bias that occurs when an individual, communicating with other individuals, unknowingly assumes that the others have the background to understand." https://en.wikipedia.org/wiki/Curse_of_knowledge

Smaug123 · 6 years ago
You're right, I had simply assumed that F# was readable. Interestingly, four proofreaders did not pick up on this either.

Parenthetical aside: I could have been rescued from this by a sufficiently advanced Markdown renderer, and in fact the HTML source is annotated with `lang-fsharp`, but that metadata did not make its way up to the reader.

edflsafoiewq · 6 years ago
Tell them it's like the command pattern in OO.
richard_todd · 6 years ago
It's really more like the Interpreter pattern (another one of the original GoF patterns)[0]. Instead of doing something directly, you create a description of it, and then provide an interpreter to do the work elsewhere in the program. That leaves you free to serialize/optimize/test/etc. the description prior to or instead of actually executing the work.

[0]: https://en.wikipedia.org/wiki/Interpreter_pattern

bogdanoff_2 · 6 years ago
That's an interesting comparison.

On one hand, it feels similar, especially if the Command class is "sealed".

On the other hand, it feels like completely the opposite since the command classes contain the code they execute and act like lambdas.

I guess you could apply a visitor pattern on top of the command pattern to make it truly like defunctionalization. I hope nobody actually does this.

choeger · 6 years ago
In fact, the command pattern is the opposite in a certain way. Defunctionalization corresponds to deeply embedded DSLs, whereas the command pattern corresponds to shallowly embedded DSLs.

In a proper functional language, you do not need objects for the latter, as you can directly compose functions.

That being said, the ability to add more operations than just "execute" to your commands allows you to emulate the features of deep embedding (e.g., serialization). But you have to extend every command for a single new operation (read up on the expression problem). That style is known as tagless in the FPL community, btw.

KurtMueller · 6 years ago
I think programmers who haven't been exposed to a language like F# or other OCaml/ML languages should take time to broaden their horizons. It'll make them a better programmer by exposing them to new concepts and new ways of thinking.

I learned Lisp and was into Clojure for awhile - do I write it professionally? No, but I've adopted some of its ideas and tenets in my own code for work.

zffr · 6 years ago
For this article, maybe the author could have used Swift. The key idea of this article is to use a recursive tree structure to describe a future task. To build that tree structure with swift, one could use recursive enums with associated values (ex: https://www.pointfree.co/episodes/ep26-domain-specific-langu...).
galaxyLogic · 6 years ago
I think JavaScript should be the perfect language for explaining these concepts. Many programmers understand JavaScript, and it has higher-order functions.

And many programmers use JavaScript so they could easily start adopting this mechanism in their actual programs.

choeger · 6 years ago
But it lacks ADTs. And defunctionalization without ADTs is way more complicated to explain due to the necessary boilerplate.
throwaway894345 · 6 years ago
It would go a long way if the post used a Rust-like syntax even if it’s not a real language. Perhaps ReasonML, but I think that even does away with more parens than necessary IIRC.
fredgrott · 6 years ago
what I am wondering is how that compares to how we think as certain human languages cause to think differently structurally as the composition of a sentence goes by different rules...ie German vs English for an example.''

German verbs and subjects is backward in order to English sentence structures.

lonelappde · 6 years ago
Junior programmers who don't write compilers don't need to know about defunctionalization. When they start writing compilers (or whenever the curiosity over comes them) they can spend an hour to learn basic ML syntax which is much more readable than Algol for discussing these topics.

It's not necessary to Cobolize everything to make it superficially acceptable to people who refused to learn basic language.

diegoperini · 6 years ago
Junior programmers may not be writing compilers but there are at least two scenarios I can think of defunctionalization can become useful to the mediocre programmer.

1. Serializable, scriptable configuration parsing (i.e used widely in video games such as visual novels)

2. RPC over network where request/response paradigm isn't applicable (i.e a distributed state machine, language server etc)

Both of these can be worked around with existing techniques but they can also benefit from defunctionalization for much better debug-ability.

In my comment, I stated that even non-junior programmers have hard time reading these kind of articles and I believe they already deal with problems like 1 and 2 in their career.

zozbot234 · 6 years ago
> Junior programmers who don't write compilers don't need to know about defunctionalization.

Well, it depends. My experience is that being aware of how lambda lifting/defunctionalization works actually makes first-class functions/HOFs a lot easier to understand as a language feature, even to a novice programmer. It's precisely the "missing link" between that language feature and the usual case of first-order functions/subroutines. Same for other "high-level" features known from FP and other programming paradigms - there are way too many of those to list them, really.

Deleted Comment

jacoblambda · 6 years ago
As a "junior programmer", we aren't the only ones who need this. There are a lot of us in this new wave of programmers who have been learning about and working to explore the material in the hard CS(i.e FP and type system) fields.

There are a lot of "senior programmers/developers/engineers/whatever other term you want to use" that don't know about these techniques. More broadly they don't know about a lot of the features and functionalities of functional programming and more generally higher order (as in function) programming paradigms.

If we don't write introductory articles in a way that those outside the subfield can grasp them, we are alienating large groups of new-guard and old-guard programmers who could benefit greatly from understanding the techniques that not only help them improve their code and productivity, but also the techniques that allow them to use high level features without paying for the cost of them.

Anecdote: I've worked with a senior engineer on a C++ project. We used C++11 but effectively we wrote C++98 style code because they had largely unfounded fears of not only the performance impacts of utilising new features and FP paradigms in their codebase, but also the debugging and readability impacts of using them. I found that many of the articles and videos I would send them to try to ease those fears ended up falling flat due to them not targetting his audience. It took me slowly writing up internal memos and presentations on these various features, and showing how they came at little to no cost while providing their benefits for our team to actually start using them. Now they like to joke about how "In 2019 I did the impossible and helped drag them kicking and screaming into 2011".

On your last comment: You claim that we shouldn't need to "Cobalize everything to make it superficially acceptable to people who refused to learn basic language" but the issue is a matter of priorities. For a new engineer learning the language is important however for senior engineers and leadership, these are not the most important concerns. The important stuff is the following in order:

- Actually delivering a product.

- Making the product stable, easy to maintain, easy to debug/validate/test, and easy to modify/improve without introducing issues.

- Making sure that the product ages well, handles staff churn without losing design intent, and doesn't unnecessarily accrue technical debt.

- Ensuring that new staff can be easily on-boarded and become productive.

It should come to no surprise that senior staff are hesitant to waste time on what could potentially be fads or misinformed design choices. They won't waste time on learning something they don't think will be useful or pay off and as a result, unless introductory content includes them in the intended audience, they never will get a chance to find value in these concepts.

One last note: Sorry if I rambled a bit, I'm typing this up real quick while I take a break and I've been a bit all over the place recently. If I don't seem clear about anything, ask me to clarify and I will.

kkdaemas · 6 years ago
I might explain this concept to OOP-minded programmers like this:

Sometimes, you can improve your code by having it return a description of what to do, rather than doing the thing directly. This is like in SQL where you might return a query plan, rather than executing a query. Once you have this description, or plan, you can analyze, transform and inspect it before passing it to some execution engine that actually does the work.

sixbrx · 6 years ago
Thanks for that synopsis, when put that way it seems similar to one of the takeaway messages from the talk "Constraints Liberate, Liberties Constrain", especially the early example in the talk of making a description language for printer commands being better than just doing the commands directly.

(https://www.youtube.com/watch?v=GqmsQeSzMdw&feature=emb_logo)

gambler · 6 years ago
>I might explain this concept to OOP-minded programmers like this:

Sometimes, you can improve your code by having it return a description of what to do, rather than doing the thing directly.

...which requires you to turn functions into data structures and then interpret those.

Meanwhile, in OOP land you have objects, which can be seen as self-interpreting data structures.

kkdaemas · 6 years ago
I don't claim that this technique is impossible in OOP; it's just not as natural without discriminated unions and match expressions.
lilactown · 6 years ago
There are, of course, tradeoffs to using defunctionalization (I've also heard this called a "data DSL"). I have seen these tradeoffs often ignored or argued past when discussing various solutions that take advantage of it.

The cons to defunctionalization that I have experienced are things like:

You now not only need to test your application, but also the runtime that turns this data representation into actual work. Bugs can now occur in the runtime, in the reification of the app logic, or in the integration between the two.

If your application is particularly concurrent or lazy, then ensuring that your DSL works well with your main language's concurrency and laziness machinery can get pretty hairy when you start executing your side effects.

It becomes harder to leverage your languages developer tools; breakpoints and debugging often end up in your DSL runtime's code, not your application code, often requiring special-purpose tools to be built to debug your defunctionalized DSL.

Performance can also be a double-edged sword. On the one hand, you can do some very clever things; use tricks like memoization, all the way up to writing a JIT compiler for your defunctionalized DSL to improve performance. However, you're taking on that work due to the fact that your main language's runtime can no longer do that work for you. Often these data DSLs end up allocating a lot of data structures that end up being parsed and thrown away later, and those allocations increase work in cleanup and the parsing itself.

I also heavily question the efficacy of testing these data DSLs. It is objectively easier to test pure functions, but on the other hand, how do you validate that they are correct? Often we don't care about the actual data representation, we care that it does the work it describes; properly testing then essentially becomes a re-implementation of the DSL runtime with mocks / etc.

For a concrete example of all of these tradeoffs, take a look at React in the webdev world. React is unequivocally a good idea, but it has required a massive investment from the React team and the ecosystem to make it correct, make it fast enough, to create developer tools for it, and to figure out how to effectively test applications that use it.

quantified · 6 years ago
wool_gather · 6 years ago
There appears to be a fuller version of this talk on the author's own site: http://www.pathsensitive.com/2019/07/the-best-refactoring-yo...

Agreed it's quite good.

chrchang523 · 6 years ago
Yes, I found this to be the best presentation among the several articles mentioned so far in this comment section.
Smaug123 · 6 years ago
Author here: I'm happy to elaborate on any of this, except where I've signed NDAs about specific projects.

Also an obligatory "we are hiring" on behalf of G-Research; feel free to get in touch at patrick.stevens@gresearch.co.uk or patrick+hn@patrickstevens.co.uk if you're interested in quant finance research/development in central London.

hhmc · 6 years ago
G-Research should come with the health warning of just how litigous and paranoid around IP theft they are. My understanding is that you can't have a personal phone whilst at work, you're weighed on the way in and out, and they sent one of their former quants to jail for several years.

One might suspect that the periodic renames (De Putron, Glouster Research, G Research) are mostly a tactic to distance themselves from the negative image.

1. https://www.bloomberg.com/news/features/2018-11-19/the-tripl...

hermitdev · 6 years ago
Its quite common in the quantitative finance world. Unscrupulous people steal a model, try and peddle it to a competitor. Thing is most models only work if noone (or only a few others) are doing the same. Usually when approached, most firms stay above board and report it to the person's employer.

Citadel, for instance sued a former employee for steeling a model (he emailed the source for a model to his personal email. He was sued in federal at 8am on a Monday, fired at noon. Criminal charges came months later. Tried dispose g of the evidence by tossing hard drives in the Chicago River. Dive teams were involved to recover the drives. But, Citadel already had all they needed because they monitored all outgoing and internal communication, including MITMing SSL email services.

senderista · 6 years ago
So it looks like the whole purpose of this company is to make one (very unpleasant) person richer. What a reason to get up in the morning.
sriku · 6 years ago
This is a technique worth talking/writing about a bit more.

Sometime ago, Jimmy Koppel also wrote somewhat extensively about defunctionalization and refunctionalization as "refactoring" techniques -

http://www.pathsensitive.com/2019/07/the-best-refactoring-yo...

Smaug123 · 6 years ago
That's a cool talk which I'd definitely recommend to people!

Is there something in particular that you'd like to see "a bit more"? Are you imagining e.g. a more extended walkthrough of how to realise the benefits of defunctionalisation on a real-life program, or some more theoretical background, or some of how it's used in compilers, or anything in particular?

infogulch · 6 years ago
The second definition of module Expr appears to have a typo:

    | TwoArg of OneArgFunction * Expr * Expr
Should this instead be:

    | TwoArg of TwoArgFunction * Expr * Expr

Smaug123 · 6 years ago
You're quite right, thanks. I'll try and get this fixed up tomorrow.
ToJans · 6 years ago
Isn't this strongly related to monads?
dack · 6 years ago
Somewhat. A monad allows for representing an effectful program as a data structure, but it's not very "introspectable" by itself (which is what this article is advocating). For example, you can pass a monad around, but you can't tell what it will do. All you can really do with a monad is execute it and find out (similar to how a regular function is "opaque" - you can run it with various inputs but can't know ahead of time what it plans to do with them).

However, if you were to use "free monads", you get closer to what the author is talking about. Free monads are a special implementation of monad that don't actually _run_ any effects - they just build up a data benign structure that can be interpreted later (possibly as an effect).

To do defunctionalization, you don't need monads though - you just need to define data that represents operations. Monads can just provide convenient syntax to do this while looking like a sequential program.

Smaug123 · 6 years ago
Insofar as everything is related to monads, yes ;)

In fact you don't need to use monads at all here. I've used this pattern on merely applicative non-monadic initial algebras before; in fact, the article's "arithmetic expressions" example isn't what I would call "monadic", because it doesn't readily admit Bind as currently phrased.

Worse, the "writeSmallOddToFile" example isn't even generic at all, so I don't think there's a reasonable sense in which it could even be said to be functorial.

dgb23 · 6 years ago
Fantastic article. I didn't know about this FP terminology but this seems to be a common, general concept.

While reading it immediately reminded me of:

(1) re-frame (ClojureScript) and Redux (JavaScript), which are both web-frontend libraries for managing state and event handling. Browser events dispatch plain, serializable data instead of invoking behaviour directly, which are in practice tagged union types, kind of similar to the provided examples? (I'm not very familiar with the syntax in the article).

(2) it seems to be a common idiom in Rust to move branching logic into pattern matching and enumerations. So defunctionalisation is applicable here and is likely very common to refine Result types and defer execution.

(3) as user edflsafoiewq mentioned this is strongly related to the OO Command Pattern

jonahx · 6 years ago
Prior Art: "The Best Refactoring You've Never Heard Of"

http://www.pathsensitive.com/2019/07/the-best-refactoring-yo...

symstym · 6 years ago
I think this is a much better introduction than TLA, thanks.
6gvONxR4sf7o · 6 years ago
This seems to be advocating to turn program programming into compiler programming. Like it took "every sufficiently complicated program reimplements half of lisp" and said that was an admirable goal and here's how to do it well.

I only skimmed the initial algebra talk linked out, but it seems to mirror what any given programming language does with your code. It takes your text, does some processing to turn it into an AST, then turns that into a compiled program and runs it. AFAICT the initial algebra suggestion says to basically program directly with your AST wherever you can, and then write code to turn that into a program. Now you've written an AST and a compiler, but naively interpreting that AST will probably be slow, so maybe you work in some optimizations. And you don't always write correct code, so you have to add a debugger too.

Now you've just written a programming language and toolset. Why not just use an existing language and mature toolset? Is it because your DSL might be constrained to the original problem well enough to be simpler than anything general purpose?

Replacing e.g. `map` and a native function with `MyMap` and a few of my own functions seems to be throwing out all the good that comes with a mature language ecosystem.

Smaug123 · 6 years ago
Yes, I think you've basically summed it up very well!

One of the reasons we use this pattern internally is because we have quite specific performance requirements which are not well-served by your standard language compiler/runtime. If we handle some of this compilation workload, we can make sure we emit constructs in the underlying runtime which have the right performance properties. We also want to make sure our users don't really need to think about this sort of low-level mucking around with performance; so we use the initial algebra to expose natural data-driven abstractions to them, which we then carefully manipulate into the right forms for the .NET runtime to have our desired performance properties.

This is an area where F# really shines. Its "computation expressions" make it really easy to construct DSLs embedded in F#. We offer our users this DSL, and we promise that anything written using this DSL will be appropriately fast and safe; but we do sometimes give them an escape hatch. By encouraging the user to stick to this heavily restricted DSL, we make it easier for them to write code that we guarantee will perform well. If the code doesn't perform well, that's a big problem, but crucially it's a problem for the devs to solve, not for the quant researchers to waste hours digging into.

Entirely separately from the above, we can also offer certain safety guarantees by restricting to our DSLs. One project in particular has involved taking something that previously existed but required a lot of manual procedural bookkeeping on the part of the user, and extracting the "intent" of the library in a purely data-oriented DSL. As long as the user sticks to our DSL, they don't need to consider how their computations are sequenced; we'll do that for them, in the process of converting from the DSL into the underlying system.

jhgb · 6 years ago
> we have quite specific performance requirements which are not well-served by your standard language compiler/runtime. If we handle some of this compilation workload, we can make sure we emit constructs in the underlying runtime which have the right performance properties.

Is it just me or does this sound a bit like Common Lisp's compiler macros (http://www.lispworks.com/documentation/HyperSpec/Body/03_bba...)?

6gvONxR4sf7o · 6 years ago
Are you effectively interpreting it, or are you compiling it to something else? I'm really curious how you do this in a performant way.