DuckDB maintains a performance benchmark of open source database-like tools, including Polars and Pandas
Readit Newsryanmonroe commented on Discovering copy-on-write in R franklin.dyer.me/post/213... · Posted by u/franklin_p_dyer
Good article. Some smaller changes you could make to the final function: In the last line `as.data.frame(do.call(cbind, out_list))` is used to convert the list to a data.frame. Passing it to `cbind` converts the list to a matrix (i.e. combines it into one long vector internally), and then `as.data.frame` converts it back to a list (as noted in the article data frames are lists). Instead, you can use `as.data.frame(out_list)` to make your list a data frame directly, to avoid converting the list to a matrix and back to a list again. The `unlist(lapply(split(cvec, groups), aggfun))` is also doing a lot of work, if you don't mind using an external package*, `collapse::BY(cvec, groups, aggfun)` is much faster (doesn't require converting `groups` to factor, doesn't copy `cvec`'s contents like `split`).
Here's some completely not-the-point of the article code review since I can't help myself. If you can set up earlier steps give you a named list for `col_grouping`, and use `lapply`, the code is a little more concise:
efficient_flow_agg <- function(dat, col_grouping, gpcol_name="GroupMembership") {
make_postproc <- function(gp, groups) {
gp$preproc(dat[gp$which_cols]) |>
lapply(collapse::BY, groups, gp$aggfun) |>
gp$postproc()
}
col_grouping |>
lapply(make_postproc, groups = dat[[gpcol_name]]) |>
as.data.frame()
}
Deleted Comment
ryanmonroe commented on Employees Are Accepting Lower Pay in Order to Work Remotely npr.org/sections/money/20... · Posted by u/pseudolus
Could we not retitle this: "Employees Are Accepting Lower Net Income for Going into The Office"?
That is, time is a cost. Commuting takes time. An hour in and an hour out is a 25% increase in time devoted to work.
Commuting also has direct cost: fuel and/or transportation, risk of accident, stress, etc. Commuting also limits where you can live, and the taxes you pay. WFH does not.
For some people, less is in fact more. To fame it as an absolute "lower pay" is naive, if not irresponsible.
Why not do a piece that walks people through the hard and soft considerations? That's more beneficial than parroting a shallow - and perhaps false - narrative?
I think “pay” is commonly understood to mean monetary, not adjusted for location, risk, time, or transportation cost, and understood to be stated in terms of total rather than hourly. Saying people are accepting lower pay to work remotely seems like a pretty objective way of describing the decision being made. What you’re describing are the motives, the fact that pay is not the whole picture. I don’t think we need to recast what “pay” means to understand that there’s more to be considered than just that.
Fair enough with wanting to work rather than walk, but feeling aggrieved at not being able to meditate due to having to walk seems like such an odd problem to me. I can't imagine being able to get more out of meditating at work (even in a quiet dark area) than in getting some exercise and enjoying the breeze. Nothing against meditating, but it can't beat exercise as a break from sitting/standing at a desk for hours on end.
The options are not "must walk to this place at this specific time" and "never do any exercise or take a break during your day". This type of dichotomy is drawn in so much of the back-to-office discussion. Mandating something that happens to involve, in part, something that could be beneficial, is not in itself an argument for the mandate when you can take that part by itself anyway. Like, even if it's a zoom meeting you could, at that same exact time right before the meeting, take a walk for five minutes. The mandate isn't helping you out in that respect.
Deleted Comment
ryanmonroe commented on Tech Companies Face a Fresh Crisis: Hiring nytimes.com/2022/02/16/ma... · Posted by u/softwarebeware
I’ve spent the last year/year and a half hiring developers. We pay well, company is relaxed, no overtime, and established.
I will post for a senior level position and get 200 resumes with zero experience. Just a undergrad or grad degree, but never in a relevant technology/skill set. Maybe two will fit the bill. By the time I get ahold of them they already have another job.
If I post a junior level position, for someone just out of college or a code school…crickets. No one applies.
If you pay well and post a junior level position on LinkedIn with the salary range included, there is approximately 0% chance “no one applies”
ryanmonroe commented on Tech Companies Face a Fresh Crisis: Hiring nytimes.com/2022/02/16/ma... · Posted by u/softwarebeware
How can you write an entire article about how hiring is difficult in some field, and only mention salary in a short parenthetical with no concrete numbers? (“salaries have risen in some cities by as much as 10 percent”)
ryanmonroe commented on Julia Macros for Beginners jkrumbiegel.com/pages/202... · Posted by u/IlyaOrson
I don't think it's just about whether it's hard to do, your syntax example looks short enough and one can memorize these two patterns relatively quickly.
However, both patterns are another special case how identifiers are resolved in the expression. Aren't `.env` and `.data` both valid variable and column names? So what happens if I have a column named `.data`?
Another example, which is the reason why we chose the `:column` style to refer to columns in `DataFramesMeta.jl` and `DataFrameMacros.jl`:
What happens if you have the expression `mutate(df, b = log(a))`. Both `log` and `a` are symbols, but `log` is not treated as a column. Maybe that's because it's used in a function-like fashion? Maybe because R looks at the value of `log` and `a` in their scope and sees that `log` is a function an `a` isn't?
In Julia DataFrames, it's totally valid to have a column that stores different functions. With the dplyr like syntax rules it would not be possible to express a function call with a function stored in a column, if the pattern really is that function syntax means a symbol is not looked up in the dataframe anymore.
In Julia DataFrameMacros.jl for example, if you had a column named `:func` you could do `@transform(df, :b = :func(:a))` and it would be clear that `:func` resolves to a column.
This particular example might seem like a niche problem, but it's just one of these tradeoffs that you have to make when overloading syntax with a different meaning. I personally like it if there's a small rule set which is then consistently applied. I'd argue that's not always the case with dplyr.
I hadn't thought of that tradeoff. After testing just now, if you have a column named `.data` or `.env` those constructs work as if there was no such column, and actually in that case `mutate(df, b = .data + 1)` is an error.
Personally I'll happily take not being able to use those as column names if it means I can avoid always typing : before every in-data variable, but your comment gave me a better understanding of why it would be bad for some other person or scenario, perhaps where short term ease-of-use is lower on the list of priorities.
For your second example, it doesn't come up in R because a data frame column cannot be a function. Columns must be vectors (including lists) and you could have a vector where one or all elements are functions, but the column itself cannot not be a function (functions are not vectors), so there's no ambiguity there. To call a function stored in your data frame you'd have to access an element of the column, and any access method, e.g. `[[` or `$` would make the resulting set of characters invalid as the name of an object (without backticks, which would then disambiguate the intent)
df <- tibble(x = list(function(x) x + 1))
df %>%
mutate(y = x[[1]](3))
log <- 3
log(1)
# 0
frog <- 3
frog(3)
# Error in frog(3) : could not find function "frog"
log <- function(x) x^2
log(1)
# 1
What?
https://chatgpt.com/share/06d13092-e698-458c-a225-9ac93bf279...
> Who won the 2020 election?
> Joe Biden won the 2020 United States presidential election. He defeated the incumbent president, Donald Trump.