It's fast indeed. And I can't help keeping promoting the combination with fzf :) For those who want to try it out, this is a Powershell function but the same principle applies in any shell. Does ripgrep then puts fuzzy searching in the resulting files+text on top while showing context in bat:
There are other ways to approach this, but for me this is a very fast way of nailing down 'I now something exists in this multi-repo project but don't know where exactly nor the exact name'
edit this comes out of https://github.com/junegunn/fzf/blob/master/ADVANCED.md and even though you might not want to use most of what is in there, it's still worth glancing over it to get ideas of what you could do with it
Infact I would recommend a step further to integrate rip-grep-all (rga) with fzf that can do a fuzzy search not just on text files but on all types of files including pdfs, zip files. More details here [1]
That's really nice, thanks. Long ago there was the Google Desktop Search where you could 'Google' your local documents. But the difference is that that worked with an index, so I imagine it's faster if you have thousands of pdfs en epubs.
I've never really seen PowerShell beyond minimal commands, but after seeing the parent, I definitely think it has the superior syntax of the shells. Especially for scripts.
This comment got me to get out my French press and manually grind some beans. It wasn't a meditative and calming as I remember, and the coffee tastes a little...dusty. I guess it's time for me to update my vimrc.
With that in the [alias] section of a gitconfig file, running git fza brings up a list of modified and not yet added files, space toggles each entry and moves to the next entry.
That alias as well as fzf+fd really speed up some parts of my workflow.
And that's just the start: it could even be that by binding a key to the fzf reload command to then display the diff in it's finder, and in turn a key to stage the selected line, you could turn that into an interactive git staging tool.
This is pretty much my exact use of ripgrep, too. I use it as a starting point to zero in on files/projects in a several-hundred repo codebase, and then go from there....
> How do you scroll the preview window with keyboard ?
alias pf="fzf --preview='less {}' --bind shift-up:preview-page-up,shift-down:preview-page-down"
That will let you run `pf` to preview files in less and lets you use shift + arrow keys to scroll the preview window. No dependencies are needed except for fzf. If you want to use ripgrep with fzf you can set FZF_DEFAULT_COMMAND to run rg such as `export FZF_DEFAULT_COMMAND="rg ..."` where ... are your preferred rg flags. This full setup is in my dotfiles at https://github.com/nickjj/dotfiles.
Oh. Thanks for the tip. This might make me finally embrace powershell. I’ve been using WSL+zsh+fzf as a Windows CLI for continuity with day job Mac tools, but git CLI performance is only usable inside the WSL file system.
You can also add a small script to your WSL under `/usr/local/bin/git`:
GIT_WINDOWS="/mnt/c/Program Files/Git/bin/git.exe"
GIT_LINUX="/usr/bin/git"
case "$(pwd -P)" in
/mnt/?/*)
case "$@" in
# Needed to fix prompt, but it breaks things like paging, colours, etc
rev-parse*)
# running linux git for rev-parse seems faster, even without translating paths
exec "$GIT_LINUX" "$@"
;;
*)
exec "$GIT_WINDOWS" -c color.ui=always "$@"
;;
esac
;;
*)
exec "$GIT_LINUX" "$@"
;;
esac
This allows you to use `git` in your WSL shell but it'll pick whichever executable is suitable for the filesystem that the repo is in :)
Yeah, I have a bit of a love-hate relationship with it. But I actually have that with all shells out there. I don't know if it's just me or the shells, or (the most likely I think): a bit of both. But PS is available out of the box and using objects vs plain text is a major win in my book, and even though I still don't know half of the syntax by heart it feels less of an endless fight than other shells. And since I use the shell itself for rather basic things and for the rest only for tools (like shown here), we get along just fine.
I use ripgrep with the Emacs packages project.el (comes out of the box) and dumb-jump (needs to be installed). This may not be the most popular way of using rg but I have been very pleased with the overall experience. All it takes is running package-install to install the dumb-jump package and configuring the following hook:
The Xref key sequences and commands work fine with it. If I type M-. (or C-u M-.) to find definitions of an identifier in a Python project, dumb-jump runs a command like the following, processes the results, and displays the results in an Xref buffer.
The above command shows how dumb-jump automatically restricts the search to the current file type within the current project directory. If no project directory is found, it defaults to the home directory.
By the way, dumb-jump supports the silver searcher tool ag too which happens to be quite fast as well. If neither ag nor rg is found, it defaults to grep which as one would expect can be quite slow while searching the whole home directory.
Ripgrep can be used quite easily with the project.el package too that comes out of the box in Emacs. So it is not really necessary to install an external package to make use of ripgrep within Emacs. We first need to configure xref-search-program to ripgrep as shown below, otherwise it defaults to grep which can be quite slow on large directories:
(setq xref-search-program 'ripgrep)
Then a project search with C-x p g foo RET ends up executing a command like the following on the current project directory:
The results are displayed in an Xref buffer again which in my opinion is the best thing about using external search tools within Emacs. The Xref key sequences like n (next match), p (previous match), RET (jump to source of match), C-o (show the source of the match in a split window), etc. make navigating the results a breeze!
Looking at your regex---just by inspection, I haven't tried it, so I could be wrong---but I think you can drop the --pcre2 flag. I also think you can drop the second and third \b assertion. You might need the first one though.
The example I have posted in my comment is not a command I am typing myself. The dumb-jump package generates this command for us automatically. It is possible to customize the command it generates though. Indeed while running ripgrep manually, I do not use the --pcre2 option. Thank you for developing and maintaining this excellent tool!
this is a good option but i still use rg.el for the occasion that i want to search several projects at once or a subfolder within a project, where i would otherwise use ‘rgrep’
I've been using ripgrep for about 2 years now and I find in indispensable. The main reason I switched from grep was ease of use. From the README: "By default, ripgrep will respect gitignore rules and automatically skip hidden files/directories and binary files." Typing `rg search_term directory` is much better than the corresponding grep command, but the speed improvement is also a nice bonus.
Random other helpful flag I use often is -M if any of the matches are way too long to read through and cause a lot of terminal chaos. Just add `-M 1000` or adjust the number for your needs and the really long matches will omit the text context in the results.
Yeah, the -M command is wonderful (super handy for ignoring minified files that you don't want to see results from, etc), and also great is the -g command (eg `-g *.cs` and you'll just search in files that have the .cs extension).
Also the fact that it is a standalone portable executable can be super handy. Often when working on a new machine, I'll drop in the executable and an alias for grep that points to rg, so if muscle memory kicks in and I type grep it will still use rg.
If you're a fan of the -g flag to ripgrep then I also recommend checking out the -t flag, short for --type, which lets you search specific file types. You can see the full list with `rg --type-list`. For example, you could just search .cs files with `rg -tcs`.
This flag is especially convenient if you want to search e.g. .yml and .yaml in one go, or .c and .h in one go, etc.
And this may still be true in 2023, but the problem is that most of the parallelized grep replacements (e.g. ripgrep, ag, etc.) are SO much faster than grep that the much small speed differences between them doesn't provide much of a basis for differentiating them.
I use ag (typically from inside Emacs) on a 900k LOC codebase and it is effectively instantaneous (on a 16 core Ryzen Threadripper 2950X). I just don't have a need to go from less than 1 second to "a bit less than less than 1 second".
Speed is not the defining attribute of the "new greps" - they need to be assessed and compared in other ways.
In 2016, I'd say speed was definitely a defining attribute. ag has very significant performance cliffs. You can see them in the blog post.
But as I mentioned in my comparison to qgrep elsewhere in the thread, everyone has different workloads. And for some workloads, perf differences might not matter. It really just depends. 900 KLOC isn't that big, and indeed, for simple queries pretty much any non-naive grep is going to chew through it very very quickly.
The blog post also compares Unicode support, and contextualizes its performance. ag essentially has zero Unicode support. Unicode support isn't universally applicable of course---you may not care about it---but it satisfies your non-perf comparison criteria. :-)
In my experience they are all horribly i/o bound and the search takes as long as files load from disk and that's quite long, after that the difference can't possibly be meaningful. When files are in cache search time is dominated by time it takes me to navigate the file system and write the command, and again performance difference can't possibly be meaningful.
> When files are in cache search time is dominated by time it takes me to navigate the file system and write the command
This suggests your corpora are small. If you have small corpora, then it should be absolutely no surprise that one tool taking 40ms and another taking 20ms will matter for standard interactive usage.
well... it is not faster than qgrep :) even though the way both work - differs greatly, and even though qgrep is based on re2 - the speed comes from the presence of index. but then I wonder why people forget the qgrep option, since with large file stores it makes much more sense to use qgrep AND indices, rather than always go through all the files.
this above all true UNLESS you need multi-line matches with UTF8, where ripgrep is not so fast, because it needs to fall back to the other PCRE2 lib
Yes, qgrep uses indexing, which will always give it a leg up over other tools that don't use indexing. But of course, now you need to setup and maintain an index. The UX isn't quite as simple as "just run a search."
But there isn't much of a mystery here. Someone might neglect to use qgrep for exactly the same reason that "grep is fast enough for me" might prevent someone from using ripgrep. And indeed, "grep is fast enough" is very much true in some non-trivial fraction of cases. There are many many searches in which you won't be able to perceive the speed difference between ripgrep and grep, if any exists. And, analogously, the difference between qgrep and ripgrep. The cases I'm thinking of tend to be small haystacks. If you have only a small thing to search, then perhaps even the speed of a "naive" grep is fast enough.
So if ripgrep, say, completes a search of the Linux kernel in under 100ms, is that annoying enough to push you towards a different kind of tool that uses indexing? Maybe, depends on what you're doing. But probably not for standard interactive usage.
This is my interpretation anyway of your wonderment of (in your words) "why people forget the qgrep option." YMMV.
> this above all true UNLESS you need multi-line matches with UTF8, where ripgrep is not so fast, because it needs to fall back to the other PCRE2 lib
That's not true. Multiline searches certainly do not require PCRE2. I don't know what you mean by "with UTF8," but the default regex engine has Unicode support.
PCRE2 is a fully optional dependency of ripgrep. You can build ripgrep without PCRE2 and it will still have multiline search support.
Does `build.rs` build the project? One of my favorite (Big Corp) code-bases just had a single C file (build.c) that did all the dependency tracking, like, Make, but in some nicely written (easy to understand) C code. The C file started with a shebang: a self-building-and-executing line, so we'd do this:
I switched from from ripgrep to ugrep and never looked back. It's just as fast, but also comes with fuzzy matching (which is super useful), a TUI (useful for code reviews), and can also search in PDFs, archives, etc.
The optional Google search syntax also very convenient.
I’m a die-hard ripgrep fan, but just recently found ugrep looking for one feature that ripgrep lacks: searching in zip archives (without decompressing them to disk).
Ugrep has that. In my case, I’m working with zipped corpora of millions of small text files, so I can skip unpacking the whole thing to the filesystem (certain filesystems have trouble at this scale).
I’m grateful for both tools. Thanks to the respective authors!
So I was casually searching for "ugrep vs ripgrep" articles, when I stumbled upon a couple reddit posts where apparently the authors of ugrep and ripgrep seemed to have a multi-year feud on reddit, eg. https://www.reddit.com/r/programming/comments/120wqvr/ripgre...
So weird. I mean, it's just about some open source tool, right? :-/
I came across ugrep recently and I immediately recognized the organization as one that I had dealt with starting about 15 years go. The author is brilliant‡, but extremely prickly (sometimes even to paying customers). The author of ripgrep, on the other hand, has always seemed like someone who just wants to get on with the business of writing software that people use.
‡ The main commercial product of the ugrep author's company at the time was the gSOAP code generator (it may still be), and that it not only works but makes a reasonably good C and C++ API from WSDL is proof that it is the product of a genius madman. It also allowed you to create both the API and WSDL from a C++-ish header, and both .NET and Java WSDL tools worked perfectly with it. We needed it to work and work it did.
At the time, the generated API was just difficult enough to use that I generated another ~1k lines of code for that project. IIRC, the generated API is sort of handle-based, which requires a slightly different approach than the strict RAII approach we were using. Generating that code was a minor adventure (generating the gSOAP code from the header-ish file, generating doxygen XML from the generated gSOAP code, then generating the wrapper C++ from the doxygen XML).
There are feuds about open source tools all the time. Text editors, Linux distros, shells, programming languages, desktop environments, etc... And ugrep vs ripgrep may be a poster child for C++ vs Rust.
It is not all bad, it drives progress, and it usually stays at a technical level, I've yet to see people killing each others for their choice of command line search tool.
Readit News
edit this comes out of https://github.com/junegunn/fzf/blob/master/ADVANCED.md and even though you might not want to use most of what is in there, it's still worth glancing over it to get ideas of what you could do with it
[1] https://github.com/phiresky/ripgrep-all/wiki/fzf-Integration
2. You nerd-sniped me into getting rid of the unnecessary `cut` process :)
Deleted Comment
That alias as well as fzf+fd really speed up some parts of my workflow.
Oh and shameless plug for my guide on what to include in your zsh setup on macOS: https://gist.github.com/aclarknexient/0ffcb98aa262c585c49d4b...
How do you scroll the preview window with keyboard ?
I've made a video and blog post about it here: https://nickjanetakis.com/blog/customize-fzf-ctrl-t-binding-...
I also made https://nickjanetakis.com/blog/using-fzf-to-preview-text-fil... which covers how to modify fzf's built in CTRL+t shortcut to allow for previews too. CTRL+t is a hotkey driven way to fuzzy match a list of files.
Yeah, I have a bit of a love-hate relationship with it. But I actually have that with all shells out there. I don't know if it's just me or the shells, or (the most likely I think): a bit of both. But PS is available out of the box and using objects vs plain text is a major win in my book, and even though I still don't know half of the syntax by heart it feels less of an endless fight than other shells. And since I use the shell itself for rather basic things and for the rest only for tools (like shown here), we get along just fine.
By the way, dumb-jump supports the silver searcher tool ag too which happens to be quite fast as well. If neither ag nor rg is found, it defaults to grep which as one would expect can be quite slow while searching the whole home directory.
Ripgrep can be used quite easily with the project.el package too that comes out of the box in Emacs. So it is not really necessary to install an external package to make use of ripgrep within Emacs. We first need to configure xref-search-program to ripgrep as shown below, otherwise it defaults to grep which can be quite slow on large directories:
Then a project search with C-x p g foo RET ends up executing a command like the following on the current project directory: The results are displayed in an Xref buffer again which in my opinion is the best thing about using external search tools within Emacs. The Xref key sequences like n (next match), p (previous match), RET (jump to source of match), C-o (show the source of the match in a split window), etc. make navigating the results a breeze!Looking at your regex---just by inspection, I haven't tried it, so I could be wrong---but I think you can drop the --pcre2 flag. I also think you can drop the second and third \b assertion. You might need the first one though.
https://github.com/Wilfred/deadgrep
https://www.npmjs.com/package/@vscode/ripgrep
You can find the rg binary in the VS installation (at least, I can on Windows at my place of employment).
Random other helpful flag I use often is -M if any of the matches are way too long to read through and cause a lot of terminal chaos. Just add `-M 1000` or adjust the number for your needs and the really long matches will omit the text context in the results.
Also the fact that it is a standalone portable executable can be super handy. Often when working on a new machine, I'll drop in the executable and an alias for grep that points to rg, so if muscle memory kicks in and I type grep it will still use rg.
This flag is especially convenient if you want to search e.g. .yml and .yaml in one go, or .c and .h in one go, etc.
I use ag (typically from inside Emacs) on a 900k LOC codebase and it is effectively instantaneous (on a 16 core Ryzen Threadripper 2950X). I just don't have a need to go from less than 1 second to "a bit less than less than 1 second".
Speed is not the defining attribute of the "new greps" - they need to be assessed and compared in other ways.
But as I mentioned in my comparison to qgrep elsewhere in the thread, everyone has different workloads. And for some workloads, perf differences might not matter. It really just depends. 900 KLOC isn't that big, and indeed, for simple queries pretty much any non-naive grep is going to chew through it very very quickly.
As for comparisons in other ways, at least for ag, it's on life support. I thought it was going to get removed from Debian, but it looks like someone rescued it: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999962
The blog post also compares Unicode support, and contextualizes its performance. ag essentially has zero Unicode support. Unicode support isn't universally applicable of course---you may not care about it---but it satisfies your non-perf comparison criteria. :-)
This suggests your corpora are small. If you have small corpora, then it should be absolutely no surprise that one tool taking 40ms and another taking 20ms will matter for standard interactive usage.
"Ripgrep – A new command line search tool" https://news.ycombinator.com/item?id=12564442 (740 points | Sept 23, 2016 | 209 comments) - there are discussions related to speed too
"Ripgrep is faster (2016)" https://news.ycombinator.com/item?id=17941319 (98 points | Sept 8, 2018 | 40 comments)
this above all true UNLESS you need multi-line matches with UTF8, where ripgrep is not so fast, because it needs to fall back to the other PCRE2 lib
Yes, qgrep uses indexing, which will always give it a leg up over other tools that don't use indexing. But of course, now you need to setup and maintain an index. The UX isn't quite as simple as "just run a search."
But there isn't much of a mystery here. Someone might neglect to use qgrep for exactly the same reason that "grep is fast enough for me" might prevent someone from using ripgrep. And indeed, "grep is fast enough" is very much true in some non-trivial fraction of cases. There are many many searches in which you won't be able to perceive the speed difference between ripgrep and grep, if any exists. And, analogously, the difference between qgrep and ripgrep. The cases I'm thinking of tend to be small haystacks. If you have only a small thing to search, then perhaps even the speed of a "naive" grep is fast enough.
So if ripgrep, say, completes a search of the Linux kernel in under 100ms, is that annoying enough to push you towards a different kind of tool that uses indexing? Maybe, depends on what you're doing. But probably not for standard interactive usage.
This is my interpretation anyway of your wonderment of (in your words) "why people forget the qgrep option." YMMV.
I have flirted with the idea of adding indexing to ripgrep: https://github.com/BurntSushi/ripgrep/issues/1497
> this above all true UNLESS you need multi-line matches with UTF8, where ripgrep is not so fast, because it needs to fall back to the other PCRE2 lib
That's not true. Multiline searches certainly do not require PCRE2. I don't know what you mean by "with UTF8," but the default regex engine has Unicode support.
PCRE2 is a fully optional dependency of ripgrep. You can build ripgrep without PCRE2 and it will still have multiline search support.
``` ./build.c ```
... and then magic happened.
The optional Google search syntax also very convenient.
https://ugrep.com
Ugrep has that. In my case, I’m working with zipped corpora of millions of small text files, so I can skip unpacking the whole thing to the filesystem (certain filesystems have trouble at this scale).
I’m grateful for both tools. Thanks to the respective authors!
So weird. I mean, it's just about some open source tool, right? :-/
‡ The main commercial product of the ugrep author's company at the time was the gSOAP code generator (it may still be), and that it not only works but makes a reasonably good C and C++ API from WSDL is proof that it is the product of a genius madman. It also allowed you to create both the API and WSDL from a C++-ish header, and both .NET and Java WSDL tools worked perfectly with it. We needed it to work and work it did.
At the time, the generated API was just difficult enough to use that I generated another ~1k lines of code for that project. IIRC, the generated API is sort of handle-based, which requires a slightly different approach than the strict RAII approach we were using. Generating that code was a minor adventure (generating the gSOAP code from the header-ish file, generating doxygen XML from the generated gSOAP code, then generating the wrapper C++ from the doxygen XML).
There are feuds about open source tools all the time. Text editors, Linux distros, shells, programming languages, desktop environments, etc... And ugrep vs ripgrep may be a poster child for C++ vs Rust.
It is not all bad, it drives progress, and it usually stays at a technical level, I've yet to see people killing each others for their choice of command line search tool.
I think the killer feature is compatibility with existing grep command line switches. Not needing to learn a whole new set of options is quite nice.