Cool to see a new general purpose search engine with its own crawler! I tried a couple of queries and it worked fine for me. Searching for for inetdex (https://www.tuxdex.com/?q=inetdex) returns 1,442 documents that apparently just reflect the crawler's user agent though. Might need some tuning. Can you (assuming OP is affiliated with the site) say something about the team and business model, if any? Would also be interesting to hear about what infrastructure you're running on and if there's any plans to open source the crawler.
I really like the minimal/90s look and feel too--keep up the good work, assuming it's not a honeypot! :^)
Yup, a plus from me about the clean 90's look, hope it will stay that way.
Searched for a term, gives +400k results, up to page 80 and still going. Another big plus from me since I just love to search low ranked results, sometimes you find true gems in obscure pages.
The search results come very fast, but there's some low hanging fruit QOL optimizations as far as I can see, mostly to do with search result ranking.
It doesn't seem to weigh the terms properly, there is some pull toward relevant terms but not as much as I'd expect from BM-25. I also think it doesn't promote results with adjacent terms enough. As a result the relevance of the search results sort of undulate up and down as you go through the results.
The de-duplication also seems non-existent. You can easily get a page full of duplicates of the same two links. You come a pretty long way if you just keep a simple hash of the HTML of each website or whatever.
This is made worse by what I think is a bug where the crawler follows redirects, and reports them as the original URL and not the redirected URL. I actually had the same bug in my crawler when I started out. Nasty anti-synergy with Wikipedia which has a lot of redirect.
These are relatively minor issues that shouldn't be hard to fix.
Thank you for the many posts and for visiting the TUXDEX.COM.
TUXDEX has received an update of both the software and the indexes, thanks bacon-and-stars we have been able to fix the error, TUXDEX is safe from XSS and other attacks.
About the results: We are working on our algorithm and other methods that will improve the results.
Please keep in mind that the well-known search engines sometimes employ thousands of people and have been on the market for years.
This. I'm glad these guys are running their own crawler, but their story doesn't add up.
Most likely this is an outfit that does a lot of webscraping and wants to minimize the amount of manual banning they incur. So they set up a plausible indie search engine as a front. Of course the search engine has to actually work when the scrapees notice and try it out.
I dont remember exactly, but it's possible to create a completely anon search engine. User encrypts the query using their key and sends it to search engine, search engine now doesn't know the query, but then it encrypts its entire database using public key of user and see if hashes meet for keywords?... something like that
Our search server receives the request from the web server and returns the results. The web server, which could store the user IP, does not know the search query, since the POST method is used for transmission. The search server that processes the request does not know the user IP, only the web server.
The logs will be deleted after the visitor numbers have been determined.
Your search query remains anonymous at https://tuxdex.com.
Readit News
I really like the minimal/90s look and feel too--keep up the good work, assuming it's not a honeypot! :^)
Searched for a term, gives +400k results, up to page 80 and still going. Another big plus from me since I just love to search low ranked results, sometimes you find true gems in obscure pages.
It looks like the html tags from one of the search results messed up the page
The search results come very fast, but there's some low hanging fruit QOL optimizations as far as I can see, mostly to do with search result ranking.
It doesn't seem to weigh the terms properly, there is some pull toward relevant terms but not as much as I'd expect from BM-25. I also think it doesn't promote results with adjacent terms enough. As a result the relevance of the search results sort of undulate up and down as you go through the results.
The de-duplication also seems non-existent. You can easily get a page full of duplicates of the same two links. You come a pretty long way if you just keep a simple hash of the HTML of each website or whatever.
This is made worse by what I think is a bug where the crawler follows redirects, and reports them as the original URL and not the redirected URL. I actually had the same bug in my crawler when I started out. Nasty anti-synergy with Wikipedia which has a lot of redirect.
These are relatively minor issues that shouldn't be hard to fix.
TUXDEX has received an update of both the software and the indexes, thanks bacon-and-stars we have been able to fix the error, TUXDEX is safe from XSS and other attacks.
About the results: We are working on our algorithm and other methods that will improve the results. Please keep in mind that the well-known search engines sometimes employ thousands of people and have been on the market for years.
http://tuxdex.com (Anonymous Search Engine)
Deleted Comment
Most likely this is an outfit that does a lot of webscraping and wants to minimize the amount of manual banning they incur. So they set up a plausible indie search engine as a front. Of course the search engine has to actually work when the scrapees notice and try it out.
And the search engine is actually pretty decent (from my test).
You could easily get 100 not requests for every 1 user request.
The logs will be deleted after the visitor numbers have been determined. Your search query remains anonymous at https://tuxdex.com.
Problem solved.