My current AI chat benchmark is to ask about LLVM APIs. I’ve asked to both Bing Chat and Bard three questions:

- What info can the LLVM lazy-value-info analysis pass analyze?

- Can you give me code samples to use the info?

- Can you give me a list on important LVI methods?

Both fared well on the first question, but on the second question, Bard either didn’t produce LLVM code at all (showing examples of LLVM IR), or hallucinate non-existent methods not even close to what existed.

Bing chat, in comparison produced correct answers that are very helpful. In my experience Bing chat almost always produces something that is both coherent and useful; Asking LLVM questions, Bing chat can search the Doxygen documentation, find out common LLVM patterns on its own (like using a Worklist while iterating), and I’ve succeeded in writing whole LLVM passes that compiled in one go, from a very light description of an algorithm.

Maybe I’m spoiled with Bing chat, but I don’t think I’ll be using Bard that much if it’s quality doesn’t improve. Very disappointed personally. (I’d have expected Bard to work much better mostly just because Google searches better than Bing.)

My vanity domain. Now off to find my admin credentials and likely discover that the workspace setting isn't available yet.

So, in the first 7 messages I exchanged with it every single response it gave was a lie or a hallucination. Not exactly a promising start. If only there was a quick way to look up factual information that didn’t rely on opaque language models

When I quoted the man page on -F, it said this:

> You're right. The nmap man page does say that `nmap -F` scans fewer ports than the default scan. However, it does not say that `nmap -F` scans any ports at all.

1. It's arguing with me. 2. Sure, "fewer" could mean "zero", but no human would say "fewer" when they mean "zero".

ChatGPT, it should be noted, correctly answered the question (and didn't argue over silly semantics).

I use ChatGPT 4 to keep track of my task list and as a "procrastination coach". I tried this with Bard when they first announced it and it just couldn't do it. Now, though, it is pretty much on par with ChatGPT, and even has some niceties, like also automatically keeping track of my completed tasks.

> Write me a python function that accepts a list of filenames and uses threads and queues to load them in parallel with the pickle library

…

> Also, please output the results to a queue as well

Final result

  import pickle
  import threading
  import queue
  
  def load_files_in_parallel(filenames, results_queue):
    """Loads a list of files in parallel with the pickle library and outputs the results to a queue.
  
    Args:
      filenames: A list of filenames to load.
      results_queue: A queue to hold the results of the file loads.
  
    Returns:
      None.
    """
  
    # Create a queue to hold the file handles.
    file_handles = queue.Queue()
  
    # Create a thread pool to load the files.
    threads = []
    for _ in range(len(filenames)):
      thread = threading.Thread(target=load_file, args=(file_handles, filenames.pop(), results_queue))
      threads.append(thread)
      thread.start()
  
    # Wait for all threads to finish.
    for thread in threads:
      thread.join()
  
    # Return the list of objects loaded from the files.
    # No need to return anything since the results are already in the queue.
  
  def load_file(file_handles, filename, results_queue):
    """Loads a file and adds it to the results queue.
  
    Args:
      file_handles: A queue to hold the file handles.
      filename: The name of the file to load.
      results_queue: A queue to hold the results of the file loads.
  
    Returns:
      None.
    """
  
    # Open the file.
    with open(filename, 'rb') as f:
      obj = pickle.load(f)
  
    # Add the object to the results queue.
    results_queue.put(obj)

Lol, I'm in Germany :D I was wondering where all the people saying the same came from. Apparently not any country with an embargo, that was my theory.

ChatGPT was able to completely write this perfectly. Absolutely mind blowing. Saved me probably 1-2 hours of looking things up and fiddling with it.

Bard failed multiple times in multiple places. It only found AMIs that exactly, down to the second, were two weeks old. The filter json looks wrong ('Key' isn't valid AFAIK). It didn't bucket them and just kept the newest 10.

In either case both of these are mind blowing. If ChatGPT didn't exist, Bard would still be very helpful at getting me started. ChatGPT did it all without me making any changes though.

ChatGPT log: https://gist.github.com/mdeeks/de297f1bc8cbd00fe2db01e6232aa...

Bard log: https://gist.github.com/mdeeks/1404e09da8879b94469166927ddff...