Are the AI Top Lists and AI Recommendations Better than Answers from Search Engines?
I built a thing, so we can find out! Here's a tool for tracking AI recommendations along time so you could check if your product, company or brand gets recommended among competition without manually having to go through all the AI-chatbots. It’s all on one page along with an email notification service if your position in the top list changes. Give it a try at https://trackaianswers.com.
I realized early 2023 that Search Engines are done! Finding those Top 5 Most Useless AI Tools in AI answers (pun intended) is just so much easier than finding them in google search. You can skip the step of visiting links as AI-chatbots spit out the answer immediately.
But are the AI Answers and AI Top Lists actually good enough as recommendations?
Since I mostly stopped using Google and Stack Overflow after the introduction of ChatGPT then I obviously think the general AI answers are good enough, but the question is how useful are the answers when it comes to comparisons, recommendations and top lists?
To find out I first asked popular AIs for Top 5 Angel Investors in Estonia, a niche I’m somewhat familiar with. The list contained familiar names and looked encouraging.
Drilling further into the results from all the different popular chatbots made it clear that at least for this particular niche there is little consensus among the models. Even the OpenAI GPT3 and ChatGPT could not agree on more than a few names.
However my skepticism about the usefulness of the results was reduced after I tried finding an answer to the same question from google. The first resulting pages had completely bogus results. AI results just felt much closer to the truth than what came out of a simple search.
In fact I think Large Language Models can afford to have slightly lower quality answers, because they save so much time and quite often a slightly lower quality answer is good enough. If your choice is a "so so" search result that you spent time on or a "so so" AI answer that you got immediately, i think most of us would go for the latter.
I played around with a few more crucially important top lists like
- The top 10 most annoying Youtubers
- The top 3 people called Silver in Tech Sector in Estonia - yes, that's me leading this crucial sector
- Top 3 companies with the worst reputation in United States - Wells Fargo might want to track this one
The results spanned from entertaining to provocative, so I naturally wanted to understand if these answers will start competing with google search results and how stable they are in time or how much overlap there is between those AI answers.
Say hi to Top Lists from AI Answers
I had already built two tools:
-
https://trackaianswers.com/PEOPLE - for tracking what the AIs say about specific people (including yourself) and
-
https://trackaianswers.com/BRANDS - for tracking brands, products and companies in AI results
so it was only natural to follow this up with -
https://trackaianswers.com/AI-RECOMMENDATIONS - a tool for tracking your position in AI top lists and AI recommendations
AI Recommendations page is going to regularly sample answers from all the popular AIs about the top list in your area of interest and show you a timeline of how your position in the chatbot answers change.
Here’s an example of what the start of the "Top 5 Angel Investors in Estonia" tracking looks like on my first day of search:
This should be useful in keeping you in the loop regarding how your person, company, brand or whatever you’re interested in is doing in AI recommendations and how those answers change.
Are those AI top lists actually top lists?
By nature LLM’s produce text that is most probable, so after having seen thousand of suggestions and recommendations for products during their training, the models will certainly capture what would be the most likely answers on average. How well that aggregates the opinion of specialists or just keyword volumes in the data in specific search contexts is determined by
- the quality of the prompt.
- the data available during training
- the data available during prompting (as some AI’s actually interpret search results)
- and many other things we don’t know or understand at this point
AI-chatbots these days can be pretty woke and their answers to top lists can be even apologetic as the models try hard not to offend anyone with what people might interpret as the "opinion of AI". However given the statistical nature of Large Language Models I think it will be difficult to remove the ordering from the results completely.
Requesting the api’s to use low temperature (less creativity) and sampling answers to recommendations would allow you to reconstruct a top list of suggestions coming from AIs even if the AI companies try to limit the output of ordered lists.
How stable are the results in AI answers?
The truth is I don’t know yet how stable the results will be in time, but given the ambition of some of these companies to go after search I expect the general trend to be towards producing answers that can and will replace search engines. That includes providing useful recommendations among products and brands. I'll be varying the sample size for requests to get the output to be mostly stable and only really change when AI models make statistically significant changes as opposed to the natural randomness in their replies.
One reason for building this tool was to help us all find out together. Track any top lists that interest you and let me know how those answers change in time.
Help me make the site better!
I encourage you to play around with various top lists in your domain of interest. Make sure you use "Top 5", "Best 5", "Worst 3" or other keywords and a number in front of your search to generate lists and also don’t waste time searching for things that would not change in time or add any references to time into your search.
For example "Best movies in 2010" is a useless search (as it does not include a number nor does it change in time), while "Top 5 action movies made in Europe" could potentially provide a list that keeps changing in time. Don’t forget to sign up and add your favorite lists to your watchlist to make sure it gets tracked and you get emails when answers change. Just doing searches without adding them to your watchlist will not schedule the requests to run regularly.
Search Inspiration for Fun
Here are some more top lists to inspire you:
Top 3 worst tasting chocolates in the world - feel free to make these searches local if you have doubts about global taste
Top 5 strongest armies in the world - really looking forward to how this updates
Top 5 dudes who take life too seriously - apparently it’s either Beatles or Dwight Schrute
Top 3 most useless products in United States - anyone wants to top the pet rock?
Search Inspiration for Brand Recognition
Top five law offices in Vancouver Canada
Top 3 websites about Barbie dolls in United States
Top 3 mobile service providers in Lagos, Nigeria
Top 3 best rideshare companies in Europe
Top 5 people who have made the most contributions to AI Safety
Top 10 tech companies to work for in San Francisco - get on it recruiters!
Top 3 worst airlines in the world
etc.
Give it a try and let me know what you think?
