Microsoft: This is how we integrated ChatGPT-style tech into Bing search

Microsoft has offered some details about Prometheus, the proprietary AI model it created to integrate ChatGPT-like functionality into Bing search.  

Microsoft has already said the new Bing is “much more powerful” than OpenAI’s GPT-3.5 AI model behind ChatGPT. Public experimentation, however, quickly showed the chatbot feature was prone to unexpected outbursts, which prompted Microsoft to limit the number of “chat turns” to five per session to stop it becoming confused. 

Also: What is ChatGPT? Here’s everything you need to know

Now, Microsoft’s CVP of search and AI, Jordi Ribas, has shared some details about Prometheus — Microsoft’s proprietary model that meshes Bing with OpenAI’s GPT model to generate more relevant results in Bing — and how it went about building the new Bing. 

Ribas’ explanation comes as users discovered that Microsoft had been testing Sydney (the internal codename for Bing Chat) with users in India in late 2021. Users there reported in November 2022 that Sydney was “rude” in almost exactly the same way users found it to be after Microsoft released the new Bing.      

According to Ribas, OpenAI shared its “game changing” post-GPT-3.5 model with Microsoft in summer 2022. Microsoft president Brad Smith said it was shown the OpenAI technology “early last summer” in his blogpost about Microsoft’s “responsible AI”.

OpenAI opened up ChatGPT to the world in November while Microsoft was reportedly in talks to invest $10 billion in OpenAI in December, which it confirmed in late January. It then released the new Bing on February 7, a day after Google rushed out its ChatGPT rival, Bard.  

“Seeing this new model inspired us to explore how to integrate the GPT capabilities into the Bing search product, so that we could provide more accurate and complete search results for any query including long, complex, natural queries,” explains Ribas. 

As Ribas points out, large language models like GPT-3.5 are trained on data up to a given point in time. In ChatGPT’s case, its training data stopped at some point in 2021. To make the user experience “richer, more relevant and accurate,” Microsoft decided to combine it with Bing’s back-end via Prometheus. 

Prometheus combines Bing index, ranking, and answer results with the version of GPT that Bing uses to deliver the Chat answer, according to Ribas.  

“Prometheus leverages the power of Bing and GPT to generate a set of internal queries iteratively through a component called Bing Orchestrator, and aims to provide an accurate and rich answer for the user query within the given conversation context. All within a matter of milliseconds. We refer to this Prometheus-generated answer as the Chat answer,” explains Ribas. 

Ribas adds that Bing search results, based on user search queries and chats, feed fresh information to Prometheus to help it answer recent questions and reduce inaccuracies. The process is called “grounding” — a technique Google used when it developed a large language model for intelligent robots. In Bing’s case, the GPT language model is grounded in Bing data, which is how Bing adds citations to Bing Chat responses.  

“Even though grounding is a remarkable innovation, it is a new technology and we need to think about how we apply it responsibly, including further reducing inaccuracies and preventing offensive and harmful content. That’s why a key goal of the new Bing preview is to collect user feedback to help us improve Prometheus further,” explains Ribas.

To improve chat answer accuracy, Microsoft is planning to increase the grounding data it sends to the model by fourfold. 

Ribas also says the Bing team were divided over how it should integrate chat into the familiar search query and response user interface. One side wanted the Chat answer to appear like search results, with answers placed on relevancy to the query. Others wanted a new “interactive, chat-based way of searching.”

Also: The best AI art generators

In the end, Microsoft combined Search and Chat in a single interface that allows users to switch between Search and Chat mode by clicking on an element of the page or by scrolling up and down.