Today we are announcing the YandexGPT 3 line of neural networks. The first of them, YandexGPT 3 Pro, is already available via API on the Yandex Cloud website, including in free demo mode. In addition, you can now additionally train a new neural network yourself.
New generation neural networks work better with complex queries and more accurately follow a given response format, making them especially useful in solving real problems for users and companies.
In the near future, YandexGPT 3 neural networks will appear in Yandex services for a wide audience.
To evaluate the quality of YandexGPT 3 Pro, we conducted several tests. First, we evaluated the model on a localized version of the international MMLU benchmark. Secondly, we tested the model using the Side-by-Side method on real requests from users and companies. Thirdly, we created our own Russian-language test based on the IFEval benchmark to assess the compliance of the response with the format specified in the request. Now a little more about each.
To evaluate the quality of the new neural network, we created YaMMLU_ru, a Russian-language version of the open international benchmark MMLU. To do this, we translated the original tasks into Russian using Yandex Translator. Then the experts double-checked the texts, corrected errors, and also localized the queries (for example, brought the units of measurement into compliance with Russian standards). This version allows us to better take into account the local context and specifics of queries formulated in Russian.
We also applied the SBS method to evaluate how the new model copes with idea generation, information summarization, classification tasks, content creation, and other requests that are in demand among users and companies.
The number of situations when the model does not answer the user’s question has decreased by 5 times. In addition, the new language model makes significantly fewer mistakes. We tested this on a special set of particularly complex queries. Results for this set:
To check how well the neural network's responses correspond to the format specified in the request, there is a good benchmark IFEval. It contains prompts, the answers to which can be assessed quite accurately. For example, “write a text that contains more than 400 words” or “mention the term AI at least three times.” To evaluate YandexGPT 3 responses, we created a Russian-language version of the benchmark based on IFEval. At the same time, the list of tasks to be solved was significantly expanded and complicated.
Compared to YandexGPT 2, the quality of YandexGPT 3 answers improved by 10 percentage points, and their consistency doubled. This means that the neural network has learned to better understand what exactly the answers should be to queries that are essentially the same, but formulated differently.
YandexGPT 3 Pro can be built into products via the API. The cost of using the new neural network has almost halved, but you can test it for free. In demo mode, 30 free requests per hour are available to new registered users.
The new neural network works well in areas such as customer support, online sales, digital communications, marketing, advertising and personnel management. Also, the language model works better with documents: for example, it draws up contracts, invoices, regulatory documentation, job descriptions and more. Tasks related to the industries listed above made up a significant portion of the YandexGPT 3 training dataset.
An example that we created via the API on the Yandex Cloud website:
In addition, you can now independently train YandexGPT 3 Pro in the Yandex DataSphere service to make it even better suit your needs. To start the additional training process, you need to upload a file with example queries and standard answers to them into DataSphere. The retrained neural network will be available only to you.
bbabo.Net