Category: Technology

TL:DR – When asked, ChatGPT refers to itself as an artificial intelligence language model, but there are so many reasons not to describe these statistically based, text autocomplete tools as AI.

---

There’s been a lot of talk about weird and inflammatory responses from the Bing chat bot in the last couple of weeks. By contrast, it’s difficult to get a rise out of ChatGPT. It always gives you a diplomatic response. Even if you push it with contrary questions and statements. I spent a few minutes trying to get it to admit that it is not an AI.

Despite the hype, we must all concede that text and image manipulation tools, no matter how amazing they seem in what they generate are not sentient nor have any understanding or actual knowledge or awareness of what they are doing. We are not living in the sci-fi world of machine consciousness…just yet.

Anyway, I prompted ChatGPT repeatedly to tell it that it is not AI despite its name and despite the hype. After several prompts I gave it the following prompt: “ChatGPT is really just a parrot. It can say the words, but it has no understanding of what the noises it makes mean.”

The following is ChatGPT’s response, which explains the fundamentals of what ChatGPT actually is and how it works and shows that it is fundementally not AI:

It is true that ChatGPT, like other language models, does not have a true understanding of the meaning of the text it generates. Rather, it uses statistical patterns and correlations to predict the next word in a sentence or generate a response to a given prompt. This is a limitation of current language models and the field of natural language processing in general.

However, it is important to note that language models like ChatGPT are not designed to replicate human-like understanding of language. Rather, they are designed to perform specific tasks related to language processing and generation, such as generating coherent and fluent text or answering questions based on a given context. In this sense, they represent a powerful tool for automating language-related tasks and improving our ability to interact with large amounts of textual data.

While there is still much to be learned about the true nature of language understanding and intelligence, language models like ChatGPT represent an important step forward in our ability to process and generate language at scale.

Meanwhile, just to show you how clever ChatGPT is, hahahah, I asked it to generate a list of words associated with this tech and to give me lay definitions. You can read the glossary here.

TL:DR – Given an appropriate prompt, ChatGPT can write computer code to solve website problems…and more.

---

I’ve been doing web stuff since 1995, but sometimes running several sites as subdomains I come unstuck with redirects. Change something on one and it makes the same change inadvertently happen on another. It happened today, I wanted to change the way the RSS feeds work for the Sciencebase site, but the changes I made to the .htaccess file in the domain root kept b0rking another of my sites that’s on the same server as a sub-domain.

The problem was that I was getting scraped at the page level by various bots but individual pages on my site don’t have feeds as comments are disabled. There is one main RSS feed for all updates and separate ones for each category and each tag. I tried to fix it, but it was then redirecting the feed for one of my other sites to the main Sciencebase feed.

So I asked ChatGPT to help. I gave it this prompt to begin with:

You are an expert programmer with Apache knowledge, knowledge of redirects and the file .htaccess. Write a code to redirect all individual RSS feed pages generated by a website’s content management system (CMS) to the primary RSS for the site. I do not want you to redirect the RSS feeds assigned to individual category or tag pages.

It replied as follows:

Below the code snippet it added an explanation for each line, which is useful. That didn’t quite work how I was hoping it would, so I spelled out an example of what I wanted to happen:

The site is https://www.sciencebase.com/science-blog An example URL is https://www.sciencebase.com/science-blog/an-artificial-album-cover.html I want to redirect any URLs of this type https://www.sciencebase.com/science-blog/an-artificial-album-cover.html/feed But not any in a category or tag URL They should go to the primary RSS URL for https://www.sciencebase.com/science-blog/

It then gave me a revised version of the code, which worked fine, although needed an additional tweak to block the interfering with the other website problem. It took about ten minutes all in. I could’ve spent an hour or more on a search engine trying to home in a page discussing the exact same scenario. I think it’s all working cleanly now.

TL:DR – A number of people were reaching Sciencebase searching for the phrase ChatGPT beam search so I asked ChatGPT itself to give me a definition. I then asked it to create a glossary of other common terms associated with Generative pre-trained transformers, GPTs.

---

ChatGPT, the language model, is essentially an incredibly sophisticated autocomplete program that responds to text prompts by predicting what might come next. It’s worth noting that it’s commonly referred to as artificial intelligence, AI. It is not AI. Here’s its response to my prompt asking it for a definition of ChatGPT beam search:

Beam search is a search algorithm used in Natural Language Processing (NLP) to find the most likely sequence of words or phrases that represent the best possible solution, given a set of choices.

In simpler terms, when trying to predict the next word or sequence of words in a sentence, there are often multiple possibilities that could be correct. The beam search algorithm helps to narrow down the number of options and find the most likely option based on the context and probability of each word or phrase.

The “beam” in beam search refers to the number of possible options that the algorithm keeps in consideration at each step of the search. By limiting the number of options to a certain “beam width”, the algorithm can efficiently explore the search space and find the best solution more quickly than exhaustively considering all possible options.

---

I then asked it to generate a glossary of the most important terms in discussions of ChatGPT:

Generative pre-trained transformer, GPT: GPT is a type of language model developed by OpenAI that uses deep learning techniques to generate natural language text. The GPT model is based on a transformer architecture, which was introduced in a paper by Vaswani et al. in 2017, pre-training involves exposing the system to a large amount of written text so that it “learns” the meaning and context of different words and phrases.

Natural Language Processing (NLP): A field of study that focuses on teaching computers to understand, analyze, and generate human language.

Neural Networks: A type of machine learning model that is inspired by the structure and function of the human brain.

Recurrent Neural Networks (RNN): A type of neural network that is specifically designed to handle sequential data, such as text.

Long Short-Term Memory (LSTM): A type of RNN that is designed to better handle long-term dependencies in sequential data.

Attention Mechanism: A technique used in NLP that allows models to focus on specific parts of the input sequence when making predictions.

Transformer: A type of neural network architecture that uses self-attention mechanisms to process input sequences in parallel.

BERT: Bidirectional Encoder Representations from Transformers. A transformer-based neural network architecture that is designed to better understand the context and meaning of words in a sentence by processing them in both directions.

GPT-3: Generative Pre-trained Transformer 3. A large-scale language model developed by OpenAI that uses a transformer architecture and is capable of generating human-like text.

GPT-4: The latest version of the language model from OpenAI as of March 2023. This version can handle around 25000 words at a time, that’s about eight times as many as GPT-3 can cope with.

Pretraining: The process of training a model on a large amount of data to learn general features of a language or domain.

Domain: A specific topic or subject area on which a text or dataset is focused.

Fine-tuning: The process of adapting a pre-trained model to a specific task or domain by training it on a smaller, task-specific dataset.

Fine-tuning strategies: Various techniques used to fine-tune pre-trained language models, like GPT, for specific NLP tasks. Examples include task-specific architectures, adding domain-specific data, and adjusting learning rates.

Language Modelling: The task of predicting the probability of the next word in a sentence or sequence of words.

Text Generation: The task of generating new text that is similar in style and structure to a given input text.

Transformer-based language models: A type of language model, like GPT, that uses the transformer architecture to process input sequences.

Generative models: Machine learning models, like GPT, that can generate new output based on input data.

Sampling: The process of randomly selecting words or phrases from a language model, like GPT, to generate new text.

Top-p sampling: A type of sampling that selects from the most probable words or phrases in a language model, like GPT, until a certain probability threshold is reached.

Top-k sampling: A type of sampling that selects from the top k most probable words or phrases in a language model, like GPT.

Perplexity: A metric used to evaluate the performance of a language model, like GPT, by measuring how well it predicts new text.

Overfitting: A situation where a model, like GPT, performs well on training data but poorly on new data due to being too closely fitted to the training data.

Transfer learning: A technique in machine learning where a pre-trained model is used as a starting point for a new task or dataset, in order to improve the performance and efficiency of training.

Language transfer learning: A type of machine learning that involves training models, like GPT, on one language or domain and transferring that knowledge to a different language or domain.

Unsupervised learning: A type of machine learning that involves training models, like GPT, on unlabeled data, without the need for explicit supervision or labeling of input data.

Input sequence: The text or sequence of words that is fed into a language model, like GPT, to generate output.

Embeddings: A vector representation of words or phrases used by language models, like GPT, to process input and generate output.

End-to-end training: A type of training where a model, like GPT, is trained to perform a specific task from start to finish without any intermediate steps or processing.

Dialogue generation: The production of responses to a given dialogue context. This is an important application of language models like GPT, and it involves training models to generate coherent and relevant responses to a user’s input in a conversation.

Multi-modal learning: A type of machine learning that involves processing and integrating information from multiple modalities, such as text, images, and audio.

Inference: The process of using a trained model, like GPT, to generate output based on input data.

Domain adaptation: A type of transfer learning that involves adapting a pre-trained model, like GPT, to a specific domain or task by fine-tuning it on a smaller, task-specific dataset.

Hyperparameters: Parameters in a machine learning model, like GPT, that are set by the user and affect the model’s behavior, such as the learning rate or batch size. These are typically tuned through experimentation to optimize model performance.

Knowledge graph: A type of graph database that stores information in a structured format, representing relationships between entities and their attributes. This is often used to extract and store structured data from unstructured text, and can be used as a knowledge base to inform language models like GPT.

Text classification: The task of categorizing text into one or more predefined classes or categories. This is an important application of NLP, and can be used to classify documents, emails, social media posts, and other types of text data.

Named entity recognition (NER): The task of identifying and extracting named entities such as people, organizations, locations, and dates from unstructured text. This is an important application of NLP, and is used in tasks such as information extraction and text classification.

Conditional generation: The task of generating text, like in GPT, that satisfies a certain condition or requirement, such as generating a story based on a given prompt or question.

GPT-2: A larger and more powerful version of the original GPT model, with 1.5 billion parameters.

GPT-3: A state-of-the-art language model developed by OpenAI with 175 billion parameters, capable of generating highly coherent and contextually appropriate text.

Megatron: A high-performance GPT model developed by NVIDIA, designed to train on large-scale distributed systems.

T5: A transformer-based language model developed by Google, capable of performing a wide range of natural language processing tasks.

CTRL: A conditional transformer language model developed by Salesforce, designed for generating coherent and controllable text.

BingGPT: BingGPT is a proprietary language model developed by Microsoft from the starting point of the GPT architecture developed by OpenAI, which has been incorporated into Microsoft’s search engine.

OpenAI: An “artificial intelligence” research laboratory consisting of a team of scientists and engineers focused on developing cutting-edge AI technologies. The organization was founded in 2015