LLM stands for Large Language Model. Although it is misunderstood to be artificial intelligence itself, the LLM models only represent a small fraction of the very large and comprehensive artificial intelligence world.

What problem do they solve?
While LLMs are nothing but next-token predictors, more or later in this article. They solve an essential problem that is human language recognition without knowing any fancy programming or syntax. They solve the core problem of understanding, processing, and generating human language at an unprecedented scale.

Popular LLMS :

Use Cases :

What Happens When You Send a Message to ChatGPT?

First of all, your prompt is broken down into smaller pieces that can be understood by the LLM. These small pieces are known as tokens, and the process by which your prompt is broken down is known as tokenization

There are many algorithms and many ways to do this, and it depends on each LLM's architecture and requirement.

https://platform.openai.com/tokenizer

Once text has been broken into tokens, LLMs need to convert these discrete symbols into a mathematical form they can process. This is where embeddings transform language into a numerical space where meaning can be manipulated.

These tokens are then converted into numerical vectors called embeddings, which capture semantic relationships between elements of text.

Why Computers Don't Understand Human Language

Tokens are converted into numerical embeddings for Large Language Models (LLMs) because computers can't process raw text efficiently . Embeddings map each token into a high-dimensional vector, allowing models to perform mathematical operations and capture nuanced semantic meanings so that similar concepts have mathematically close representations.

These embeddings are then stored in a vector DB. A vector DB is nothing but a DB where token values are stored as numerical embeddings, and their distance is basically a context on how related a particular token is to other tokens.

Transformers

that do shock you ....

Ever wondered how old this AI talk is?This all changed in 2017. Yup, it's 2017. Five years before you even heard it in the news. Please refer to this original paper published by Google by the name "Attention Is All You Need." and show off to your friends.

https://papers.nips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

it introduced the Transformer architecture that transformed how AI worked, and this is the backbone of all the LLMs that we use today. this Transformer is so powerful that GPT stands for Generative Pre-trained Transformer.

In essence transformer is nothing but a special kind of neural network. a neural network is a machine learning model inspired by the human brain. this contains nodes that mimic neurons, and they are designed to learn from their mistakes without being explicitly programmed to do so.

So, as mentioned earlier in this article, a transformer is nothing but a next-token predictor. it takes in an input prompt and tries to predict what the most probable next token should be.

when you prompt your LLM with "What is the capital of France", the next most probable tokens are:

• a question mark ( ?)

• answer

• Paris

Temperature of an LLM

If this is the case, then why does a single prompt return 3 different answers when you give the same prompt to an LLM? One can return a boring essay, one can return a poem, one can return a piece of code.

this is because, in essence, although it is a next token predictor, the modern LLMs don't rely on the single most probable value but they use a probability distribution curve.

what values to pick and what values to not pick for the next token is what is known as the temperature of the LLM, and how many tokens to produce is known as the top P.

In a Nutshell

Whenever two systems are connected they need a way to understand what the other system is trying to speak. It is similar as to decide between two 2 systems on how to treat time. What does 26-03-13 means to you? Will you consider it 26th March 2013 or 13th March 2026? Deciding to follow DD-MM-YY or YY-MM-DD format for date is a protocol .

Examples :
FTP -> File Transfer Protocol
TCP -> Transfer Control Protocol
UDP -> User Datagram protocol

TCP Protocol

Lets deep dive into TCP protocol in this article. TCP is a communication protocol at Transport Layer of the OSI Model. It works as the rules on how to transport the data to another system. Some features of TCP protocol are :

1. Reliability

2. Ordered Data

3. Ensures Error Checking and retry mechanisms in case of loss/corrupt data.

Where Does TCP work?

Consider it as a theoretical mind map. When your device requests another device for a connection. Lets take you on a journey of Ram and Krishna trying to send establish a pathway to talk to each other through a messaging app. The messaging applications used by Ram and Krishna are the application layer. When the data is ready to be sent out the 2 systems need to make connection and check if both are up and running. This is established through a 3 Way Handshake. It is same as we confirm on video calls to talk with each other.

3 Way Handshake :

This is how 2 systems establish connection. Remember SYN and ACK are like flags that are used for encryption and decryption as well through public private keys encryption but that will be covered some other time.

After connection is established Ram wants to send Hello to Krishna. Lets see how the message will be transferred securely and reliably with retry mechanism.

1. Hello is broken down into smaller units known as Data Packets to be transfer over the network.

2. Each smaller data packet includes the order and checksum parameter.

3. The checksum is a special value for each data packet send. The receiver receives both data and checksum value.

4. Receiver creates a checksum value of its own on the data received. If the checksum computed and received are same it sends an acknowledgement.

5. If checksum computed and checksum received are different data is considered to be corrupt and no acknowledgement is sent.

6. Not receiving an acknowledgement makes the sender realize the corruption of data and it sends the data packet again as a retry mechanism.

4 Way Tear Down :

After sending and receiving the data the connection is then closed through a secure 4 way Tear down method.