What is a transformer architecture?

The transformer is the neural network architecture underpinning most modern LLMs. Its key innovation is self-attention — each token can attend to every other token in the context window simultaneously, capturing long-range dependencies far more effectively than older RNNs or LSTMs.