AI-ANNE
GENERATIVE TRANSFORMER
(1) Initial Settings
Architecture Hyperparameters
Embedding Dimensions
14
few (2) or many (24) dimensions
(2) Data
Training Data
Example Sentences as Training Data
The queen sees the dog. The queen calls the dog. The queen feeds the dog. The queen praises the dog. The dog sees the queen. The dog hears the queen. The dog follows the queen. The dog guards the castle. The dog searches for the bone. The dog finds the bone. The gardener sees the dog. The gardener calls the queen. The gardener brings the food. The queen gives the dog food. The queen gives the gardener the bone. The dog sleeps in the castle. The queen sits in the garden. The gardener works in the garden.
Tokenize
Example Data
Suggested Training Hyperparameters
Vocabulary Size
–
Batches
–
Context Length
–
Attention Heads
–
(3) Training
Training Hyperparameters
Vocabulary Size
60
few (5) or many (80) words
Context Length
10
few (3) or many (20) tokens
Attention Heads
one (1) or four (4) attention heads
Epochs
few (50) or many (1000) epochs
Learning Rate
0.04
slow (0.01) or fast (1)
Start Training
Stop
Multi-Head Attention (max. 2 visualized)
Loss Function (Cross-Entropy)
(4) Embeddings
Mathematical Representations
Token Vector Matrix
Token
Vector
(5) Sentence Generator
Sampling Hyperparameters
Maximum Sentence Length
Safety limit (if the end of the sentence is missing)
Temperature
0.8
low (0.2) or high (2.0) creativity
Top-k (Number)
5
few (1) or many (10) alternatives
Top-p (Probability)
0.90
low (0.1) or high (1.0) probability
Generate Sentence
Result
–