We use cookies to ensure you get the best experience on our website.
Accept
Deny
Glossary
Batch inference

Batch inference

Copy link
www.titanml.co/glossary/batch-inference

Batch inference is the process of running models with batched inputs to increase throughput.

Related Articles

No items found.
Learn More
Join Beta

No Next Term!

Check out our other Terms

No Previous Term!

Check out our other Terms

Tensor Parallelelism

Context Length

Rate Limits

HIPPA

Docker

Llava

Containerized

Public Cloud

Virtual Private Cloud (VPC)

Self-hosted models

Compression

Bandwidth

Autoscaling

API-based large language models

API

CI/CD Pipelines

Kubernetes

Node

Inference

Inference Server

Mixture of Expert Models (MoE)

Continuous batching

Multi-GPU inference

Zero shot learning

Unsupervised learning

Weight

Turing Test

Transformer

Training set

Transfer learning

Training data

TPU

Top P

Top K

Tokenization

Token

Throughput

Titan Takeoff Inference Server

Synthetic data

Supervised learning

Serving

Speculative decoding

Sentiment analysis

Sampling temperature

Rust

Recurrent neural network (RNN)

Repetition penalty

RAG (Retrieval Augmented Generation)

Quantization aware training

Quantization

Pruning

Prompt engineering

Pretrained model

On-prem

Perplexity

Natural language processing (NLP)

Ngram

Natural language understanding (NLU)

Neural networks

Model serving

Model parallelism

Human in the loop

Model monitoring

Model

Model compilation

Mistral

Machine learning (ML)

LLaMA

Latency

Large language model

Language model

Kernel

Inference optimization

Instruction tuning

Machine learning inference

Hallucination

GPT

GPU

Falcon

Generative AI (GenAI)

Few shot prompting

Foundation model

Fine tuning

F-score

Few shot learning

Epoch

Encoder

Distillation

Data parallelism

Dynamic batching

Deep neural network (DNN)

Deep learning

CPU

Cloud computing

Classification model

Classification

Big data

Chain of thought prompting

BERT

Batch size