Batch

Copy link

A batch of inputs is a set of inputs which are processed by a machine learning model in parallel. This is an effective technique to increase throughput when using GPUs.

No items found.

Learn More

Join Beta

Previous Term

No Next Term!

Check out our other Terms

Next Term

No Previous Term!

Check out our other Terms

No Next Term!

No Previous Term!

Tensor Parallelelism

Context Length

Rate Limits

HIPPA

Docker

Llava

Containerized

Public Cloud

Virtual Private Cloud (VPC)

Self-hosted models

Compression

Bandwidth

Autoscaling

API-based large language models

API

CI/CD Pipelines

Kubernetes

Node

Inference

Inference Server

Mixture of Expert Models (MoE)

Continuous batching

Multi-GPU inference

Zero shot learning

Unsupervised learning

Weight

Turing Test

Transformer

Training set

Transfer learning

Training data

TPU

Top P

Top K

Tokenization

Token

Throughput

Titan Takeoff Inference Server

Synthetic data

Supervised learning

Serving

Speculative decoding

Sentiment analysis

Sampling temperature

Rust

Recurrent neural network (RNN)

Repetition penalty

RAG (Retrieval Augmented Generation)

Quantization aware training

Quantization

Pruning

Prompt engineering

Pretrained model

On-prem

Perplexity

Natural language processing (NLP)

Ngram

Natural language understanding (NLU)

Neural networks

Model serving

Model parallelism

Human in the loop

Model monitoring

Model

Model compilation

Mistral

Machine learning (ML)

LLaMA

Latency

Large language model

Language model

Kernel

Inference optimization

Instruction tuning

Machine learning inference

Hallucination

GPT

GPU