Mathematics for AI — Ch.05: Information Theory [KR]

Entropy measures the average information of random variable outcomes, linking probability theory to communication, data compression, statistical learning, and AI. Scaling AI systems expands computational capacity, service throughput, data processing capability, and operational reliability as workload grows; scalable architecture must handle increasing users, models, devices, and requests without unacceptable performance loss, requiring integrated design across infrastructure, networking, storage, software, and resource management. Compute optimization seeks to minimize execution time, memory consumption, energy use, and operational cost while maximizing computational efficiency, integrating algorithm, hardware architecture, memory systems, and execution strategies.