g5g vs g6 for LLM Serving: the Same Code, and 3.7x the Throughput

A comparison of AWS GPU instance families for serving a small language model reveals that g6.2xlarge delivers 3.7x the throughput of g5g.2xlarge. The g6.2xlarge utilizes an x86_64 host with an NVIDIA L4 GPU, while the g5g.2xlarge pairs a Graviton2 host with an NVIDIA T4G GPU. Findings indicate the older g5g family loses 87% of decode performance to dtype conversion, a degradation that remains undetected by standard logs, metrics, or health checks during spot instance operations.