Vishakha Ramani

Research Staff Member, IBM T. J. Watson Research Center

prof_pic.jpg

I am a Research Staff Member at IBM T. J. Watson Research Center in Yorktown Heights, New York. I work on performance modeling and analysis of computing systems. My current focus is scheduling and autoscaling large language model inference at production scale.

Research

Queueing models of LLM inference. Three parameters per model–GPU pair predict time to first token and inter-token latency closely enough to drive an autoscaler. Paper · Blog post

Autoscaling in llm-d. I co-architected the Workload Variant Autoscaler, which sizes LLM inference deployments against latency targets. Code

Age of Information. My PhD at Rutgers (2024), with Roy Yates, on keeping data fresh as it is stored, retrieved, and processed. Thesis

More on my research →

News

Outside research, I follow professional road cycling closely (what I’m watching now) and play piano on occasion.

selected publications

  1. Preprint
    An Approximate Queueing Model of LLM Inference Serving for SLO-Driven Autoscaling
    Vishakha Ramani and Asser N. Tantawi
    arXiv preprint arXiv:2609.20957, 2026
    Abs arXiv PDF

    We present a tractable approximate queueing model, built on Markovian assumptions, of how LLM inference servers interleave prefill and decode work. Each model-accelerator pair is described by three parameters: a fixed per-iteration overhead, a per-token compute cost, and a per-token KV-cache access cost. Mean-value analysis of per-iteration work is combined with a state-dependent Markov chain over batch occupancy to predict mean time to first token (TTFT) and inter-token latency (ITL), and the parameters can be fitted online from observed latencies. On an H100 GPU under light to moderate load, mean ITL error is about 5% for Llama-3.1-8B and 8% for Qwen2.5-14B, with TTFT errors of 14% and 16%. An autoscaling controller built on the model tracks a fourfold load ramp on an OpenShift H100 cluster, missing the SLO in 7 of 127 control cycles with in-loop median prediction errors of at most 5% for TTFT and 9% for ITL, compared with 27 of 128 cycles for a decode-throughput analyzer without a latency target.

  2. WVA: A Global Optimization Control Plane for llm-d
    Abhishek Malvankar, Vishakha Ramani, and others
    In IEEE International Conference on Cloud Computing (CLOUD), 2026
  3. Bottlenecks of AI Inference: A Performance-Modeling Perspective
    Tarek Abdelzaher, Vishakha Ramani, and others
    Real-Time Systems, 2025
  4. Lock-Based or Lock-Less, Which is Fresh?
    Vishakha Ramani and Roy D. Yates
    In IEEE INFOCOM, 2023