Juncheng Yang

I am an Assistant Professor in Harvard John A. Paulson School of Engineering and Applied Sciences. I received my Ph.D. in Computer Science from Carnegie Mellon University working with Rashmi Vinayak. I am broadly interested in computer systems with particular focus on workload analysis, design of efficient, reliable and sustainable data storage and machine learning systems.

My works have received Best-Paper Award or Honorable Mention at NSDI'24, NSDI'21, SOSP'21, VALUETOOLS'24, and SYSTOR'16 and have been deployed in production at Google, VMware, Twitter, Redpanda with many open-source libraries contributed by the community. I am a 2020 Meta Fellow, a 2023 Google Cloud Research Innovator, and a 2023 Rising Star in Machine Learning and Systems. And I received the CMU SCS Doctoral Dissertation Award, the ACM SIGOPS Dennis M. Ritchie Doctoral Dissertation Award.

I lead the Harvard Measurements and Design of Computer Systems Group (MadSys Group) where we find problems instead of just solving them. And I am fortunate to work with several amazing students.

Storage systems and machine learning systems with a focus on efficiency, scalability and robustness:

• Efficient, sustainable and scalable storage systems
  • Measurement studies [OSDI'20] [HotOS'23] [VLDB'26 ]
  • New cache system designs [NSDI'21 ] [SOSP'21 ] [NSDI'22] [EuroSys'23]
  • New replacement algorithm designs [FAST'23] [SOSP'23] [NSDI'24 ] [VLDB'26] [OSDI'26]
  • Storage for Large Models [NSDI'26][SOSP'26]
• Efficient LLM serving and democratizing access to AI
  • Measurement and analysis of LLM and agent serving
  • Serving Prefill workloads [NSDI'27]
  • Hybrid Inference and Serving bursty workloads [MLSys'26] [Eurosys'27]
  • Prefix-cache management
• Robust and reliable learned systems [VLDB'23] [OSDI'26]

• FreeInference: democratizing access to large language models, currently serving more than 10B tokens daily to users from more than 50 countries speaking 25 different languages using 20+ agents.
• SIEVE (NSDI'24): the first cache eviction algorithm simpler than LRU but yet more effective than state-of-the-art algorithms for web caches. Adopted by software and systems such as Android API, BIND 9, ImmuDB, TiDB, PostgREST Implemented in many open-source libraries, e.g., Golang, Python, JavaScript, Rust, Java, Swift, Ruby, Nim, and Zig. Find more details on SIEVE website.
• S3-FIFO (SOSP'23): a simple and scalable cache eviction algorithm composed of only FIFO queues. Implemented or deployed at companies including Google, VMware and Redpanda, and many open-source libraries. Find more details on S3-FIFO website.
• Segcache (NSDI'21): received a community best-paper award, and deployed at Twitter and Momento.

We release open artifacts so that others can reproduce, extend, and build on our work. Most of them are led and maintained by students in the lab.

• Open-source libraries
  • libCacheSim — a high-performance library for building cache simulators, used by almost 100 research institutes and companies
  • libCacheSim-python — Python bindings for rapid cache simulation experiments (pip install libcachesim)
  • RouteWise — a library for LLM routers to optimize cost and latency across multiple providers
• Open Datasets
  • Cache trace dataset — a large open collection of key-value, object/CDN, and block cache traces (2007–2025) from Meta, Twitter, Alibaba, Tencent, and others
  • Year-long LLM serving dataset (coming soon)
  • FreeInference Agentic dataset (with full sanitized prompts and responses) (coming soon)
• Free Services
  • FreeInference — a free, OpenAI- and Anthropic-compatible inference API for frontier open-source models, aimed at research, teaching, and prototyping
  • traceViz — a web tool for visualizing and exploring storage and cache workloads
• Open infrastructure
  • A public dashboard for the GPUs and serving engine behind FreeInference (coming soon)

Juncheng Yang is an Assistant Professor in Harvard John A. Paulson School of Engineering and Applied Sciences. He received his Ph.D. in Computer Science from Carnegie Mellon University in 2024. His research interests broadly cover the efficiency, performance, reliability, and sustainability of large-scale data systems.

Juncheng's works have received best paper awards or honorable mention at VLDB'26, VALUETOOLS'24, NSDI'24, NSDI'21, SOSP'21, and SYSTOR'16. Juncheng was a Facebook Fellow, recognized as a Rising Star in machine learning and systems, and a Google Cloud Research Innovator. His dissertation on designing efficient and scalable cache management systems received the CMU SCS Doctoral Dissertation Award and the ACM SIGOPS Dennis M. Ritchie Doctoral Dissertation Award.

His works have been widely adopted. S3-FIFO and SIEVE are adopted for production at hundreds of companies with more than 60 open-source libraries and packages in 18 programming languages. Moreover, his group maintains libCacheSim, the most popular cache simulation library, and freeinference, a free LLM inference service.