ToolsCompaniesAbout

Results for “llm serving”

66 tools in this category · Updated

“llm serving”CodingRuns on Command lineShowing 66 tools
  • vLLMby vLLMLLM inference and serving library.Free
  • LLM Compressorby IBMOpen-source Python library for quantizing and pruning LLMs to run faster in vLLMFree
  • Orq.aiby Orq.aiAI gateway and platform for routing, governing and observing LLM and agent traffic across 500+ models.Freemium
  • NVIDIA Dynamoby NVIDIAOpen-source framework for serving LLMs across multi-GPU, multi-node clustersFree
  • Opikby CometOpen-source observability and evaluation for LLM and agent applicationsFreemium
  • LMCacheby LMCacheOpen-source KV-cache infrastructure for storing and reusing LLM inference cache.Free
  • Red Hat AI Inferenceby IBMSelf-managed vLLM and llm-d inference stack for running open models on your own acceleratorsContact sales
  • .txt APIby .txtStructured outputs API that enforces JSON Schema, regex, and grammar constraints on LLM responses.Pay as you go
  • Haystackby deepsetOpen-source Python framework for building RAG, agent, and multimodal LLM applicationsFreemium
  • Sakana AIby Sakana AISakana AI API for Fugu orchestration models and the Japanese-focused Sakana Namazu LLM.Pay as you go
  • DeepEvalby Confident AIYC W25Open-source LLM evaluation framework for testing AI agents and appsUnknown
  • AI SDKby VercelOpen-source TypeScript toolkit for adding LLMs and agents to apps through one provider-neutral API.Free
  • Langfuseby LangfuseYC W23Open-source platform for tracing, evaluating, and improving LLM and agent applicationsFreemium
  • NVIDIA Cosmosby NVIDIAOpen world foundation models, datasets and tools for building physical AI.Free
  • Prime Intellectby Prime IntellectTrain, evaluate, and serve your own AI models on hosted RL infrastructureUnknown
  • Basetenby BasetenHosted model APIs and dedicated GPU inference for production AI appsPay as you go
  • Inference Endpointsby Hugging FaceManaged hosting for deploying AI models on dedicated, autoscaling infrastructurePay as you go
  • Rayby AnyscaleOpen-source Python framework for scaling AI and ML workloadsFree
  • Cerebriumby CerebriumYC W22Serverless GPU infrastructure for real-time AI workloadsFreemium
  • Magnitudeby MagnitudeYC S25Local open-model inference engine with hardware-tuned kernels, desktop app, and CLI.Free
  • RunAnywhereby RunAnywhereYC W26Hosted open models behind one OpenAI-compatible API key and CLIPay as you go
  • llamafileby Mozilla.aiRun local language models from portable executable filesFree
  • CodeHealth MCP Serverby CodeSceneLocal MCP server that gives AI coding agents CodeHealth feedbackPaidFree trial
  • Pydantic Logfireby PydanticFull-stack observability and evaluation for AI appsFreemium