AI Card Bao AICARDBAO.COM

Industry News

LiteLLM rewrote /v1/messages in Rust this week, and also launched a portal for calculating inference costs by customer.

The open source model gateway LiteLLM has updated a number of features this week, two of which are more practical for teams that are using it to build products.

The first is a beta version of the customer portal. It can attribute call traffic and inference costs to customers, and calculate how much money each customer spent. For companies that package large model capabilities and sell them to enterprise customers, this is the basis for reconciliation and pricing. In the past, many teams had to write another layer of statistics outside the gateway.

The second one is performance. The /v1/messages interface, the Anthropic-style messaging interface, has been completely rewritten in Rust. The LiteLLM ontology is written in Python. The delay under high concurrency has always been a point of attack by competing products. This time we will start with the hottest interface.

The other two items: Day 0 support for new models. As soon as the new models of Claude and Gemini are released, calls on mainstream cloud platforms can be connected on the same day; it also integrates video, image and voice models on Replicate.

A background note. LiteLLM had a supply chain incident in March this year. The two versions 1.82.7 and 1.82.8 on PyPI were implanted with malicious code that stole credentials and hung for about three hours before being quarantined. After that, the project team rotated maintainer accounts and began to disclose verification values ​​for released versions.

If you are still using it, it is standard practice for this project to check the checksum before upgrading.

Related