Carton now supports running ML models from C and C++

As a reminder, Carton is an open-source library with the goal of letting you run any machine learning model from any programming language.

I'm excited to share that Carton now supports running inference from C and C++!

This brings the list of supported languages to:

Python
Rust
C
C++

This means you can run any of the supported model types from any of these languages.

(Carton also supports JavaScript/TypeScript via WebAssembly, but that interface doesn't have inference support yet)

Run GPT2 from C++

As an example, we'll write some C++ code to load GPT2 from the model registry and run it.

int main()
{
    // Load a model, wait for the future to complete, get the result
    auto model = carton::Carton::load("https://carton.pub/openai/gpt2-medium")
        .get()
        .get_or_throw();
    
    // Create an input tensor
    uint64_t shape[]{1};
    auto tensor = carton::Tensor(carton::DataType::kString, shape);
    tensor.set_string(0, "Hello! I like to write about");

    // Add it to a map of inputs
    std::unordered_map<std::string, carton::Tensor> inputs;
    inputs.insert(std::make_pair("input", std::move(tensor)));

    // Run inference, wait for the future to complete, get the result
    auto out = model.infer(std::move(inputs)).get().get_or_throw();

    // Get the model output
    const auto output = out.get_and_remove("output");

    std::cout << "Got output: " << output.get_string(0) << std::endl;
}

In the code above, load and infer return std::futures, but if you don't want to use futures, Carton also supports callbacks and notifications as ways of more deeply integrating with async code.

See the documentation for more details.

Contributions

It's been a few weeks since I open-sourced Carton and in that time, we've had new contributors work on some cool things:

@leifu1128 is working on a runner for Wasm binaries (initially targeted at Candle)
@bramhoven is working on C# Bindings
@arthurmelton is working on OCaml support

Help wanted!

Because Carton now supports C and C++, there are more opportunities to contribute even if you're not familiar with Rust!

Specifically, it would be very helpful if anyone is interested in working on support for:

Go (via cgo)
Java (via JNI or something else)

Additionally, adding support for Ludwig can be done entirely in Python code.

You can also check out the "good first issues" tag or the "help wanted" tag on the repo for more ways to contribute. As always, feel free to create a discussion or issue if you have an idea for something else!

Roadmap

There are a few other things currently in progress that should land soon:

JAX support
NodeJS bindings

If you haven't already, please give Carton a try, and if you find it interesting, I'd appreciate it if you'd consider starring the repo!