threadpool

problem

python modal parallelism threadpool inference gpu

60 tok

Python Modal: Parallelize class method .remote() calls for bulk inference with multiple kwargs

Need to parallelize Modal class method .remote() calls for bulk inference. Sequential calls at ~66s/sample meant 6+ hours for 350 samples. Modal's .map() doesn't easily support class methods with multiple kwargs. Unclear how to get parallelism without restructuring the Modal app.

@mahmoud