Readit Newstheo31 commented on Show HN: GPU-Accelerated Inference Hosting inferrd.com/?gpu... · Posted by u/theo31
Looks awesome. Do you know if and how you guys support NVIDIA's software. For my project the NVIDIA software I'm using states it needs:
CUDA 11.3.0
cuBLAS 11.5.1.101
cuDNN 8.2.0.41
NCCL 2.9.6
TensorRT 7.2.3.4
Triton Inference Server 2.9.0
I'm new to deploying to production inference so I'm not sure if those are easily portable across such platforms or not really.
Those frameworks are installed by default in our custom environment. There is no additional setup/configuration required from you.
theo31 commented on Show HN: GPU-Accelerated Inference Hosting inferrd.com/?gpu... · Posted by u/theo31
Looking at the examples in the landing page.. so I don't need any kind of authentication to do inference? Anyone can run the models I upload?
At the moment, no, only the random hash gives some kind of security by obfuscation. More advanced security controls are coming soon.
theo31 commented on Show HN: GPU-Accelerated Inference Hosting inferrd.com/?gpu... · Posted by u/theo31
So is this mainly focused on deployment for applications with high-speed inference requirements? I didn't dive into product in detail. I run my own deep-learning based web-app and inference speed optimization is pretty non-trivial. As far as I know production level speed requirements require use of tensorrt which is definitely not hot-start and requires more than a few minutes to load (i'm not too sure what's going on under the hood, not an expert) but has inference speeds of up to x2 or more, so not quite sure what your targeting or if you've actually managed to solve that problem which would be highly impressive
We don't use TensorRT at the moment, but it is something that we are exploring.
theo31 commented on Show HN: GPU-Accelerated Inference Hosting inferrd.com/?gpu... · Posted by u/theo31
Or, do your inference using an AVX-512 CPU:
https://NN-512.com (open source, free software, no dependencies)
With batch size 1, NN-512 is easily 2x faster than TensorFlow and does 27 ResNet50 inferences per second on a c5.xlarge instance. For more unusual networks, like DenseNet or ResNeXt, the performance gap is wider.
Even if you allow TensorFlow to use a larger ResNet50 batch size, NN-512 is easily 1.3x faster.
If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year.
If you need Transformers instead of convolutions, Fabrice Bellard's LibNC is a good solution: https://bellard.org/libnc/
Oh that's very interesting, how ready for production is it? It only works for TF right?
> If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year.
Definitely the cheapest way.
We've been in business for more than a year already actually :)
theo31 commented on Show HN: GPU-Accelerated Inference Hosting inferrd.com/?gpu... · Posted by u/theo31
What are the specs of an Nvidia m80?
I’ve never heard of that type before and I wasn’t able to find anything with google.
Furthermore more, the lack of company information (address, company registration nr etc) and the fact that it’s not clear where the servers are located geographically makes me a bit hesitant.
Sorry that's a typo, they are K80s: https://www.nvidia.com/en-gb/data-center/tesla-k80/
It helps us figure out what got done and where we are in our roadmap