Yonatan G. on LinkedIn: #nlp #computervision #deeplearning | 10 comments (2024)

Yonatan G.

Building AI software and foundation models at Nvidia

  • Report this post

Will language models shift to edge deployment?I estimate that 80-90% of the NLP models are currently deployed in the cloud and data center. In contrast, in CV about 60% of models deployed at the edge. Will we see a similar trend in NLP of shifting workloads to the edge (and when)?The benefits of deploying at the edge:- Compute is free- deploying on the consumer edge doesnโ€™t require expensive cloud computing.- Secured - Data doesnโ€™t leave where it has been collected/generated.- Low latency - reducing communication time can help build low-latency real-time applications. We are all familiar with the 1-2 seconds that it takes Siri to respondโ€ฆThe challenges in edge deployment- Computational complexity - edge hardware is usually limited by its computing power- Model size - downloading 1GB of model weights to the edge could be extremely heavy and fill the device's memory.- Diverse HW - The HW in consumer environments is very diverse, and models should target the lowest denominator. I remember trying to run Zoom's virtual background on i5 CPU and getting a message that this feature is not supported for my laptop.We are now at a point similar to 2014 in computer vision when Resnet-152 was introduced and showed that larger models could bring better accuracy. However, no one uses Resnet-152. Today, we have models that are 10X smaller and 5X faster with better accuracy, and this is exactly what will happen in the NLP space during the next 2-3 years.#nlp #computervision #deeplearning

140

10 Comments

Like Comment

Eyal Betzalel

๐ŸŽ—๏ธUtilizing AI to improve bioprocesses

1y

  • Report this comment

CV applications differ in two manners: it requires RT processing for safety reasons in many tasks (mainly in automotive) and the raw data requires very high BW so it is far more economic to process it near the sensors. NLP, on the other hand, is different. the models are huge, the data BW is small (text...), and the latency constraints are less critical (at least in the applications that I can imagine) so it is far more suitable for cloud processing. I believe that edge computing will be involved in this area via RT speech-to-text models since this block on the NLP flow has a real latency benefit.

Like Reply

1Reaction 2Reactions

Dror Meiri

๐ŸŽ—๐‘ฉ๐’–๐’”๐’Š๐’๐’†๐’”๐’”, ๐‘ท๐’“๐’๐’…๐’–๐’„๐’• & ๐‘บ๐’•๐’“๐’‚๐’•๐’†๐’ˆ๐’š ๐‘ฌ๐’™๐’†๐’„๐’–๐’•๐’Š๐’—๐’† ๐š๐ญ ๐“๐ž๐œ๐ก ๐‚๐จ๐ฆ๐ฉ๐š๐ง๐ข๐ž๐ฌ

1y

  • Report this comment

Thanks Yonatan G. , NLP on the edge is definitely inevitable because of the same reasons you mentioned. What are the opportunities for the industry that you foresee, given this evolution?

Like Reply

1Reaction

Assaf Pinhasi

Data, ML Engineering and MLOps expert | hands-on consultant

1y

  • Report this comment

I disagree about the likelihood of this happening in the near future:1. Motivation is different, as payloads are small and sending over the wire doesn't add latency.Note that encoding voice to text can be done on edge and proven good enough, and then it's text over the wire with zero footprint.2. LLMs like chatGPT encode a huge amount of information about tons of areas to reach general conversation ability/assistant level..It's not a hotdog detector that can be optimized specific for a narrow task like most of the vision models running on the edge.But we will see.

Like Reply

6Reactions 7Reactions

Prithivi Da

25M+ Model โ†“ in ๐Ÿค— | Cited in NeurIPS, ICLR, ACL | 3.5K+ โญ๏ธ GitHub | Sharing wisdom from a 2 decade experience in creative problem solving.

1y

  • Report this comment

Edge deployments with differential privacy >> security. Google GBoard is the classic pioneering NLP example with edge deployments of word learning, suggesting and correcting models which got consolidated using federated learning centrally. But thanks to differential privacy inspite of that no privacy is lost.

Like Reply

12Reactions 13Reactions

See more comments

To view or add a comment, sign in

More Relevant Posts

  • Yonatan G.

    Building AI software and foundation models at Nvidia

    • Report this post

    GTC is almost here, and I'm excited to share that I'll be presenting a session on developing generative models that achieve exceptional quality and efficiency.Join me to explore the transformative potential of neural architecture search for generating resource-efficient LLMs such as DeciLM, and learn how to achieve high throughput, low latency, and cost-effectiveness on commonly available GPUs.If you're wondering whether high-performance generative AI can be cost-effectively deployed, this session on March 19 at 4:30 p.m. PDT (11:30 p.m. GMT) is a must-attend. See you there! Learn more here > https://lnkd.in/dz4mnzjg #gtc2024 #gtc #generativeai

    • Yonatan G. on LinkedIn: #nlp #computervision #deeplearning | 10 comments (10)

    92

    2 Comments

    Like Comment

    To view or add a comment, sign in

  • Yonatan G.

    Building AI software and foundation models at Nvidia

    • Report this post

    Today, Iโ€™m thrilled to announce the launch of Deciโ€™s Gen AI Development Platform. It features a new series of proprietary, ultra-efficient LLMs, an inference engine, and an AI cluster management solution.The first model being released in the series is Deci-Nano:๐Ÿ† With an average Open LLM Leaderboard score of 70.76 and a 7.41 MT-Bench score, Deci-Nano outperforms Mistral-7b-instruct-v0.2 and Gemma-7b-it.โšก Ultra Fast inference speed: Deci is off-the-charts with E2E latency (256 tokens) coming at 4.56 seconds measured on NVIDIA A100 GPUs.๐Ÿ’ฐExceptional price: $0.1 per 1 million tokens.Designed to balance quality, speed, and cost-effectiveness, our models are complemented by flexible deployment options. You can easily access them through our API or opt for deployment on your own infrastructure so your data stays secure and on site.Deci-Nano embodies our production-oriented approach, which includes a dedication not only to quality but also to efficiency and cost-effectiveness. Weโ€™re building architectures and software solutions that squeeze maximum compute power out of existing GPUs. Looking ahead, we anticipate the release of additional LLMs that promise to elevate this standard of excellence even higher.Hereโ€™s how you can get started:๐Ÿ’ป Try Deci-Nano in our playground or get your API trial token > https://lnkd.in/ddiKy8rs๐Ÿ“š Read the blog > https://hubs.ly/Q02pspZM0๐Ÿ“˜ Check out the QuickStart notebook > bit.ly/Deci-Nanonotebook#generativeai #largelanguagemodels #llm

    • Yonatan G. on LinkedIn: #nlp #computervision #deeplearning | 10 comments (15)

    136

    4 Comments

    Like Comment

    To view or add a comment, sign in

  • Yonatan G.

    Building AI software and foundation models at Nvidia

    • Report this post

    Generative AI is transforming the software development landscape. It has the potential to offer innovative solutions that impact every stage of the development process:โ€ข Automating coding tasksโ€ข Improving quality and fostering innovation while reducing costsโ€ข Enabling personalized user experiences and streamlined collaborationโ€ข Improving security and content creationโ€ข Allowing scalability and global market adaptationRead more > https://lnkd.in/eN_ynzVY๐ˆ๐Ÿ ๐ฒ๐จ๐ฎ๐ซ ๐จ๐ซ๐ ๐š๐ง๐ข๐ณ๐š๐ญ๐ข๐จ๐ง ๐ข๐ฌ ๐ฅ๐จ๐จ๐ค๐ข๐ง๐  ๐ญ๐จ ๐ข๐ง๐ญ๐ž๐ ๐ซ๐š๐ญ๐ž ๐ ๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐€๐ˆ ๐ข๐ง๐ญ๐จ ๐ฒ๐จ๐ฎ๐ซ ๐ฌ๐จ๐Ÿ๐ญ๐ฐ๐š๐ซ๐ž ๐๐ž๐ฏ๐ž๐ฅ๐จ๐ฉ๐ฆ๐ž๐ง๐ญ ๐ฉ๐ซ๐จ๐œ๐ž๐ฌ๐ฌ, ๐ฒ๐จ๐ฎ ๐œ๐š๐ง ๐œ๐ก๐ž๐œ๐ค ๐จ๐ฎ๐ญ ๐จ๐ฎ๐ซ ๐œ๐จ๐๐ž ๐‹๐‹๐Œ, ๐ƒ๐ž๐œ๐ข๐‚๐จ๐๐ž๐ซ-6๐.DeciCoder-6B balances accuracy and cost-efficiency and excels in 8 code languages. Released under Apache 2.0, it surpasses CodeGen 2.5 7B, CodeLlama 7B, and StarCoder 7B in most supported languages on HumanEval, and leads by 3 points in Python over StarCoderBase 15.5B. DeciCoder-6B achieves 19x higher throughput vs CodGen 2.5 7B on Qualcommโ€™s Cloud AI 100.The enhancements in inference and memory efficiency brought by DeciCoder-6B have a wide array of positive implications:โ€ข Improved user experience for applications built using the model, directly benefiting end-usersโ€ข A significant reduction in inference costs, primarily due to the modelโ€™s ability to perform optimally on more cost-effective hardware.If you want to learn more, our technical blog has all the details > https://lnkd.in/e_gw4bikHave you integrated generative AI into your software development process yet?#generativeai #codegeneration #llms

    A New Class of Non-standard Probability Distributions - DataScienceCentral.com https://www.datasciencecentral.com

    30

    Like Comment

    To view or add a comment, sign in

  • Yonatan G.

    Building AI software and foundation models at Nvidia

    • Report this post

    What do the initial confusion and slow adoption of the digital spreadsheet in the 1970s and generative AI today have in common? https://lnkd.in/ddnh3pZpBoth technologies have the ability to collapse time, completing tasks that once took hours or days in mere seconds. While the spreadsheet automated many accounting functions, it did not decimate the profession; instead, it augmented it, much like generative AI might enhance various industries.However, the author, Tim Harford, expresses concern about the pace of change in AI development. Drawing a cautionary parallel with the misuse of spreadsheets when tools become ubiquitous, he emphasizes the need for careful understanding and management of generative AI to avoid unintended consequences.A case in point: After months of resisting, Air Canada was recently forced to pay a passenger hundreds of dollars in damages due to inaccurate information provided by its online chatbot prior to his flight booking. (See comments for a link to the article).As practitioners in the field of AI, what are you most excited and also apprehensive about generative AI?#generativeai #llms #ai

    What the birth of the spreadsheet can teach us about generative AI ft.com

    19

    1 Comment

    Like Comment

    To view or add a comment, sign in

  • Yonatan G.

    Building AI software and foundation models at Nvidia

    • Report this post

    LLMs like OpenAI's GPT and Mistral AI's Mixtral are playing an increasingly important role in AI applications.However, LLMs face the challenge of hallucinations โ€“ inaccuracies that arise when they fabricate information in response to uncertainty. These errors can be influenced by the training dataset and inherent biases within the models.The good news is there are techniques to reduce hallucinations and improve LLM efficiency and speed. Check out these articles:๐Ÿ‘‰ From KDnuggets - https://lnkd.in/dW7wktHn๐Ÿ‘‰ From Harpreet Sahota ๐Ÿฅ‘ - https://lnkd.in/d6WmzKqS๐Ÿ‘‰ From Najeeb Nabwani - https://lnkd.in/d2VbM-WKWhatโ€™s the prompting method have you tried that works best for you?#generativeai #llms #largelanguagemodels

    3 Research-Driven Advanced Prompting Techniques for LLM Efficiency and Speed Optimization - KDnuggets kdnuggets.com

    51

    1 Comment

    Like Comment

    To view or add a comment, sign in

  • Yonatan G.

    Building AI software and foundation models at Nvidia

    • Report this post

    Massachusetts Institute of Technology researchers looked at whether or not AI was more cost-effective in 1,000 visually assisted tasks in 800 occupations. They found that in many cases it was more expensive to install and maintain AI systems than for a human to perform the same tasks.Deci AI works with organizations that have computer vision use cases, and there are two lenses from which we look at the research findings:First, while weโ€™re able to deliver better performance and reduce the costs of AI, at the end of the day, itโ€™s not about replacing humans, but improving the way we do business โ€“ from enhancing the user experience to increasing productivity.Second, AI is here to stay. Working with early adopters across industries including retail, automotive, manufacturing, and more, continuous innovation is tantamount to not only staying competitive but also, although it sounds a bit clichรฉ, contributing to the common good. One example of this is how we at Deci can lower the carbon footprint of AI systems by building, optimizing, and deploying highly accurate and cost-efficient deep learning and generative AI applications.Learn more about MIT's findings here: https://lnkd.in/d984nq-q#computervision #deeplearning #artificialintelligence

    Beyond AI Exposure: Which Tasks are Cost-Effective to Automate with Computer Vision? futuretech.mit.edu

    27

    Like Comment

    To view or add a comment, sign in

  • Yonatan G.

    Building AI software and foundation models at Nvidia

    • Report this post

    As more companies aspire to develop and deploy applications based on LLMs, the question of whether to use open source models or closed models is always going to pop up.In the article (link in the comments), VentureBeat and experts argue that open source LLMs could have a greater impact on generative AI in enterprise settings compared to closed models.While itโ€™s challenging to prove as of the moment, the publication did find 16 companies that have shipped generative AI applications using open source models โ€“ and industry observers say the number of cases is going to pick up strongly later this year.At Deci AI, weโ€™ve always viewed that open source as the better choice, especially for enterprises looking for the following, among others:โ€ข High control over model quality and customization, direct access to model architecture, ability to fine-tune, adjust content filters, and perform counterdeployment.โ€ข Full in-house control over data privacy, which is ideal for enterprises and for handling sensitive data.โ€ข Highly cost-efficient at scale, without the API service provider markup and cloud vendor โ€œlock-in.โ€To learn about the other factors to consider when deciding between open source vs closed source models, we have a comprehensive breakdown on the blog here > https://lnkd.in/dENj2hbbAt the end of the day, thereโ€™s no right way to build and run generative AI applications as it depends heavily on the specific use case.What are your thoughts on the open source vs closed models debate?

    Open-Source LLM vs. API: 7 Crucial Factors to Decide Your Generative AI Strategy | Deci https://deci.ai

    45

    2 Comments

    Like Comment

    To view or add a comment, sign in

  • Yonatan G.

    Building AI software and foundation models at Nvidia

    • Report this post

    Navigating the challenges of scaling video analytics on edge devices demands a multifaceted approach, blending advanced algorithmic solutions with efficient runtime and pipeline optimization tactics.In our upcoming webinar, explore how to select efficient models designed for object detection on edge hardware, and discover cutting-edge tools and techniques that can boost throughput and hardware utilization.If achieving exceptional real-time analytics performance at the edge is your goal, you wouldnโ€™t want to miss this session. Join now > https://lnkd.in/d695h-Vw

    26

    Like Comment

    To view or add a comment, sign in

Yonatan G. on LinkedIn: #nlp #computervision #deeplearning | 10 comments (41)

Yonatan G. on LinkedIn: #nlp #computervision #deeplearning | 10 comments (42)

8,620 followers

  • 145 Posts

View Profile

Follow

Explore topics

  • Sales
  • Marketing
  • Business Administration
  • HR Management
  • Content Management
  • Engineering
  • Soft Skills
  • See All
Yonatan G. on LinkedIn: #nlp #computervision #deeplearning | 10 comments (2024)
Top Articles
Latest Posts
Article information

Author: Mr. See Jast

Last Updated:

Views: 5897

Rating: 4.4 / 5 (55 voted)

Reviews: 86% of readers found this page helpful

Author information

Name: Mr. See Jast

Birthday: 1999-07-30

Address: 8409 Megan Mountain, New Mathew, MT 44997-8193

Phone: +5023589614038

Job: Chief Executive

Hobby: Leather crafting, Flag Football, Candle making, Flying, Poi, Gunsmithing, Swimming

Introduction: My name is Mr. See Jast, I am a open, jolly, gorgeous, courageous, inexpensive, friendly, homely person who loves writing and wants to share my knowledge and understanding with you.