{"id":1039,"date":"2021-10-15T08:38:26","date_gmt":"2021-10-15T08:38:26","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/10\/15\/how-imperva-expedites-ml-development-and-collaboration-via-amazon-sagemaker-notebooks\/"},"modified":"2021-10-15T08:38:26","modified_gmt":"2021-10-15T08:38:26","slug":"how-imperva-expedites-ml-development-and-collaboration-via-amazon-sagemaker-notebooks","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/10\/15\/how-imperva-expedites-ml-development-and-collaboration-via-amazon-sagemaker-notebooks\/","title":{"rendered":"How Imperva expedites ML development and collaboration via Amazon SageMaker notebooks"},"content":{"rendered":"
\n

This is a guest post by Imperva, a solutions provider for cybersecurity.\u00a0<\/em><\/p>\n

Imperva<\/a> is a cybersecurity leader, headquartered in California, USA, whose mission is to protect data and all paths to it. In the last few years, we\u2019ve been working on integrating machine learning (ML) into our products. This includes detecting malicious activities in databases, automatically configuring security policies, and clustering security events into meaningful stories.<\/p>\n

As we\u2019re pushing to advance our detection capabilities, we\u2019re investing in ML models for our solutions. For example, Imperva provides an API Security service. This service aims to protect all APIs from various attacks, including attacks that traditional WAF can\u2019t easily stop, such as those described in the OWASP top 10<\/a>. This is a significant investment area for us, so we took steps to expedite our ML development process in order to cover more ground, efficiently research API attacks, and expedite our ability to deliver value for our customers.<\/p>\n

In this post, we share how we expedited ML development and collaboration via Amazon SageMaker<\/a> notebooks.<\/p>\n

Jupyter Notebooks: The common research ground<\/h2>\n

Data science research processes raised the attention of big tech companies and the development community to new heights. It\u2019s now easier than ever to kick off a data-driven project using managed ML services. A great example for this is the rise of citizen data scientists, which according to Gartner<\/a> are \u201c\u2018power users who can perform both simple and moderately sophisticated analytical tasks that would previously have required more expertise.\u201d<\/p>\n

With the expected growth of ML users, sharing experiments across teams becomes a critical parameter in the development velocity. Among the many common steps, one of the most important steps for data scientists kicking off a project would be to open up a new Jupyter notebook and dive into the challenge ahead.<\/p>\n

Jupyter notebooks are a cross between an IDE and a document. It provides the researcher with an easy and interactive way to test different approaches, plot the results, present and export them, while using a language and interface of their choice such as Python, R, Spark, Bash, or others.<\/p>\n

Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy ML. SageMaker includes this exact capability and more as part of its SageMaker notebooks feature.<\/p>\n

Anyone who has tried to use Jupyter Notebooks in a team has probably reached a point where they attempted to use a notebook belonging to someone else, only to find out it\u2019s not as easy as it sounds. Often, you just don\u2019t have access to the required notebook. On other occasions, notebooks are used locally for research, and so the code is often littered with hardcoded paths and isn\u2019t committed to any repository. Even if the code is committed to a repository of some sorts, (hopefully) the data it requires isn\u2019t committed. To sum things up, it ain\u2019t easy to collaborate with Jupyter Notebooks.<\/p>\n

In this post, we show you how we share data science research code at Imperva, and how we use SageMaker notebooks with additional features we\u2019ve added to support our custom requirements and enhance collaboration. We also share how all these efforts have led to a significant reduction in costs and time spent on housekeeping. Although this architecture is a good fit for us, you can choose different configurations, such as complete resource isolation with a separate file system for each user.<\/p>\n

How we expedited our ML development<\/h2>\n

Our workflow is pretty standard, we take a subset of data, load it into a Jupyter notebook, and start exploring the data. After we have a decent understanding of the data, we start experimenting and combining different algorithms until we come up with a decent initial solution. When we have a good enough proof of concept (POC), we proceed to validate the results over time, experimenting and adjusting the algorithm as we go. Eventually, when we reach a high level of confidence, we deliver the model<\/a> and continue to validate the results.<\/p>\n

\"\"<\/a><\/p>\n

At first this process made perfect sense. We had small projects that didn\u2019t require much computing power, and we had enough time to work on them solo until we reached a POC. The projects were simple enough for us to deploy, serve, and monitor the model ourselves, or in other cases, deliver the model as a Docker container. When performance and scale were important, we would pass ownership of the model to a dev team using a specification document with pseudo-code. But times are changing, and as the team and projects grew and developed, we needed a better way to do things. We had to scale our projects when massive computing resources were required, and find a better way to pass ownership without using dull and extensive specification documents.<\/p>\n

Furthermore, when everyone is using some remote virtual machine or Amazon Elastic Compute Cloud<\/a> (Amazon EC2) instance to run their Jupyter notebooks, their projects tend to lack documentation and get messy.<\/p>\n

SageMaker notebooks<\/h2>\n

In comes SageMaker notebooks: a managed Jupyter Notebooks platform hosted on AWS, where you can easily create a notebook instance\u2014an EC2 (virtual computer) instance that runs a Jupyter Notebooks server. Besides the notebook now being in the cloud and accessible from everywhere, you can easily rescale the notebook instance, giving it as much computing resources as you require.<\/p>\n

Having unlimited computing resources is great, but it wasn\u2019t why we decided to start using SageMaker notebooks. We can summarize the objectives we wanted to achieve into three main points:<\/p>\n