Engineering at Barracuda

The story behind Barracuda Active Threat Intelligence

A while ago at one of our brainstorming sessions, as teams discussed the next level of evolution for our products, it became evident that detection and protection against new and emerging threats required intensive data analysis at scale. The analysis would need to predict clients’ risk and do that efficiently and quickly if we wanted to prevent hostile action from taking place.

As we analyzed the requirements, we realized that to protect against advanced attackers like bots, we needed to build a platform that could analyze traffic for web sessions, correlate it with data across sessions, and for many things, across the entire customer base. We also figured out that many parts of the system needed to be real-time, some near real-time, and others could have a much longer analysis phase.

A few years ago we introduced Barracuda Advanced Threat Protection (BATP) for zero-day malware attack protection across the Barracuda product line. This capability—analyzing files to detect malware using multiple engines in addition to sandboxing—was introduced in Barracuda’s application security products to secure applications like order processing systems where files were uploaded by third parties. This was the first attempt to use a cloud-based layer for advanced analysis that would have been difficult to build into web application firewall appliances.

While the BATP cloud layer could handle millions of file scans, we needed a system that could store large amounts of meta information so it could be analyzed to figure out new and evolving threats. This started us on the journey toward the next threat intelligence platform.

How Active Threat Intelligence works

The Barracuda Active Threat Intelligence platform is our answer. The platform is built on a massive data lake,  which can handle stream processing as well as batch processing of data. It processes millions of events per minute, across geographies, and provides intelligence that is used for detecting bots and client-side attacks, as well as providing information to protect against those threat vectors. Barracuda Active Threat Intelligence is built with an open architecture to be able to evolve rapidly to address newer threats.

Today, Barracuda Active Threat Intelligence platform receives data from the security engines in the Barracuda Web Application Firewall and WAF-as-a-Service, as well as other sources. As the events are received, they are augmented using crowd-sourced threat feeds and other intelligence databases. Detailed analysis of these events, both individually and as a part of a user session, is used to categorize the clients as humans or bots.

The data analysis pipelines use various engines and machine learning models to analyze multiple aspects of the traffic and reach their recommendations, which are finally reconciled to produce the final verdict.

Ways Active Threat Intelligence helps protect your apps

In addition to supporting all the analysis required for Advanced Bot Protection, the Active Threat Intelligence platform is being used for our latest offerings: Client-Side Protection and the Automated Configuration Engine.

Active Threat Intelligence tracks any external resources that may be used by the app, such as an external JavaScript or a stylesheet. Keeping track of external resources ensures that we are aware of the threat surface, and we can protect against attacks such as MageCart and more.

Because the meta data that is collected is extremely rich, we are able to derive additional information from it to assist administrators by providing configuration recommendations based on the real traffic coming to their apps.

This platform has been instrumental in helping us build the next generation of protection capabilities that our customers require. We continue to leverage this scalable platform to gather deep insights into traffic patterns, application consumption, and more. Stay tuned for blogs from our engineering teams that will talk about how we built Barracuda Advanced Threat Intelligence.

Anshuman Singh

Anshuman Singh is Senior Director Product Management, Barracuda. Connect with him on LinkedIn here.

www.barracuda.com

Highly scalable event logging on AWS

Most applications generate configuration events and access events. It is important for administrators to have visibility into these events. The Barracuda Email Security Service provides transparency and visibility into many of these events to help administrators fine-tune and understand the system. For example, knowing who logged into the account and when. Or knowing who added, changed, or deleted the configuration of a particular policy.

To build this distributed and searchable system, many questions come to mind, such as:

• How should you write these logs from all the applications, services, and machines to a central location?
• What should the standard format of the log files be?
• How long should you retain these logs?
• How should you correlate events from different applications?
• How do you provide a simple and quick searching mechanism via a user interface for the administrator?
• How do you make these logs available via an API?

When you think of a distributed search engine, the first thing that comes to mind is Elasticsearch. It is highly scalable with near real-time search and available as a fully managed service in AWS. So, the journey started with the thinking of storing these event logs in Elasticsearch and all the different applications pushing logs to Elasticsearch using Kinesis Data Firehose.

Components involved in this architecture

1. Kinesis Agent – Amazon Kinesis Agent is a standalone Java software application that offers an easy way to collect and send data to Kinesis Data Firehose. The agent continuously monitors event log files on EC2 Linux instances and sends them to the configured Kinesis Data Firehose delivery stream. The agent handles file rotation, checkpointing, and retry upon failures. It delivers all of your data in a reliable, timely, and simple manner. Note: If the application that needs to write to Kinesis Firehose is a Fargate container, you will need a Fluentd container. However, this article focuses on applications running on Amazon EC2 instances.
2. Kinesis Data Firehose – Amazon Kinesis Data Firehose direct put method can write the JSON formatted data into Elasticsearch. This way it doesn’t store any data on the stream.
3. S3 – An S3 bucket can be used to back up either all records or records that fail to be delivered to Elasticsearch. Lifecycle policies can also be set to auto-archive logs.
4. Elasticsearch – Elasticsearch hosted by Amazon. Kibana access can be enabled to help query and search the logs for any debugging purpose.
5. Curator – AWS recommends using Lambda and Curator to manage the indices and snapshots of the Elasticsearch cluster. AWS has more details and sample implementation that can be found here
6. REST API interface – You can create an API as an abstraction for Elasticsearch which integrates well with the User interface. API-driven microservice architectures are proven to be the best in many aspects such as security, compliance, and integration with other services.

Scaling

• Kinesis Data Firehose: By default, firehose delivery streams can scale up to 1,000 records/sec or 1MiB/sec for US East (Ohio). This is a soft limit and can be increased up to 10,000 records/sec. This is region specific.
• Elasticsearch: The Elasticsearch cluster can be scaled both in terms of storage and compute power on AWS. Version upgrades are also possible. Amazon ES uses a blue/green deployment process when updating domains. This means that the number of nodes in the cluster might temporarily increase while your changes are applied.

Advantages of this Architecture

1. The pipeline architecture is effectively completely managed and requires very little maintenance.
2. If the Elasticsearch cluster fails, Kinesis Firehose can retain records for up to 24 hours. In addition, records that fail to deliver are also backed up to S3.

The chances for data loss are low with these options available.

3. Fine-grained access control is possible to both Kibana and Elasticsearch API through IAM policies.

Shortcomings

1. Pricing needs to be carefully considered and monitored. The Kinesis Data Firehose can handle large amounts of data ingestion with ease, and if a rogue agent starts logging large amounts of data, the Kinesis Data Firehose will deliver them without issues. This can incur large costs.
2. The Kinesis Data Firehose to Elasticsearch integration is only supported for non-vpc Elasticsearch clusters.
3. The Kinesis Data Firehose currently cannot deliver logs to Elasticsearch clusters that are not hosted by AWS. If you would like to self-host Elasticsearch clusters, this setup will not work.

Conclusion

If you are looking for a solution that is completely managed and (mostly) scales without intervention, this would be a good option to consider. The automatic backup to S3 with lifecycle policies also solves the log retention and archival problem easily.

Sravanthi Gottipati

Sravanthi Gottipati is Engineering Manager Email Security at Barracuda Networks. You can connect with her on LinkedIn here.

www.barracuda.com

DJANGO-EB-SQS: An easier way for Django applications to communicate with AWS SQS

AWS services like Amazon ECS, Amazon S3, Amazon Kinesis, Amazon SQS and Amazon RDS are used extensively around the world. Here at Barracuda, we use AWS Simple Queue Service (SQS) to manage messaging within and among the microservices that we have developed on the Django framework.

AWS SQS is a message queuing service that can “send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.” SQS is designed to help organizations decouple applications and scale services, and it was the perfect tool for our work on microservices.  However, each new Django-based microservice or decoupling of an existing service using AWS SQS required that we duplicate our code and logic to communicate with AWS SQS. This resulted in lot of repeat code and encouraged our team to build this GitHub library: DJANGO-EB-SQS

Django-EB-SQS is a python library meant to help developers quickly integrate AWS SQS with existing and/or new Django based applications. The library takes care of the following tasks:

• Serializing the data
• Adding delaying logic
• Continuous polling from queue
• De-serializing the data as per AWS SQS standards and/or using third-party libraries to communicate with AWS SQS.

In short, it abstracts all the complexity involved in communicating with AWS SQS and lets developers focus only on core business logic.

The library is based on Django ORM framework and boto3 library.

How we use it

Our team works on an email protection solution that uses artificial intelligence to detect spear phishing and other social engineering attacks. We integrate with our customer's Office 365 account and receive notifications whenever they receive new emails. One of the tasks is to determine if the new email is clean from any fraud or not. On receiving such notifications, one of our services (Figure1: Service 1) talks to Office 365 via Graph API and gets those emails. For further processing of those emails and to make the emails available for other services, those emails are then pushed to AWS SQS queue (Figure1: queue_1).

Figure 1

Let’s look at a simple use case on how we use the library in our solutions. One of our services (Figure 1: Service 2) is responsible to extract headers and feature sets from individual emails and make them available for other services to consume.

Service 2 is configured to listen to queue_1 to retrieve the raw email bodies.

Let's say that Service 2 performs the following actions:

# consume email messages from queue_1

…

# extract headers and feature sets from emails

…

# submit a task

process_message.delay(tenant_id=, email_id=, headers=, tenant_id=, feature_set=, ….)

This process_message method won’t be called up synchronously, instead it will be queued up as a task and will get executed once one of the workers picks it up. The worker here could be from same service or from a different service. The caller of the method need not worry about the underlying behavior and how the task will get executed.

Let’s look at how the process_message method is defined as a task.

from eb_sqs.decorators import task

@task(queue_name='queue_2′, max_retries=3)
def process_message(tenant_id: int, email_id: str, headers: List[dict], feature_set: List[dict], …) :

try:

# perform some action using headers and feature sets
# also can queue up further tasks, if required

except(OperationalError, InterfaceError) as exc:

try:

process_message.retry()

except MaxRetriesReachedException:

logger.error(‘MaxRetries reached for Service2:process_message ex: {exc}')

When we decorate the method with the task decorator, what happens underneath is that it adds extra data like the calling method, target method, its arguments, and some additional metadata before it serializes the message and pushes it to the AWS SQS queue. When the message is consumed from the queue by one of the workers, it has all the information needed to execute the task: which method to call, which parameters to pass, and so on.

We can also retry the task in case of an exception. However, to avoid an infinite loop scenario, we can set an optional parameter max_retries where we can stop processing after we reach the maximum number of retries. We can then log the error or send the task to a dead letter queue for further analysis.

AWS SQS gives the ability to delay the processing of the message up to 15 mins. We can add similar capability to our task by passing the delay parameter:

process_message.delay(email_id=, headers=, …., delay=300) # delaying by 5 min

Executing the tasks can be achieved by running the Django command process_queue. This supports listening to one or more queues, reading from the queues indefinitely and executing the tasks as they come in:

python manage.py process_queue –queues

We just saw how this library makes communication within service or between services easy via AWS SQS queues.

More details on how to configure the library with Django settings, and ability to listen to multiple queues, development setup and many more features can be found here.

Contribute

If you wish to contribute to the project, please refer here: DJANGO-EB-SQS

Rohan Patil

Rohan Patil is a Principal Software Engineer at Barracuda Networks. Currently he is working on Barracuda Sentinel, an AI based protection from phishing and account takeover. He has worked last five years on cloud technologies and the past decade in various roles around software development. He holds a Masters in Computer Science from California State University and Bachelors in Computers from Mumbai, India.

Connect with Rohan on LinkedIn here.

www.barracuda.com

Using GraphQL for robust and flexible APIs

API design is an area where there can be a lot of contention between client application developers and backend developers. REST APIs enabled us to design stateless servers and structured access to resources for over two decades, and it continues to serve the industry, mainly due to its simplicity and moderate learning curve.

REST was developed around the year 2000 when client applications were relatively simple and the development pace was not as fast as it is today.

With a traditional REST-based approach, the design would be based on the concept of resources a given server manages. Then, we would typically rely on HTTP verbs such as GET, POST, PATCH, DELETE to perform CRUD operations on those resources.

Since the 2000s, several things have changed:

• Increased usage of single-page applications and mobile applications created a need for efficient data loading.
• Many of the backend architectures have turned from monolithic to µservice architectures for faster and more efficient development cycles.
• A variety of clients and consumers are needed for APIs. REST makes it hard to build an API that supports multiple clients, as it would return a fixed data structure.
• Businesses expect to rollout features faster to the market. If a change needs to be done on the client-side, it often requires a server-side adjustment with REST, leading to slower development cycles.
• Increased attention on user experience often leads to designing views/widgets that need data from multiple REST API resource servers to render them.

GraphQL as an alternative to REST

GraphQL is a modern alternative to REST that aims to solve several shortcomings, and its architecture and tooling are built to offer solutions for contemporary software development practices. It allows clients to specify exactly what data is needed and allows fetching data from multiple resources in a single request. It works more like RPC, with named queries and mutations instead of standard HTTP-based mandatory actions. This puts control where it belongs, with the backend API developer specifying what is possible, and the client/API consumer specifying what is required.

Here is a sample GraphQL query, which was an eye-opener for me when I first came across this. Assume that we are building a microblogging website, and then we need to query for 50 recent posts.

query recentPosts(count: 50, offset: 0) {
id
title
tags
content
author {
id
name
profile {
email
interests
}
}
}

The above one GraphQL query aims to request:

• Recent 50 posts
• Id, title, tags, and content for each blog post
• Author information containing id, name, and profile information.

If we have to use a traditional REST API approach for this, the client would need to make 51 requests. If posts and authors are considered separate resources, that’s one request for getting 50 recent posts and then 50 requests for getting author information for each post. If author information can be included in the post details, then this could be one request with REST API as well. But, in most cases, when we model our data using relational database normalization best practices, we would manage author information in a separate table, and that leads author information to be a separate REST API resource.

Here's the cool part with GraphQL. Say in a mobile view, we don't have real estate on screen to show both the post content and author profile information. That query could now be:

query recentPosts(count: 50, offset: 0) {
id
title
tags
author {
id
name
}
}

The mobile client now specifies the information it desires, and GraphQL API provides the exact data it’s asked for, nothing more and nothing less. We didn't have to make any server-side adjustments, nor did our client-side code have to change significantly, and our network traffic between client and server is optimal.

The point to note here is that GraphQL lets us design flexible APIs based on client-side requirements rather than from a server-side resource management perspective. A general perception is GraphQL makes sense only for complex architectures involving several dozens of µservices. This is true to some extent, given that there is some learning curve with GraphQL compared to REST API architectures. But, that gap is closing, with significant intellectual and financial investment from the emerging vendor-neutral foundation.

Barracuda is an early adopter of GraphQL architectures. If this blog has piqued your interests, please follow this space for my subsequent blogs where I will delve into more technical details and architectural benefits.

Vinay Patnana

Vinay Patnana is the Engineering Manager, Email Security Service, Barracuda.  In this role, he assists in the design and development of scalability services of Barracuda email solutions.

Vinay holds a Masters in Computer Science from North Carolina State University and a Bachelor of Engineering from BIT Mesra, India.  He has been with Barracuda for several years and he has over a decade of experience working with several varieties of technical stacks.  You can connect with him on LinkedIn here.

www.barracuda.com