CodeIsGo.com

Uber has one of the largest deployments of Apache Kafka® in the world. It empowers a large number of real-time workflows at Uber, including pub-sub message buses for passing event data from the rider and driver apps, as well as financial transaction events between the backend services. By Prateek Agarwal, Ryan Turner, and KK Sriramadhesikan.

During our performance runs, we saw ~2.5X degradation in the p99 latencies when enabled security feature on the clusters. On further investigation, we discovered that Java 11 has significant SSL/TLS performance improvements over Java 8 because of faster cryptographic operation handling in Java 9+.

The article deep dives into:

• Kafka security concepts
  • Encryption
  • Authentication
  • Authorization
  • Security provider
  • Uber PKI (uPKI) framework
• Architecture
• Charter IAM framework
  • Key/Certificate retrieval flow
  • Authorization flow
• Enabling Security Feature on Clusters

… and much more. In this blog we showed the essential components to enable security features on a Kafka cluster. We then showed how Kafka interacts with uPKI and Charter (IAM) systems together to attain security on the Kafka clusters. We discussed how we incrementally enabled security features on Kafka clusters and topics without any degradation. Nice one!

The built-in functionality of SQS and Lambda provided us serverless, scalable and fault-tolerant basis, but while running the solution we also learned some important lessons. In this blog post I will discuss the issue of valid messages ending up in dead-letter queues (DLQ) and correctly configuring your DLQ to catch only erroneous messages from your source SQS queue. By Miia Niemelä.

In brief, Amazon SQS is a lightweight, fully managed message queueing service, that enables decoupling and scaling microservices, distributed systems and serverless applications. With SQS, it is easy to send, store, and receive messages between software components, without losing messages.

The article deals with:

• What are Amazon SQS and Lambda?
• Problems with valid messages ending up in DLQ
• Lambda scales automatically with the number of messages arriving to SQS – up to a limit
• Configuring DLQ to your SQS and Lambda combination

When testing our solution with pushing thousands of messages rapidly to the queue, we observed many of the messages ending up in a dead-letter queue, even though they were not erroneous.From the CloudWatch metrics, we found no execution errors during the given period, but instead there was a peak in the Lambda throttling metric. We had configured a DLQ to catch erroneous messages, but ended up having completely valid and unprocessed messages in the DLQ.

With carefully tuning the parameters of our SQS queue, mainly by increasing the maxReceiveCount and VisibilityTimeOut, we were able to overcome the problems with Lambda functions throttling. Excellent read!

Microservices and APIs are becoming famous everywhere in the software development space with the increasing need to produce scalable, secure, and flexible applications at faster turnaround times. By Durga Prasad Acharya.

But many people confuse between them, and when it comes to developing a software application, they don’t know what will be suitable for them This article will compare microservices vs API with an aim to put an end to all your confusion so you can decide the best way to build and deploy your application.

The article contains information on:

• What are microservices?
• What is an API?
• Microservices vs API: How do they work?
• Microservices vs API: Benefits of each
• Microservices vs API: What are they used for?
• Microservices vs API: Similarities and differences
• Can microservices and API work together? How?

Microservices and API can work together in an application. Although they can exist separately, using both together in your application can help organizations effectively implement the microservices architecture. Many companies face difficulties deploying microservices architecture when they already have other architectures deployed. In addition, integrating multiple, smaller services and benefitting from them is problematic. Interesting read!

Automation has found its way into almost all industries. Many industries have adopted automation to reduce human efforts and boost productivity to meet customer demands. Recently, automated farming robots are being used for various agricultural practices. They minimize the intensive hard work required for farming and save a lot of time. By Amna Faryad.

As a result, the agriculture industry has adapted advanced approaches to tackle different uncertain situations like reducing human errors and costs. The article then explains:

• Smart agriculture
• Robots for planting
• Irrigation drones
• Harvest robots
• Weeding robots
• Robots for pruning

With the use of these automated farming approaches, it has become easy for farmers to increase their land’s productivity even in the absence of enough labor. Automated farming robots use artificial intelligence technology and computer vision to increase their efficiency for the effective use of available resources. Interesting read!

As robots make their way into a variety of real-world environments, roboticists are trying to ensure that they can efficiently complete a growing number of tasks. For robots that are designed to assist humans in their homes, this includes household chores, such as cleaning, tidying up and cooking. By Ingrid Fadelli , Tech Xplore.

Stir-fry, the cooking style that the team focused on in their recent paper, involves complex bimanual skills that are difficult to teach to robots. To effectively do this, Liu and his colleagues first tried to train a bimanual coordination model known as a “structured-transformer” using human demonstrations.

Researchers at the Idiap Research Institute in Switzerland, the Chinese University of Hong Kong (CUHK) and Wuhan University (WHU) have recently developed a machine learning-based method to specifically teach robots to master stir-fry, the Chinese culinary cooking technique. Their method, presented in a paper published in IEEE Robotics and Automation Letters, combines the use of a transformer-based model and a graph neural network (GNN).

Food preparation and cooking are two crucial activities in the household, and a robot chef that can follow arbitrary recipes and cook automatically would be practical and bring a new interactive entertainment experience. The researchers assessed their model’s performance both in simulations and on a physical two-handed robotic platform, known as the Panda robot. In these tests, their model allowed the robot to successfully and realistically reproduce the motions involved in stir-fry. Interesting read!

It’s common sense that being part of a thriving startup community can bring immense advantages to your own project. Yet, not all startup founders have the opportunity or desire to move to Silicon Valley, New York, London, Beijing, or any of the other big startup hubs. By Abdo Riani.

If you are serious about building a tech startup, it’s a great idea to create a strong connection with at least one investor, mentor, or partner located in an active startup hub.

In this article, we’ll talk about how to find a productive compromise. The article captures:

• Access to knowledge
• Connectedness and access to talent
• Access to capital

Being connected to a thriving ecosystem is a crucial factor for the success of any startup project. It’s hard to be successful in a high-value-added field if you have to start by reinventing the wheel, so access to fellow successful startup founders, mentors, and knowledgeable partners is an advantage that’s hard to compensate for. Good read.

Learning how to install and manage Kubernetes clusters can be a big obstacle between your organization and all the potential magic of a container orchestration platform. You have to consider tasks like provisioning machines, choosing OS and runtime, and setting up networking and security. By Fer De Oliveira @google.

When adopting containers, you should always ask yourself: Does my startup or tech company have the skills, resources, and time to maintain, upgrade, and secure my platform?

The main sections of the article:

• The benefits of embracing managed services
• Always be building towards an open future

Containers and managed services based on open standards are a powerful combination that allow you to take advantage of best-of-breed capabilities on any platform, while simultaneously standardizing skills and processes. As a leader at a startup or tech company, you’re always looking for ways to move faster, work more efficiently, and make the most of the technical talent you have. You want to spend more time on roadmap priorities and spend the minimum amount of resources on maintaining your infrastructure. Good read!

Since automating IAM remediation requires recurring parsing of JSON documents, modulating your parsing function(s) will save you time as you work on each control. If you have controls that pertain to trust policies, you will need a separate function to parse those out since the format differs from that of permission policies. By Cody Bench.

Automating IAM policy remediation is no walk in the park. A development team must:

• Identify the controls to be remediate
• Build a program that pulls down and parses out the roles and policie
• Validate the permission and trust policies against the necessary control
• Perform the required remediation - which may include checking action history
• Rebuild the JSON document
• Deploy the remediated artifacts per your company’s or client’s workflow

Having any way to access the historical data for the roles and/or policies you are remediating is a huge win when implementing least privilege. A stakeholder will need to decide the cutoff for the last time an action was used or a resource was accessed, whether that be a few months or a year prior to remediation. Excellent read!

A guide to mathematical optimization with Google OR-Tools. Linear programming is a technique to optimize any problem with multiple variables and constraints. It’s a simple but powerful tool every data scientist should master. By Maxime Labonne.

Fortunately for us, there is a method that can solve our problem in an optimal way: linear programming (or linear optimization), which is part of the field of operations research (OR). In this article, we’ll use it to find the best numbers of swordsmen, bowmen, and horsemen to build the army with the highest power possible.

You can run the code from this tutorial with the following Google Colab notebook.

The article then describes:

• Solvers
• Variables
• Constraints
• Objective
• Optimize!

In Python, there are different libraries for linear programming such as the multi-purposed SciPy, the beginner-friendly PuLP, the exhaustive Pyomo, and many others. Excellent read!

Open source software is being vilified once again and, in some circles, even considered a national security threat. Open source software risk has been a recurring theme: First it was classified as dangerous because anyone could work on it and then it was called insecure because nobody was in charge. By Josh Bressers.

Let’s consider where open source stands today. It’s running at minimum 80% of the world. Probably more. Some of the most mission-critical applications and services on the planet (and on Mars) are open source. The reality is, open source software isn’t inherently more risky than anything else. It’s simply misunderstood, so it’s easy to pick on.

The article then discusses:

• Myth 1: Open source software is a risk because it isn’t secure
• Myth 2: Open source software is a risk because it isn’t high quality
• Myth 3: Open source software is a risk because you can’t trust the people writing it
• The true risk of open source software

In an era where the use of open source software is only increasing, the true risk in using open source — or any software for that matter – is failing to understand how it works. In the early days of open source, we could only understand our software by creating it. There wasn’t a difference between being an open source user and an open source contributor. Nice one!