CodeIsGo.com

In mid-2019, the Linkerd project’s continuous integration (CI) took 45 minutes, all tests were serialized on a single Kubernetes cluster, and multi-hour backups were common. A migration onto one-off Kubernetes in Docker (kind) clusters and GitHub Actions got CI below 10 minutes, and made it parallelizable. By Andrew Seigner.

While the goal of this article is to detail an end user story of how to efficiently test Kubernetes applications in CI, some background on Linkerd is helpful. Linkerd is an open source service mesh, and a CNCF member project.

Source: https://buoyant.io/2020/09/16/linkerds-ci-kubernetes-in-docker-github-actions/

In the article the author captures following:

• What is Linkerd?
• Testing Linkerd
• Prioritized CI requirements
• CI tech evaluation
• Lessons learned

Kind is a great tool, not only for CI but for local development as well. There are similar flavors of Kubernetes, such as Minikube and k3d. We selected kind because it’s used heavily by the Kubernetes community for testing Kubernetes itself. Also a huge shout out to the #kind channel in Kubernetes Slack. Nice one!2

Learn the basics of abstraction in object-oriented programming with example code and practice challenges. By Lee Nathan.

Along with inheritance, abstraction is an important concept in object-oriented programming. The theory is that every object should deliver simple and predictable results. Objects should also only share what needs to be shared.

The article dives into:

• Abstraction keeps code and data hidden when appropriate
• Abstraction is closely tied to encapsulation
• Why classes should sometimes be abstract
• What is abstract and what is concrete?
• Abstraction practice
• But what is the point of a class you can’t instantiate?

At first, abstract classes may seem unnecessary. After all, you can’t actually use them to make an object. And it’s not like they pass down their functions. Functions have to be rewritten in every child class. You will get code examples and link to the sandbox to practice abstraction with JavaScript. Good read!

Serverless functions are great for lightweight cloud architecture and rapid provisioning. However, sometimes serverless introduces additional complexity to the deployment process. Author compares Python and Go with respect to the ease of deployment when setting up a simple data factory on AWS Lambda. Written by Dick Abma.

Serverless functions are a great way to simplify your cloud stack and quickly produce new functionality. Serverless contributes to scalability, transferability, and flexibility of the entire stack. For author as a data engineer, this means that he can produce applications that will be understood by clients and can adapt to evolving requirements.

The article describes:

• Case study - bunch of IoT devices add data to a database, which is accessible via a REST API
• CI/CD: Platform compatibility
• Python in AWS Lambda
• Golang on AWS Lambda
• AWS limits, concurrent requests and validation
• Conclusion deserialization

The article covers more realistic setup with a Lambda Layer, in which you can place the needed packages. The layer is loaded by the function when it is invoked. Packages are thus removed from the function itself.

Serverless functions like AWS Lambda are great for lightweight cloud architecture. Though we should be careful not to introduce complexity in our CI/CD and Terraform code, which may outweigh the advantages of Lambda. Code is provided in this GitHub repository. Great read!

Staying on top of ephemeral environments is a challenge many organizations face. This blog post describes the process we undertook at Thought Machine, a cloud-native company with environments spanning across multiple cloud providers, to identify a solution able to detect, identify, categorize, and visualize all the cloud assets being deployed in an organization. Written by Marco Lancini, Security Engineer.

If you look for common root causes of recent security breaches, you will see that a considerable amount of them can be traced back to assets that slipped out of the organization’s inventory management, or that have been misconfigured. “The scale of cloud infrastructure is essentially infinite”: the server fleet really has no practical limit since the cloud providers operate at enormous scale. We can then gather that, nowadays, one of the core problems many organizations are facing is to keep track of all their assets.

Cartography is described as: A Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database. Cartography is not the only security graph tool out there, but it differentiates itself by being fully-featured yet generic and extensible enough to help make anyone better understand their risk exposure, regardless of what platforms they use.

Source: https://thoughtmachine.net/blog/mapping-moving-cloudshow-to-stay-on-top-of-your-ephemeral-environments-with-cartography-mb6d8/

The article content is split into:

• The challenges posed by ephemeral environments
• Enter Cartography
• Cartography’s value proposition
• Real world setup
• Multi-cloud auditing
• Deployment on Kubernetes
• Data consumption

In this blog post we saw what benefits Cartography could have on the security posture of your organization, and author walked through the process we undertook to deploy it in a multi-cloud environment, from inception to self-service dashboards for data consumption. But there is still much to do, and Cartography still has much more to offer. Great read!

In this blog series, we would like to introduce you gently to the topic on which we were working over last 2 years. Improving how InnoDB locks data (tables and rows) in order to provide illusion to clients that their queries are executed one after another, while in reality there is a lot of concurrency. By Kuba Łopuszański.

Do you remember Excel? You know, columns, and rows, perhaps in multiple tabs. And then you might have multiple files. You probably do not appreciate it consciously, but it is a nice feature, that whatever you put inside a cell stays there if you save. Perhaps sometimes you need to add a new row, or remove it expecting that to simply work, even if there were some formulas, like SUM(A1:A100) involving these rows, right? And have you ever had to collaborate on a single spreadsheet with another person? How did you go about not overwriting each others work?

In this post author will introduce following concepts:

• Serializability of transactions (ability to explain states observed over time with a convincing story about relative order of parallel operations)
• Amdahl’s law (limits of how much parallelism can help in presence of tasks which must be done one at a time)
• Timeouts (for misbehaving lock owners, and to resolve deadlocks)
• Reader-writer lock (shared/exclusive access rights)
• Starvation (permanent inflow of readers starving a writer waiting for its turn) queueing (FIFO, or priority)
• Read views (read-only snapshots which allow stale reads concurrent to new writes)
• Wall clock time out of band communication and their possible inconsistency with serialization order
• Deadlocks caused by escalation
• Granularity of a lock (access right to everything vs. only the resources needed)

… and more. This is klong read with explaining the concepts rea;ted to locking on the Excel spreadsheet example. Enjoyable!

Have you ever deployed a feature which interrupted your application because the schema change on the MySQL database has locked the table? Then I have good news for you! There are some tools like the Percona Toolkit which can apply the operation without a downtime. By Michi Lehr.

When you have a MySQL database and want to apply a scheme change on a table (e.g. an alter table), the table needs to be rebuild on some DDL (Data Definition Language) operations. This depends on the MySQL version and DDL operation.

The tool pt-online-schema-change of the Percona Toolkit will take another strategy when altering a table. It will create a new table and add triggers to the existing table so that changes will be copied to the new table. Then the existing data will be copied. After this, the tables will swapped, and the triggers will be dropped.

# Example of dropping a column.
pt-online-schema-change --alter "drop column my_column" D=my_database,t=my_column --alter-foreign-keys-method auto --execute

Pretty straightforward. You will also get the links to similar tools. Nice one.

The polling system acts like the impatient customer. In a commerce-related scenario, it keeps polling the system for new updates such as orders and payment authorizations while the event driven system relies on asynchronous event handlers to notify it of updates in the system. By Sajjad Heydari.

Event based systems in computer science go back to the 1950s where they were first designed to handle asynchronous events such as Input/Output. Edsger Wybe Dijkstra designed the first interrupt handler on the Electrologica X-1 that is still used in modern systems to this day. The modern day’s events are the same concept, only they are used in different environments.

The event driven system is easier to develop and more efficient, but it requires special infrastructure to work. The good news is that some software and platforms like Fabric support event driven systems. In commerce, these systems create more streamlined operations so customers get their orders faster.

The article then describes:

• Example of an event driven system
• Parts of an event driven system
  • The event
  • The event router
• Benefits of event driven systems

In this post we provided an overview of how event driven systems streamline commerce. For a more detailed view of how events and event routers allow different services to communicate better in a commerce tech stack, check out thisvideo from AWS. Nice one!

Building a real-time web or mobile application is a bit more challenging than building a standard service. That’s because the protocol you choose to deliver data from the server to the client — and back — will have a significant impact on the overall experience. By Kieran Kilbride-Singh.

When you need data to flow from the server to the client quickly, a persistent connection is necessary. You do this by using long polling, WebSockets, Server-Sent Events (SSE) or HTTP2 Push.

Source: www.ably.io/blog/mqtt-vs-sse/

Today, we’re going to take a look at MQTT and SSE, specifically, to see how they compare:

• A brief history of MQTT and SSE
• How are they different?
• An overview of MQTT in action
• MQTT: Pros and cons
• An overview of SSE in action
• SSE: Pros and cons

The most important thing to consider when comparing MQTT and SSE is their connection capabilities. SSE, for example, is mono-directional and specifically designed for services or applications that need to pull information from a server. MQTT, on the other hand, is a robust, bi-directional communication option that can facilitate data transfers back and forth between the server and client(s). Excellent information here!

Today’s users expect your app to be accessible from their computer, mobile phone, tablet, or any other device! This transition to Software as a Service (SaaS) as the norm requires developers to effectively integrate with robust tools that scale to handle thousands (or even millions) of requests every second. Apache Kafka is one of the most effective tools for handling those high throughput environments. By Daniel Pereira.

In this tutorial, you’ll learn the basic concepts behind Apache Kafka and build a fully-functional Java application, capable of both producing and consuming messages from Kafka.

The tutorial reads about:

• A brief overview of Apache Kafka
• Distributed streaming platform
• Publish/Subscribe messaging system with durable messages
• Kafka topics
• Create a Java + Kafka application
• Push messages to a Kafka topic in your Java app
• Consume messages from a Kafka topic in a Java App
• Secure your Java Kafka application

This is very hand on tutorial. you will get all the code, plus explanation how to set up secure authentication via Okta API using free Okta account. The Okta Identity Cloud provides secure identity management with Single Sign-On, Multi-factor Authentication, Lifecycle Management (Provisioning), and more. Great read!

Without Continuous Integration or CI, developers would need to manually coordinate, communicate, and test while contributing code to the end product each time. This, in turn, affects the production and causes a delay in release. By Salman Khan.

With Bitbucket CI coming into the picture, it will go a long way in helping developers with their daily operations. To ensure faster and smoother delivery, organizations use Selenium Grid Cloud to automate their test suite.

The article then pays attention to:

• What is bitbucket pipelines? Why do we need it?
• Why choose selenium cloud to run the tests?
• How to run test scripts with bitbucket pipelines in selenium?
• Forking or cloning the bit bucket pipeline repository
• Setting up the environment variables
• How to configure the pipeline in bitbucket ci?
• How to execute the pipeline?
• How to view the build & test logs?

… and much more. Screenshots and all the code for the pipelines provided. To perform parallel testing, author provided in the bitbucket-pipeline.yml file with instructions how to execute parallel and cross-browser testing using cloud services such as LambdaTest and its uses to integrate bitbucket pipelines. Nice one!