CodeIsGo.com

A comparison piece by Yan Cui about underlying plumbing when deploying Lambda function to AWS. One needs to be able manage the configuration for functions as well as other related resources such as API Gateway, CloudWatch log groups and IAM policies.

Depending on the event sources you would like to use, you also need to provision the necessary EventSourceMapping in order to use Lambda with the likes of Kinesis Streams and SQS.

Engineers want to work with a layer of abstraction that shields developers from these incidental complexities related to CloudFormation and Lambda. This is where deployment frameworks come in.

The article then compares:

• Serverless framework
• Aws SAM
• Terraform
• Claudia.js
• Aws Chalice
• Zappa
• Appex
• Up
• Architect

In terms of features, all the frameworks fulfill the basic requirement of packaging and deploying your code to Lambda very well. You will also get with categorized customizability and opinionatedness. Good job!

An article by Nikolaos Tsipas from Skyscanner with help of colleagues Omar Kooheji and Michael Okarimia about how to solve the puzzle when there is a need to import datasets from external sources, and make them available for querying. Examples of imported data include: analytics metrics, advertising data, and currency exchange rates, all of which are used by Skyscanner engineers and data scientists.

The main entry-point for data at Skyscanner is their Kafka-based streaming platform which allows near real-time processing and archiving.

For a solution that scales across the company they approached the problem with an emphasis on:

• Minimizing dependency on the Data Platform engineers when onboarding new datasets – preventing our availability from becoming a blocker
• Moving ETL pipeline ownership to the user – allowing users to own the data they produce
• Automating boilerplate code and config generation – ensuring that infrastructure, permissions and deployment setup are abstracted away from users
• Scalability and cost management – maintaining flexibility and cost efficiency, and ensuring the solution is future-proof

They use managed services provided by AWS to enable the various ETL stages and have adopted Cookiecutter for project templating.

Read the rest of this excellent article to learn if they opted for AWS Batch, Glue or both. A high-level illustration of the third-party data ETL pipeline is also provided. Nice one!

This is a detailed guide on hot topic – how to collect, customize, and centralize Python logs. From pens of Emily Chang and Nils Bunge. this guide will show you how to configure this module to log all the data you need, route it to your desired destinations, and centralize your logs to get deeper insights into your Python applications.

Authors flashed out some best practices for configuring Python’s standard logging library to generate context-rich logs, capture exception tracebacks, and route logs to the appropriate destinations.

The guide dives into:

• Customize the priority level and destination of your logs
• Configure a custom setup that involves multiple loggers and destinations
• Incorporate exception handling and tracebacks in your logs
• Format your logs in JSON and centralize them for more effective troubleshooting

The article then provides detailed instructions with advanced usage of logging module that is included in Python’s standard library. Code examples are included. Even describes how to configure multiple loggers. Well thought and thorough article!

This is well written guide about dependency injection on Scala. Dependency Injection (DI) is a popular pattern which encourages loose coupling between a services’ clients and service implementations.

Dependency Injection is a simple concept, and it can be implemented using relatively few simple constructs. We should avoid over-complicating and over-using containers or frameworks, without thoroughly analysing the costs.

This guide describes how to do Dependency Injection using the Scala language constructs as much as possible, while remaining practical, with the help of the MacWire library where necessary.

DI is all about decoupling client and service code (the client may happen to be another service). Services need to expose information on what dependencies they need. Instead of creating dependent service implementations inside the service itself, references to dependent services are “injected”. This makes the code easier to understand, more testable and more reusable.

The guide then explains:

• What is Dependency Injection?
• Other approaches
• Manual Dependency Injection
• Using MacWire for wiring
• Simple scoping
• Modularising object graph creation
• Multiple implementations
• Testing
• Interceptors
• Advanced scoping
• Factories
• Accessing the object graph dynamically
• Multiple instances
• DI in Akka

Well organised and informative resource for any Scala lover. Top notch work!

This article is based on the talk Scale in / Scale out with Kafka-Streams and Kubernetes from Xebicon'18. Loïc Divad is the author of this article and he aims to present a few advantages that come with specific practices related to Kafka-Streams like containerization and orchestration, and especially autoscaling.

Kafka consumers subscribe to topics and then regularly poll messages from it. When it comes to Kafka-Streams, polling intervals are generally short enough to get a real time processing effect.

The article then explains:

• What is consumer lag, a reason to scale
• Message consumption
• The consumer protocol
• Kubernetes and the custom metrics support
• Expose the JMX metrics in a Prometheus format
• Building the Docker image of the streaming app
• Metrics aggregation to Stackdriver
• MetricServer Setup
• Horizontal Pod Autoscaler configuration

… and much more. With existing tools like Stackdriver and Kubernetes, it was simple to enrich the capabilities of streaming application. There is also a list of useful readings on the same subject. Excellent!

Google has developed a methodology to make the design process fast and still offer valuable insight. Forget minimum viable products and focus on prototypes and build and test in a week! An article by interaction-design.org.

The Google design sprint operates in a 5 phase process. Each phase takes approximately 1 day to perform (8 hours) and all 5 phases take approximately 40 hours to execute in full.

The 5 phases of Google’s Design Sprint:

• Unpack
• Sketch
• Decide
• Prototype
• Test

You are strongly encouraged to make revisions based on your first sprint and then re-iterate the last two phases.

Google design sprints should help you take a process that currently takes months and make it lean and efficient. It is not a substitute for all design processes but one that lets you ideate and test ideas incredibly quickly.

Links to further resources also provided! Very well done!

Ben Kehoe wrote a blog post which is a general critique of Cloud Run relative to FaaS and managed API services – and how this is different from AWS Lambda.

Google’ s Cloud Run allows you to hand over a container image with a web server inside, and specify some combination of memory/CPU resources and allowed concurrency. The logic inside your container must be stateless.

Cloud Run then takes care of creating an HTTP endpoint, receiving requests and routing them to containers, and making sure enough containers are running to handle the volume of requests. While your containers are handling requests, you are billed in 100 ms increments.

All of the web logic is inside your container; auth*, validation, error handling, the lot of it.

What is good about Cloud Run?

• Very simple for people who are running containerized, stateless web servers
• Better scaling and fine-grained billing
• Dead simple to test locally (container)

What is bad about Cloud Run?

• Serverless should focus mainly on business value (outsource everything else to managed services)
• It is not FaaS

Find out more in this first overview of Cloud Run!

An article by Josefina Blattmann. If you ever had a business idea you probably wondered which steps you should follow in order to discover if it’s actually a product, this article walks you through the process.

To achieve success, there must be a validated idea that will increase your chances of building an MVP that won’t be rejected by the market.

If you want to build a product, a feature or anything, you need to get the story straight and that’s where a Product Manager comes in. They need to make sure that there is a clear roadmap that makes sense to everyone.

Article then provides a great summary of the stages every product goes through:

• Conceive – the unique value proposition of the product
• Plan – market research of the features
• Development – build the product
• Iterate – learn from users

Article also goes briefly over possible scenarios in which you may consider a product finished. Good read!

This is comparison between the largest serverless vendors (and some open source alternatives) has all the right metrics. From the pen of Ihor Lobastov published in March 2019. According to the RightScale 2018 State of the Cloud report, serverless architecture penetration rate increased to 75 percent.

In a nutshell, FaaS is a form of serverless computing that uses an infrastructure fully managed by a provider to upload functions and run them on a pay-per-request basis. Serverless completely abstracts developers from servers and allows them to focus on business logic.

Benefits of Serverless Architecture include lower costs and scalability, faster development and deployment, reduced expenses on human resources, high availability and auto-scaling, focus on business needs.

The article then gives detailed overview of and comparison of serverless by various cloud providers, including:

• Pricing models and billing factors
• Programming languages supported
• Trigger types
• Deployment methods
• Monitoring and logging
• Other options to consider

All the major serverless stack providers propose relatively equal infrastructure services. AWS Lambda and Azure Functions are still the most complete and diverse services to work with. Well written!

Good UX requires that all information is logical and within reach. Check out this complete guide and leave no user lost behind! Great article by Rebeca Costa.

Rule of thumb: Pay attention to detail, and don’t bombard your users with more information or more decisions than they can handle.

Whenever we use any website or mobile app, we are faced with an interface that lays down information to us as we need it or look for it.

This guide then analyzes:

• What is information architecture?
• Elements of information architecture
• Eight principles of information architecture

Information architecture can be surprisingly difficult to define. This is in part because while other subjects such as content writing can be narrowed to a specific job title – the writer – the field of IA stretches across many professions.

Information architecture is vital if you want users to understand your product enough to enjoy it. Find more about UX and design in the series of articles by clicking on link below.