Welcome to curated list of handpicked free online resources related to IT, cloud, Big Data, programming languages, Devops. Fresh news and community maintained list of links updated daily. Like what you see? [ Join our newsletter ]

How to build a serverless web crawler

Categories

Tags serverless continuous-delivery containers web-development

James Beswick wrote this piece about how to use serverless to scale an old concept for the modern era. It describes a client project which involved the need to crawl a large media site to generate a list of URLs and site assets.

For Node users, there’s a package that does this elegantly called Website Scraper, and it has plenty of configuration options to handle those questions plus a lot of other features. The package is mostly configuration-driven.

The article is split into:

  • Let’s crawl before we run
  • Serverless Crawler — Version 1.0
  • Serverless Web Crawler 2.0 — now slower!
  • Show me the code!
  • Serverless Web Crawler 3.0

There are many reasons to crawl a website – and crawling is different to scraping. The article describes evolution of lambda aws based solution with detailed experience and learnings. Great!

[Read More]

Full text search with Node.js and ElasticSearch on Docker

Categories

Tags search servers apache web-development

An article by Michele Riva about building the search engine for simple REST API using Node.js and ElasticSearch. ElasticSearch is a search engine server built on top of Lucene with an amazing distributed-architecture support.

Full-text search can be both scary and exciting. Some popular databases such as MySQL and PostgreSQL are amazing solutions for storing data. But when it comes to full-text search performances, there’s a little competition with ElasticSearch.

Author guides you through building REST application called The Quotes Database which will allow us to store and search as many quotes as we want. He has prepared a JSON file containing 5000+ quotes with their authors, it is a starting data for populating ElasticSearch.

The article sections:

  • Setting up Docker
  • Setting up docker-compose
  • Bootstrapping the Node.js app
  • Populating ElasticSearch with quotes
  • Creating a RESTful API
  • Launching the application

Detailed guide / tutorial with all the code for Nodejs application, including ElasticSearch connector, CURL commands and how to for indexing operations available. Excellent!

[Read More]

Tour of Akka Typed: Message Adapters, Ask Pattern and Actor Discovery

Categories

Tags akka java scala programming messaging queues

An article in series by Manuel Bernhardt in which he explores the basics of the Akka Typed API, why it was created and what are its benefits in comparison to the classic Actor API, how to build typed actor systems via protocols and behaviors and how to create typed actors.

The author continues to go further down the route of building typed actor systems by looking at fundamental concepts necessary for the interaction between actors. He is also using the functional style of the typed API to create the actor.

The article deals with:

  • Adapted Responses
  • The ask pattern
  • Discovering actors with the Receptionist
  • Concept comparison table

The ask pattern allows two actors to interact in such a way that there’s a 1 to 1 mapping between the request and the response. Since there’s no guarantee that the responding actor will in fact ever respond, the ask pattern requires to define a timeout after which the interaction is considered to fail.

Code examples are includes for you to test. Nice one!

[Read More]

Managing hundreds of millions of orders with Table Store

Categories

Tags mysql database web-performance how-to cloud

This article discusses how Table Store can help you store and manage massive amounts of data by using a fictional e-commerce platform as an example.

Ordering systems are a type of general-purpose systems that are widely used in various industries, such as e-commerce orders, bank journal entries, and phone bills from communication service providers. Traditional practices for these systems have been established over decades of development. However, the development of Internet and increasing importance attached to data by enterprises lead to an increasing number of orders that need to be stored and persisted. The increasing data importance and data scale bring new challenges.

The article’s content:

  • Scenario and technical considerations
  • Query capability, data volume, service performance
  • Solution evolution, including ordinary
    • MySQL database/table sharding
    • MySQL + HBase
    • MySQL + ElasticSearch
  • Table Store
    • Overview of the order system based on Table Store

… and much more. The SearchIndex solution from Table Store can perfectly support an order system holding 100 million orders. Great overview for architects working with successful e-commerce shop!

[Read More]

A developer's guide to setting and smashing career goals

Categories

Tags agile teams career how-to

An older article by Isabel Nyo about setting developer’s career goals. There is little information available for developers and software engineers on how to set these effectively.

These days, I help developers and software engineers set and smash career goals that are specific, measurable, achievable, timely, and most importantly, that align with their values and desires in life.

Find an accountability buddy who will check in with you periodically. It is better if your accountability buddy has similar goals so you can help each other out.

Author goes through five essential strategies to effectively set and smash career goals for a developer, using relatable examples and best practices:

  • Look at what is out there
  • Understand what you want
  • Set achievable goals
  • Create a personal brand
  • Be accountable

A lot of developers give up on their career goals way too early. They are either too busy in their current job, or everyday life activities like relationships, children, marriage, and paying bills had taken over and they lost sight of their career goals. Great advice!

[Read More]

Choose boring technology

Categories

Tags programming software-architecture miscellaneous how-to

The perspective of seasoned developer on development process and how big companies can have a branded way of doing engineering. Compared with situations where “the way engineering is done” is actively being figured out.

The essay basically focuses on:

  • First, how do you pick the tools you’re going to use to get your work done?
  • How do you make developers happy?

If you go up to a developer and say, “hey, what would make you happy?” you often get highly specific answers. “I would be happy if I could write Clojure at work,” they’ll say, “so you should let me do my work in Clojure.” I don’t discount that when they do this, developers are genuinely describing some kind of endorphin-rich experience they have had at some point. – The answer for many: chase shiny objects

If you think about innovation as a scarce resource, it starts to make less sense to also be on the front lines of innovating on databases. Or on programming paradigms. Software that’s been around longer tends to need less care and feeding than software that just came out. Good read!

[Read More]

Step Zero: Kubernetes on Microsoft Azure Kubernetes Service (AKS)

Categories

Tags devops azure programming software-architecture

Step to step guide how to create a Kubernetes cluster using the Azure command line tools. It describes the commands required to setup a Kubernetes cluster using the command line. By JupyterHub.

This is straightforward guide and you have two options: one is to use the Azure interactive shell, the other is to install the Azure command-line tools locally. Instructions for each are included.

The guide then lists and explains how to do these:

  • Prepare your Azure shell environment
  • Activate the correct subscription
  • Create a resource group
  • Choose a cluster name
  • Create an SSH key to secure your cluster
  • Create an AKS cluste
  • Install kubectl, a tool for accessing the Kubernetes API
  • Get credentials from Azure for kubectl to work
  • Check if your cluster is fully functional

If you prefer to use the Azure portal see the Azure Kubernetes Service quickstart. For detailed commands followe the link below, easy!

[Read More]

Mono repo vs one per service

Categories

Tags devops microservices programming agile aws software-architecture teams

An article by Yan Cui about his take on using code repository when working with microservices and serverless. With AWS Lambda, we can deploy and scale individual functions. However, we engineers still like to think in terms of services and maintain a mapping between business capabilities and service boundaries.

The service boundaries allow the teams to be autonomous and empowers them to choose their own technology stacks. Where AWS Lambda is a good fit, a service would be made up of one or more Lambda functions as well as event sources such as API Gateway. The fact that the functions that make up a service are scaled independently is merely an implementation detail, albeit one that can work very well to our advantage.

Before we delve into it, I should clarify that within a monorepo you can still deploy services independently. Functions are still organized into services, and each service would reside within a subfolder in the repository.

The article explains:

  • What is a monorepo
  • How one repository per service works
  • Which approach should you choose?

The monorepo approach offers small teams a great opportunity to gain early momentum. But you need to keep in mind that the cost of this approach would skyrocket as the organization grows beyond a certain point. Nice read!

[Read More]

The basics of Kubernetes networking

Categories

Tags devops programming kubernetes containers

Kamil Lelonek explains everything you need to know about Kubernetes networking in this article. When you deploy your application to a Kubernetes cluster, you usually want to have it accessible from the outside. For real-world production applications, one of the important questions to ask is how to get external traffic into your container.

A ClusterIP is an internally reachable IP for the Kubernetes cluster and all Services within it. For NodePort, a ClusterIP is created firstly and then all traffic is load balanced over a specified port. The NodePort itself is just an iptable rule to forward traffic on a port to a ClusterIP.

The article details:

  • Terminology – pod, node, containers, labels, service
  • NodePort
  • Ingress

In Kubernetes, an Ingress is an object that allows accessing your services from outside a cluster. Boths charts explaining the concepts and example configuration are also included. Excellent!

[Read More]

Monkey patching TLS in Node.js to support self-signed certificates with custom root certificate authorities

Categories

Tags nodejs security web-development apps

David Barral wrote this detailed guide on updating TLS (Transport Layer Security) in Node.js and how to consume services that have been secured with self-signed certificates.

A self-signed SSL Certificate is an identity certificate that is signed by the same entity whose identity it certifies.

The self signed certificate in certificate chain error means that our client sees the certificates but does not know the CA that issued the server certificate and cannot trust it. That’s cool, node is helping us to protect against a man in the middle attack.

The article describes and explains:

  • Self-Signed certificates
  • How to test HTTPS server with self-signed certificates
  • Client Error: self signed certificate in certificate chain
  • How to add extra root certificates to node
  • Monkey patching tls.createSecureContext

A monkey patch is a way for a program to extend or modify supporting system software locally (affecting only the running instance of the program). good read.

[Read More]