Skip to main content

Posts

Design a serverless web scrapper

Greetings In a previous article , I explained how we synced the WordPress website into our microservices architecture. We use AWS EKS for our deployments and managing microservices. But what if we use a serverless architecture? Here is my attempt at the serverless solution on my serverless journey. Note that this is about converting your own website into a serverless architecture. Always check ethicality in use. Note: This is about the architecture, not about actual coding. Requirement Convert existing website into a serverless microservice. Technologies By sticking into Nodejs and AWS I would go with the below stack. AWS Lambda AWS S3 AWS Eventbridge AWS RDS AWS API gateway AWS Cognito AWS CloudFront AWS Amplify AWS SES Nodejs JavaScript Axios Cheerio React I would consider AWS Step functions in the future as one lambda is doing too many things (save in database, store images in S3, send emails). However, my AWS Step functions knowledge is not as great as today. Architecture Let's

Redis Cache with Nodejs, AWS

Greetings Modern applications are highly data-driven, and distributed and performance is a top priority. Redis has become one of those critical components in most applications. In this article, we will briefly look into Redis and how we can use it in a NodeJS application. "Power real-time applications with sub-millisecond latency" Redis Fast, open source in-memory, a key-value data store for use as a database, cache, message broker, and queue (from the doc). It stands for Remote Dictionary Server. We usually think of Redis as a cache because it is the most used use case. But Redis is more than a cache. As an example, Redis is an ideal option to store user sessions in web applications. Here are a few use cases but for more details better to read the documentation. Caching Session store Messaging Queue Gaming leader boards Geospatial Caching is a must in Microservices architecture to improve performance. We utilize Redis for this purpose in our e-commerce ap

Play with Elasticsearch basic queries

Greetings In a previous article , we set up our environment to play with Elasticsearch. Let's try basic queries in this article by building a movie search. Unlike other databases, Elasticsearch gives full searching power. Let's slowly dig into that power. Hence this is not a deep dive. I use the top 250 movies from  imdb-api in this exercise. Movie database Let's build a movie database using imdb-api and play with queries. Save movies as documents Movies to contain text, numbers, arrays, full text Structured search using movie ratings, year Full-text search (title, actors) Phrase search (plot, title) Return highlighted search snippets Note: You can either use Kibana UI or a direct call to Elasticsearch. Hence I skip the URL path. curl -XPUT -H "Content-Type:application/json" localhost:9200/movies/_doc/tt0068646 -d ' { "title": "The Godfather", "year": 1972, "rating": 9.2 }' Indexing mov

Setup Elasticsearch playground

Greetings When learning new technology, installation is the first burden and it is irritating. However, in modern-day development, we developers usually don't have to bother too much with operations and there are dedicated teams assigned for that purpose. Let's see how we can get started with Elasticsearch. For someone who starts exploring Elasticsearch, there are two main options. Cloud free trial Docker container Cloud free trial This is the easiest way for someone new to Elasticsearch. You just register for the free trial without a credit card. The only drawback is it expires in 14 days. But that is enough for initial experience. If you know how to use emails with plus at the end, you can have endless accounts. https://www.elastic.co/cloud/elasticsearch-service/signup Docker container The other option is to spin up a Docker container. That is also an easy option. However, you need to know just enough Docker to use it. With Elasticsearch 8 security is enabled by d

Architecting seasonal offers using AWS serverless

Greetings! When I started writing lambda and serverless, I wanted to design a real-world project. The example that was in mind was a geolocation extractor using an image upload that my friend did. I thought to tweak it to an Excel file upload with another example that is based on real-world e-commerce requirements. What I want to focus on here is the architecture. Seasonal offers Offers are common in any business. Now the marketing team decides that having a separate section to indicate seasonal offers with additional descriptions would increase the orders. As they are experts in excel they say hey Devs, we don't want a UI for us. We will give you an excel to make your lives easier. Requirement Design a restful API to expose seasonal offers that are coming as an excel sheet. Technical breakdown We can store the excel in our code and directly read it. But that is not an expandable solution. Hence we store the file in an S3 bucket and separate out the extraction logic. Architecture