a dev blog
-
Silver model part II
We was analyzing our tables in bronze layer to clean in silver layer, we saw that we have some problem with the tables staff and staff_schedule because we have different id_staff from staff schedule and staff table, also we have people that exits in staff_schedule and didn’t exist in staff table, in other hand the…
-
Silver model part I
We will continue our example, we will analyze the data of the files , clean and transform the data from bronze schema. we will analyze the tables staff, staff_schedule and services_weekly. staff we will select the role unique values from staff table : we will select the service unique values from staff table : staff_schedule…
-
How to create DBT Models
In this post we will explain how to create DBT models, we will continue the hospital example and we will transform the data from bronze schema and load to silver schema, first we will analyze the data, make some queries filter and transform the raw data from Bronze schema. After filter the silver schema, we…
-
How to use dbt seed
In this post we describe how to used dbt seed, is a command that load csv files to the database, this function was created for load small files, for example : Example of DBT seed In our case, for only a example we use dbt seed to load the data from csv files. The files…
-
How to setup dbt core ?
Here we will explain how to setup DBT core in Dev container, in the other post https://juliovaldiviamarin.com/dev-container-dbt-postgresql-metabase-part-1/ we see how to install DBT in vs code deb container, now inside in the dev container we will setup dbt core like: dbt project name, profile database(hostname, port, username, password,dbname, schema, threads) dbt init dbt init is…
-
Dev Container + DBT + Postgresql + Metabase Part 1
I gonna to explain in many post how to create a project using DBT-Core to transformm data, postgresql to save the data and Metabase to show the data in dashboards. For this example I gonna to use VS code y Dev container to develop, first i gonna to explain how to setup the Dev Container.…
-
Amazon DynamoDB Basic Apis
Amazon DynamoDB Basic Apis Writing Data Amazon DynamoDB Basic Apis Reading Data 1 GetItem Reading Data (Query) 2. Returns: 3. Abaility to do pagination on the results 4. can query table a local secondary index or a Global Secondary index Reading Data (Scan) Deleting Data Batch operations PartiQL Example of update and delete , query…
-
Amazon DynamoDB (RCU & WCU)
Read Write Capacity Modes to control the tables capacity we have 2 methods Provisioned Mode You specify the number of reads / writes per secondneed to plan capacity beforehandPay for provisioned read and write capacity units On demand Mode Read and write automatically scale up/down with your workloadsNo capacity planning neededpay for what you see,…
-
Amazon Database
Amazon Dynamo DB Traditional databasesVertical Scaling (getting a more powerful CPU / RAM /IO) Horizontal Scaling (increasing reading capability by adding EC2/ RDS)g NoSQL databases we have Mongo DB and Amazon Dynamo DB and No query joins all the data that we needed is present in one row no aggregation such a SUM AVG databases…
-
AWS Backup
AWS Backup fully managed servicecentrally manage and automate backups across AWS servicesno need to create custom scripts and manual processes supported services AWS Backup vault lock enforce worm write once read manystate for all the backups that you store in your AWS backup vault additional layer defense to protect backups against inadvertent or malicious delete…