#python

Testing and Comparing 5 Rest Api Clients While I was working on a performance test of my folaris server, I wanted to find out how well it performs. The performance obviously depends on the available network bandwidth and CPU power. I was happy to see it performs well without using a lot of memory and processor time. First I tested with Powershell/Invoke-RestMethod on localhost interface, I got an average 606 calls per second. [Read More]

sqlalchemy is widely used. For example, Pandas read_sql uses sqlalchemy to connect to a supported database. In this test, I want to transfer all records between two identitical tables. It is useful when you need to move data between 2 non-production environments. This is not the most efficient way to move data, but I will show here that it’s very a convenient way to move small data sets under 500K records. [Read More]

Code are on github Summary Apache Airflow is Python based workflow scheduler. You have to know Python to use it. I will focus on how writing a DAG, the workflow definition in Python. Then I will adapt the code into an Airflow plugin to make the DAG file more readable. References: Apache Airflow Github Airflow Docker image DAGS The Python workflow definition files are stored in a subdirectory defined in airflow. [Read More]

Source code: My Test-Drive github repo Summary build a Postgres Docker image start a new container get data from College Scorecard web site perform an ETL to get a subset of the data install PostgresSql client on Windows load data to server Build a custom Docker image based on Postgres 11 Dockerfile and create user/db script FROMpostgres:11.2COPY ./create-bruno-db.sh /docker-entrypoint-initdb.d/create-bruno-db.sh create-bruno-db.sh This file must be executable If you edit it on Windows, make sure you convert Windows CR LF to Linux new line charater LF [Read More]