Python, Amazon, and Rust

API on AWS Project 

I have been working on several projects the last couple months. I wrote a serverless application in .net the first functions are to fetch various amazon accounts so that we can iterate over them for some future projects. I went down a few rabbit holes but ended up with a webapi c# serverless project that connects to a postgres database on Amazon RDS.  
 

The whole project was pretty illumninating because I built it with several different AWS services API Gateway, RDS Postgres, SSM Parameter Store, SSM Secrets, Cloudfront, EC2 serverless deployment, private vpc, network loadbalancers and a few other things. I also started using the AWS toolbox in Rider, VSCode, and Visual Studio. I can see where someone with a lot of IT experience has a huge leg up on the young developer on setting up this type of infrastructure for the first time. 
 

Amazon Advirtising API to S3 Files 

From there the next project was to build a some lambda functions in Python 3. I found the workflow to slow from ideas/psuedo code to the working python code, so for the initial programming I used colab.research.google.com which was a lot of fun. You can do !pip install boto3 and it will install the boto3 library into your workspace. Only downside is that you need to pass in your credentials to the client and session objects. When you then deploy to production you have to remove this code. 
 

Once I got everything working I refactored some of the code into other modules and setup a layer for the needed modules that are not included by default in the AWS Lambda Functions. For this project I got to use a few new AWS services SQS, Step function, event bridge, S3, lambda on a queue, serverless config file, and had to setup some special IAM roles and policies. This project itterates over our Amazon accounts and then downloads various snapshots, and reports from the Amazon API. In the coming months we will be taking the data downloaded from amazon and importing it into our dimensional and fact tables in SnowFlake which will be a little bit new to me. I have tons of SQL experience but it was all pretty standard database work to support custom apps and websites. 
 

So I will be learning more about PowerBI, and Snowflake, the datalake and datawarehouses in the near future. 
 

Rust pains 

Learning Rust is going slowly because I don't have a lot of time for it and no real projects. However I have read several books, completed several courses on Udemy and started one on Pluralsight.com. I have completed the rustlings with out to much trouble also. I do have an idea for an application that would be good, I am thinking about creating a rust cli to gather data from a mongodb database where we record our rate shopping for shipping and maybe also connect to a mysql database to get the awarded rate and shipping information so that I can then export it to a csv or xslx file for use by logistics. The rust app could then be deployed to our rundeck server to email the report to various recipients. 
 
gatherShippingRates [-i IFNUMBER | -d yyyymmdd] [-o csvfilename] [-h|--help]  

Rust for sure has a steep learning curve but it certainly in some points is functional with the iterators and other functions. The library eco system is vibrant although the 3rd party packages are deeply concerning from a security standpoint, and that may be one of the biggest struggles developers and companies have in the coming years. Every week I hear of new exploits or backdoors in 3rd party libraries, I would hate to see us go back to just using paid libraries because that was not a great experience back in the 90s and early 2000s when I had to buy them for VB6 or C# and .net. 
 

Enhancing this blog 

Also I have been using Amplify with static site generatorrs to rebuild my personal blog and website.