Rust. AWS S3. Scripts. Python. Datetimes. Files. Etc.

I was recently sucked into doing something like a caveman … writing Rust by hand with the wonderful Zed IDE to solve a Python problem. Ahh … what a wonderful world we live in. It felt strangely good to do such things; it brought me back to the days pre-LLM when I would fight tooth and nail with that borrow checker and compiler. Like it or not, those days are mostly done as far as the future of Software Development goes … I’m sure there will be little hobbit holes of bearded Rust developers hacking away by hand long into the future, but that will be the exception, not the rule.

Anywho, the entire process did remind me of why such hand-writing of Rust is important as a mental exercise, at least.

Writing code and solving problems slowly might not be the most economical way to produce software anymore, but that sort of process is important for those in charge of large systems and software. Sure, having been writing code since Perl was a thing, I’ve maybe learned a few things about how to produce good products both technically and soft(skill) wise. I mean, we are paid to produce digital widgets, and that is a mix of both technical (now cheap) and philosophy.

For example, recently I reviewed a bit of code that was put forward into an existing codebase to add a simple digital widget, one of many pre-existing widgets. I ended up explaining why, although the solution would indeed work, the addition of 1000+ lines(vibe) to the codebase wasn’t the best approach when a simple 150 lines would do well enough. Well, this has always been the problem well before AI/LLMs; it’s simply just accentuated now and accepted practice. AI simply makes people more of what they are today.

I’m not here to bemoan the change that has swept us software writers away into a new and unknown ocean. We must accept change and do our best to fit into the new world, find a spot where we can do what we love at least some of the time … and be paid to do so. No amount of hand-wringing and wailing will change the spot in which we find ourselves. We aren’t the ones holding the purse strings (for the most part), and so we must do what we are bid.

But even the most ardent prompters of The Vibes must recognize that it’s always been a slippery slope when producing digital widgets, and prompt or no-prompt, sticking to good design principals is important.

What are some things that writing a simple S3 Rust script instead of Python (best suited for simple scripts) reminded me of?

Performance is important … it saves us time and compute money, uses less memory etc. This matters to the bottom dollars.
Being forced to write simple, clean code that does no more and no less than intended is true Gold indeed.
Reminders of async code and how network I/O is a large bottleneck these days
Some things like timestamp and string manipulation never change … for decades they never change.
AWS makes wonderful to use SDKs for many languages, in today’s cloud world, you should know them and use them.

The problem I recently ran into was that I had an S3 bucket simply chock-full of tar files, some large, some small, each tar file filled with maybe 10 other gz flat files. I also had another S3 location that had the unpacked files from inside those tars. I’m talking thousands and thousands of tar files over days, leading to millions of files, and 10s of millions of records in those gz files daily.

Just enough to be annoying.

The simple question was … are we missing some individual .txt.gz files? All those wonderful AWS lambdas doing the daily work of unpacking tars … yeah sometimes they puke for various reasons, usually memory-related. Of course what I am not mentioning is the inherent technical debt from years and business logic strangeness in which those S3 buckets and tar files and .txt.gz files exist. No one lives in a perfect world, neither do I.

I knew beforehand that there was going to be timestamp antics, tar unpacking antics, I/O antics waiting for results. Look, all problems are easy on the surface; it’s the details and bumps that make it hard to do right. Sure, an LLM could pump out a Python script in 30 seconds to do that work … which it did, but painfully slow.

Do you know how good it felt to watch those lines of Rust appear from fingers into the Zed IDE? Just grabbing some simple CLI args?

#[tokio::main]
async fn main() -> Result<(), s3::Error> {
    let args: Vec<String> = env::args().collect();
    let mut tars = false;
    let user_s3_bucket = &args[1];
    let user_prefix_filter = &args[2];
    if user_prefix_filter.contains("raw_tars") {
        let tars = true;
    }

So simple and nice, clean.

The AWS-provided Rust SDKs are simply a joy to work with.

use aws_config::BehaviorVersion;
use aws_sdk_s3::{self as s3, Client};
...
let config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let client = s3::Client::new(&config);
let existing_files =
        gobble_up_exisiting_files(client, user_s3_bucket, user_prefix_filter, tars);

I mean, such nice code in Rust? Heck, looks almost like Python … *gasp.

Of course Rust can’t help being a little verbose at times; some functions look dirty, like your 10-year-old self in the back of the house playing in the mud pit your Mom told you to stay out of.

async fn gobble_up_exisiting_files(
    client: Client,
    user_s3_bucket: &String,
    user_prefix_filter: &String,
    tar: bool,
) -> Vec<String> {
    let mut paginator = client
        .list_objects_v2()
        .bucket(user_s3_bucket)
        .prefix(user_prefix_filter)
        .max_keys(1000)
        .into_paginator()
        .send();
    let mut existing_files = Vec::new();
    while let Some(page_result) = paginator.next().await {
        match page_result {
            Ok(output) => {
                for object in output.contents() {
                    if let Some(key) = object.key() {
                        if tar == true && key.to_string().contains(".tar") {
                            existing_files.push(key.to_string());
                            println!("{:?}", key.to_string());
                            sniffing_tar_file_contents(&client, user_s3_bucket, key.to_string());
                        }
                        if tar == false {
                            existing_files.push(key.to_string());
                        }
                    }
                }
            }
            Err(err) => {
                eprintln!("Error fetching page: {:?}", err);
            }
        }
    }
    return existing_files;
}

I mean, yes, it’s ugly code because I write ugly Rust code like it’s Python or something. But heck, when it compiles, you sleep good at night. I mean, it’s Rust after all. It makes you think about Vectors and Strings and actually dealing with errors, doing a match when you don’t have to. It requires you to use your brain to think about what you really want to accomplish at the end of the day, and just write code to do that one thing and nothing else. Screw the edge cases!

Heck, the simple discovery of a wonderful Rust crate for tar files was worth the entire agonizing process itself.

use tar::Archive;
...
async fn sniffing_tar_file_contents(client: &Client, bucket: &String, key: String) {
      println!("trying to sniff tar file");
      let response = client
        .get_object()
        .bucket(bucket)
        .key(key)
        .send()
        .await
        .unwrap();

     let bytes = response.body.collect().await.unwrap().into_bytes();
     let mut archive = Archive::new(Cursor::new(bytes));
     for entry in archive.entries().expect("failed to get tar archive") {
        let file = entry.unwrap();
        println!("{:?}", file.path());
    }
}

Of course, who doesn’t want to do strange date and string wonders to meet the strangeness of the technical debt you run into?

use chrono::Local;
use chrono::{Months, Utc};
...

fn get_strange_date() -> (String, String) {
    let today = Local::now().date_naive();
    let one_month_ago_today = &today
        .checked_sub_months(Months::new(1))
        .expect("failed converstion");
    let strange_today: String = today.to_string().replace("-", "")[0..6].to_owned();
    let strange_last_month: String =
        one_month_ago_today.to_string().replace("-", "")[0..6].to_owned();
    return (strange_today, strange_last_month);

This sort of code seems, and probably is, purpose-built for an LLM-puked Python script. It’s not particularly important or useful to the business in any meaningful way other than a novelty that might get used once a year if you are lucky. It’s hard to argue with the green-eyed AI psychosis that has gripped the CTOs and C-suite at large … our paychecks come from profits in the short term, at least … we need to use automation where it makes sense … let The Vibes ride when they should ride.

But, just like you always must be learning and growing to stay relevant in your career, you should stop and smell the roses at certain points, in this case something like Rust, C, Golang, Scala … whatever, and be reminded why simplicity matters and first principals can be helpful when examining LLM generated codebases.

It’s a fine line to walk, but I think we should try to walk it as tightly as possible. Who knows what the future holds, but be just the best possible version of yourself technically; don’t reject the future as unfair or stick your head in the sand and pretend like the LLM wave isn’t coming. Be a survivalist. Vibe when they want to vibe, write the Rust by hand when you feel spicy, and don’t forget from whence you came.

Rust. AWS S3. Scripts. Python. Datetimes. Files. Etc.

Interesting links

Pages

Categories

Archive