I want to quickly share a little gem of a tool that I have been using for some time now. It's a cute, simple notification tool, and I think it's invaluable for anyone that does any form of self-hosting, but could potentially also be useful in small organizations.

---

## What is ntfy ?

In simple terms, it's a utility that will let you send and receive notifcations via HTTP(S). You can create different "topics" (basically feeds) to subscribe to, and impose access control on each topic using accounts and api-keys (or forego it altogether, and have it be a public channel). You can learn more about the project on its [website](https://ntfy.sh).

It has a bunch of delivery mechanisms you can hook it up to as the web app, text messages, phone calls, and my favorite, the mobile app.

## The Mobile App

What makes ntfy worth having for me, has to be the fact that it has a mobile application that can be found on both Android and iOS platforms.

This means you can hack any of your existing services to use ntfy and delivery push notifications to your mobile device, giving a quasi-native app experience.

There is one caveat, though. If you're reading this, you're most likely interested in running the service yourself. This means that you will need to re-compile the mobile applications yourself.

Well, technically you don't *have* to. But an important aspect of notifications is their availability and punctuality. Who wants a notification that's an hour late?

The TL;DR is, you need to compile it yourself in order to let your device use dedicated polling utilities to efficiently listen for incoming notifications.

## Some Usecases

Here's some ideas for how to use ntfy:

1. Server health checks
2. Custom email notifications
3. Content feed monitoring
4. **Bonus:** My absolute favorite way to use ntfy is as a cross-platform clipboard.

review

FOSS Highlight: ntfy

It has been over 4 years since the release of ChatGPT 3--arguably the catalyst responsible for popular interest in generative AI--and it remains more polarized than it should be.

Before I start, I want to make it clear I am by no means an "AI bro", nor do I (at the time of writing) have any financial associations with a business in the AI space. There's been a lot of arguments against the usage of AI (despite the fact that it's not going away), and while some of them are valid critiques, many of them just seem more representative of some form of luddism, tribalism, or elitism.

The list of arguments I address is by no means comprehensive, but it is representative of some of the arguments I find most egregious or am just tired of seeing.

---

## Counterpoint:  Generative AI takes jobs away from artists

First off, this is peak [Luddism](https://en.wikipedia.org/wiki/Luddite). That is not to say it isn't true, there have been many people displaced from their careers by AI, especially writers, and they all have my deepest sympathy.

But, consider that humans have been innovating for thousands of years now. It's the same song and dance we have been through as a civilization multiple times. Technological breakthroughs always end up displacing some careers, but they end up creating new ones.

> But this is unprecedented

Big innovations are always unprecedented, that's what makes them innovations.

An extremely similar event was the invention of the printing press, which eliminated the need for scribes altogether. I hope we're all in consensus that this was a *good* thing.

Secondly, **you can't make this argument while also saying that generative AI works are inferior** (without further clarification)

Let's imagine a simple, but realistic scenario: An artist makes some amount of money doing commissions for people to make posters or avatars, or whatever. Now that people can generate an infinite amount of anime profile pictures, that will of course, reduce some amount of orders that artist will get.

And now two sub-scenarios:

**1. The GenAI content is superior to the artist's works**

It's generaly agreed that this is not true, but suppose that it was--what's the problem here? Is it reasonable to expect someone to *pay more for an inferior product*? I don't think so.

**2. GenAI content is inferior to the artist's works**

Okay, so now what's the issue? That people aren't paying for something they don't need? This would be like being Ferrari and getting mad at Honda because they're making cheaper, more affordable vehicles. People might not need the quality of hand-made art. This is not even a novel issue, **people already buy mass manufactured art**.

If you have a superior product, then you shouldn't get upset that a cheaper alternative is now available to others. Your market of high-quality art isn't threatened. Yeah, you're losing *some* customers, but those customers were only customers because they didn't have an *alternative*. To get upset at this is straying close to gate-keeping art. And speaking of Generative AI content being inferior ...

---

## Counterpoint: Generative AI Content Has No Soul

You're right. It doesn't. But I don't think that's a problem. Believe it or not, I completely understand how meanings and emotions that enriches art. I took a performance art class in college that changed how I perceived art completely.

Yeah, when an AI makes a picture of a sunset, it doesn't come from a sense of awe or appreciation of nature, and thus the created work is void of emotional depth. **But sometimes, people just want a pretty picture**. The human story behind an art piece is a significant component of the piece, but it's not *all of it*. A picturesque scene of a meadow can still be picturesque even if it didn't come from a person.

It's also ridiculous that sometimes I see people that ask whether AI was used to produced media in some video or game, and then if they find out that it does, then they exclaim something along the lines of "Oh, that explains why the art seemed so generic and lifeless". If you had to ask, that means you couldn't tell. And if you couldn't tell, why does it matter?

---

## Counterpoint: Generative AI can be used maliciously to disseminate harmful information

You know what? This is actually pretty valid. But still, my response is "*and?*". What's the goal here? If you're saying this to spread awareness and advocate for more research and development in fighting misinformation, then great! We're on the same side.

But I see some people use this as an argument against the existence of Generative AI completely. They're probably a minority, in which case, I'm probably wasting my time, but whatever, I see it enough that I'm sick of seeing it.

**Reality check:** Generative AI is not going *anywhere.*

The invention of online services like emails and banking made it easier than ever to scam people.

Did we get rid of emails? No, we create spam filters and advocate for public awareness.

Did we get rid of online banking? No, we implemented additional safety checks to mitigate the issue.

Did any of those get rid of the issue completely? No, but we still use them.

Same goes for Generative AI works. Let's focus on thinking up counter-measures and systems to protect against the spread of misinformation. *That's the solution.* You don't throw the baby out with the bath water.

---

## Conclusion

Generative AI content is not leaving anytime soon. As with anything in the world, it has its pros and its cons. But it's a nothing but a pipe dream to get rid of it.

I see so much fearmongering or doom-speaking about how its drawbacks are going to be the end of human creativity.

opinion

Stop the AI Fearmongering Please

I recently got into a small discussion on Hacker News with someone who suggested that "Using python ... is almost always a better choice than trying to write even one loop, if statement, or argument parser in a bash script."

I've written a fair share of Python and Bash, and it is my opinion that this thought is misguided, and that Bash and Python have their own niches to fulfill, and they both excel in their own domains.

So if you're someone who writes only Python scripts, maybe this post will encourage you to explore Bash.

<div id="blog-sections">
## Content

1. [Two Use Cases for Bash](#two-use-cases-for-bash)
    - [Quick and dirty scripts](#quick-and-dirty-scripts)
        1. [External dependency](#external-dependency)
        2. [Verbosity](#verbosity)
        3. [Complexity](#complexity)
    - [Interacting with system configuration](#interacting-with-system-configuration)
2. [ Best of Both Worlds ](#best-of-both-worlds)
</div>

## Two Use Cases for Bash
---

Here are two strong use cases for Bash. I was going to write more, but felt that the article was getting too long for what I thought was a relatively straightforward topic.

1. Quick and dirty scripts
2. Interacting with system configuration and tools


#### Quick and dirty scripts
---
If you know exactly what you need to do for a script, and only need to run it once, it just makes sense to make a Bash script. The pipe operator is an extremely powerful operator and massively reduces the amount of text you need to write. Take for example, scraping images off a website.

This how I would do it in a Bash script.

```bash
#!/usr/bin/env bash

set -euo pipefail
URL="https://example.com"
curl -s "$URL" \
	| pup 'img.myClass attr{src}' \
	| grep -v 'myFilter' \
	| xargs -n 1 -- wget -P imageDir
```

Look at that beauty. Now let's say I want to do it in Python. Maybe this is just a skill issue thing, but this is how I would do it.

```python
from requests import get
from bs4 import BeautifulSoup
import os

URL="https://example.com"
IMAGE_DIR="imageDir"

res = get(URL)
soup = BeautifulSoup(res.text, 'html.parser')

for img in soup.select('img.myClass'):
    url = img['src']
    if 'myFilter' in url:
        continue
    filename = url.split("/")[-1]
    with get(url, stream=True) as r:
        with open(os.path.join(SAVE_DIR, filename), "wb") as f:
            for ck in r.iter_content(chunk_size=8192)
                f.write(ck)
```

Boy that was a handful! And I didn't even bother to add error handling with `try-except`!

Let's look at some issues:

##### 1. External dependency

The Python script depends on the external dependency `requests` and `BeautifulSoup`. Technically, you could make the same script using the `HTMLParser` and the `http.client` that is part of Python's stdlib, but that would make the script _even more complex_.

> Aha! You have committed a folly, for you have forgotten the Bash script depends on `pup` 

Well, the difference lies in how each dependency is implemented. Firstly, the Python dependencies must be installed per interpreter. Now, unless you're trying to create a development environment for each Python script you write, that means installing the dependency globally. 

And because these tools do not follow KISS, it is very likely that some time down the road, a version will be released that will break your script. 

>But I'm only using the simple APIs, those aren't going to change! 

True, but the other issue is down the road, you might need to use a different library that depends on a specific version of these dependencies, and that's going to drop you down into shit creek without a paddle, as they say.

`pup` is a tool made with the UNIX philosophy in mind, and does a very specific job. Breaking changes are hard to fathom here, though I'm sure it's possible.

Most tools for the CLI are purpose-built and have minimal dependencies, if any. Whereas the design of Python inherently relies on importing modules, and while it's thankfully not too common, this can cause dependency conflicts to arise.

##### 2. Verbosity

I don't think I need to say much here.

##### 3. Complexity

The great thing about tools that follow KISS is that it's hard to get them wrong. The tools are designed for a very specific, common task, and as such their APIs are extremely accessible.

Python on the other hand, being an extremely powerful language, comes with a lot of the associated complexity. Everything is an object, and all those objects have a bazillion methods. Who has time to remember all the specific methods and their options? Why is there a `res.json()` and a `req.text` and a `req.raw` and a `req.content`?

And thank god the filter string was a simple string instead of a regex, because now we would have to interact with the `re` library and _its_ objects and methods.

#### Interacting with system configuration
---

On a Linux system, the command line is King. In any way you want to configure your system, you will be able to do it faster on the CLI than with a GUI. A lot of services and daemons will have CLI tools to interact with them. 

I cannot fathom why anyone would want to interact with say, `cron`, `docker`, `nginx`, or `ssh` with Python as a middleman. You would need to use the `subprocess` module, and then decide between using `Popen()` versus `run()`, and in both cases it's a PITA to access STDOUT and STDERR. And forget piping if you're going that route.


#### Best of both worlds
---

Now, what Python has over bash in terms of simple APIs, is data manipulation. For example, working with arrays, maps, and floats just isn't fun in Bash. Let's do something that would really suck, let's square an array of floats! We can employ Python inside our Bash script.

```bash
UGLY_FLOAT_ARR=("1.1" "2.2" "3.3" "10.123")
SQUARED_FLOAT_ARR=($(python <<EOF
x=[$(printf '%s,' "${UGLY_FLOAT_ARR[@]}"| sed 's/,$//')]
print(list([a**2 for a in x]))
EOF
))
echo "${SQUARED_FLOAT_ARR[@]}"
# Output:
# [
#  1.2100000000000002,
#  4.840000000000001,
#  10.889999999999999,
#  102.47512899999998
# ]
```

Ah, nice to see you again my old friend floating-point inaccuracies. Now, it's also true that you could also embed bash commands inside a Python script like so:

```python
import subprocess

bash_command = """
cat /etc/os-release | grep -o "^NAME=.*"
"""

result = subprocess.run(
    bash_command,
    shell=True,
    capture_output=True,
    text=True
)
print(result.stdout)
# Output: 
# NAME=NixOS
```

And I think that works decently, too. I haven't explored Bash-in-Python as much, so it's possible there are some additional benefits here that I'm not recognizing, but I believe the most significant factor boils down to whether most of your script is suited for native Bash or native Python. Any how, hopefully this helps some people in their scripting endeavors.


Bash vs Python for Scripting

Diving into the terminal, you will find that lines of command share a surprising kinship with firearm safety. Picture this: every keystroke is a potential bullet, and just as a responsible shooter takes deliberate aim, developers and administrators can adopt a similar mindset to safeguard their digital landscapes. Let's explore the unexpected parallels between command craftsmanship and muzzle discipline, uncovering valuable lessons that go beyond the screen and into the realm of responsible coding practices

Imagine a situation: you're working on a server when disk space starts going low. You check what's using it and find out some directory in `/tmp` has been accumulating junk. All right, let's clear it out.

> `rm: cannot to remove '/tmp/junk': Permission denied`

All right, think *carefully* about what you would do in this situation. What do you need to type?

...

Obviously, it's `sudo rm -rf /tmp/junk`.

But what's *actually* important is **how** you type, not **what** you type.

#### Firearm rule: Flagging

In firearm safety, there's a general rule that you don't point the muzzle at anything you don't intend to completely destroy. If you accidentally do at, some point, have your weapon aimed in an unsafe direction (e.g. at a friendly person) that's called **flagging**. In this post, I propose that typing out dangerous (i.e. irreversible commands) in the traditional fashion (i.e. left-to-right) is the command-line equivalent of 'flagging'.

Whilst typing `sudo rm -rf /tmp/junk`, there is a brief moment where what you have typed is `sudo rm -rf /`, which for the uninitiated, nukes your computer. Now while you'll carry on typing just fine *probably 99.9%* of the time, who knows what freak accident might happen? Maybe your pinky taps the ENTER key too early--maybe your cat jumps onto your keyboard. Uh oh. So here's what I propose as a habit to do instead: type everything without `sudo` first (`rm -rf /tmp/junk`), and write it back in once the command has been constructed. For *extra* added safety, you could also inspect the target file or directory beforehand--this would be the equivalent to confirming that what you're pointing your weapon at is *positively* what you want to destroy.

This isn't limited to linux shells either. 

#### Firearm rule: The "Safety"

Consider SQL commands (particularly **`UPDATE`** and **`DELETE`**). As a contrived example:

```sql
DELETE FROM users
WHERE name IS 'john'
  AND email ILIKE '%john%@example.com';
```

Some tools, like SQL, have a benefit that system shells typically don't have, which is the requirement of the semi-colon at the end--this is equivalent to the safety on a gun. However, firearm safeties should also not be relied upon as an infallible defense against negligent discharges. In this case, prior to completing your **`WHERE`** filters, you essentially have a query threatening to nuke your table. In addition, *even if* you type the full query out, there's still the risk of having an inaccurate **`WHERE`** filter. For this, I propose constructing your query initially with **`SELECT`** instead. The structure of the query remains exactly the same, you don't flag the entirety of your table, and at the end you can get confirmation that the rows you have selected are indeed the rows you are trying to delete.

#### Conclusion

There's a surprising amount out of parallels you can draw between command-line and firearm practices. The two I covered here seem to be the most applicable. There's a lot of additional layers of safety practices you can employ, but I feel like these are the bare minimum you should do in order to not have an incident. Some people might say this is too tedious, but to me it just seems like it should be common sense--and common practice.



tech

guide

The Command Line is a Gun

When you need to utilize a [ Relational Database Management system ]([https://en.wikipedia.org/wiki/Relational_database]) ( RDBMS ) in your application, should you use an [ Object Relational Mapper ](https://stackoverflow.com/questions/1279613/what-is-an-orm-how-does-it-work-and-how-should-i-use-one) ( ORM ) library? 

Well, as usual, the answer is ... <br/>*it depends* (but technically you can't avoid using an ORM anyways)*
<style>
h3 {
    text-align:center;
}
</style>

#### Table of Contents:

* [ Quick Comparison ](#quick-comparison)

* [ Why ORMs Are Evil ](#why-orms-are-evil)

* [ Why I Still Prefer Using Prisma ](#why-i-still-prefer-using-prisma)

* [ So What Should You Use ](#so-what-should-you-use)

* [Conclusion](#conclusion)




### Quick Comparison

For a quick intro, a RDBMS organizes information in tables and columns, which is inherently different than objects in object-oriented programming languages. ORMs seek to expose an interface in which you can interact with the data in the columns/tables using the native structures of the language.

Let's say I have a simple class `User` and I want to store one in the database. This would mean I need to implement the typical CRUD operations for it. For brevity I'll just do the find method.

#### Custom SQL

```python
db = ... # Connect your SQL db
class User:
    name: str
    age: int

    @staticmethod
    def find(name):
        with db.cursor() as cur:
            cur.execute("""
                SELECT * FROM users
                WHERE
                  name = %s
                  AND
                  email IS NOT NULL;
                """,
                (name) # Don't get SQL injected ya'll
            )
            user = cur.fetchone()
        return user

# Create a user
john = User.find("John")
```

Now let's see what it would like using an ORM library, like [Prisma](https://www.prisma.io/) (it's originally written for NodeJS, there's also a [python client](prisma-client-py.readthedocs.io))

#### ORM Libraries

<h5><center>Prisma</center></h5>

```python
from prisma import Prisma
db = Prisma()
# db.user is automatically generated from a prisma.schema file
async def find_user(name: str):
    user = db.user.find_first(
        where={
            "name": "John",
            "NOT": [ {"email": None} ]}
        )
```

However the implementation can vary across libraries. Here's what Django's ORM would look like.
<h5><center>Django</center></h5>

```python
from models import User

john = User.objects.filter(name="John", email__is_null=False)
```

Now, the ORM implementations may seem straightforward enough, but whereas SQL has a standard approach to more complex queries, ORMs generally don't, and just become a wild guessing game where you rely on your language server's autocomplete. The documentation can't cover this, because the arguments and types are totally dependent on the columns of your table--such as its data type and constraints.


### Why ORMs Are Evil

Okay, so maybe not evil, but potentially full of trickery.

The principal issue is due to how we don't know how the library works under the hood, and this can lead to *a lot* of unexpected issues. 

```python \
# This is a quick example of what a model
# looks like in Django for reference
from django.db import models

class Address(models.Model):
    location = models.CharField(max_length=50, blank=True, null=True)  # Field .

class User(models.Model):
    name = models.CharField(max_length=50, blank=True, null=True)  # Field . 
    age = models.IntField(null=False)  # Field . 
    address = models.ForeignKey(Address, to_field="name", db_column="name")
```

**1\. Implicit Behavior Instead of Explicit Defintion** 

When you create a model without a declared primary_key column, Django will automatically create an INT column `id`and make that the primary key. Although having an `id INT PRIMARY KEY` column is pretty standard in RDMBS, I firmly believe that any modifications to the database absolutely need to be explicit. If there is no primary key defined, Django should error fast instead of doing things on the user's behalf unknownst to them.


**2\. Compatibility**

Django's ORM *requires* a single column primary key. I don't see why you would ever not have a primary key, but this also means you can't have a composite primary key. This makes usage very awkward has you have to pretend like one of the fields is a single primary key, and treating the other field(s) in a simple WHERE clause.


**3\. Overfetching/Sub-optimal queries**

If we have something like 
Additionally, when you declare a foreign key like

```python
users = User.objects.filter(age__gt=30)
# No database hit yet

for user in users:
    print(user.address.location)
    # Database hit happens here
```

This will actually create not one query, but 1 query + an additional query for each user. This is because `address` is a foreign field and does not exist in the initial query. You need to use the `select_related` method to include the foreign fields in your query. This issue won't make itself visible until you start getting performance issues, and unless you were already aware of this, it might be a while before you figure out where the performance hit is coming from.
****
In an ironic way, my terrible experience here actually led to me preferring ORMs, because it pushed me to writing my own SQL for a few applications, which had me experiencing how inconvenient that was--leading me back to trying out ORMs. And here's ...

### Why I Still Prefer Using Prisma
 

1. **Convenience**<br/> The prevailing reason is simply because it reduces the amount of boilerplate code I need to write. The benefit of this during early development stages of a project cannot be overstated. There can be many tables to create models after, and it's nice to not have to create or update the interfaces after each minor change.
2. **Manages Database**<br/> Although I'm perfectly capable of managing the database, Prisma allows for a declarative approach to the database schema via its `prisma.schema`. And as a NixOS user you already know anything with "declarative" is going to get my attention. 
3. **Database Layer Abstraction**<br/>What's also exceptionally nice is having your code be pseudo-database agnostic. While SQL is mostly standardized, there can be minor differences between the syntax of the different databases. Furthermore, there are also differences in the syntax of the database structure itself. For example, Postgres uses `AUTOINCREMENT` for its primary keys while MySQL uses "AUTO_INCREMENT". Not having to worry about these syntax differences makes life a lot easier, and also makes it very forgiving when you want to make your application compatible with multiple databases.
4. **Multi-Language Support**<br/> Currently there are Prisma clients for Dart, Go, Python, Rust, and of course, JavaScript/TypeScript. This solves the issue where knowledge of an ORM doesn't transfer (e.g.knowing how to use TailwindCSS will only help you with specifically using TailwindCSS in a NodeJS project.)
5. **Escape Hatch**<br/>This is a pretty prevalent feature across ORM libs, but in a way it's like having the best of both worlds--as you are still free to write raw SQL if your use case isn't covered by the library.


### So What Should You Use

Well, I think it boils down to where you're at and what you're doing.

Here are a few questions to help decide:

1. **Is the ORM tightly coupled with a framework you're using?**<br/>
If you're new to the framework, it's probably better to follow idiomatic practices. So despite my disinclination with Django's ORM, if you're learning Django for the first time, using its ORM will help you understand the framework as a whole better.<br/>
However, if you are experienced, then you should have the judgement to decide if using the ORM truly fits the structure of your project, or if you should opt for an alternate.
2. **How big is your application?**<br/>
I find that the benefits greatly outweight the inconveniences if your application contains more than a handful of data models. This reduces the boilerplate code greatly. In addition, having a declarative source of truth for the database schema is very nice, and allows for easy migration and schema changes.
3. **Do you know exactly what you want to do?**<br/>
If you know *exactly* what your data models should look like, and how you expect them to be updated, then it probably doesn't really matter unless  what you want to do isn't supported by the ORM. I would still recommend using Prisma just for convenience of managing the schema, and you can always use the raw SQL escape hatch if you don't trust the query implementation.
4. **Do you want to support multiple databases?**<br/>
Definitely an ORM, unless you *really* want to implement the appropriate interfaces for each database. 
5. **Is your application data-driven or task-driven?**<br/>
If your application is tied closely to data, it probably makes more sense to write raw SQL as that brings you closer to your data. If you're task-driven, that layer of abstraction will come in nicely and help you with development speed and flexibility in the underlying database technology that you choose to implement.

### Conclusion

While both have their merits, I think for most developers using an ORM library is a no-brainer--it just saves *sooo* much time. And if you use Rust, Dart, TypeScript/JavaScript, Go, or Python then I highly recommend checking out Prisma.

****

\* This is because if you're binding your native SQL to functions and objects, you're basically just making your own ORM library


On Prisma and Whether You Should Use ORMs

Typical Python usage doesn't call for asynchronous code, but it's nonetheless an essential feature for libraries like discord.py or FastAPI. Even though the implementation of asynchronous code in Python has its merits, every time I am writing it I can't help but wish I was writing JavaScript promises instead. To illustrate the reason behind my preference, let's take a look at some examples. Let's say I'm writing some code that will:

1. Fetch Resource1 from ExternalServer1
2. Fetch Resource2 from ExternalServer2
3. Perform some expensive computation on Resource1 to get Processed1
4. Process Processed1 and Resource2 together to yield FinalResult and return

So let's look at how this might look in Javascript:
```javascript
async function main() {
    const Resource1Promise = fetch("ExternalServer1").then((res)=>{
        return processResource(res)
    })
    const Resource2Promise = fetch("ExternalServer2")

    // This is Processed1 because of the .then() that we invoked that immediately
    // started processing Resource1 as soon as it was retrieved
    Processed1 = await Resource1Promise

    Resource2 = await Resource2Promise
    return combineResources(Processed1, Resource2)
}
main()
```

What I find beautiful about this is that async functions (like `fetch`) can be invoked with or without `await` (the example invokes them without `await`). Either way, the function will start its execution and return a `Promise` object. Then, `await` is used to retrieve the final result of the function call stack-- so if you invoke the async function with `await` then you're basically saying you need the result of the function *now* and this parent function should wait for the async execution to finish before continuing. But the important thing is that the implementation is pretty straightforward:

**Call the async function and it'll start executing. Then `await` it *if* you want the result of the function**

Now let's see at a similar typical python implementation:

```python
import aiothttp
import asyncio
async def main():
    async with aiohttp.ClientSession() as session:
        Resource1Response = await session.get("ExternalServer1")
        Resource2Response = await session.get("ExternalServer2")
        # Assume the content we want is a json, but it could be anything
        Resource1 = await Resource1Response.json()
        Resource2 = await Resource2Response.json()
        Processed1 = processResources(Resource1)
    return combineResources(Processed1, Resource2)
    
asyncio.run(main())
```

Initially, one might think these are equivalent--at least I did when I first started out--but that could be farther from the truth. It's deceptive because of the syntax similarity and brevity. To understand the significance of this implementation, the important thing to focus on is all the `await` keywords that show up. In the python implementation. This means that each line has to finish resolving before the next one is finished. Meaning that even though retrieving Resource1 and Resource2 have nothing to do with each other, we still need to wait Resource1 to resolve first. 

> So why not just leave out the `await`? 

Because that's simply not how Python works. In python, calling `async` functions return a `Coroutine` object. Do not make the mistake that this is equivalent to JavaScript's `Promise`. A coroutine does not start executing its underlying function until it's `await`-ed. 

To get the similar behavior to our JavaScript implementation, we need to utilize the `Task` object.


```python
import aiothttp
import asyncio
async def main():
    async with aiohttp.ClientSession() as session:

        # When invoked with create_task, the data retrieve starts happening immediately
        Resource1Task = asyncio.create_task(session.get("ExternalServer1"))
        Resource2Task = asyncio.create_task(session.get("ExternalServer2"))

        # Assume the content we want is a json, but it could be anything
        Resource1Response = await Resource1Task
        Resource1 = await Resource1Response.json()
        Processed1 = processResources(Resource1)
        Resource2Response = await Resource2Task
        Resource2 = await Resource2Response.json()
    return combineResources(Processed1, Resource2)
    
asyncio.run(main())
```

 In Python's asyncio library, a `Task` is what I know to be the closest thing to a JavaScript `Promise`. Here, the we start requesting the resources from the external servers immediately. This means by the time we're done with processing `Resource1` that `Resource2` might already be ready to be parsed.

> *What's the big deal*?

Well just look at it! It takes up nearly double the lines of code, and adds additional layers of complexity that a beginner might not know about ( how would you know you need to create a Task when your code works just fine when you `await` your async function? ) 

> So what can be done?

Well, I believe a formal solution would involve modifying the python interpreter, so that's a bit out of question. Really, the solution is to just learn the python syntax. However,  I've decided to play around and see if I could create something that would replicate the implementation of the Javascript `Promise`. It's not really meant to be used, but just an experiment to see what is achievable.

First, since a `Task` is pretty close to a `Promise`, all we really need to do is create a wrapper.

```python
import asyncio
import functools
from typing import Coroutine

def promise(func_or_coroutine):
    if asyncio.iscoroutinefunction(func_or_coroutine):
        # If the input is a coroutine function, return a decorator
        @functools.wraps(func_or_coroutine)
        def wrapper(*args, **kwargs):
            return asyncio.create_task(func_or_coroutine(*args, **kwargs))

        return wrapper
    elif isinstance(func_or_coroutine, Coroutine):
        # If the input is a coroutine, wrap it in a task directly
        return asyncio.create_task(func_or_coroutine)
    else:
        raise TypeError("promise requires an async function or a coroutine")

# Usage as decorator:
@promise
async def some_function():
    await asyncio.sleep(1)
    return 1

# Imperative usage:
async def other_function():
    await asyncio.sleep(1)
    return 2
# other_function starts executing immediately
res = promise(other_function())
# get the result
print(await res)

```

So that's pretty sufficient. But then a key feature of a `Promise` is fluent-style chaining, meaning you can declare functions that will be ran successively on the results of the previous functions. You can also add a `.catch` function to handle any errors along the way.

```javascript
actionOne().then(actionTwo).then(actionThree).catch((error)=>errorHandler(error))
```

Since `.then` is a method, we need to create a class to put these methods on. Here is the full code of my implementation:

```python
class Promise:
    def __init__(self, coro):
        self.current_task = asyncio.create_task(coro)
        self.error_handler: Optional[Callable] = None
        self.pending_error = None


    def then(self, func, error_func=None):
        async def continuation(original_task):
            try:
                result = await original_task
                if asyncio.iscoroutinefunction(func):
                    res = await func(result)
                else:
                    res = func(result)
                return res
            except Exception as e:
                self.pending_error = e
                if error_func:
                    return error_func(e)
                if self.error_handler:
                    return self.error_handler(e)
                raise e

        # This critical line doesn't actually work
        if self.pending_error:  
            return self

        original_task = self.current_task
        self.current_task = asyncio.create_task(continuation(original_task))
        return self

    def catch(self, error_func):
        self.error_handler = error_func
        if self.pending_error:
            return self.error_handler(self.pending_error)
        return self

    def __await__(self):
        # Delegate to the internal awaitable coroutine
        return self.current_task.__await__()

# Rewrite the promise function as a wrapper for the class
def promise(func_or_coroutine):
    if asyncio.iscoroutinefunction(func_or_coroutine):
        # If the input is a coroutine function, return a decorator
        @functools.wraps(func_or_coroutine)
        def wrapper(*args, **kwargs):
            return Promise(func_or_coroutine(*args, **kwargs))

        return wrapper
    elif isinstance(func_or_coroutine, Coroutine):
        # If the input is a coroutine, wrap it in a task directly
        return Promise(func_or_coroutine)
    else:
        raise TypeError("promise requires an async function or a coroutine")
```

With this implementation, our old python code now looks like this:

```python
import aiothttp
import asyncio

# Kind of silly but necessary for python implementation, 
# since `.json` is a method on the response, and we don't have access to that
async def get_json(r):
    return await r.json()


async def main():
    async with aiohttp.ClientSession() as session:
        Resource1Promise = promise(session.get("ExternalServer1")).then(get_json).then(processResource)
        Resource2Promise = promise(session.get("ExternalServer2")).then(get_json)

        Processed1 = await Resource1Promise
        Resource2 = await Resource2Promise

    return combineResources(Processed1, Resource2)
    
asyncio.run(main())
```

It's pretty close, but there's a few issues. My implementation of the `.catch` method doesn't work perfectly--when an error is reached, the function chain should stop and the error should be raised. However, I haven't found a solution to checking for an error prior to scheduling a Task. I think this is because the Tasks are all scheduled before they are ran, so the error only happens after scheduling. I'd be delighted to see if anyone could make improvements on this.

Finally, if anyone is reading this that is new to Python, I want to make it clear that *actually* implementing this is probably a no-go since your code will be extremely non-idiomatic and Pythonistas are probably going to raise their pitchforks if they see you write code this way. This was simply an educational experiment to see if I could make the async Python code function similar to JavasScript's, and how much the code would benefit from a theoretical implementation.



python

javascript

exploratory

Promising Python Coroutines

In the <a href="https://dvytran.dev/blog/2023/09/19/x3framework:%20Powering%20NixOS%20with%20docker%20compose-5" target="_blank">previous post</a> I talked about <a href="https://github.com/agentx3/nix-expressions" target="_blank">my nix framework</a> that uses <a href="https://docs.docker.com/compose/" target="_blank">docker compose</a> to run services. In this post, I'll show how to write your own NixOS modules to extend my `x3framework`. It should also give you the fundamental principals in creating NixOS modules in general.

**NOTE**: This particular implementation is for a development environment, and I **advise against directly extending `x3framework` in production** as upstream changes might break your configuration. Instead, I recommend forking my repo and using that as your input url.

**Again, do not directly extend `x3framework` in production** (unless it's from a personal fork)

For this post, I will assume you have already included my flake and imported its `default` or `dockerModules` into your NixOS configuration.

```nix
# flake.nix
{
  inputs = {
    # Other inputs ...
    nixpkgs.url = "github:nixos/nixpkgs/nixos-23.05";
    x3framework.url = "github:agentx3/nix-expressions";
  };
  outputs = {
    # Other inputs ...
    nixpkgs
    , x3framework
    , ...
  }: {
      nixosConfigurations = {
          yourSystem.nixpkgs.lib.nixosSystem = rec {
              system = "your system architecture";
              # ... 
              modules = [
                 # Either 'default' or 'dockerModules' work
                 x3framework.nixosModules.${system}.default
              ];
          };
      };
  }
```

At the core of my system, all that really needs to be done is:
  1. Setup a docker network
  2. Define the compose file
  3. Define your desired options and config output
  4. Configure the system to use the module

#### 1. Setup a docker network:
The docker containers are configured to operate on a docker network. As part of my `dockerModules`, I include an extension for the nixpkgs docker module that allows declarative networks. In my framework, I've included a wrappear for simplicity. To use: 

```nix
# configuration.nix
{config, ...}: 
{
    x3framework.docker.networks.default = {
        enable = true;
        name = "x3framework-net"; # Choose whatever you want here
        # 172.16.0.0 - 172.31.255.255 is a private IP only range so
        # it's smart probably to choose something between there.
        # Avoid 172.17.x.x because that's what docker uses for its
        # default bridge network.
        subnetBase = "172.30.0.0";
        # Allows for 172.30.0.X , but you can modify it per your requirements.
        subnetMaskLength = "24";
    };
}
```

#### 2. Define the compose file:
For this tutorial I'll cover a simpler service that I have a <a href="https://roundcube.net/" target="_blank" rel="noopener noreferrer">Roundcube</a> container.
Let's look at a normal docker-compose.yml for roundcube:

```yaml
# docker-compose.yml
version: '3.9'

services:
  roundcubemail:
    image: roundcube/roundcubemail:latest
    container_name: roundcubemail
    restart: unless-stopped
    volumes:
      - ./www:/var/www/html
      - ./db/sqlite:/var/roundcube/db
    ports:
      - 9002:80
    environment:
      - ROUNDCUBEMAIL_DB_TYPE=sqlite
      - ROUNDCUBEMAIL_SKIN=elastic
      - ROUNDCUBEMAIL_DEFAULT_HOST=tls://mail.example.org
      - ROUNDCUBEMAIL_SMTP_SERVER=tls://mail.example.org
```

YAML relies on line-dentation for parsing, which doesn't work well with variable interpoliation in Nix multi-line strings. Fortunately, YAML is a superset of JSON, so we can use nix's `builtins.toJSON` to generate our config file.

First, make a separate file for your module. Then let's define some variables that will be used often through the configuration and then write the compose file.  Because `roundcube` already exists as part of my framework, I will call this `roundcubeDemo`.

```nix
# roundcubeModule.nix
{config, lib, ...}:
let
  inherit (lib) mkEnableOption mkOption mkIf types mdDoc;
  service = "roundcubeDemo";
  # Define this to be the container IP suffix. 
  # If your subnet mask was /16, then you can also do "X.X" e.g. "12.1"
  IP4 = "234"; 
  cfg = config.x3framework.docker.services.roundcubeDemo;
  # This function merges the subnet with the provided IP suffix
  # E.g. `mkIp 172.30.0.0 12.1` becomes `172.30.12.1`
  mkIp = config.lib.x3framework.mkIPFromSubnetAndSuffix;
  SERVICE_IP = mkIp cfg.network.subnetBase IP4;

  # This will be used specifically just for Roundcube to help us
  # bind additional config files into the container. 
  mkExtraConfig = extraConfig: pkgs.writeTextFile {
    name = "${service}-extra-config.inc.php";
    text = extraConfig;
  };

  composeFile = pkgs.writeTextFile {
    name = "${service}-compose.yml";
    text = builtins.toJSON {
      version = "3.9";
      services = {
        roundcube = {
          restart = "unless-stopped";
          container_name = "${service}";
          image = cfg.image;
          environment = {
            # See https://github.com/roundcube/roundcubemail-docker/blob/master/README.md
            # for explanations of the environment variables
            ROUNDCUBEMAIL_DEFAULT_HOST = cfg.host;
            ROUNDCUBEMAIL_DEFAULT_PORT = toString cfg.port;
            ROUNDCUBEMAIL_SMTP_SERVER = cfg.smtpServer;
            ROUNDCUBEMAIL_SMTP_PORT = toString cfg.smtpPort;
          } // cfg.environment;
          networks."${cfg.network.name}".ipv4_address = SERVICE_IP;
          volumes = [
            "${service}-db:/var/roundcube/db"
            "${service}-config:/var/roundcube/config"
            "${service}-temp:/tmp/roundcube-temp"
            "${service}-html:/var/www/html"
          # These next few aren't strictly necessary, but are useful if
          # if you want to utilize features like sieves and plugins
          ] ++ lib.optionals (cfg.extraConfig != "") [
            "${mkExtraConfig cfg.extraConfig}:/var/roundcube/config/extra-config.inc.php"
          ] ++ lib.optionals (cfg.extraOauthConfigFile != "") [
            "${cfg.extraOauthConfigFile}:/var/roundcube/config/extra-oauth.inc.php"
          ];
        };
      };
      volumes = {
        "${service}-db" = { };
        "${service}-config" = { };
        "${service}-temp" = { };
        "${service}-html" = { };
      };
      networks."${cfg.network.name}".external = true;
    };
  };
in
{
 # Next parts go here
}
```
<br/>
#### 3. Define desired and necessary options
We need to define all the variables that were used in the compose file (anything that's an attribute of `cfg`). Additionally, to be compatible with the `x3framework` the **following options must also be created**:
 1. `containerIp`
 2. `network`

Now, let's create the options in the actual attribute set portion of the file.

```nix
# roundcubeModule.nix
{config, ...}:
let
  # previous section
in
{
  # This part is what creates the new options to be used in the rest of the system configuration
  options.x3framework.docker.services.roundcubeDemo = {
    enable = mkEnableOption "" // {
      description = ''
        Whether to enable the container task. Enable this if the configuration
        is deployed to a NixOS container.
      '';
    };
    containerIp = mkOption {
      type = types.str;
      default = SERVICE_IP;
      description = mdDoc "IP address of the container. Needs to be in subnet of the assigned network";
    };
    smtpServer = mkOption {
      type = types.str;
      description = mdDoc "Location of the smtp server";
    };
    host = mkOption {
      type = types.str;
      description = mdDoc "Location of the host mail server";
    };
    port = mkOption {
      type = types.int;
      default = 143;
      description = mdDoc "Port to use for IMAP. Usually 993 for forced TLS, 143 for STARTTLS";
    };
    image = mkOption {
      type = types.str;
      description = mdDoc "Image to use for the container";
      default = "roundcube/roundcubemail";
    };
    smtpPort = mkOption {
      type = types.int;
      default = 587;
      description = mdDoc "Port to use for SMTP. Usually 465 for forced TLS, 587 for STARTTLS";
    };
    environment = mkOption {
      type = types.attrs;
      default = { };
      description = mdDoc "Additional environment variables to pass to the container";
    };
    envFile = mkOption {
      type = types.nullOr types.str;
      default = null;
      description = mdDoc "Path to an environment file for the docker container.";
    };
    extraConfig = mkOption {
      type = types.str;
      default = "";
      description = mdDoc "Additional configuration to pass to the roundcube php config file";
    };
    extraOauthConfigFile = mkOption {
      type = types.oneOf [ types.path types.str ];
      default = "";
      description = mdDoc "Additional configuration file to pass to the roundcube oauth config file";
    };
    network = mkOption {
      type = config.lib.x3framework.types.x3NetworkType;
      default = config.x3framework.docker.networks.default;
      description = mdDoc "The docker network to use for the container";
    };
  };

  # Now this next part is used to define how the system should be configured based on the options.
  config = mkIf cfg.enable {
    # You can add additional assertions to enforce here. These will cause errors be thrown
    # during the system build process if the assertion doesn't resolve to a boolean true.
    assertions = [
      {
        assertion = cfg.network.enable;
        message = "The framework's docker network must be enabled to use this";
      }
    ];

    # This generates a helper script for root to use as shell alias
    x3framework.helper.${service}.composeFile = composeFile;

    # This generates a systemd unit that automatically starts the container as well as
    # introduces a dependency on the network, so that the container will automatically be 
    # stopped and removed if the docker-network systemd services needs to stop or change.
    # The container will start back up afterwards after the docker-network service unit is re-established.
    systemd.services."${service}-x3framework" = config.lib.x3framework.mkSystemdUnit {
        inherit config service composeFile; network = cfg.network;
    };
  };
}
```

Now to use roundcubeDemo, import the module in your configuration.nix and configure the module options.


#### 4. Configure the system to use the module

```nix
# configuration.nix
{config, ...}:
{
  imports = [
      # If using flakes, this needs to be a relative import
      ./path/to/roundcubeModule.nix
  ]
  # Configure the module
  x3framework.docker.services.roundcubeDemo = {
    enable = true;
    # Use ssl:// prefix if using implicit SSL
    # If using STARTTLS, use tls://
    host = "ssl://<YOUR IMAP MAIL SERVER HERE>";
    port = 993; # Typically 993 for IMAPS, 143 for IMAP
    smtpServer = "ssl://<YOUR SMTP SERVER HERE>";
    smtpPort = 465; # Typically 465 for SMTPS, 587 for STARTTLS
    environment = {
      ROUNDCUBEMAIL_PLUGINS = "archive,zipdownload";
    };
    env_file = "/path/to/envFile.env";
  };
}
```
Do a system rebuild and if all is well, then a systemd unit for the network will start, followed by the systemd unit for the docker container. Roundcube will now be listening on port 80 on its containerIp. So if using nginx, you should proxy_pass to `${config.x3framework.docker.services.roundcubeDemo.containerIp}:80`.

As an final exercise to the reader: see if you can bind the container to a port of your choice on your system's localhost.

Feel free to open pull requests if you have useful docker compose services you'd like to add. Thanks for reading!


nixos

How to: Extend x3framework (and write NixOS modules)

In this article, I'll go over briefly what NixOS is, discuss some problems I encountered with it while trying to setup some services, and how I solved it.

<a target="_blank" rel="noopener noreferrer" href="https://nixos.org/">NixOS</a> is an incredibly powerful tool. It's a declarative OS managed by configuration files, in a way it's like Terraform but on an individual OS level. I've been using it as a daily driver for almost a year now. I'll have to write a dedicated post for NixOS one day. But for the uninitiated, the TL;DR on what makes NixOS so unique, is that essentially the entire OS is managed by json-like configuration files, meaning everything, from users, system packages, nginx settings to filesystems, grub, and even the kernel can be managed via these files. It's great because once you figure out how to setup your desired system, you won't ever have to go through that process again, of trying to remember what you did--it's all there in your config files. Let's move on now  to the <a target="_blank" rel="noopener noreferrer" href="https://github.com/agentx3/nix-expressions">nix "framework"</a> I made. 

Some definitions before we get started:
* **Derivation**: You can essentially consider this an "atomic unit" in Nix. Basically, to make a derivation, you:
    1. Declare all the necessary inputs
    2. Declare all necessary build steps/commands
    3. Declare desired output from the build results
* **Module**: A broad term that technically encompasses any general sort of self-contained nix configuration. However, in conversation, *generally* means a purpose-built nix configuration that specifically adds new options to use in the configuration files.


Nix has an enormous cache of applications and packages ready to be used, and Nix makes it easy(ish) enough to write your own derivations for any packages you want. However, not all packages are meant to be, well, packaged with Nix. Unfortunately, some applications write to files in their source directory as port of normal operations, instead of using an external data/config location to store stateful data. This makes things very challenging, as the `/nix/store` is read-only. An example of this is `services.matomo`, which if I recall correctly, tried to write to one of its config files during normal operation. This made for me, the service unusable.

So, what to do? Download the source and run it directly with PHP? Sounds messy and imperative. Write a Dockerfile or compose file, and run it manually? Clean, but still imperative. What if I told you this was all I needed to get an instance of matomo running?

```nix
# config.nix
x3framework.docker.services.matomo = {
  enable = true;
  envFile = config.age.secrets.matomo-env.path;
  domain = domain;
};
```

Let's explore the module to see what is happening. There's two core files: <a target="_blank" rel="noopener noreferrer" href="https://github.com/agentx3/nix-expressions/blob/main/nixos/modules/dockerModules/services/matomo.nix">matomo.nix</a> and <a target="_blank" rel="noopener noreferrer" href="https://github.com/agentx3/nix-expressions/blob/main/nixos/modules/dockerModules/services/mkSystemdUnit.nix">mkSystemdUnit.nix</a>. There's a third supplementary file <a target="_blank" rel="noopener noreferrer" href="https://github.com/agentx3/nix-expressions/blob/main/nixos/modules/dockerModules/helper.nix">helper.nix</a> that just adds shell aliases.

Inside `matomo.nix` I have a docker-compose file defined similar to the <a target="_blank" rel="noopener noreferrer" href="https://github.com/matomo-org/docker/blob/master/.examples/nginx/docker-compose.yml">example compose</a> file from matomo. The containers are bound to the newly available declarative docker network, which then get referenced inside the systemd unit generated by `mkSystemdUnit.nix`. All the systemd unit needs to do is spin up and spin down the containers. The extra addition that I made is since the docker network is also tied to systemd, I'm able to reference the container and the network services to each other and establish a dependency hierarchy, enabling a clean tear-down and spin-up of containers and networks as necessary.

```nix
# mkSystemdUnit.nix
{   
   # ...
  requires = [ "docker.service" "docker-network-${network.name}.service" ];
  after = [ "network.target" "docker.service" "docker-network-${network.name}.service" ];
  script = /* bash */ ''
    compose="${composeFile}"
    command="${docker} compose -f $compose -p ${service}"
    $command up -d
  '';
  preStop = /* bash */ ''
    compose="${composeFile}"
    command="${docker} compose -f $compose -p ${service}"
    $command down
  '';
}
```

And for quality of life, my matomo module also creates an nginx virtualhost for the specified domain. 

Using services with Docker might not be the best practice of Nix philosophy since it's not immutable nor exactly reproducible. However, I think it's a fair trade off, since I would rather have a functioning serve that might break later, than a completely non-functional one. 


x3framework: Powering NixOS with docker compose

 *Hey you. You're finally reading this. You were trying to find some good tech content, right? Stumbled right onto that search result, same as me, and that google web crawler over there.*

Boy it's about time I wrote an actual post. Funny enough, I've been more obsessed with constructing my own content platform rather than providing content for said  platform. *The feature creep is real my dudes.*

Anyways, this site will serve as a general platform for me to share thoughts/insights/findings that hopefully, at least one soul will find useful before the heat death of the universe.

Per the title card, I'm a <span style="color: var(--base08)">fullstack developer</span> and <span style="color: var(--base08)">systems administrator</span>. The very beginning of my journey began about 1.5 years ago. It started innocently enough, with Python discord bots. I didn't even know how to disown a shell process. About 6 months of shenanigans later, our (actual) SysAdmin of my non-profit group bailed out. It just happens I was the only one in the position to take over. Probably not very ideal when the person who's supposed to take over barely knows their bash-- made me "grow up real fast", as they say. Crash coursed myself on some containerization with <span style="color: var(--base0C)">Docker</span>, dived into some nginx, bashed myself with <span style="color: var(--base0F)">bash</span>, and developed Stockholm syndrome with <span style="color: var(--base0A)">Nix</span>/<span style="color: var(--base0A)">NixOS</span>. Of course, there's like about half-a-million other things that I learned (*and an infinite amount of things to be learned, I learned*), but I can save that for another day.

If you've read this far, thanks! Have this seal of thanks.

<img style="width: auto; height: 10em" aria-label="A seal of thanks (not the animal)" src="https://res.cloudinary.com/dbtfu4e73/image/upload/v1694424658/sealofthanks_6b989cbacc.jpg"/>





Blogs

Tags