Yesterday was the first day of my much-needed and much-deserved (if I may say so myself) 2-week vacation. We’re not\ngoing anywhere. Instead, this will be a staycation in which I allow myself to indulge in all of my hobbies outside\nof work, ranging from going on trips with my wife and dog, reading books, playing video games, working on side-projects and generally taking life as it comes. And these\nstaycations are usually a great opportunity to pay my social debts as well by seeing friends and family I haven’t\nseen for way too long.
The first day of a typical staycation for me starts with sleeping in - preferably an outrageous amount - after which\nI will slowly get into some of the aforementioned activities. Yesterday started in such a way, but a couple of hours\nafter I got up, I was sitting on the couch bawling my eyes out, surrounded by my wife and dog who were trying to\nconsole me.
This is what happened.
One of the first things I used to do after waking up, is to open the Apollo app to see what’s happening on Reddit.\nFor better of for worse, through /r/news and /r/worldnews, I tried to keep myself up-to-date on what is\nhappening in the world. After the 2023 Reddit API I\nrefuse to use Reddit anymore, so I looked for other outlets to consume news and being curious about local news as\nwell, I ended up frequenting nu.nl, a Dutch news outlet.
Everybody who’s been on the modern internet for more than 2 seconds knows this: comment sections on websites are the\nworst. And social media are basically just giant comment sections without useful content (which is why I don’t have\nFacebook/Instagram/Twitter accounts anymore). For some reason, I thought it would be a good idea to create an\naccount on nu.nl to have friendly discussions with my fellow citizens on the news of the day.
I was dumb
Each day I got confronted more and more by how excruciatingly unempathetic, selfish, unreasonable and unkind people\nhave become. You used to be able to explain this by virtue of The Greater Internet Fuckwad Theory, which states:
Normal Person + Anonymity + Audience = Total Fuckwad
\n
- John Gabriel (2004)
But a bunch of things have changed to make this even worse:
Yesterday, I read 2 news articles that in particular triggered emotional distress with me - and especially the\ncomments underneath the reporting on nu.nl on those topics:
I care deeply about LGBTQI+ issues, not in the least because my father is gay, I have a transgender niece and I have\nmany people in my social circles that identify as LGBTQI+. And what makes me so sad is that - despite years of\nfighting for LGBTQI+ rights - I’m noticing a clear movement backwards on these topics. People seem to have this\nidea that the fight for equality, equity, diversity and inclusion is over now and they’re using this to argue we should\nsweep any expression of \"otherness\" under the rug.
I’m not against gay people, but why do you have to kiss in public?
\n
- Paraphrased from a comment under an article about the Amsterdam Pride event
If you cannot accept self-expression of other people different than you, you are effectively against those\npeople being different
The sentiment of people towards issues I consider to be wrong in the world - and that I naively believed were\ngenerally considered to be wrong - can be summed up by me, me me. This sentiment pervades both on personal and\nsocietal level:
The first point is not only prevalent on places like Twitter, Facebook, Reddit and the comment sections of the\naverage news outlet. Even LinkedIn - a professional networking website - is rife with people that love to\n\"challenge\" ideas to improve the global optimum just so they can optimize for their own local happiness. I lament the\npassing of the days when LinkedIn was about professionals discussing professional things professionally with each\nother.
But I’m happy to contribute to the downfall of professionalism on LinkedIn by posting this opinion piece once\nI’m done writing itIt’s the second point that makes my blood truly boil, especially because the acceptance of oppression is done under\nthe guise of reasonableness. In the comment section on the article of The 1975 being banned from Malaysia, a\ncommon sentiment could be summed up as: you go to their country, you follow their laws. This completely ignores\nthat there is such a thing as Universal Human Rights.
When you accept oppression - even under the guise of \"law\" - you let society slide downwards on a slippery slope and it\nwill not end well. I truly don’t think I’m being hyperbolic when I say this kind of acceptance is pretty much\nexactly what happened in Europe by the end of the 1930ies.
We should always keep fighting for the rights of our fellow citizens and borders do not matter in that regard
Society turning \"harder\" and \"colder\" by the day, causes depression in me. Reading how unreasonable and intolerant\nmy fellow citizens are, makes me immensely sad. It also makes me feel lonely, because it sometimes feels I’m the\nonly one who still cares about their fellow citizens that are not friends or family.
To be honest, I had some pretty dark thoughts yesterday.
If only the whole of humanity would just die, it would be better for everyone, especially the rest of the species\nwe share our planet with
\n
- Me, after reading one inane comment too many
I feel a strong juxtaposition of the happiness induced by my wife, my pets, my friends and my job\nand my increasing unhappiness with the world I live in. I’m sure other people have this as well, and part of me\nhopes that by writing this blog post, I’ll encounter people saying \"Hey, I feel this too!\" which would make me feel\nless lonely. But it is honestly increasinly hard to deal with.
I resolve to talk more about these feelings I have with others. Maybe we can form a counter-movement. Maybe we can\nrebel against the tide of \"not caring\".
I considered not reading these comment sections anymore. I’m not sure if I should or shouldn’t, because I also think\nthat if I can turn only one mind towards positivity and caring about others, I will have \"won\" a little. And not\nspeaking up is exactly what I’m trying to rebel against here.
It is really simple. We (Society) should strive to not be a dick.\nYou should care about your fellow citizens. We should agree to uphold\nUniversal Human Rights.
Here is one of its most simple tennets and it is to me the core of what each person is entitled to:
Everyone is entitled to all the rights and freedoms set forth in this Declaration, without distinction of any kind,\nsuch as race, colour, sex, language, religion, political or other opinion, national or social origin, property,\nbirth or other status. Furthermore, no distinction shall be made on the basis of the political, jurisdictional or\ninternational status of the country or territory to which a person belongs, whether it be independent, trust,\nnon-self-governing or under any other limitation of sovereignty.
\n
- Article 2 of the Universal Declaration of Human Rights
One important thing to note here, is that this only works if freedom of one group is not abused to limit the\nfreedoms of another group. This means for example that you cannot limit the rights of LGBTQI+ people because you\nwant to express your own religion.
Furthermore, I think that the time of being accepting of all opinions should come to an end. Any opinion that\ninfringes on universal human rights, should not be tolerated. If we tolerate everything, we tolerate nothing.
It is time that news outlets and social media - such as nu.nl - take responsibility and limit the ability of people\nto express thoughts that lead to oppression. It is for example not ok to express your \"opinion\" that same-sex\ncouples kissing is \"icky\" and should not be done in public. And news outlets and social media should not let you\nexpress such opinions in public.
I personally would even go so far to state that scientifically proven facts should not be allowed to be contradicted\nwithout proper evidence to the contrary. Everyone should be able to challenge the status quo, but there is the scientific method you can follow to do this in a rigorous and\nfair way.
When I started the new version of my blog, I promised that I would write more. That hasn’t really materialized and I\ndon’t know if it will. I felt so strongly about the topic of this blog post, that I wanted to write down my thoughts.\nLet’s see if that will happen more often.
Peace!
", "url": "https://www.daan.fyi/writings/mental-health", "title": "How the State of the World Affects My Mental Health", "summary": "I sometimes have a hard time coping with the ugliness of people's opinions on the internet and in the world at large.", "date_modified": "2023-07-23T00:00:00.000Z" }, { "id": "https://www.daan.fyi/writings/iam", "content_html": "All access management in AWS is done through AWS Identity and Access Management\n(or IAM in short). IAM provides a way to configure who can do what in an AWS account.\nThe \"who\" in this case can be humans, machines/software or other AWS services.
IAM is quite complex and there are several concepts you need to understand in order to know how to\ngive people, software and machines the right access to the right resources in your AWS Accounts.\nThis can become complicated quite fast if you’re managing multiple AWS Accounts with many different\nresources and services, as explained in the previous post in this series.
In this blog post I’m going to attempt to explain the terminology in a clear and concise way. Over\nthe past months, we worked hard at Source.ag to create a good setup for our\nAWS-hosted infrastructure. I feel that this gave me a solid understanding of how IAM works\n(among other things). This blog post will help me personally as a kind of cheat sheet for later\nrecall and hopefully be of use to others to whom these concepts are still new.
A Resource in AWS is basically any piece of infrastructure or any AWS service you can pay for\nand use. This also represents the things you want to control access to. Some examples of this: EC2 servers and S3 buckets.
For all these resources, you can control who or what can access them and to what extent.
An Account in AWS is a container for resources. So it’s not a representation of a human\nusing AWS services. In that sense, it’s quite different from the typical definition of an account\nin online services like Twitter or Facebook, where accounts are digital extensions of human beings.
An AWS Account is used to create, manage and pay for resources. It can be similar to an\nenvironment or project, with servers, databases, S3 buckets etc.
Even though an account does not represent a user, it still has — confusingly — a unique email\naddress attached to it and a password to log in. This login represents the AWS account root user of\nan AWS account.
So if Accounts in AWS do not represent human users, how is identity established so that\naccess can be granted to resources in AWS Accounts?
This is done through Users, Roles and Groups. All three are a form of Identity in AWS.\nAn Identity is something that can be granted access to an AWS service or resource.
An IAM User represents a human that can log into AWS and/or leverage the AWS API to use\nAWS services and access resources on AWS. An IAM User has 2 sets of credentials:
Two key things to understand about IAM users:
As we’ll see later, there are special kinds of users you can create that do have an email address\nand can access resources in multiple AWS Accounts. These are created through AWS SSO.
IAM Users can have permissions (policies, see below) that dictate what they can and cannot do on\nAWS. These permissions are used to allow/deny access to AWS services and resources.
Users usually represent humans but can also represent software and machines outside of AWS that\nyou want to give programmatic access to certain AWS services and resources through the AWS API.\nIf you want to grant software running inside AWS access to other AWS services and resources, you\ncan use Roles.
A Role in IAM is similar to a User in that it is an Identity with permissions that determine\nwhat it can and cannot do within AWS. The difference with a User is that a Role is not tied to a\nspecific human or machine. Instead, a role can be assumed by users or AWS services to\ntemporarily gain access to certain AWS services and/or resources.
Roles do not have permanent login/API credentials. Instead, a role provides temporary credentials\nwhen the role is assumed.
In IAM, Groups are used to group Users together so the same permissions (policies, see below)\ncan be applied to multiple Users at once.
Now that we have established that we have Resources *\nand Identities* that can potentially access those\nResources, how exactly do we determine who can access what?
This is done through Policies. In IAM, Policies are basically permissions on resources
A policy defines access (or denial) to an AWS service and/or resource. There are 2 types of policies:
A policy contains the following elements:
The Principal is only specified in the case of Resource-based policies. That makes sense if you\nthink about it, because in the case of Identity-based policies, the Principal is the Identity itself\nthe policy is attached to!
Let’s say that we want to give IAM User kiara access to read all files in the confidential-data\nS3 bucket. What would the Effect, Resource and Action look like for our policy? The policy would\nlook something like this:
{\n\"Effect\": \"Allow\",\n\"Action\": [\n\"s3:List*\",\n\"s3:Get*\"\n],\n\"Resource\": [\n\"arn:aws:s3:::confidential-data\",\n\"arn:aws:s3:::confidential-data/*\"\n]\n}\n This policy would then be attached to the IAM User kiara to grant her access.
As mentioned before, for Identity-based policies, you don’t need to specify a Principal. But what\nexactly is a Principal? Is it the same as an Identity? It turns out it’s more than that:
A Principal is an AWS identity or Service. To understand why services can be used as Principal,\nwe have to understand where Principals are specified, which is in 2 places:
When specifying a Principal in a Role, we can say that instead of allowing a regular (human) User\nto assume the Role, we can allow an AWS Service (e.g. AWS Lambda) to assume the role for us. An\nexample of how this can be useful: we could allow a function run on AWS Lambda to send a message on\nan SNS topic for us whenever something important happens during its execution. To do this, we would\ncreate a special Role for this Lambda function with a Policy attached that would allow this Role\nto publish messages on an SNS topic. Then we would allow AWS Lambda to assume this Role.
What is perhaps counter-intuitive, is that Identity-based policies are more common than\nResource-based policies for defining access to resources. The common way of doing things, is to\nattach policies to Identities to specify what Resources they have access to and in what ways.
Part of the reason is that Resource-based policies are not supported by all services and\nresource-types in AWS, whereas you can use any Resource identifier in an Identity-based polity.
So in what situations are Resource-based policies being used?
Resource-based policies are often used when you want to grant an Identity outside your AWS\naccount access to a Resource in your AWS account. To understand how this works, let’s assume the\nfollowing scenario:
secret-stuffsecret-stuff bucketsecret-stuff bucket that Company A\nas granted them access toTo make this happen you have to do 2 things:
secret-stuff bucket): on the\naforementioned S3 bucket, we would create a Resource-based policy that has AWS Account B as\nprincipalsecret-stuff bucket, which happens to live in AWS Account A.What is interesting here, is that Company A would have no say in exactly what Identity is used\nby Company B — in AWS Account B — to access their resource. The only thing they would control, is\nthe fact that AWS Account B can access their resource in AWS Account A. It is Account B where it is\ndetermined exactly what Identity gets to access the secret-stuff bucket.
As explained in the previous blog post, you can leverage AWS Organizations to\ngroup multiple AWS Accounts together to consolidate billing but also access control.
How does IAM work in the context of AWS Organizations? This happens through 2 additional concepts:
AWS Single Sign-on lets you define Users and Groups in a central place — usually in the\nmanagement account of your AWS Organization — and control access to resources in all AWS Accounts\nin an Organization for those Users and Groups.
SSO Users have a few fundamental differences compared to regular IAM users:
SSO Groups are similar to IAM Groups in that they’re used to group together Users so that the\nsame policies can be applied to all users within a group. A key difference here is that — just\nlike with SSO Users — policies are not attached to SSO Groups directly.
If you cannot attach policies to SSO Users & Groups directly, how can you allow those Users & Groups\naccess to resources in the AWS Accounts in your Organization? This is done through\nPermission Sets.
Permission Sets are basically the AWS SSO alternative to regular IAM Policies. They are set up in a\nvery similar way — with Effects, Actions and Resources. The big difference is that Permission Sets\nare not applied to an SSO User/Group alone, but they are applied in combination with an AWS Account.
A Permission Set is applied to an SSO User/Group for a specific AWS Account
This means that when you apply a Permission Set to an SSO User/Group, you choose an AWS Account so\nthat the policies defined in the Permission Set are applied to the SSO User/Group for the selected\nAWS Account.
Example: let’s say you have created a Permission Set that allows read-only access to all resources.\nNow you can grant kiara@examplecorp.com read access to all resources in Account A in your\nOrganization by applying the aforementioned Permission Set to kiara@examplecorp.com specifically\nfor Account A.
Permission Sets encourage reuse of similar policies across AWS Accounts and SSO Users.
If you apply a Permission Set to an SSO User for a specific AWS Account, how does that work\ninternally? In the end, AWS will still use IAM to do the actual access management. For each\nPermission Set you apply to a certain AWS Account — for 1 or more SSO Users/Groups — AWS will\ncreate a dedicated Role in that AWS Account. Let’s say that in the example above where we want to\ngrant read-only access, we called the Permission Set ReadOnlyPermissions. When we apply this\nPermission Set to Account A for kiara@examplecorp.com, AWS will do 3 things:
ReadOnlyPermissions Permission SetReadOnlyPermissions\nPermission Setkiara@examplecorp.com as Principal to the created Role so she can assume that Role\nin Account ANow when kiara@examplecorp.com logs into AWS using SSO, she will be presented with a choice to\nassume the ReadOnlyPermissions Role in Account A — next to all the other Roles she can assume\nin the same or other AWS Accounts in the Organization, all based on the Permission Sets applied\nto her for specific Accounts.
When I started writing this post, I hoped it would be concise. I guess I failed a bit on that\naccount, which just goes to show how complex AWS IAM really is. I still hope that this gives you an\noverview that is easier to digest than reading through pages of AWS documentation.
Thanks to Dan for proofreading this yet again! ❤️
", "url": "https://www.daan.fyi/writings/iam", "title": "Aws Iam Demystified", "summary": "AWS Identity and Access Management (IAM) provides a way to configure who can do what in AWS Accounts. It's powerful but also hard to understand. This post explains the basics.", "date_modified": "2022-03-21T00:00:00.000Z" }, { "id": "https://www.daan.fyi/writings/aws-multi-account", "content_html": "Don’t you just love the freedom of working at an early stage startup, with a relentless focus on\nproduct and a blatant disregard for any non-functional distractions? I do.
When I started working at Source about 9 months ago, it was just me — the\nfirst employee — and the 2 founders. The only code written, were a couple of demos for investor\npitches which we threw away quickly. I was in the luxurious position to be able to start building\nour product from scratch. I deployed everything I built into a single AWS root account.
Fast-forward to today, and we’re with more than 30 people — most of whom are software engineers\nand data scientists writing and deploying code. In December, we finished migrating all of our\ninfrastructure to a multi-account setup in AWS. In this blog post I want to explain why we\ndid this and what our multi-account setup looks like.
\n![\"Buy](\"https://www.daan.fyi/_next/image?url=%2Fimages%2Fbutwhy.gif&w=1920&q=75\"/)
As mentioned, when we started everything was deployed into a single AWS account.\nWhen the moment came that we wanted to have separate development and production environments, I\ncreated 2 VPCs in the same account and deployed an identical setup in both, each consisting of:
It looked something like this:
\n![\"All](\"https://www.daan.fyi/_next/image?url=%2Fimages%2Faws-single-account.png&w=3840&q=75\"/)
I deployed our applications and services (except for our frontend, which is deployed on Vercel) in\ntwo identical VPCs — still living in the same AWS account — and called it a day.
When we started building our team and the first software engineers and data scientists joined, I\ncreated IAM users for everyone and solved access control by defining groups with specific IAM\npolicies (permissions) attached to them and adding the IAM users to those groups. The more people\njoined the team, the more diverse the access requirements became, for example:
And there are many more subtle differences in the kind of access to infrastructure people need and\nshould have.
In the end, our single-account setup made it difficult to build a secure and resilient\ninfrastructure:
Many of these problems can actually be solved — even in a single-account setup — by \"properly\"\ntagging all your AWS resources and creating elaborate IAM policies for users and resources based on\nthose tags. But such a tagging strategy will quickly become really complex and will cost a lot of\ntime to maintain. Needless to say that it’s quite error-prone as well.
Aside from the security and safety implications, our setup had some other problems:
When I was researching a way forward, I found many\nblog posts\nextolling the virtues\nof putting your infrastructure into multiple AWS accounts. But before getting a better\nunderstanding of how this would work, I had some reservations about using multiple AWS accounts:
AWS offers a nice solution to these problems (and others) in the form of AWS Organizations
According to AWS:
\nAn organization is an entity that you create to consolidate a collection of accounts so that you\ncan administer them as a single unit.
\n
![\"What](\"https://www.daan.fyi/_next/image?url=%2Fimages%2Fwhatdoesthatevenmean.gif&w=640&q=75\"/)
AWS Organizations lets you organize a bunch of related AWS accounts in a hierarchical, tree-like\nstructure. It will let you manage all those accounts from one central management account. With\nConsolidated Billing, all bills from all member accounts will be aggregated to the management\naccount where you can pay them as one.
Arguably the most powerful feature of AWS Organizations is that you can group the member accounts\ntogether in Organizational Units (OUs) and apply common security policies to those. This lets you\ndefine permissions and control access in broad strokes — and from a central place — while keeping\nthe natural separation of concerns that comes from using separate AWS accounts to organize your\ninfrastructure.
When it comes to user management, AWS Organizations offers the convenience of letting users log in\nwith one user account (not to be confused with AWS account), using AWS Single Sign-on. AWS SSO lets you create users in\nthe root account (also known as management account) of your organization and\ngive them login credentials within AWS itself or link them to an external identity provider\nlike Google or Active Directory. You can then give these users access to specific AWS accounts within\nyour AWS Organization. Users can be assigned specific permissions within these accounts. (I will explain more on how this works in a later blog post)
This has several advantages compared to using regular IAM users:
Well, I don’t know. But I do know what works for us so far
When deciding what AWS accounts to create and how to group them, I had the following requirements:
This was aside from solving the aforementioned problems we had with our single-account setup.
This led me to create the following accounts, organized by Organizational Unit:
\n![\"AWS](\"https://www.daan.fyi/_next/image?url=%2Fimages%2Faws-org-design.png&w=3840&q=75\"/)
Some notes and highlights:
Using AWS Organizations we have now set up our infrastructure at Source in such a way that it is\nmore secure and also more scalable and easier to maintain from a security perspective.\nOur Software Engineers and Data Scientists have easier access to the infrastructure, services and\nresources they need — using only one set of credentials — without exposing them to\nsensitive data and/or critical infrastructure.
In future blog posts I will elaborate on the steps taken to set all of this up and share some\npractical guides on how to use a multi-account setup on a day-to-day basis.
Thanks to Dan for proofreading this ❤️
", "url": "https://www.daan.fyi/writings/aws-multi-account", "title": "Setting up an Aws Multi-Account Strategy for Fun nor Profit", "summary": "Using multiple AWS accounts is not fun, nor is it profitable, but it can help you set up your infrastructure in a more secure way. It also helps with keeping your sanity as your company/team grows in size.", "date_modified": "2022-02-09T00:00:00.000Z" }, { "id": "https://www.daan.fyi/writings/pyenv-hooks", "content_html": "I use pyenv to manage different Python versions on my laptop. It\nalso comes with an official plugin that lets you manage\nvirtualenvs through pyenv, which I find very convenient. Basically, virtualenvs are treated as\njust different Python versions by pyenv.
One thing that bothered me is that, whenever I create a new virtualenv and use pip in it, I am\ninevitably greeted by a message telling me pip is out-of-date:
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.\nYou should consider upgrading via the '~/.pyenv/versions/3.10.0/envs/test/bin/python3.10 -m pip install --upgrade pip' command.\n So I end up always having to upgrade pip after creating a new virtualenv. Wouldn’t it be nice if\nthis could be automated?
Turns out, we actually can by leveraging pyenv hooks.
pyenv hooks are scripts that are executed by pyenv whenever certain commands are run. These can be\nregular pyenv commands like pyenv install or pyenv rehash for example. But what is not\napparent from the pyenv documentation, is that you can also create hooks for plugins, like\npyenv virtualenv.
You can create a hook by creating a script at the following location:
\n $PYENV_ROOT/pyenv.d/<hook-name>/<whatever>.bash\n hook-name here can be any of: exec, rehash, which, install — which are all regular pyenv\ncommands — but it can also be a plugin command, like virtualenv. The filename of the script doesn’t\nmatter, and neither does the extension. I use .bash here to make it explicit that this is a bash\nscript, but pyenv hooks can be written in any language.
To create a hook that upgrades pip and some other default packages, you can create a new script\nas follows:
mkdir -p $PYENV_ROOT/pyenv.d/virtualenv/\n$EDITOR $PYENV_ROOT/pyenv.d/virtualenv/after.bash\n Where $EDITOR is your favorite editor (like vim, RIGHT?!)
Then add the following contents:
\n after_virtualenv 'PYENV_VERSION=\"$VIRTUALENV_NAME\" pyenv-exec pip install --upgrade pip setuptools wheel'\n after_virtualenv is the command that tells pyenv when to execute the following command. In this\ncase its defined by the pyenv virtualenv plugin. First we set the pyenv version to the name\nof the virtualenv we just created. This is set by pyenv virtualenv as $VIRTUALENV_NAME. Then we\ninstall/upgrade pip itself and setuptools and wheel.
That is all there is to it! Now any time you create a new virtualenv using pyenv virtualenv, the\naforementioned packages will be automatically upgraded after the virtualenv was created.
At Source, we write most of our code in Python. It’s a language that both our\nSoftware Engineers and Data Scientists are equally at home in. It’s easy to be productive in Python,\nin part due to its dynamic nature. Not having to think too much about the types of your variables\nand functions, can make it easier to experiment, especially if you’re not entirely clear yet\non how you’re going to solve a particular problem.
When moving our code to production however, we want to have more guarantees about the behaviour of\nour code. Writing (unit) tests is one way to get those guarantees, but we also make heavy use of type hints to give us more confidence in our code.\nType hints can also provide a productivity boost, because not only humans can reason better about\ntype hinted code, your editor can as well!
Sometimes though, using type hints everywhere can feel like you’re losing out on a lot of the magic\nand speed that a dynamic type system brings you. One particular trait of dynamic typing that is\npretty idiomatic in Python, is duck typing.
Duck typing is a philosophy in programming where you\ncare more about the behaviour and properties of an object than its stated type to determine if that\nobject is useful in a certain situation. Duck typing is inspired by the duck test:
If it walks like a duck and it quacks like a duck, then it must be a duck
\n
In practice this means that when you write a function that receives a certain input, you care only\nabout the behaviour and/or attributes of that input, not the explicit type of that input.
One interesting question that arises is: if you don’t want to be strict about the type of the\nparameters a function receives, are there still any static type guarantees to be had?
And the other way around is interesting as well: if you have a function with statically typed\ninputs, can you loosen up those parameters to make the function more universally useful, the way\nduck typing does?
As it turns out, Python provides a neat way to have our cake and eat it too!
When reviewing some code recently, I came across a function that looked roughly like this:
\n def calculate_windowed_avg(\nmeasurements: Union[List[TemperatureMeasurement], List[HumidityMeasurement]],\nwindow_size: timedelta,\nfield_name: str\n) -> Dict[datetime, float]:\nwindow_upper_bound = measurements[0].timestamp + window_size\ncurrent_window = []\nwindow_averages = OrderedDict()\nfor m in measurements:\n# various calculations happen here\n# based on the timestamp of each measurement\n...\nreturn window_averages\n The goal of this function is to calculate the average of a certain field (identified by field_name) in a rolling window. At the time of writing this function, we were using it for\nTemperatureMeasurement and HumidityMeasurement, but it is very likely we’ll want to use it for\ndifferent types of measurements in the future.
If we look closely at how the function uses the input, it turns out that the only thing we want to\nbe guaranteed of, is that the items we pass into the function have a timestamp field. So instead\nof specifying each different type that has adheres to this contract, we’d like to tell the type\nchecker that we only care about having a timestamp field to work with.
Protocol from the typing module lets us do that. Just like with duck typing, Protocols let\nyou specify the behaviour or attributes you expect, without caring about the type. Here is what\nthat looks like:
from typing import Protocol, List, Dict\nfrom datetime import datetime\n\nclass MeasurementLike(Protocol):\n timestamp: datetime\n\ndef calculate_windowed_avg(\n measurements: List[MeasurementLike],\n window_size: timedelta,\nfield_name: str\n) -> Dict[datetime, float]:\nwindow_upper_bound = measurements[0].timestamp + window_size\n...\n Now the type checker doesn’t know exactly what the type is of whatever is provided as measurements but it does know what those items have a timestamp field because they adhere to\nthe MeasurementLike Protocol.
In a sense, a Protocol acts like one side of an Interface as we know it from Java or Typescript.\nInstead of having to specify the behaviour and properties both on a type and on the functions\nthat use it, we only have to specify it on a function, without caring about the types of the objects\nthat are provided to the function.
You can also use Protocols together with TypeVar for even more generic functions that are still\ntype checked to some extend. One use-case that comes to mind, is when you don’t care about the\ninput type to a function, as long as it follows a protocol, but you also want to guarantee that\nthe output of the function is of the same type as the input, no matter what the exact type is.
This works as follows:
\n from typing import Protocol, TypeVar\nfrom datetime import datetime\n\nclass MeasurementLike(Protocol):\ntimestamp: datetime\n\nM = TypeVar('M', bound=MeasurementLike)\n\ndef measurement_as_timezone(measurement: M, tz: tzinfo) -> M:\nmeasurement.timestamp = measurement.timestamp.astimezone(tz)\nreturn measurement\n Here we create a function that takes any object that has a timestamp field and guarantees that\nthe output will be of the same type as the input.
Protocols in Python provide a nice way to use duck typing while still having some static type\nguarantees. You can define contracts for your functions without caring too much about the actual\ntypes of your inputs.
Update on 2021–11–23: There was a wrong type annotation in this article, as pointed out by\nragebol on Hacker News, which is now fixed
", "url": "https://www.daan.fyi/writings/python-protocols", "title": "Static Duck Typing in Python with Protocols", "summary": "Duck typing is considered to be one of Python's strengths. If you want to have the benefits of duck typing but also want your types statically checked, Protocols offer an excellent solution.", "date_modified": "2021-11-19T00:00:00.000Z" }, { "id": "https://www.daan.fyi/writings/rss", "content_html": "Today, I released a new version of my website with only one new feature: syndication through RSS, Atom and JSON Feed. Even\nthough by some accounts, RSS\nseems to be dead,\nI strongly believe RSS is an important feature in the fight to keep ownership over your own content\nwhile also increasing exposure. The way I approach publishing and sharing my content, is called\nPOSSE:
Publish (on your) Own Site, Syndicate Elsewhere.
\n
RSS is an important part of that strategy.
Luckily, I’m not the only one who values RSS. There\nare others
In this article I want to share how I implemented syndication feeds in my NextJS-powered website.
I did some Googling to see how other NextJS users were generating RSS feeds, and there turned out\nto be many people that had solved this particular problem and wrote about it\nI found a couple of different approaches to generating and rendering RSS feeds in NextJS:
Generating feeds:
Rendering feeds:
Content-Type header for statically generated pages, so you can’t serve\nthose pages as application/rss+xml. I’m not how big of a problem this is and what black\nmagic Vercel applies when serving my NextJS siteAfter looking at the options, I decided on the following requirements for my feeds:
As per my requirements, I wanted to generate the feeds at build time. But as mentioned before,\nNextJS doesn’t support setting the Content-Type header for statically generated pages.\nThe alternative that many people use, is to have a separate script generate your feeds and\nwriting them to the public folder where all other static assets such as images are stored as well.\nThat way, the feeds would be served as static assets instead of statically generated pages — which,\nfrom the browsers perspective doesn’t make a difference!
I found a good explanation by Ashlee Boyer\nof this technique.
My plan:
postbuild step so it would always be invoked after building the site\nusing npm run build (this happens not only locally, but also when Vercel deploys my site)I immediately hit a snag with (1), because I couldn’t manage to use ES6/Typescript modules in a\nscript run outside of my normal website code.
I’m using Typescript, and apparently ts-node, the tool to run Typescript files, doesn’t support\nmodules. Writing the script in Javascript wasn’t really an option for me because I wanted to reuse\na lot of logic that I already wrote for reading and parsing MDX files in the website itself.
I decided to follow the route that Ashlee Boyer suggests in her blog post and sneak in the\nfunction to generate my feeds as a \"stowaway\" in the getStaticProps function of my index\npage. This works beautifully!
export const getStaticProps: GetStaticProps = () => {\ngenerateMainFeeds();\nconst lastPosts = getAllPostsFrontMatter('blog', 3);\nreturn {\nprops: {\nlastPosts,\n},\n};\n};\n The code of my website already supported translating MDX files into valid JSX to be rendered\nby React. But how to generate valid HTML from that content and include it in the feeds?
I couldn’t find many examples of this, but did find out about ReactDOMServer.renderToStaticMarkup.\nThis function will take a bunch of React components and render them into HTML. This is what is\nused by many React server-side rendering solutions (maybe also by NextJS?) and works perfectly\nhere as well.
One caveat: if your content contains internal links — which are often relative links — then you\nhave to be mindful that those relative links are meaningless in the context of an RSS feed.\nThe way I solved this is by doing some regex-based replacements on the generated HTML.
The complete content generation part looks like this:
\n const url = `${baseUrl}/blog/${post.frontMatter.slug}`;\nconst htmlContent = ReactDOMServer.renderToStaticMarkup(\n<ChakraProvider resetCSS theme={theme}>\n<MDXRemote {...post.mdxSource} components={MDXComponents} />\n</ChakraProvider>\n.replace(/href=\"\\/#/g, `href=\"${url}#`)\n.replace(/href=\"\\//g, `href=\"${baseUrl}/`)\n.replace(/src=\"\\//g, `src=\"${baseUrl}/`);\n)\n My site uses Chakra UI for theming, which uses\nEmotion — a CSS-in-JS library — under the hood. Emotion will happily\nrender tons of tags when statically generating HTML from your React components. For most\nuse cases where you render React on the server (statically or not), this is desirable. In the case\nof RSS/Atom feeds, this is pretty useless.
The solution here is to strip all the and The One whose Name cannot be expressed in the Basic Multilingual Plane\nby trying to use regex here, I found this library\nto help me with this task:
const cleanHtmlContent = stripHtml(htmlContent, {\n onlyStripTags: ['script', 'style'],\n stripTogetherWithTheirContents: ['script', 'style'],\n}).result;\nI now have serve RSS, Atom and a JSON Feed\nfor your reading pleasure. Most of the relevant code can be found here
\nAt some point I want to diversify my writing output by not only writing about tech. And\neven within the topic of tech there are many sub-topics I could write about, not all of which\nare equally interesting to every reader (all 5 of them, including my mom 👩👦).\nI’m planning to introduce tags to allow\nfiltering content once I have enough of it.
\nOnce I have tags, I would like to start supporting dynamic feeds so readers can subscribe only\nto the stuff they actually want to read. I imagine building an endpoint like this:
\n/feeds/by-tags.xml?tags=tag1,tag2\nI’m curious if others are doing this!
", "url": "https://www.daan.fyi/writings/rss", "title": "How to Add an Rss Feed to a Nextjs Blog", "summary": "RSS is an important feature to give your content maximum exposure while also retaining control over your content. This post shows how to add syndication feeds to a statically generated NextJS site", "date_modified": "2021-10-05T00:00:00.000Z" }, { "id": "https://www.daan.fyi/writings/tools", "content_html": "A couple of weeks ago someone shared an article on Hacker News that\ndiscussed the drawbacks of using an IDE for programming. Needless to say, when you touch\npeople’s tools, you can be sure to\nspark a hefty discussion.
To be fair, the author did clarify that they’re not against IDE’s in general and they actually\nuse one on a daily basis.
When reading the discussion, one of the first things I wondered is if there are places on the\ninternet where carpenters or plumbers furiously criticize each other’s choice of tools.\nBut even that has a place on Hacker News
In this post I want to explore the fundamental problems with this type of discussion and how you\ncan embrace freedom of choice.
When discussing someone else’s choice of tools — in this case the choice of editor or IDE — it’s\ngood to be mindful of what we don’t know about this person and their reasons for choosing certain\ntools. Aside from personal preference and/or familiarity, there can be lots of extra factors to\nconsider:
I think it’s good form to voice an opinion only when we can see the whole picture. I also\nthink that people should be encouraged to choose the right tool for the job. And to put it\nsimply: the right tool for the job is the one that fits the problem and makes you the most\nproductive.
Another issue with this kind of discussion is that everyone has a different idea of what really\nconstitutes an \"IDE\". We’re inclined to frame the choice of editor as a binary choice — \"pure text\neditor\" vs \"IDE\", almost like a false dichotomy — while it really is a spectrum. On this spectrum\nyou’ll not only find different editors, but even different configurations of the same editor with\nvarying degrees of \"smartness\" applied through plugins.
So when someone says \"why use an IDE when you can do everything an IDE does in VIM if you install\nthe following 20 plugins\", is this really so different from an actual IDE (that was branded as\nsuch)?
When looking at how people setup their development environments, I basically see 2 different\napproaches:
To me, both approaches have their merits and uses!
I think the most important question around editors is not
How to choose the right editor?
\n
but rather
How to enable each member of my team to choose the editor that makes them most productive,\nwhile enabling effective collaboration?
\n
Here’s my recipe to do that:
My basic starting point in teams: everyone can use the editor/IDE of their choice. To enable\neffective collaboration, the following rules must be followed:
I have to admit that in some ways I also limit choice within my teams: building and running\nprojects often involves shell scripts which are portable between Linux and MacOS, but not to\nWindows. This can be solved by using WSL on Windows.
My personal setup involves quite a lot of tools.
For any \"serious\" development work (i.e. long stretches of coding and big projects) I take the\naforementioned \"subtractive\" approach. My basis is one of the Jetbrains tools, depending on the\nlanguage(s) I’m working with. I find the Jetbrains tools make me really productive and their All products pack offers great value for money, especially with\ntheir huge loyalty discounts after 1 or more years.
I still use the terminal and external tools for many things in this case. I put quite some time in\nimproving and maintaining my dotfiles as well.
For quick and short edits and viewing files where speed is preferred and conveniences like syntax\nhighlighting and intellisense are not important, I use Sublime Text or VIM. Example of this:\nediting my dotfiles.
Sometimes, but rarely, I use VSCodium for editing or viewing files which are not part of a project\nand which I want to reformat/reindent. This doesn’t work well for me in Sublime Text. For example:\nviewing/editing minified JSON files.
To wrap this post up, I just want to reiterate: use whatever tools make you productive and help you\nsolve the problems you’re trying to solve and please let others do the same.
And here’s some additional closing advice: should you ever grow unhappy with the tools you’re\nusing, just write a blog post extolling the virtues of your current tools. You can be sure people\non the internet will tell you how much better their tools are, thereby providing you with potential\nalternatives to your current setup 😇
Thanks to Dan for proofreading this
", "url": "https://www.daan.fyi/writings/tools", "title": "Use Tools That Suit You and the Problem", "summary": "Don't let anyone tell you what tools to use. Make sure that your builds are tool-agnostic", "date_modified": "2021-09-10T00:00:00.000Z" }, { "id": "https://www.daan.fyi/writings/amplify", "content_html": "Have you ever gone above and beyond to give a product a fair chance, only giving up way past your\nown sanity and patience? I did. The product was AWS Amplify
When I started developing the primary (web) UI for the product we’re building at Source, I quickly decided to use the NextJS framework as\nthe basis for the frontend. One thing I really like about NextJS is that it does filesystem-based\nrouting. Even though I like to diss on PHP as much\nas the next person, this particular feature of NextJS reminds me a bit of PHP in a good way.
Another very compelling feature of NextJS, is the ability to have different rendering strategies\nfor each different page. You can decide to use:
This particular feature requires a specific hosting setup that supports the different flavours\nof rendering. You basically want:
The NextJS documentation mentions a couple of options for\ndeployment:
At Source however, we’ve made a conscious decision to use AWS for all of our infrastructure needs. We’re\ntoo early in the lifecycle of both our product and company to consider a hybrid cloud setup and\nAWS offers all the features that we need for prices that are reasonable for now. Through one of\nour investors we also got a boat-load of AWS credits to get started, which helps.
I didn’t want to manually set up a hybrid deployment with NodeJS and CDN - which I could probably do\nusing Cloudfront and ECS - so I looked for alternatives. AWS Amplify claims to be the
Fastest, easiest way to build mobile and web apps that scale
\n
And they claim to support NextJS\nas well as server-side rendering\nby virtue of Lambda@Edge
Sounds great! So how does that all work?
The idea behind Amplify is that it lets you focus on building your app without having to worry\nabout the infrastructure. This means that it will not only let you build and deploy your frontend\neasily, it will also let you hook-up backend components like authentication through\nCognito,\na GraphQL API using\nAppSync\nand a DataStore.
The way Amplify manages all of this, is through the idea of \"backends\". A backend contains one or\nmore of the aforementioned components. You can setup different backend environments in Amplify and\ncouple these to different (git) branches of your frontend.
We are bringing our own database and API and wanted to use a pre-existing Cognito user pool.\nAmplify lets you import an existing Cognito user pool instead of creating one for you, and this\nworked fine… until I tried to create a second \"backend environment\" with a different Cognito\nuser pool.
I want to allow different users on our development environment than our production\nenvironment. These environments correspond to the main branch in Git (production) and any other\nbranches in Git (development).
Interaction with Amplify is done through the web console or the Amplify CLI. According to the documentation, we can create a new\nAmplify backend environment like this:
\n amplify env add\n But nowhere is it made clear how this is linked to our frontend branch. The Amplify CLI tries to\nmimic the git CLI by having amplify push and amplify pull. So I assume we need to push our\nnew environment to the cloud. Does this command depend on the git branch I’m on? I don’t know.\nI could just as easily overwrite my existing backend environment when using this command.
Eventually I was able to create a main environment and a dev environment, but no matter how hard\nI tried, I could not get those environments to use different Cognito user pools.
The Amplify CLI will create some configuration files, some of which you need to commit to git, and\nsome of them you need to ignore. These files should make the build reproducible. It is unclear\nhowever, what a new developer needs to do to be able to run an existing Amplify project.\nThe documentation mentions multiple Amplify CLI commands to setup a project:
\n amplify pull\n\n amplify init\n\n amplify configure project\n\n amplify env checkout \n To be run in arbitrary order?
Three\nsimilar\nquestions\non StackOverflow give 3 different answers on this topic.
When I was on-boarding new team-members, we basically tried some commands in different order until\nwe were able to run the project locally for that engineer.
In short: you don’t. Or you enlist\nthe help of AWS Support (which you will have to pay for). The Amplify web console doesn’t share\ndetailed build logs, so you’ll have to fix your problems by trial and error.
AWS Amplify seems to have quite some bugs. For example: when you use Gitlab, you\ncannot put the source code of your project in a sub-group. This will silently\nfail your build, with no clear error message in sight.
The aforementioned problem of using multiple Cognito user pools for different\nenvironments of one Amplify frontend, I also consider a bug.
Not having access to the actual build logs, I also consider a bug.
The most painful issue I have with AWS Amplify - and AWS in general - is the complete lack of\nsupport. You are paying a lot of money for their services, but that does not come with any form of\nsupport, unless you pay extra. And when you make use of that paid support, it goes something like\nthis:
This means that in practice, AWS Amplify is not usable unless you are willing to pay extra for\nsupport and have a lot of patience.
As a side note, I find it really frustrating that with big corporations like Amazon in general, you\nquickly feel helpless and ignored when you have problems. Stories about Google customer support (or\nlack thereof) are rampant on Hacker News, and I think Amazon is no different. You’d think that with\nso much revenue they should be able to set up actual proper customer support 🙄
I think that these problems with Amplify are endemic of a larger problem with AWS: their PaaS and\nSaaS solutions suck. Aside from pricing (which is quite high compared to competitors like\nDigital Ocean), I think AWs’ infrastructure-as-a-service offerings are fine. If you want to have a\nbunch of servers (EC2), a database here and there (RDS) and some serverless functions (Lambda), AWS\nwill serve you just fine.
But if you want something on top of that, like authentication (Cognito), or a PaaS solution like\nAmplify, be prepared for a sub-par offering compared to the competition. I think Amazon should\nfocus on strengthening their core offerings instead of offering everything and the kitchen sink.
In the end, I dropped Amplify and went with Vercel. $20/user/month is a small price to pay for\nactually being able to work on the product instead of fighting AWS services and support.
", "url": "https://www.daan.fyi/writings/amplify", "title": "The Pitfalls of Deploying a Nextjs Frontend on Aws Amplify", "summary": "Half-baked products lead to much frustration", "date_modified": "2021-08-23T00:00:00.000Z" }, { "id": "https://www.daan.fyi/writings/new", "content_html": "I don’t really know how to say this, but: I’m going to try and blog again! After building this site,\nI didn’t even know how to begin actually writing again. Looking for inspiration, I studied some\nfamous tech people’s websites but came up empty trying to find examples of \"first blog posts\" there.\nMost people like to just dig right in with some on-topic tech content it seems…
But HERE IT IS!
My first blog post on my new website.
It’s been nigh on 4 years since I last wrote something on my old crib dandydev.net (by the time you’re reading this, I’ll have most probably\nredirected everything from there to here). I don’t really know why I stopped blogging in the first\nplace, but it probably had something to do with life/work/… getting in the way (or me letting it\nget in the way anyways).
So what has been going on?
After working for KLM Royal Dutch Airlines for 4.5 years I got a dream position at Source.ag, a new startup co-founded by a friend, focusing on Agriculture & AI.\nAs VP of Engineering I help build both the tech and the team and I’m loving it to bits!
Keeping with the theme of this post, it’s really exciting to start something so brand new. I’m\nactually employee #1 which gives me ample opportunity to have a major impact. There are no\npre-established rules, processes, dynamics. For the first couple of months, it was just the 2\nco-founders and me figuring out what to build, how to build it and then actually building it while\nat the same time trying to grow the company. And now, 5 months after I started, we’re 12 lifeforms strong\nand growing!
And after my previous job veered more and more into organizational politics instead of tech, it\nfeels great to be building an actual product again. It’s both amazing and terrifying to code most\nof an MVP all by yourself in 5 months 😱
If you’re interested in bringing positive change to the food system, you can join us
Interlude: why is it that I find starting new things so exciting? I often wonder about this.
I think it comes down to wanting to learn new things as much as possible and I guess I just love\nexploring the unknown.
But I’m gonna be honest: the challenge is to stick with things. For each side-project I finished,\nthere are many left in the dust. They were never time badly spent, because each time I learnt\nsomething new. But I think there is value in finishing things as well.
I hope that a different approach with this blog will make it easier for me to stick with it.
I want to sound off this rambling first blog post by sharing how things that are (going to be)\ndifferent with this website:
When building this website, there were a few websites that inspired me and/or helped me with actual\nsolutions to coding problems (yay for open source!):
See you next time!
", "url": "https://www.daan.fyi/writings/new", "title": "The Joy of Starting Something New", "summary": "The start of a new blog/job/...", "date_modified": "2021-08-16T00:00:00.000Z" } ] }