Numerous startups, especially those specializing in DevOps and/or platform engineering, have contributed valuable insights on enhancing developer productivity. While there's a wealth of knowledge in these fields, the creation of development environments presents unique challenges that might not be as widely recognized. The process of setting up these environments requires a nuanced understanding and a tailored approach to keep pace with developers' needs. It's a nuanced skill that extends beyond the realm of traditional coding, demanding a deep appreciation for the dynamics of developer workflows. Recognizing this, it's crucial for those in the DevOps and platform engineering sectors to explore these distinctions further, ensuring development environments are as efficient and supportive as possible.
Development environments serve a specific purpose, focusing on facilitating efficient coding and immediate problem-solving rather than demonstrating the capabilities of GitOps or CI/CD. While these practices offer significant benefits in the right contexts, they may not always align with the core objectives of development environments. It's important to consider the unique needs of developers and adapt our tools and processes accordingly to support rather than hinder the development workflow.
Many times, it takes a lot of time (hours/days) to implement a feature or find solution for a problem/bug. Once you are ready with the code or changes, you want to test it in a local/remote dev environment as quickly as possible to confirm if those changes are going to work or not. Many times, developers need to do multiple iterations before we reach on the final working solution. As the developers struggle with poor documentation around (new age) frameworks and other tools, the least they wish is that they are not made to wait for hours by CI/CD pipelines before they can validate if their changes are working or not.
There is nothing which can beat the convenience of a local development environment where your changes reflect in real-time, and you can debug the code with ease. However, many times it is not possible to test things in a local development environment because -
You are using a laptop which has only 16 gb ram but the application which you are working on, your IDE, opening a number of chrome tabs etc require a laptop which should have a min of 24/32 gb ram. This is the case which I have seen in the India offices of many large (India / US based) IT services companies.
The application has dependency on a number of managed services which are either not available outside a public cloud or difficult to run outside a private network in the cloud.
You don't want to take the headache of setting up an application in the local development environment because it comes with some documentation which is either incomplete or runs into 10s of pages which is very difficult to follow for someone who is new to the application.
So, you come with the idea of remote development environments to overcome these challenges but then there are other challenges for remote development environments -
A 24/32 gb development environment in the cloud comes with a huge price tag.
Egress changes in the cloud may spoil your fun.
If it is a managed service (e.g. a managed k8s cluster) then there is no option to shut it down to save money outside working hours.
People working in tier 2/3 cities may suffer due to high latency while accessing these cloud-based environments or managed services which these development environments are connecting to.
You devops/platform engineering team is not allowing you to push the code to remote development environments using anything other than CI/CD.
So, then what is the solution of this problem?
While a baremetal server with 32 gb ram may cost a fortune in the cloud, getting a desktop with same amount of ram is relatively cheaper. I have one with 16 gb ram and another with 32 gb ram is on its way to my personal datacenter running in my home. With these desktops, I can have a laptop with as little ram as 8 gb, edit my code on the laptop and see my changes getting reflected on k8s servers in real-time with a single digit milliseconds latency and without building any docker images. For (small and bootstrapped) startups, such setup can significantly reduce your investment in the laptops and help a team to test their k8s application(s) on a shared baremetal machine without any fear of incurring high data egress charges and with a very low latency.
Here is how my development environment looks for MechCloud SaaS application which has some microservices running on cloudflare pages functions / workers while a dozen of python based microservices are running on a number of k8s clusters bootstrapped using kubeadm in different regions -
Eclipse has been my preferred IDE since the beginning of my career when I used to be a Java developer. However, now I am using it for developing and debugging python based microservices.
In eclipse you can start a group of microservices by grouping these under a launch group and then starting this launch group. So a launch group can be used to simulate a namespace in k8s.
A launch group can have another launch groups in addition to microservices. This can be used to simulate a setup where you want to start microservices in a namespace before starting microservices in a dependent namespace in k8s.
I use VS Code for developing vuejs applications and javascript based microservices.
Docker is the biggest innovation I have seen since the beginning of my career. I couldn't have imaged developing MechCloud with supersonic speed without this amazing technology.
With docker, managing application dependencies (e.g. mongodb, redis etc in my case) is a trivial task.
Docker has given a tool in the hand of developers where they can run application dependencies on a linux OS of their choice without worrying about package managers which fedora, debian etc based operating systems come with.
However, I don't build docker images in my local development environment for every single change because if I do that then I will be wasting half of my day in building docker images only. I do it only once in a week / two weeks or when I am ready to release a feature to MechCloud stage environment consisting of a few k8s servers spread across the globe. The reason is that building docker images for every change slows down the development speed which is crucial is my case. However, to make sure that this setup works, and I can deploy MechCloud on k8s without repeating the whole testing, I had to make few tough choices -
All MechCloud python based microservices run on the same version of python and a REST Api framework for developing python based microservices.
This setup allows me to upgrade python and/or REST Api framework for all the microservices by just pulling a new version of python, shutting down all microservices, upgrading REST Api framework in a common repo and then starting all the microservices again in eclipse. So, upgrading python and rest framework takes hardly 15 seconds across a dozen microservices which can be scaled to 10s of services and it will still take 15-20 seconds only. Imagine the amount of time wasted in doing it for 20 odd microservices by building docker images for all these microservices every time there is a change in the code which is common across all the microservices and then deploying it on a k8s cluster just to see if it is working or not.
Conclusion
While I agree that GitOps, CI/CD should be the preferred/only way to push the code in non-development environments, using these practices in development environments when developers are under immense pressure to achieve a lot in limited time is bad for both the developers and the customers. Development environments require innovations from time to time because things like build tools for client-side frameworks etc are evolving at a much faster pace than what was seen in the past. So, it is very unlikely that someone who is not facing these challenges (because they are not coding) will be able to do the justice with the development environments.
Join MechCloud discord server for learning more about great developer experience.