Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you may not be able to execute some actions.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Growing local development environment issues

N
Nykeriab last edited by

Where I work we've been adding microservices for different purposes and the local development environment is becoming difficult to setup. Services have too many environment variables to configure and usually there's not enough memory avaiable to run them.

We plan to fix these issues. I understand it's a matter of architecture and DevOps mostly. One way we've thought of is to create a proper service registry that allows easier setup and opens the door to, for example, have some services running locally and others in the cloud. All wired together with the service registry.

Another option could be to stub some of the dependencies with something like https://wiremock.org/ but it seems too limited and difficult (?).

I wanted to ask, what other strategies are there to manage growing development environments?
Reply Quote 0
1 Reply Last reply
C
courtlanda last edited by

I recommend running all the dependent microservices on cloud and deploy only the microservice that is under active development in your laptop.
Reply Quote 0
1 Reply Last reply

environments 19

2
Posts

0
Views

Suggested Topics

R

How to mitigate issues in an IIS on-prem application calling an unreliable cloud vendor API?
Continuous Integration and Delivery (CI, CD) • windows api servers • • Raziyah00

2

0
Votes

2
Posts

1
Views

M

I'd make the invocation safe, use some try/catch on the invocation itself. That way, if the API is unreachable or HTTP error code is returned - the app can handle it and present the "error" to the user in a more UX-y way Second thing, I'd make a returned data validator of some sort. For example data types, structure etc. Especially on key fields. That way we know that nothing that comes from the vendor API is breaking our app and make sure that no data returned won't crash the app on our side.
R

What are best practices for over-committing resources in a staging environment?
Continuous Integration and Delivery (CI, CD) • continuous integration deployment continuous deployment autoscaling • • Raziyah00

2

0
Votes

2
Posts

1
Views

Very good question; I'm interested to see what comes up here. In terms of cost, obviously each company and team is different. Ideally, everyone should be resource and cost conscious... but most people really don't seem to penny pinch by heavily monitoring and tuning their workloads (probably as that takes a certain skill level and tool set). My Thoughts / Habits: Unlike most on-prem environments, it is very easy to spawn up dynamic environments in the cloud (if you're automating properly). So, when cost is prohibitive, you can just spin up full-scale staging environments in order to do load testing and then shut them down. This will work for most use cases. Obviously some things do only break when systems are alive a certain period of time under a certain load. We usually keep identical but smaller environments in staging. E.g. linearly scaled down presto clusters/spark clusters, k8s micro-services with less copies, and the like. We try to keep the actual memory/cpu requests (or config) the same on any tool. This allows for proper load testing without config changes for anything that scales. Additional Thought I'm pretty fond of the DevOps Toolkit book series by Victor Farcic. If you're using kubernetes, the 2.5 book (https://www.devopstoolkitseries.com/posts/devops-25/) goes into detail on using prometheus and alert manager to detect over-sized or undersized resource requests, among other things. If you're really into saving resources and testing well simultaneously, it may make most sense to implement and use this in staging and prod. That way you can tune your alerting and automate your response so you can keep both environments sensibly sized for themselves in a consistent and enforced manner. I think you'd still want to ensure you could set the staging environment (or another dynamic environment) to the prod settings to load test effectively though.
C

Error when using 'ansible-local' provisioner with Docker container
Continuous Integration and Delivery (CI, CD) • docker ansible packer • • Cleofas

2

0
Votes

2
Posts

1
Views

T

It seems that there is some bug in Hashicorps Packer 1.2.0, consider downgrading or upgrading to v1.2.1, comments in GH issue page confirm that the deployed fix is working.
E

How to handle resource creation?
Continuous Integration and Delivery (CI, CD) • azure devops tfs microservices release management environments • • emran

2

0
Votes

2
Posts

1
Views

T

You can use Terraform for azure to handle the dynamic creation of azure resources in the release pipeline in Azure DevOps. Terraform maintains a state file for all the created resources after a run. If the script is run multiple times, it ignores already created resources unless there are any properties updated and created new resources. FYI, follow below link for a quick example Terraform example
R

VM Hosts: Single Large Host or Few Smaller Hosts?
Continuous Integration and Delivery (CI, CD) • servers environments • • rielyn

2

0
Votes

2
Posts

1
Views

B

A good rule of thumb is to try and avoid using one bit of logical kit (be it a single VM or a single physical host) for more than one thing. If a physical server is going to be a VM host then that's all it should be. If a VM or a physical host is going to be a source code repository then that's all it should be etc. You can squeeze a bit extra out of your infrastructure if you consider everything to be a VM host and you then partition up those VMs to be single purpose hosts themselves. The rationale behind this is that A single host going down shouldn't impact your entire stack. Trust me this will hurt you badly as soon as you need downtime for an upgrade. The cost to scale any individual component increases drastically if you want to try and scale on multiple similar servers (which will save you management dollars by reducing complexity) or it will cost you to scale out adhoc hardware as needed because you then need to deal with many different-sized hosts that are a pain to manage and require anyone managing them to know the platform intimately. recommendation: split your infrastructure into as small a unit as is logical. If you can only afford to buy a single large bit of kit then make sure you split it into logically separate VMs or containers so that if you need to pop a bit of your infrastructure onto another server, or you need to scale stuff at different rates you can easily expand out but investing in a smaller upgrade. Obviously the realities of software licencing, especially for windows OS's might make this cost prohibitive but, while I haven't caught up on windows licensing models in a while, I'm sure there are development centric licensing models that can suit your specific needs.
Z

Where to Store Container Configuration
Continuous Integration and Delivery (CI, CD) • configuration deployment containers environments • • zymir

2

0
Votes

2
Posts

1
Views

S

we do this with .conf files and loading them via compose env_file: /path/to/our.conf they can also be used with docker run --env-file /path/to/our.conf ourcontainer:version In some cases we use the variables defined in our conf files directly, meaning the code we are running reads them direct, in others we need to modify other files on a per environment basis so we use sed or other tools to make changes to a file we have turned into a template
R

Preventing accidental production deployments
Continuous Integration and Delivery (CI, CD) • ansible environments ansible vault • • Rossere

2

0
Votes

2
Posts

1
Views

T

The best practice is to separate production and staging(testing) inventory. See Staging vs Production. This way you pick with -i what you are targeting. To make it more robust you might want to clone a separate structure for production and restrict access to it. Pull from staging before deploying to production. In detail, this would mean to 1) put the directories into git, 2) make sure staging is committed, pushed and merged, and 3) then pull to the production and deploy. The workflow is up to you.
E

Options to build a production server as a solo python developer?
Continuous Integration and Delivery (CI, CD) • python servers • • emran

2

0
Votes

2
Posts

52
Views

A

One option that you could consider is using a continuous deployment tool like Jenkins, Travis CI, or GitLab CI. These tools can automatically deploy your code to a production server whenever you push your code to a certain branch in your git repository. This can save you the time and effort of manually deploying your code to your production server every time you make a change. Another option is to use a service like Heroku, which makes it easy to deploy and manage your web applications. With Heroku, you can simply push your code to a git repository, and Heroku will automatically build and deploy your application. Ultimately, the best option will depend on your specific needs and preferences, so it may be worth experimenting with different approaches to see what works best for you.
Is Azure's B-2 app service plan better than a S-1 for production environments?
Continuous Integration and Delivery (CI, CD) • azure • • inna

2

0
Votes

2
Posts

1
Views

N

The B-series has this name from burstable adjective. It means that you do not have 100% of the cores for 100% of the time (see Introducing B-Series, our new burstable VM size). For example B-2 you have guaranteed only 40% out of a potential 200%. The suggestion to pick a different serie for production workload, is to guarantee the CPU available.
C

Active directory integration issue with Jenkins
Continuous Integration and Delivery (CI, CD) • jenkins active directory • • chanisef

2

0
Votes

2
Posts

1
Views

N

We had to mention the specific DC on the Site field
What's the term for moving/promoting code between deployment environments
Continuous Integration and Delivery (CI, CD) • terminology deployment release management • • Laycee

2

0
Votes

2
Posts

1
Views

K

I don't think there is a generic term for promoting to production. I can tell you that within the devops community, terms are frequently interchanged. As long as the point being addressed is well thought out and made clear, the point can be understood. Some of the terms that are used when referring to the movement between environments are deploy, propagate, move, promote, and release. I will say though, whenever I hear the term stage, I usually associate it with a pre-production environment. I think most others would assume the same. Within your organization/groups, I think it is important to try to be consistent with your terminology. This eliminates any confusion and assumptions people can make. An example would be, in my organization, we use the term deploy whenever we are referencing moving between environments except when referring to the production environment which we use the term release. This is to emphasize that it is the final movement in the pipeline/process.
E

Should I shut down the staging environment when the deployment test ends?
Continuous Integration and Delivery (CI, CD) • testing deployment amazon ec2 environments • • emran

2

0
Votes

2
Posts

1
Views

You can approach this case with multiple ways. For example you can set an auto-shutdown for 20:00 pm and auto-start for 8:00 am. Use Terraform to create your environment and, after deployment/testings, you can easily destroy it, saving your costs. And many other ways. I commented on paths that you can follow, so you should choose one that brings you greater benefit and agility both in creating your environments and in testing them. Regards
F

Force script to fail if "npm install" has network issues
Continuous Integration and Delivery (CI, CD) • npm packer • • fbio

2

0
Votes

2
Posts

1
Views

G

Try piping your output to grep and then failing based off of the return code of grep: npm install 2>&1 | grep "Error: connect ETIMEDOUT" Per the grep documentation, the exit status is 0 if a line is selected, 1 if no lines were selected, and 2 if an error occurred. If needed, you can "not" the return code or just invert the conditional so that: if [ "$?" -ne "0" ]; then echo "Packer failed!" exit 1 fi becomes instead: if [ "$?" -ne "1" ]; then echo "Packer failed!" exit 1 fi
How to put production-like data into version control
Continuous Integration and Delivery (CI, CD) • data versioning environments • • morde

2

0
Votes

2
Posts

0
Views

N

it depends a bit what kind of cms you use, most cms offer a way to package data in to packages that you can check in to git and deploy to databases almost like code to webapps. For example, we work with Sitecore and it lets you use TDS or Unicorn, other big CMS use different tools but mostly the same concept. The tools let you pick items from your Database and serialise them in to code/text files that are packaged and check in to Git, lets say you have a copy of the database on your Development Workstation, you make some updates to the database and you serialise and package the change with TDS in to a package that you check in to your Development branch. When you deploy the development branch to your Staging environment the TDS package is applied and makes the same updates to the database of your Staging environments.
Z

Mitigating Docker Compliance Issues - Can I put everything in the `daemon.json`?
Continuous Integration and Delivery (CI, CD) • security docker devsecops • • zymir

2

0
Votes

2
Posts

1
Views

E

Check out the dockerd options. This will apply to all containers managed by the Docker Daemon. Docker also has quite a bit of documentation on how to enforce different standards and compliance here.

Growing local development environment issues

Suggested Topics

How to mitigate issues in an IIS on-prem application calling an unreliable cloud vendor API? Continuous Integration and Delivery (CI, CD) • windows api servers • • Raziyah00

What are best practices for over-committing resources in a staging environment? Continuous Integration and Delivery (CI, CD) • continuous integration deployment continuous deployment autoscaling • • Raziyah00

Error when using 'ansible-local' provisioner with Docker container Continuous Integration and Delivery (CI, CD) • docker ansible packer • • Cleofas

How to handle resource creation? Continuous Integration and Delivery (CI, CD) • azure devops tfs microservices release management environments • • emran

VM Hosts: Single Large Host or Few Smaller Hosts? Continuous Integration and Delivery (CI, CD) • servers environments • • rielyn

Where to Store Container Configuration Continuous Integration and Delivery (CI, CD) • configuration deployment containers environments • • zymir

Preventing accidental production deployments Continuous Integration and Delivery (CI, CD) • ansible environments ansible vault • • Rossere

Options to build a production server as a solo python developer? Continuous Integration and Delivery (CI, CD) • python servers • • emran

Is Azure's B-2 app service plan better than a S-1 for production environments? Continuous Integration and Delivery (CI, CD) • azure • • inna

Active directory integration issue with Jenkins Continuous Integration and Delivery (CI, CD) • jenkins active directory • • chanisef

What's the term for moving/promoting code between deployment environments Continuous Integration and Delivery (CI, CD) • terminology deployment release management • • Laycee

Should I shut down the staging environment when the deployment test ends? Continuous Integration and Delivery (CI, CD) • testing deployment amazon ec2 environments • • emran

Force script to fail if "npm install" has network issues Continuous Integration and Delivery (CI, CD) • npm packer • • fbio

How to put production-like data into version control Continuous Integration and Delivery (CI, CD) • data versioning environments • • morde

Mitigating Docker Compliance Issues - Can I put everything in the `daemon.json`? Continuous Integration and Delivery (CI, CD) • security docker devsecops • • zymir

How to mitigate issues in an IIS on-prem application calling an unreliable cloud vendor API?
Continuous Integration and Delivery (CI, CD) • windows api servers • • Raziyah00

What are best practices for over-committing resources in a staging environment?
Continuous Integration and Delivery (CI, CD) • continuous integration deployment continuous deployment autoscaling • • Raziyah00

Error when using 'ansible-local' provisioner with Docker container
Continuous Integration and Delivery (CI, CD) • docker ansible packer • • Cleofas

How to handle resource creation?
Continuous Integration and Delivery (CI, CD) • azure devops tfs microservices release management environments • • emran

VM Hosts: Single Large Host or Few Smaller Hosts?
Continuous Integration and Delivery (CI, CD) • servers environments • • rielyn

Where to Store Container Configuration
Continuous Integration and Delivery (CI, CD) • configuration deployment containers environments • • zymir

Preventing accidental production deployments
Continuous Integration and Delivery (CI, CD) • ansible environments ansible vault • • Rossere

Options to build a production server as a solo python developer?
Continuous Integration and Delivery (CI, CD) • python servers • • emran

Is Azure's B-2 app service plan better than a S-1 for production environments?
Continuous Integration and Delivery (CI, CD) • azure • • inna

Active directory integration issue with Jenkins
Continuous Integration and Delivery (CI, CD) • jenkins active directory • • chanisef

What's the term for moving/promoting code between deployment environments
Continuous Integration and Delivery (CI, CD) • terminology deployment release management • • Laycee

Should I shut down the staging environment when the deployment test ends?
Continuous Integration and Delivery (CI, CD) • testing deployment amazon ec2 environments • • emran

Force script to fail if "npm install" has network issues
Continuous Integration and Delivery (CI, CD) • npm packer • • fbio

How to put production-like data into version control
Continuous Integration and Delivery (CI, CD) • data versioning environments • • morde

Mitigating Docker Compliance Issues - Can I put everything in the `daemon.json`?
Continuous Integration and Delivery (CI, CD) • security docker devsecops • • zymir