Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you may not be able to execute some actions.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

What happens when two jobs produce artifacts with conflicting locations?

D
Duquan last edited by
In my GitLab CI log I pipeline with three stages
1. build creates a distribution
2. semantic-release which bumps the version
3. publish
The problem occurs in publish. Sometimes I see the version unbumped. You can see here that semantic-release has the higher number 148288 and it's running BEFORE build which overwriting the files. This means this job results in failure,
```
Downloading artifacts for semantic-release (148288)...
Downloading artifacts from coordinator... ok        id=148288 responseStatus=200 OK token=xFzfAzW6
Downloading artifacts for build (148285)...
Downloading artifacts from coordinator... ok        id=148285 responseStatus=200 OK token=zxTTXZoA
```
But sometimes the same pipeline elsewhere is succeeding,
```
Downloading artifacts for build (113579)...
Downloading artifacts from coordinator... ok        id=113579 responseStatus=200 OK token=Cfqjv8bG
Downloading artifacts for semantic-release (113665)...
Downloading artifacts from coordinator... ok        id=113665 responseStatus=200 OK token=GAyN8S9M
```
Notice on this one the build job comes before semantic-release. That means the older artifacts are replaced. What happens if the artifact in semantic-release and build conflict by path? Is this defined?
Reply Quote 0
1 Reply Last reply
inna last edited by

https://gitlab.com/gitlab-org/gitlab/-/issues/324412

With the bug, basically the rule of thumb is this,

A job can rely on an artifact produced by a previous job only so long as no jobs previous to that one produced the same artifact.

The bug report suggests using https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html to only download specific artifacts.
Reply Quote 0
1 Reply Last reply

gitlab 453

2
Posts

0
Views

Suggested Topics

What challenges did you face once you gave devs access to Jenkins jobs?
Continuous Integration and Delivery (CI, CD) • jenkins rundeck • • Alberto

2

0
Votes

2
Posts

1
Views

F

In addition, we allow devs to access Jenkin’s UI to spin up all kinds of jobs This would be the first thing that should be changed. The pipelines should only be started automatically instead of manual. DevOps is a culture and implies that all repetitive work should be automated. Why did this job fail? You should ask them whether they read the log. If the issue in the pipeline is NOT related to their code then the creator of the pipeline should investigate it and make a solid CI. Devs send me messages on slack that they got lost between all the Jenkins jobs, and they don't know which Jenkins Job is relevant for them. "I forgot the identifier of the machine I created" This indicates that the devs see unrelated pipelines. If they create microservices then there would be a dedicated pipeline. If the pipeline is broken then one knows that the issue is related to their code. I had a lot of discussions with devs back in the day. Most of the time they said that it felt superfluous to create another repository for their code, but when I referred to the 12factor app and demonstrated the effect of running a pipeline for every app, they were convinced. Devs make mistakes and deploy to the wrong environment. The one that created the pipelines decides what permissions the user will get. A best practice is to apply the least privilege. In terms of CI this means that the developer will only be capable of reading the log. Once the log is read one should be capable of solving the issue. Sometimes, devs do not run their code locally on their laptop first. Update me if you modify parameters Once the pipeline have been implemented, you should be aware that you should NOT introduce breaking changes. If this is inevitable then you should communicate it well, but you should prevent it at all costs to prevent that the devs will decide not to use the pipelines anymore. It is better to spend some more time to create a solid pipeline, before let the devs use it. Make sure we devs are not spinning to many machines This should be automated as well. You could decided to stop the instances automatically if they are running for more than half an hour. You could introduce this and inform the devs and teamlead about it. Make sure we have an audit log for every dev operation Prevent that the devs login to Jenkins. The pipeline should run automatically and the devs should only be capable of reading the log, see also Devs make mistakes and deploy to the wrong environment Jenkins seems pretty limited to allow these capabilities and I am afraid to create a tech debt and put too much pressure on Jenkins Do you have any suggestions? How do you manage it within your company? Gitlab with a gitlab-ci.yml for each microservice, Gitlab templates, helm and runners on k8s.
P

I can't copy through SSH
Software Programming • ssh github gitlab • • Pearlaqua

2

0
Votes

2
Posts

0
Views

E

Judging by the leg, the key doesn't fit.Check the public key in the designs of the guide user.
M

Jenkins Pipeline "Build when a change is pushed to GitLab"
Continuous Integration and Delivery (CI, CD) • jenkins gitlab jenkins pipeline • • magpie

2

0
Votes

2
Posts

1
Views

A

Looks like you are using the Gitlab plugin for jenkins? if so this link should be of help https://github.com/jenkinsci/gitlab-plugin#declarative-pipeline-jobs
R

Trigger downstream pipeline job via generated variable in upstream freestyle job
Continuous Integration and Delivery (CI, CD) • jenkins jenkins pipeline • • Rossere

2

0
Votes

2
Posts

1
Views

The variable GIT_AUTHOR is only visible in the build step where it's defined. You can dump the parameters for the downstream job in a properties file: cd $WORKSPACE/prancer/ GIT_AUTHOR=$(git show --format="%an" | head -1) echo '-------------' echo 'git_commit is '$GIT_COMMIT echo 'git_branch is '$GIT_BRANCH echo 'git_author is '$GIT_AUTHOR echo '-------------' cat > alex_test_pipeline.properties <<EOF GIT_COMMIT = $GIT_COMMIT GIT_BRANCH = $GIT_BRANCH GIT_AUTHOR = $GIT_AUTHOR EOF In the Trigger parameterized build... step, replace the Predefined parameters with Parameters from properties file (or something like that - I don't remember the exact wording) and select the filename used above. You might also want to check Build only if file exists.
Z

How to fix your branch origin/master have diverged
Software Programming • git gitlab • • zymir

2

0
Votes

2
Posts

0
Views

E

I understand my commits are in different branches. the git log HEAD..origin/master command shows me different commits that the git reflog command.@eftshift0, thanks for the answer, this is the result of the git log* d8782810 (HEAD -> master) Despues de pull * 116a7721 Despues de pull * 4c831a50 previo pull 1648 * a389d4c2 Restaurar commit * dab90fcb Cambios locales previo merge * 501cb77e (remoto) cambios locales previo merge * 2eb72de2 fichero previo commit * 25ca680c Añadir ficheros para el merge * 8d0ae440 commit a readme * 445d6a94 cambios patri * 84a26273 Cambios Patri * da045ac0 Install Share and social follow * fe7a9411 Modulo Wishlist * e54cdbd4 cotejamiento diferencia product-item-17 * 16e4ed89 correccion tpl merger * 5e050d7d cambios gitignore * 49f518c0 actualizacion gitignore * 80c3ae25 Cambios tema mydominio * 60d45fc7 Cambios viernes * 7b272e8c Merge branch 'master' of gldev2.mydominio.eu:dev2/dev2.mydominio.eu |\ * \ 055208b1 Merge branch 'master' of gldev2.mydominio.eu:dev2/dev2.mydominio.eu |\ \ * \ \ 4d633e86 Merge branch 'master' of gldev2.mydominio.eu:dev2/dev2.mydominio.eu |\ \ \ * | | | f51c3f6e index.tpl | | | | * 857e02b4 (origin/master) home page changes | | | | * 526dc591 prueba | | | | * 2785e382 footer and banner harcoded | | | | * 37b3b88a Update themes/mydominio/assets/css/theme.css | | | | * 317ee2d3 modulo blockausserance de la raiz y estilos del tema | | | | * 0a120d72 Merge branch 'master' of gldev2.mydominio.eu:dev2/dev2.mydominio.eu | | | | |\ | | | | | * f41e59e4 style upgrade un template of ressurance module | | | | * | 173b4c65 Merge branch 'master' of gldev2.mydominio.eu:dev2/dev2.mydominio.eu | | | | |\ \ | | | | | |/ | | | | | * 78857847 Add new file | | | | | * e4d54577 Update .gitignore | | | | * | 261de1a7 testing connection | | | | * | 6358c08f testing the repo conection | | | | |/ | | | | * 41e95634 products view on carousel | | | | * fa3d22a9 carousel, footer and styles | | | |/ | | | * 614c159c (origin/develop) Merge branch 'revert-60784ec8' into 'master' | | | |\ | | |/ / | | | * 1e664b77 (origin/revert-60784ec8) Revert "cleaning project, styles guide and carousel changed"
A

How to execute a Jenkins job until the build is green
Continuous Integration and Delivery (CI, CD) • jenkins builds • • Anderson

2

0
Votes

2
Posts

1
Views

R

This pipeline will run each day at 1 AM. It'll restart if the build fails. pipeline { agent any triggers { cron('0 1 * * *') } stages { stage ('Build') { when { expression { // When last build has failed !hudson.model.Result.SUCCESS.equals(currentBuild.rawBuild.getPreviousBuild()?.getResult()) == true } } steps { sh "./myjob.sh" } post { failure { // If the current job has failed, trigger it without // waiting. build job: "${JOB_NAME}", wait: false } } } } } ``` If you want to wait before to re-trigger the job, you can add the option quietPeriod: (seconds) to the build step (https://jenkins.io/doc/pipeline/steps/pipeline-build-step/). You might have to add these methods to the scriptApproval:
A

Unit Test AWS Glue job in CodePipeline
Continuous Integration and Delivery (CI, CD) • python amazon web services aws codepipeline unit testing • • amarantha

2

0
Votes

2
Posts

1
Views

B

Checkout this solution with codepipeline. https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/
E

Error when installing R packages in GitLab CI
Software Programming • gitlab • • emerson

2

0
Votes

2
Posts

0
Views

C

When trying to run the part of R of the code you shared (below), got a syntax error. This happens because in the files *.yml Indentation is important. O job It even started.test: script: - R -e 'install.packages(c("raster", "rgdal"))’ This GitLab CI configuration is invalid: jobs: attest config can't be blankWhen fixing the problem with identification job wheel, but does not find R and fails.test: script: - R -e 'install.packages(c("raster", "rgdal"))' /bin/bash: line 62: R: command not foundFinally, when adding the docker image R (r-base), installing the packages ran smoothly.test: image: rocker/r-base script: - R -e 'install.packages(c("raster", "rgdal"))' For a list with the various images of R in the docker, see https://hub.docker.com/r/rocker/ .The build history made for this answer https://gitlab.com/tomasbarcellos/gitlab_teste/pipelines .
S

Can Gitlab launch a bash screen on the server?
Software Programming • gitlab gitlab ci • • shizuka

2

0
Votes

2
Posts

0
Views

Starting different violin vials for the branch master and for all others through directives except and only♪ Here's the example of the file. .gitlab-ci.yml:image: ... stages: build build_branch: stage: build script: - ./build_branch.sh except: - master build_master: stage: build script: - ./build_master.sh only: - master
C

What are the pros and cons of Azure Artifacts vs. Artifactory?
Continuous Integration and Delivery (CI, CD) • artifactory azure artifacts • • chanisef

2

0
Votes

2
Posts

1
Views

T

JFrog Artifactory pros Artifactory supports 25+ package types, while Azure supports only Maven, npm, NuGet and Python. You can get Docker and Helm with Azure Container Registry but it’s a separate tool. Artifactory is fully hybrid - available both for on-premise installations and as SaaS - with full parity between the versions. With Artifactory you can avoid vendor lock-in - not depending just on one cloud vendor for all of your needs. That also means not being limited to any vendor’s stack for your database, access management, etc. Proxy any remote repository - not just the official ones (like Maven Central and PyPI.org) Artifactory is optimized for multi-site operations - Supports any network topology, over multiple, geographically distant sites and data centers, through different options for replication including pull, push, event-based and multi-push replication. Rich metadata & AQL - Artifactory provides full metadata capabilities for all major package formats for both artifacts and folders – including custom metadata. Users can tag artifacts and folders with searchable properties. Artifactory also collects build information for build artifacts as part of the CI server integration (see below). AQL gives you unprecedented flexibility in how you search for artifacts. It offers a simple way to formulate complex queries that specify any number of search criteria, filters, sorting options and output fields. Build info - Artifactory stores exhaustive metadata for all build artifacts deployed to it. With this “bill of materials” it is easy to faithfully reproduce a build, and trace the cause of issues to reduce time to resolution (TTR), even if the build is already in production. Azure Artifacts pros if your CI is already built on Azure DevOps, the initial set up for Azure Artifacts may be a bit quicker. The basic functionality of hosting artifacts, so that they can be resolved by other builds will work for you. Disclaimer: I'm a JFrog employee.
S

Spark on a Docker swarm - workers won't accept jobs
Continuous Integration and Delivery (CI, CD) • docker • • Saumya

2

0
Votes

2
Posts

1
Views

A

TLDR; set workers' memory to 4G. I reduced the test to the build-in Pi calculation: ./bin/run-example SparkPi 10 And got the same result; apparently 1G memory is insufficient even for simplest experiments. Increasing both master and workers memory to 4G and the problematic behaviour was not reproducible. However before the systems start computation the message still could appear but just once, seems it is printed in a waiting loop.
J

Gitlab CI push to docker-registry fails
Continuous Integration and Delivery (CI, CD) • docker gitlab continuous integration gitlab ci runner • • juvenalb

2

0
Votes

2
Posts

1
Views

T

Are you sure that your registry is publicly available? Checklist: Check if the domain is reachable and responding the right IP address -- nslookup myregistry Check if your port is open or not blocked any kind of software (let's say via firewalls or AWS security group) -- telnet myregistry 5050 Check if your registry is not restricted for a particular CIDR of IP Test your login locally and ensure that there is no problem with it
What is the best way to monitor job status?
Continuous Integration and Delivery (CI, CD) • monitoring • • Bogopo

2

0
Votes

2
Posts

1
Views

M

There could be multiple ways to solve this. Create a python based cronjob that queries the DB every T secs (scraping interval) for the failed jobs in the last T secs (delta). Push the relevant data into a specific index like failed-job in ES. Then use that index to set up alerts in Grafana/opsgenie.(You can also use watcher here) Don't need to read logs. Use prometheus,(if already using), create a new custom prometheus exporter.Exporter would be a Flask web server that provides an endpoint to run the query against the db and return results in a prometheus compatible format to be stored. Configure job in prometheus to scrape this endpoint every T secs. Again, use Grafana for visualization. Use a component called , prometheus-pushgateway, if possible, either push the job status into this component using the python script as in option 1. or modify the Job itself to push the data directly. Again , can be connected to Prometheus/Grafana. If you are trying to solve only a single use case as described here, option 1 would be the better choice. For a larger more complex setups that covers a lot of other use cases, Prometheus might be worth looking into. Alternatively, you can use the same design to push metrics into any datasource.(Like the one you are using for storing CPU/mem metrics, as specified.)
Where to obtain the token to register a GitLab runner?
Continuous Integration and Delivery (CI, CD) • gitlab continuous integration gitlab ci runner • • inna

2

0
Votes

2
Posts

1
Views

E

For a specific GitLab runner tied to a project, the token can be found in Step 3; in Project -> Settings -> CI/CD -> Runners in the Setup a specific Runner manually section. You will also be able to find the URL which you need to use to register the runner in Step 2.
D

Make a job easily searchable in Azure Runbook
Continuous Integration and Delivery (CI, CD) • azure devops • • derk

2

0
Votes

2
Posts

1
Views

AFAIK currently there is no way to label job so that it is easily identifiable from the Overview screen. If interested, you may raise feature request in this UserVoice / feedback forum. I see this related feature request already raised so you may even upvote it. In general, Azure feature team would check feasibility of a feature request, triage it, prioritize against existing feature backlog, add in roadmap as appropriate and would announce and/or update the related Azure document once a feature request is addressed.

What happens when two jobs produce artifacts with conflicting locations?

https://gitlab.com/gitlab-org/gitlab/-/issues/324412

Suggested Topics

What challenges did you face once you gave devs access to Jenkins jobs? Continuous Integration and Delivery (CI, CD) • jenkins rundeck • • Alberto

I can't copy through SSH Software Programming • ssh github gitlab • • Pearlaqua

Jenkins Pipeline "Build when a change is pushed to GitLab" Continuous Integration and Delivery (CI, CD) • jenkins gitlab jenkins pipeline • • magpie

Trigger downstream pipeline job via generated variable in upstream freestyle job Continuous Integration and Delivery (CI, CD) • jenkins jenkins pipeline • • Rossere

How to fix your branch origin/master have diverged Software Programming • git gitlab • • zymir

How to execute a Jenkins job until the build is green Continuous Integration and Delivery (CI, CD) • jenkins builds • • Anderson

Unit Test AWS Glue job in CodePipeline Continuous Integration and Delivery (CI, CD) • python amazon web services aws codepipeline unit testing • • amarantha

Error when installing R packages in GitLab CI Software Programming • gitlab • • emerson

Can Gitlab launch a bash screen on the server? Software Programming • gitlab gitlab ci • • shizuka

What are the pros and cons of Azure Artifacts vs. Artifactory? Continuous Integration and Delivery (CI, CD) • artifactory azure artifacts • • chanisef

Spark on a Docker swarm - workers won't accept jobs Continuous Integration and Delivery (CI, CD) • docker • • Saumya

Gitlab CI push to docker-registry fails Continuous Integration and Delivery (CI, CD) • docker gitlab continuous integration gitlab ci runner • • juvenalb

What is the best way to monitor job status? Continuous Integration and Delivery (CI, CD) • monitoring • • Bogopo

Where to obtain the token to register a GitLab runner? Continuous Integration and Delivery (CI, CD) • gitlab continuous integration gitlab ci runner • • inna

Make a job easily searchable in Azure Runbook Continuous Integration and Delivery (CI, CD) • azure devops • • derk

What challenges did you face once you gave devs access to Jenkins jobs?
Continuous Integration and Delivery (CI, CD) • jenkins rundeck • • Alberto

I can't copy through SSH
Software Programming • ssh github gitlab • • Pearlaqua

Jenkins Pipeline "Build when a change is pushed to GitLab"
Continuous Integration and Delivery (CI, CD) • jenkins gitlab jenkins pipeline • • magpie

Trigger downstream pipeline job via generated variable in upstream freestyle job
Continuous Integration and Delivery (CI, CD) • jenkins jenkins pipeline • • Rossere

How to fix your branch origin/master have diverged
Software Programming • git gitlab • • zymir

How to execute a Jenkins job until the build is green
Continuous Integration and Delivery (CI, CD) • jenkins builds • • Anderson

Unit Test AWS Glue job in CodePipeline
Continuous Integration and Delivery (CI, CD) • python amazon web services aws codepipeline unit testing • • amarantha

Error when installing R packages in GitLab CI
Software Programming • gitlab • • emerson

Can Gitlab launch a bash screen on the server?
Software Programming • gitlab gitlab ci • • shizuka

What are the pros and cons of Azure Artifacts vs. Artifactory?
Continuous Integration and Delivery (CI, CD) • artifactory azure artifacts • • chanisef

Spark on a Docker swarm - workers won't accept jobs
Continuous Integration and Delivery (CI, CD) • docker • • Saumya

Gitlab CI push to docker-registry fails
Continuous Integration and Delivery (CI, CD) • docker gitlab continuous integration gitlab ci runner • • juvenalb

What is the best way to monitor job status?
Continuous Integration and Delivery (CI, CD) • monitoring • • Bogopo

Where to obtain the token to register a GitLab runner?
Continuous Integration and Delivery (CI, CD) • gitlab continuous integration gitlab ci runner • • inna

Make a job easily searchable in Azure Runbook
Continuous Integration and Delivery (CI, CD) • azure devops • • derk