git/ci, branch v2.21.2

ci: clear and mark MAKEFLAGS exported just once

2019-02-07T19:36:28Z

Clearing it once upfront, and turning all the assignment into appending, would future-proof the code even more, to prevent mistakes the previous one fixed from happening again. Also, mark the variable exported just once at the beginning. There is no point in marking it exported repeatedly. Signed-off-by: Junio C Hamano

ci: make sure we build Git parallel

2019-02-07T19:31:33Z

Commit 2c8921db2b (travis-ci: build with the right compiler, 2019-01-17) started to use MAKEFLAGS to specify which compiler to use to build Git. A bit later, and in a different topic branch commit eaa62291ff (ci: inherit --jobs via MAKEFLAGS in run-build-and-tests, 2019-01-27) started to use MAKEFLAGS as well. Unfortunately, there is a semantic conflict between these two commits: both of them set MAKEFLAGS, and since the line adding CC from 2c8921db2b comes later in 'ci/lib.sh', it overwrites the number of parallel jobs added in eaa62291ff. Consequently, since both commits have been merged all our CI jobs have been building Git, building its documentation, and applying semantic patches sequentially, making all build jobs a bit slower. Running the test suite is unaffected, because the number of test jobs comes from GIT_PROVE_OPTS. Append to MAKEFLAGS when setting the compiler to use, to ensure that the number of parallel jobs to use is preserved. Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano

Merge branch 'sg/travis-osx-brew-breakage-workaround'

2019-02-07T06:05:28Z

The way the OSX build jobs updates its build environment used the "--quiet" option to "brew update" command, but it wasn't all that quiet to be useful. The use of the option has been replaced with an explicit redirection to the /dev/null (which incidentally would have worked around a breakage by recent updates to homebrew, which has fixed itself already). * sg/travis-osx-brew-breakage-workaround: travis-ci: make the OSX build jobs' 'brew update' more quiet

Merge branch 'js/vsts-ci'

2019-02-07T06:05:26Z

Prepare to run test suite on Azure Pipeline. * js/vsts-ci: (22 commits) test-date: drop unused parameter to getnanos() ci: parallelize testing on Windows ci: speed up Windows phase tests: optionally skip bin-wrappers/ t0061: workaround issues with --with-dashes and RUNTIME_PREFIX tests: add t/helper/ to the PATH with --with-dashes mingw: try to work around issues with the test cleanup tests: include detailed trace logs with --write-junit-xml upon failure tests: avoid calling Perl just to determine file sizes README: add a build badge (status of the Azure Pipelines build) mingw: be more generous when wrapping up the setitimer() emulation ci: use git-sdk-64-minimal build artifact ci: add a Windows job to the Azure Pipelines definition Add a build definition for Azure DevOps ci/lib.sh: add support for Azure Pipelines tests: optionally write results as JUnit-style .xml test-date: add a subcommand to measure times in shell scripts ci: use a junction on Windows instead of a symlink ci: inherit --jobs via MAKEFLAGS in run-build-and-tests ci/lib.sh: encapsulate Travis-specific things ...

travis-ci: make the OSX build jobs' 'brew update' more quiet

2019-02-04T18:27:03Z

Before installing the necessary dependencies, our OSX build jobs run 'brew update --quiet'. This is problematic for two reasons: - This '--quiet' flag apparently broke overnight, resulting in errored builds: +brew update --quiet ==> Downloading https://homebrew.bintray.com/bottles-portable-ruby/portable-ruby-2.3.7.mavericks.bottle.tar.gz ######################################################################## 100.0% ==> Pouring portable-ruby-2.3.7.mavericks.bottle.tar.gz Usage: brew update_report [--preinstall] The Ruby implementation of brew update. Never called manually. --preinstall Run in 'auto-update' mode (faster, less output). -f, --force Override warnings and enable potentially unsafe operations. -d, --debug Display any debugging information. -v, --verbose Make some output more verbose. -h, --help Show this message. Error: invalid option: --quiet The command "ci/install-dependencies.sh" failed and exited with 1 during . I belive that this breakage will be noticed and fixed soon-ish, so we could probably just wait a bit for this issue to solve itself, but: - 'brew update --quiet' wasn't really quiet in the first place, as it listed over about 2000 lines worth of available packages that we absolutely don't care about, see e.g. one of the latest 'master' builds: https://travis-ci.org/git/git/jobs/486134962#L113 So drop this '--quiet' option and redirect 'brew update's standard output to /dev/null to make it really quiet, thereby making the OSX builds work again despite the above mentioned breakage. Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano

ci: parallelize testing on Windows

2019-01-29T17:26:47Z

The fact that Git's test suite is implemented in Unix shell script that is as portable as we can muster, combined with the fact that Unix shell scripting is foreign to Windows (and therefore has to be emulated), results in pretty abysmal speed of the test suite on that platform, for pretty much no other reason than that language choice. For comparison: while the Linux build & test is typically done within about 8 minutes, the Windows build & test typically lasts about 80 minutes in Azure Pipelines. To help with that, let's use the Azure Pipeline feature where you can parallelize jobs, make jobs depend on each other, and pass artifacts between them. The tests are distributed using the following heuristic: listing all test scripts ordered by size in descending order (as a cheap way to estimate the overall run time), every Nth script is run (where N is the total number of parallel jobs), starting at the index corresponding to the parallel job. This slicing is performed by a new function that is added to the `test-tool`. To optimize the overall runtime of the entire Pipeline, we need to move the Windows jobs to the beginning (otherwise there would be a very decent chance for the Pipeline to be run only the Windows build, while all the parallel Windows test jobs wait for this single one). We use Azure Pipelines Artifacts for both the minimal Git for Windows SDK as well as the built executables, as deduplication and caching close to the agents makes that really fast. For comparison: while downloading and unpacking the minimal Git for Windows SDK via PowerShell takes only one minute (down from anywhere between 2.5 to 7 when using a shallow clone), uploading it as Pipeline Artifact takes less than 30s and downloading and unpacking less than 20s (sometimes even as little as only twelve seconds). Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano

ci: speed up Windows phase

2019-01-29T17:26:47Z

As Unix shell scripting comes at a hefty price on Windows, we have to see where we can save some time to run the test suite. Let's skip the chain linting and the bin-wrappers/ redirection on Windows; this seems to shave of anywhere between 10-30% from the overall runtime. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano

Add a build definition for Azure DevOps

2019-01-29T17:26:46Z

This commit adds an azure-pipelines.yml file which is Azure DevOps' equivalent to Travis CI's .travis.yml. The main idea is to replicate the Travis configuration as faithfully as possible, to make it easy to compare the Azure Pipeline builds to the Travis ones (spoiler: some parts, especially the macOS jobs, are way faster in Azure Pileines). Meaning: the number and the order of the jobs added in this commit faithfully replicates what we have in .travis.yml. Note: Our .travis.yml configuration has a Windows part that is *not* replicated in the Azure Pipelines definition. The reason is easy to see: As Travis cannot support our Windws needs (even with the preliminary Windows support that was recently added to Travis after waiting for *years* for that feature, our test suite would simply hit Travis' timeout every single time). To make things a bit easier to understand, we refrain from using the `matrix` feature here because (while it is powerful) it can be a bit confusing to users who are not familiar with CI setups. Therefore, we use a separate phase even for similar configurations (such as GCC vs Clang on Linux, GCC vs Clang on macOS). Also, we make use of the shiny new feature we just introduced where the test suite can output JUnit-style .xml files. This information is made available in a nice UI that allows the viewer to filter by phase and/or test number, and to see trends such as: number of (failing) tests, time spent running the test suite, etc. (While this seemingly contradicts the intention to replicate the Travis configuration as faithfully as possible, it is just too nice to show off that capability here already.) Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano

ci/lib.sh: add support for Azure Pipelines

2019-01-29T17:26:46Z

This patch introduces a conditional arm that defines some environment variables and a function that displays the URL given the job id (to identify previous runs for known-good trees). Because Azure Pipeline's macOS agents already have git-lfs and gettext installed, we can leave `BREW_INSTALL_PACKAGES` empty (unlike in Travis' case). Note: this patch does not introduce an Azure Pipelines definition yet; That is left for the next patch. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano

ci: use a junction on Windows instead of a symlink

2019-01-28T18:34:28Z

Symbolic links are still not quite as easy to use on Windows as on Linux (for example, on versions older than Windows 10, only administrators can create symlinks, and on Windows 10 you still need to be in developer mode for regular users to have permission), but NTFS junctions can give us a way out. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano