Files
unity-builder/src/model/cloud-runner/workflows/build-automation-workflow.ts
Frostebite 8abce48a48 Cloud runner v0.2 - continued quality of life improvements (#387)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo

* Always describe AWS tasks for now, because unstable error handling

* Remove unused tree commands

* Use lfs guid sum

* Simple override cache push

* Simple override cache push and pull override to allow pure cloud storage driven caching

* Removal of early branch (breaks lfs caching)

* Remove unused tree commands

* Update action.yml

* Update action.yml

* Support cache and input override commands as input + full support custom hooks

* Increase k8s timeout

* replace filename being appended for unknclear reason

* cache key should not contain whitespaces

* Always try and deploy rook for k8s

* Apply k8s files for rook

* Update action.yml

* Apply k8s files for rook

* Apply k8s files for rook

* cache test and action description for kuber storage class

* Correct test and implement dependency health check and start

* GCP-secret run, cache key

* lfs smudge set explicit and undo explicit

* Run using external secret provider to speed up input

* Update cloud-runner-aws-pipeline.yml

* Add nodejs as build step dependency

* Add nodejs as build step dependency

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* better defaults for new inputs

* better defaults

* merge latest

* force build update

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* correct new line

* quiet zipping

* quiet zipping

* default secrets for unity username and password

* default secrets for unity username and password

* ls active directory before lfs install

* Get cloud runner secrets from

* Get cloud runner secrets from

* Cleanup setup of default secrets

* Various fixes

* Cleanup setup of default secrets

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* AWS secrets manager support

* less caching logs

* default k8s storage class to pd-standard

* more readable build commands

* Capture aws exit code 1 reliably

* Always replace /head from branch

* k8s default storage class to standard-rwo

* cleanup

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* folder sizes to inspect caching

* dir command for local cloud runner test

* k8s wait for pending because pvc will not create earlier

* prefer k8s standard storage

* handle empty string as cloud runner cluster input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* fix unterminated quote

* fix unterminated quote

* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name

* remove head and heads from branch prefix

* fix reversed caching direction of cache-push

* fixes

* fixes

* fixes

* cachePull cli

* fixes

* fixes

* fixes

* fixes

* fixes

* order cache test to be first

* order cache test to be first

* fixes

* populate cache key instead of using branch

* cleanup cli

* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files

* import cli methods

* import cli files explicitly

* import cli files explicitly

* import cli files explicitly

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* Cloud runner param test before caching because we have a fast local cache test now

* Using custom build path relative to repo root rather than project root

* aws-garbage-collect at end of pipeline

* aws-garbage-collect do not actually delete anything for now - just list

* remove some legacy du commands

* Update cloud-runner-aws-pipeline.yml

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* PR comments

* Replace guid with uuid package

* use fileExists lambda instead of stat to check file exists in caching

* build failed results in core error message

* Delete sample.txt

* cloud-runner-system prefix changed to cloud-runner

* Update cloud-runner-aws-pipeline.yml

* remove du from caching, should run manually if interested in size, adds too much runtime to job to include by default

* github ephemeral pipeline support

* github ephemeral pipeline support

* Merge remote-tracking branch 'origin/main' into cloud-runner-develop

# Conflicts:
#	dist/index.js.map
#	src/model/cloud-runner/providers/aws/aws-task-runner.ts
#	src/model/cloud-runner/providers/aws/index.ts

* garbage collection

* garbage collection

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* download runner each time

* download runner each time

* download runner each time

* garbage collect all older than 1d as part of cleanup

* download runner each time

* number container cpu and memory for aws

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* Skip printing size unless cloudRunnerIntegrationTests is true

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* better aws commands

* better aws commands

* parse number for cloud formation template

* remove container resource defaults from actions yaml

* remove container resource defaults from actions yaml

* skip all input readers when cloud runner is local

* prefer fs/promises

* actually set aws cloud runner step as failure if unity build fails

* default to 3gb of ram - webgl fails on 2
2022-04-22 00:47:45 +01:00

124 lines
5.8 KiB
TypeScript

import CloudRunnerLogger from '../services/cloud-runner-logger';
import { CloudRunnerFolders } from '../services/cloud-runner-folders';
import { CloudRunnerStepState } from '../cloud-runner-step-state';
import { CustomWorkflow } from './custom-workflow';
import { WorkflowInterface } from './workflow-interface';
import * as core from '@actions/core';
import { CloudRunnerBuildCommandProcessor } from '../services/cloud-runner-build-command-process';
import path from 'path';
import CloudRunner from '../cloud-runner';
export class BuildAutomationWorkflow implements WorkflowInterface {
async run(cloudRunnerStepState: CloudRunnerStepState) {
try {
return await BuildAutomationWorkflow.standardBuildAutomation(cloudRunnerStepState.image);
} catch (error) {
throw error;
}
}
private static async standardBuildAutomation(baseImage: any) {
try {
CloudRunnerLogger.log(`Cloud Runner is running standard build automation`);
if (!CloudRunner.buildParameters.isCliMode) core.startGroup('pre build steps');
let output = '';
if (CloudRunner.buildParameters.preBuildSteps !== '') {
output += await CustomWorkflow.runCustomJob(CloudRunner.buildParameters.preBuildSteps);
}
if (!CloudRunner.buildParameters.isCliMode) core.endGroup();
CloudRunnerLogger.logWithTime('Configurable pre build step(s) time');
if (!CloudRunner.buildParameters.isCliMode) core.startGroup('build');
CloudRunnerLogger.log(baseImage.toString());
CloudRunnerLogger.logLine(` `);
CloudRunnerLogger.logLine('Starting build automation job');
output += await CloudRunner.Provider.runTask(
CloudRunner.buildParameters.buildGuid,
baseImage.toString(),
BuildAutomationWorkflow.BuildWorkflow,
`/${CloudRunnerFolders.buildVolumeFolder}`,
`/${CloudRunnerFolders.buildVolumeFolder}/`,
CloudRunner.cloudRunnerEnvironmentVariables,
CloudRunner.defaultSecrets,
);
if (!CloudRunner.buildParameters.isCliMode) core.endGroup();
CloudRunnerLogger.logWithTime('Build time');
if (!CloudRunner.buildParameters.isCliMode) core.startGroup('post build steps');
if (CloudRunner.buildParameters.postBuildSteps !== '') {
output += await CustomWorkflow.runCustomJob(CloudRunner.buildParameters.postBuildSteps);
}
if (!CloudRunner.buildParameters.isCliMode) core.endGroup();
CloudRunnerLogger.logWithTime('Configurable post build step(s) time');
CloudRunnerLogger.log(`Cloud Runner finished running standard build automation`);
return output;
} catch (error) {
throw error;
}
}
private static get BuildWorkflow() {
const setupHooks = CloudRunnerBuildCommandProcessor.getHooks(CloudRunner.buildParameters.customJobHooks).filter(
(x) => x.step.includes(`setup`),
);
const buildHooks = CloudRunnerBuildCommandProcessor.getHooks(CloudRunner.buildParameters.customJobHooks).filter(
(x) => x.step.includes(`build`),
);
const builderPath = path.join(CloudRunnerFolders.builderPathAbsolute, 'dist', `index.js`).replace(/\\/g, `/`);
return `apt-get update > /dev/null
apt-get install -y tar tree npm git-lfs jq git > /dev/null
npm install -g n > /dev/null
n stable > /dev/null
${setupHooks.filter((x) => x.hook.includes(`before`)).map((x) => x.commands) || ' '}
export GITHUB_WORKSPACE="${CloudRunnerFolders.repoPathAbsolute.replace(/\\/g, `/`)}"
${BuildAutomationWorkflow.setupCommands(builderPath)}
${setupHooks.filter((x) => x.hook.includes(`after`)).map((x) => x.commands) || ' '}
${buildHooks.filter((x) => x.hook.includes(`before`)).map((x) => x.commands) || ' '}
${BuildAutomationWorkflow.BuildCommands(builderPath, CloudRunner.buildParameters.buildGuid)}
${buildHooks.filter((x) => x.hook.includes(`after`)).map((x) => x.commands) || ' '}`;
}
private static setupCommands(builderPath) {
return `export GIT_DISCOVERY_ACROSS_FILESYSTEM=1
echo "game ci cloud runner clone"
mkdir -p ${CloudRunnerFolders.builderPathAbsolute.replace(/\\/g, `/`)}
git clone -q -b ${CloudRunner.buildParameters.cloudRunnerBranch} ${
CloudRunnerFolders.unityBuilderRepoUrl
} "${CloudRunnerFolders.builderPathAbsolute.replace(/\\/g, `/`)}"
chmod +x ${builderPath}
echo "game ci cloud runner bootstrap"
node ${builderPath} -m remote-cli`;
}
private static BuildCommands(builderPath, guid) {
const linuxCacheFolder = CloudRunnerFolders.cacheFolderFull.replace(/\\/g, `/`);
const distFolder = path.join(CloudRunnerFolders.builderPathAbsolute, 'dist');
const ubuntuPlatformsFolder = path.join(CloudRunnerFolders.builderPathAbsolute, 'dist', 'platforms', 'ubuntu');
return `echo "game ci cloud runner init"
mkdir -p ${`${CloudRunnerFolders.projectBuildFolderAbsolute}/build`.replace(/\\/g, `/`)}
cd ${CloudRunnerFolders.projectPathAbsolute}
cp -r "${path.join(distFolder, 'default-build-script').replace(/\\/g, `/`)}" "/UnityBuilderAction"
cp -r "${path.join(ubuntuPlatformsFolder, 'entrypoint.sh').replace(/\\/g, `/`)}" "/entrypoint.sh"
cp -r "${path.join(ubuntuPlatformsFolder, 'steps').replace(/\\/g, `/`)}" "/steps"
chmod -R +x "/entrypoint.sh"
chmod -R +x "/steps"
echo "game ci cloud runner start"
/entrypoint.sh
echo "game ci cloud runner push library to cache"
chmod +x ${builderPath}
node ${builderPath} -m cache-push --cachePushFrom ${
CloudRunnerFolders.libraryFolderAbsolute
} --artifactName lib-${guid} --cachePushTo ${linuxCacheFolder}/Library
echo "game ci cloud runner push build to cache"
node ${builderPath} -m cache-push --cachePushFrom ${
CloudRunnerFolders.projectBuildFolderAbsolute
} --artifactName build-${guid} --cachePushTo ${`${linuxCacheFolder}/build`.replace(/\\/g, `/`)}`;
}
}