Cloud Runner v0 - Reliable and trimmed down cloud runner (#353)

* Update cloud-runner-aws-pipeline.yml * Update cloud-runner-k8s-pipeline.yml * yarn build * yarn build * correct branch ref * correct branch ref passed to target repo * Create k8s-tests.yml * Delete k8s-tests.yml * correct branch ref passed to target repo * correct branch ref passed to target repo * Always describe AWS tasks for now, because unstable error handling * Remove unused tree commands * Use lfs guid sum * Simple override cache push * Simple override cache push and pull override to allow pure cloud storage driven caching * Removal of early branch (breaks lfs caching) * Remove unused tree commands * Update action.yml * Update action.yml * Support cache and input override commands as input + full support custom hooks * Increase k8s timeout * replace filename being appended for unknclear reason * cache key should not contain whitespaces * Always try and deploy rook for k8s * Apply k8s files for rook * Update action.yml * Apply k8s files for rook * Apply k8s files for rook * cache test and action description for kuber storage class * Correct test and implement dependency health check and start * GCP-secret run, cache key * lfs smudge set explicit and undo explicit * Run using external secret provider to speed up input * Update cloud-runner-aws-pipeline.yml * Add nodejs as build step dependency * Add nodejs as build step dependency * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * better defaults for new inputs * better defaults * merge latest * force build update * use npm n to update node in unity builder * use npm n to update node in unity builder * use npm n to update node in unity builder * correct new line * quiet zipping * quiet zipping * default secrets for unity username and password * default secrets for unity username and password * ls active directory before lfs install * Get cloud runner secrets from * Get cloud runner secrets from * Cleanup setup of default secrets * Various fixes * Cleanup setup of default secrets * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * AWS secrets manager support * less caching logs * default k8s storage class to pd-standard * more readable build commands * Capture aws exit code 1 reliably * Always replace /head from branch * k8s default storage class to standard-rwo * cleanup * further cleanup input * further cleanup input * further cleanup input * further cleanup input * further cleanup input * folder sizes to inspect caching * dir command for local cloud runner test * k8s wait for pending because pvc will not create earlier * prefer k8s standard storage * handle empty string as cloud runner cluster input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * fix unterminated quote * fix unterminated quote * do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name * remove head and heads from branch prefix * fix reversed caching direction of cache-push * fixes * fixes * fixes * cachePull cli * fixes * fixes * fixes * fixes * fixes * order cache test to be first * order cache test to be first * fixes * populate cache key instead of using branch * cleanup cli * garbage-collect-aws cli can iterate over aws resources and cli scans all ts files * import cli methods * import cli files explicitly * import cli files explicitly * import cli files explicitly * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * Cloud runner param test before caching because we have a fast local cache test now * Using custom build path relative to repo root rather than project root * aws-garbage-collect at end of pipeline * aws-garbage-collect do not actually delete anything for now - just list * remove some legacy du commands * Update cloud-runner-aws-pipeline.yml * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * PR comments * Replace guid with uuid package * use fileExists lambda instead of stat to check file exists in caching * build failed results in core error message * Delete sample.txt
2026-02-03 15:39:07 +08:00 · 2022-04-11 00:00:37 +01:00
parent 2b399b2641
commit a61c02481f
65 changed files with 10262 additions and 2259 deletions
--- a/src/model/cloud-runner/services/cloud-runner-build-command-process.ts
+++ b/src/model/cloud-runner/services/cloud-runner-build-command-process.ts
@@ -1,25 +1,27 @@
-import { BuildParameters, Input } from '../..';
+import { BuildParameters } from '../..';
 import YAML from 'yaml';
 import CloudRunnerSecret from './cloud-runner-secret';
+import CloudRunner from '../cloud-runner';

 export class CloudRunnerBuildCommandProcessor {
  public static ProcessCommands(commands: string, buildParameters: BuildParameters): string {
-    const hooks = CloudRunnerBuildCommandProcessor.getHooks().filter((x) => x.step.includes(`all`));
+    const hooks = CloudRunnerBuildCommandProcessor.getHooks(buildParameters.customJobHooks).filter((x) =>
+      x.step.includes(`all`),
+    );

    return `echo "---"
      echo "start cloud runner init"
-      ${Input.cloudRunnerTests ? '' : '#'} printenv
-      echo "start cloud runner job"
+      ${CloudRunner.buildParameters.cloudRunnerIntegrationTests ? '' : '#'} printenv
+      echo "start of cloud runner job"
      ${hooks.filter((x) => x.hook.includes(`before`)).map((x) => x.commands) || ' '}
      ${commands}
      ${hooks.filter((x) => x.hook.includes(`after`)).map((x) => x.commands) || ' '}
-      echo "end of cloud runner job
-      ---${buildParameters.logId}"
-    `;
+      echo "end of cloud runner job"
+      echo "---${buildParameters.logId}"`;
  }

-  public static getHooks(): Hook[] {
-    const experimentHooks = process.env.EXPERIMENTAL_HOOKS;
+  public static getHooks(customJobHooks): Hook[] {
+    const experimentHooks = customJobHooks;
    let output = new Array<Hook>();
    if (experimentHooks && experimentHooks !== '') {
      try {
--- a/src/model/cloud-runner/services/cloud-runner-folders.ts
+++ b/src/model/cloud-runner/services/cloud-runner-folders.ts
@@ -0,0 +1,73 @@
+import path from 'path';
+import { CloudRunner } from '../..';
+
+export class CloudRunnerFolders {
+  public static readonly repositoryFolder = 'repo';
+
+  // only the following paths that do not start a path.join with another "Full" suffixed property need to start with an absolute /
+
+  public static get uniqueCloudRunnerJobFolderAbsolute(): string {
+    return path.join(`/`, CloudRunnerFolders.buildVolumeFolder, CloudRunner.buildParameters.buildGuid);
+  }
+
+  public static get cacheFolderFull(): string {
+    return path.join(
+      '/',
+      CloudRunnerFolders.buildVolumeFolder,
+      CloudRunnerFolders.cacheFolder,
+      CloudRunner.buildParameters.cacheKey,
+    );
+  }
+
+  public static get builderPathAbsolute(): string {
+    return path.join(CloudRunnerFolders.uniqueCloudRunnerJobFolderAbsolute, `builder`);
+  }
+
+  public static get repoPathAbsolute(): string {
+    return path.join(CloudRunnerFolders.uniqueCloudRunnerJobFolderAbsolute, CloudRunnerFolders.repositoryFolder);
+  }
+
+  public static get projectPathAbsolute(): string {
+    return path.join(CloudRunnerFolders.repoPathAbsolute, CloudRunner.buildParameters.projectPath);
+  }
+
+  public static get libraryFolderAbsolute(): string {
+    return path.join(CloudRunnerFolders.projectPathAbsolute, `Library`);
+  }
+
+  public static get projectBuildFolderAbsolute(): string {
+    return path.join(CloudRunnerFolders.repoPathAbsolute, CloudRunner.buildParameters.buildPath);
+  }
+
+  public static get lfsFolderAbsolute(): string {
+    return path.join(CloudRunnerFolders.repoPathAbsolute, `.git`, `lfs`);
+  }
+
+  public static get purgeRemoteCaching(): boolean {
+    return process.env.PURGE_REMOTE_BUILDER_CACHE !== undefined;
+  }
+
+  public static get lfsCacheFolderFull() {
+    return path.join(CloudRunnerFolders.cacheFolderFull, `lfs`);
+  }
+
+  public static get libraryCacheFolderFull() {
+    return path.join(CloudRunnerFolders.cacheFolderFull, `Library`);
+  }
+
+  public static get unityBuilderRepoUrl(): string {
+    return `https://${CloudRunner.buildParameters.gitPrivateToken}@github.com/game-ci/unity-builder.git`;
+  }
+
+  public static get targetBuildRepoUrl(): string {
+    return `https://${CloudRunner.buildParameters.gitPrivateToken}@github.com/${CloudRunner.buildParameters.githubRepo}.git`;
+  }
+
+  public static get buildVolumeFolder() {
+    return 'data';
+  }
+
+  public static get cacheFolder() {
+    return 'cache';
+  }
+}
--- a/src/model/cloud-runner/services/cloud-runner-namespace.ts
+++ b/src/model/cloud-runner/services/cloud-runner-namespace.ts
@@ -2,7 +2,7 @@ import { customAlphabet } from 'nanoid';
 import CloudRunnerConstants from './cloud-runner-constants';

 class CloudRunnerNamespace {
-  static generateBuildName(runNumber: string | number, platform: string) {
+  static generateGuid(runNumber: string | number, platform: string) {
    const nanoid = customAlphabet(CloudRunnerConstants.alphabet, 4);
    return `${runNumber}-${platform.toLowerCase().replace('standalone', '')}-${nanoid()}`;
  }
--- a/src/model/cloud-runner/services/cloud-runner-provider-interface.ts
+++ b/src/model/cloud-runner/services/cloud-runner-provider-interface.ts
@@ -1,42 +0,0 @@
-import BuildParameters from '../../build-parameters';
-import CloudRunnerEnvironmentVariable from './cloud-runner-environment-variable';
-import CloudRunnerSecret from './cloud-runner-secret';
-
-export interface CloudRunnerProviderInterface {
-  cleanupSharedResources(
-    // eslint-disable-next-line no-unused-vars
-    buildGuid: string,
-    // eslint-disable-next-line no-unused-vars
-    buildParameters: BuildParameters,
-    // eslint-disable-next-line no-unused-vars
-    branchName: string,
-    // eslint-disable-next-line no-unused-vars
-    defaultSecretsArray: { ParameterKey: string; EnvironmentVariable: string; ParameterValue: string }[],
-  );
-  setupSharedResources(
-    // eslint-disable-next-line no-unused-vars
-    buildGuid: string,
-    // eslint-disable-next-line no-unused-vars
-    buildParameters: BuildParameters,
-    // eslint-disable-next-line no-unused-vars
-    branchName: string,
-    // eslint-disable-next-line no-unused-vars
-    defaultSecretsArray: { ParameterKey: string; EnvironmentVariable: string; ParameterValue: string }[],
-  );
-  runTask(
-    // eslint-disable-next-line no-unused-vars
-    buildGuid: string,
-    // eslint-disable-next-line no-unused-vars
-    image: string,
-    // eslint-disable-next-line no-unused-vars
-    commands: string,
-    // eslint-disable-next-line no-unused-vars
-    mountdir: string,
-    // eslint-disable-next-line no-unused-vars
-    workingdir: string,
-    // eslint-disable-next-line no-unused-vars
-    environment: CloudRunnerEnvironmentVariable[],
-    // eslint-disable-next-line no-unused-vars
-    secrets: CloudRunnerSecret[],
-  ): Promise<string>;
-}
--- a/src/model/cloud-runner/services/cloud-runner-query-override.ts
+++ b/src/model/cloud-runner/services/cloud-runner-query-override.ts
@@ -0,0 +1,59 @@
+import Input from '../../input';
+import { GenericInputReader } from '../../input-readers/generic-input-reader';
+
+const formatFunction = (value, arguments_) => {
+  for (const element of arguments_) {
+    value = value.replace(`{${element.key}}`, element.value);
+  }
+  return value;
+};
+
+class CloudRunnerQueryOverride {
+  static queryOverrides: any;
+
+  public static query(key, alternativeKey) {
+    if (CloudRunnerQueryOverride.queryOverrides && CloudRunnerQueryOverride.queryOverrides[key] !== undefined) {
+      return CloudRunnerQueryOverride.queryOverrides[key];
+    }
+    if (
+      CloudRunnerQueryOverride.queryOverrides &&
+      alternativeKey &&
+      CloudRunnerQueryOverride.queryOverrides[alternativeKey] !== undefined
+    ) {
+      return CloudRunnerQueryOverride.queryOverrides[alternativeKey];
+    }
+    return;
+  }
+
+  private static shouldUseOverride(query) {
+    if (Input.readInputOverrideCommand() !== '') {
+      if (Input.readInputFromOverrideList() !== '') {
+        const doesInclude =
+          Input.readInputFromOverrideList().split(',').includes(query) ||
+          Input.readInputFromOverrideList().split(',').includes(Input.ToEnvVarFormat(query));
+        return doesInclude ? true : false;
+      } else {
+        return true;
+      }
+    }
+  }
+
+  private static async queryOverride(query) {
+    if (!this.shouldUseOverride(query)) {
+      throw new Error(`Should not be trying to run override query on ${query}`);
+    }
+
+    return await GenericInputReader.Run(formatFunction(Input.readInputOverrideCommand(), [{ key: 0, value: query }]));
+  }
+
+  public static async PopulateQueryOverrideInput() {
+    const queries = Input.readInputFromOverrideList().split(',');
+    CloudRunnerQueryOverride.queryOverrides = new Array();
+    for (const element of queries) {
+      if (CloudRunnerQueryOverride.shouldUseOverride(element)) {
+        CloudRunnerQueryOverride.queryOverrides[element] = await CloudRunnerQueryOverride.queryOverride(element);
+      }
+    }
+  }
+}
+export default CloudRunnerQueryOverride;
--- a/src/model/cloud-runner/services/cloud-runner-system.ts
+++ b/src/model/cloud-runner/services/cloud-runner-system.ts
@@ -0,0 +1,45 @@
+import { exec } from 'child_process';
+import { RemoteClientLogger } from '../remote-client/remote-client-logger';
+
+export class CloudRunnerSystem {
+  public static async Run(command: string, suppressError = false, suppressLogs = false) {
+    for (const element of command.split(`\n`)) {
+      if (!suppressLogs) {
+        RemoteClientLogger.log(element);
+      }
+    }
+    return await new Promise<string>((promise, throwError) => {
+      let output = '';
+      const child = exec(command, (error, stdout, stderr) => {
+        if (!suppressError && error) {
+          RemoteClientLogger.log(error.toString());
+          throwError(error);
+        }
+        if (stderr) {
+          const diagnosticOutput = `${stderr.toString()}`;
+          if (!suppressLogs) {
+            RemoteClientLogger.logCliDiagnostic(diagnosticOutput);
+          }
+          output += diagnosticOutput;
+        }
+        const outputChunk = `${stdout}`;
+        output += outputChunk;
+      });
+      child.on('close', (code) => {
+        if (!suppressLogs) {
+          RemoteClientLogger.log(`[${code}]`);
+        }
+        if (code !== 0 && !suppressError) {
+          throwError(output);
+        }
+        const outputLines = output.split(`\n`);
+        for (const element of outputLines) {
+          if (!suppressLogs) {
+            RemoteClientLogger.log(element);
+          }
+        }
+        promise(output);
+      });
+    });
+  }
+}
--- a/src/model/cloud-runner/services/depdency-override-service.ts
+++ b/src/model/cloud-runner/services/depdency-override-service.ts
@@ -0,0 +1,21 @@
+import Input from '../../input';
+import { CloudRunnerSystem } from './cloud-runner-system';
+
+class DependencyOverrideService {
+  public static async CheckHealth() {
+    if (Input.checkDependencyHealthOverride) {
+      try {
+        await CloudRunnerSystem.Run(Input.checkDependencyHealthOverride);
+      } catch {
+        return false;
+      }
+    }
+    return true;
+  }
+  public static async TryStartDependencies() {
+    if (Input.startDependenciesOverride) {
+      await CloudRunnerSystem.Run(Input.startDependenciesOverride);
+    }
+  }
+}
+export default DependencyOverrideService;
--- a/src/model/cloud-runner/services/lfs-hashing.ts
+++ b/src/model/cloud-runner/services/lfs-hashing.ts
@@ -0,0 +1,45 @@
+import path from 'path';
+import { CloudRunnerFolders } from './cloud-runner-folders';
+import { CloudRunnerSystem } from './cloud-runner-system';
+import fs from 'fs';
+import { assert } from 'console';
+import { Cli } from '../../cli/cli';
+import { CliFunction } from '../../cli/cli-functions-repository';
+
+export class LfsHashing {
+  public static async createLFSHashFiles() {
+    try {
+      await CloudRunnerSystem.Run(`git lfs ls-files -l | cut -d ' ' -f1 | sort > .lfs-assets-guid`);
+      await CloudRunnerSystem.Run(`md5sum .lfs-assets-guid > .lfs-assets-guid-sum`);
+      assert(fs.existsSync(`.lfs-assets-guid-sum`));
+      assert(fs.existsSync(`.lfs-assets-guid`));
+      const lfsHashes = {
+        lfsGuid: fs
+          .readFileSync(`${path.join(CloudRunnerFolders.repoPathAbsolute, `.lfs-assets-guid`)}`, 'utf8')
+          .replace(/\n/g, ``),
+        lfsGuidSum: fs
+          .readFileSync(`${path.join(CloudRunnerFolders.repoPathAbsolute, `.lfs-assets-guid-sum`)}`, 'utf8')
+          .replace('  .lfs-assets-guid', '')
+          .replace(/\n/g, ``),
+      };
+      return lfsHashes;
+    } catch (error) {
+      throw error;
+    }
+  }
+  public static async hashAllFiles(folder: string) {
+    const startPath = process.cwd();
+    process.chdir(folder);
+    const result = await (await CloudRunnerSystem.Run(`find -type f -exec md5sum "{}" + | sort | md5sum`))
+      .replace(/\n/g, '')
+      .split(` `)[0];
+    process.chdir(startPath);
+    return result;
+  }
+
+  @CliFunction(`hash`, `hash all folder contents`)
+  static async hash() {
+    const folder = Cli.options['cachePushFrom'];
+    LfsHashing.hashAllFiles(folder);
+  }
+}
--- a/src/model/cloud-runner/services/task-parameter-serializer.ts
+++ b/src/model/cloud-runner/services/task-parameter-serializer.ts
@@ -1,24 +1,24 @@
-import { Input } from '../..';
+import { CloudRunner, Input } from '../..';
 import ImageEnvironmentFactory from '../../image-environment-factory';
 import CloudRunnerEnvironmentVariable from './cloud-runner-environment-variable';
-import { CloudRunnerState } from '../state/cloud-runner-state';
 import { CloudRunnerBuildCommandProcessor } from './cloud-runner-build-command-process';
+import CloudRunnerSecret from './cloud-runner-secret';
+import CloudRunnerQueryOverride from './cloud-runner-query-override';

 export class TaskParameterSerializer {
  public static readBuildEnvironmentVariables(): CloudRunnerEnvironmentVariable[] {
-    TaskParameterSerializer.setupDefaultSecrets();
    return [
      {
        name: 'ContainerMemory',
-        value: CloudRunnerState.buildParams.cloudRunnerMemory,
+        value: CloudRunner.buildParameters.cloudRunnerMemory,
      },
      {
        name: 'ContainerCpu',
-        value: CloudRunnerState.buildParams.cloudRunnerCpu,
+        value: CloudRunner.buildParameters.cloudRunnerCpu,
      },
      {
        name: 'BUILD_TARGET',
-        value: CloudRunnerState.buildParams.targetPlatform,
+        value: CloudRunner.buildParameters.targetPlatform,
      },
      ...TaskParameterSerializer.serializeBuildParamsAndInput,
    ];
@@ -27,7 +27,7 @@ export class TaskParameterSerializer {
    let array = new Array();
    array = TaskParameterSerializer.readBuildParameters(array);
    array = TaskParameterSerializer.readInput(array);
-    const configurableHooks = CloudRunnerBuildCommandProcessor.getHooks();
+    const configurableHooks = CloudRunnerBuildCommandProcessor.getHooks(CloudRunner.buildParameters.customJobHooks);
    const secrets = configurableHooks.map((x) => x.secrets).filter((x) => x !== undefined && x.length > 0);
    if (secrets.length > 0) {
      // eslint-disable-next-line unicorn/no-array-reduce
@@ -46,14 +46,14 @@ export class TaskParameterSerializer {
  }

  private static readBuildParameters(array: any[]) {
-    const keys = Object.keys(CloudRunnerState.buildParams);
+    const keys = Object.keys(CloudRunner.buildParameters);
    for (const element of keys) {
      array.push({
        name: element,
-        value: CloudRunnerState.buildParams[element],
+        value: CloudRunner.buildParameters[element],
      });
    }
-    array.push({ name: 'buildParameters', value: JSON.stringify(CloudRunnerState.buildParams) });
+    array.push({ name: 'buildParameters', value: JSON.stringify(CloudRunner.buildParameters) });
    return array;
  }

@@ -70,16 +70,40 @@ export class TaskParameterSerializer {
    return array;
  }

-  private static setupDefaultSecrets() {
-    if (CloudRunnerState.defaultSecrets === undefined)
-      CloudRunnerState.defaultSecrets = ImageEnvironmentFactory.getEnvironmentVariables(
-        CloudRunnerState.buildParams,
-      ).map((x) => {
-        return {
-          ParameterKey: x.name,
-          EnvironmentVariable: x.name,
-          ParameterValue: x.value,
-        };
+  public static readDefaultSecrets(): CloudRunnerSecret[] {
+    let array = new Array();
+    array = TaskParameterSerializer.tryAddInput(array, 'UNITY_SERIAL');
+    array = TaskParameterSerializer.tryAddInput(array, 'UNITY_EMAIL');
+    array = TaskParameterSerializer.tryAddInput(array, 'UNITY_PASSWORD');
+    array.push(
+      ...ImageEnvironmentFactory.getEnvironmentVariables(CloudRunner.buildParameters)
+        .filter((x) => array.every((y) => y.ParameterKey !== x.name))
+        .map((x) => {
+          return {
+            ParameterKey: x.name,
+            EnvironmentVariable: x.name,
+            ParameterValue: x.value,
+          };
+        }),
+    );
+    return array;
+  }
+  private static getValue(key) {
+    return CloudRunnerQueryOverride.queryOverrides !== undefined &&
+      CloudRunnerQueryOverride.queryOverrides[key] !== undefined
+      ? CloudRunnerQueryOverride.queryOverrides[key]
+      : process.env[key];
+  }
+  s;
+  private static tryAddInput(array, key): CloudRunnerSecret[] {
+    const value = TaskParameterSerializer.getValue(key);
+    if (value !== undefined && value !== '') {
+      array.push({
+        ParameterKey: key,
+        EnvironmentVariable: key,
+        ParameterValue: value,
      });
+    }
+    return array;
  }
 }