Generation support for 1.11 - build dev 1.10 pr16 #2060

sdavtaker · 2022-08-26T19:33:51Z

Issue Improvement to build, install, and distribution mechanisms #1888 *

Description of changes:

Updating Generator Java code
Adding message about REGENERATE_CLIENT removal
Refactor generate_sdks.py
Updating Generator templates for 1.10

This change doesn't include regeneration of all clients, only DDB, neither the inclussion of all of them at the generated directory level. Holding that for follow up PR since will create a lot of noise here.

Check all that applies:

Did a review by yourself.
Added proper tests to cover this PR. (If tests are not applicable, explain.)
Checked if this PR is a breaking (APIs have been changed) change.
Checked if this PR will not introduce cross-platform inconsistent behavior.
Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

All cmake modules in cmake were moved to cmake_legacy. Most of the main CMakeLists.txt was moved to cmake_legacy/legacy_main.cmake Option for disabling LEGACY_BUILD was created. But, no implementation yet.

…irectories

New src directory contains core and core unit tests New test directory contains integration tests for selected autogenerated service clients New tools directory contains tools for code generation, special builds, documentation generation, etc... prefetch_crt script removed, it will not longer be needed at 1.10 Some scripts moved to tools will be later consolidated in the build-system This commit is a breaking change but it is too noisy, so making it back to healthy in next commit by only submitting the real changes to keep things together as before.

Build fails from mixed use of types from CRT and SDK that was fixed after the creation of this branch. The rebase is going to be huge, so breaking the push in 2 steps befor continuing.

Introduced generated exported visibility header. Added all core-tests to the build for ctest. All tests build, not all pass, fixes coming up next.

Build and export header changes to use as reference for the required code-generation changes in following commits.

Moved Testing-Resources target to be a library since we use it in all the tests, integ and unit. Fixed missed inclussion in platform testing that prevented symbols to be accessible for the consumers of the library. Added generated Testing-Resources export. Added build of DynamoDB integration tests.

sbiscigl · 2022-08-26T19:50:05Z

...ain/java/com/amazonaws/util/awsclientgenerator/domainmodels/codegeneration/ServiceModel.java

+        if(metadata.getSignatureVersion().equals("v4") || metadata.getSignatureVersion().equals("s3v4")) {
+            return true;
+        }
+        return operations.values().parallelStream().anyMatch(operation -> operation.getSignerName().equals("Aws::Auth::SIGV4_SIGNER"));


parallel steams are generally advised against in the java world because they use the common fork-join pool, meaning they share an executor pool with the rest of the world. I would either use sequential streams which is often faster. or alternatively it can have its own executor pool.

sbiscigl · 2022-08-26T19:52:52Z

...r/src/main/java/com/amazonaws/util/awsclientgenerator/generators/cpp/CppClientGenerator.java

-           if(retryableErrors.contains(error.getName())) {
-               error.setRetryable(true);
-           }
+        for (Error error : serviceModel.getServiceErrors()) {


could be simplified to

error.setRetryable(serviceModel.getServiceErrors() .findAny(error -> retryableErrors.contains(error.getName())))

sbiscigl · 2022-08-26T19:56:09Z

...r/src/main/java/com/amazonaws/util/awsclientgenerator/generators/cpp/CppClientGenerator.java

+        VelocityContext context = createContext(serviceModel);
+
+        List<String> files = new ArrayList<>();
+        for (int i = 0; i < fileList.size(); i++) {


could be simplified using streams to

fileList.stream() .filter(Objects::nonNull) .map(File:: getPathRelativeToRoot) .filter(filePath -> filePath.endsWith("cpp") && filePath.contains("/model/") .map(filePath -> Paths.get(filePath).getFileName().toString()) .collect(Collectors.toList())

sbiscigl · 2022-08-26T19:57:22Z

...ava/com/amazonaws/util/awsclientgenerator/generators/cpp/s3/S3RestXmlCppClientGenerator.java

@@ -123,6 +125,11 @@ public SdkFileEntry[] generateSourceFiles(ServiceModel serviceModel) throws Exce
            replicationStatus.getEnumValues().set(indexOfComplete, "COMPLETED");
        }

+        // Some S3 operations have embedded errors, and we need to search for errors in the response.


i recognize this code 🙂

SergeyRyabinin · 2022-08-26T20:05:52Z

tools/scripts/generate_sdks.py

@@ -1,152 +1,190 @@
-# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.


this is a wrong script to modify.
You could just delete it.

I'm going to do a clean up of all no longer scripts in this directory and merge this back to the right place.
Moving to draft in the mean time.

sdavtaker and others added 30 commits August 19, 2022 11:06

Setting cmake_minimum_required to 3.12 for v1.10

b4ddaa7

Updating README.md with v1.10 new install instructions

4f59ae2

Created LEGACY_BUILD behavior

313978f

All cmake modules in cmake were moved to cmake_legacy. Most of the main CMakeLists.txt was moved to cmake_legacy/legacy_main.cmake Option for disabling LEGACY_BUILD was created. But, no implementation yet.

Moving generated libraries to generated directory

7c80a98

Regeneration of clients using new generated code location

cacd019

Making Docs directory to be lowercase to keep consistent with other d…

2dd9598

…irectories

fixes build all target (#1930)

a27f047

Setting up project version and setting plan for next steps

8f60f9b

Adding support for inferring version validating build_types

d12a5a3

Preparing toolchain and dependencies

cdefc38

Creating the branch paths for Legacy/Non legacy in Core library build

6ba80af

Adding basic CPack support for Archive packages

26260ee

Improving main CMakeLists code organization

34c9853

Removing dependencies from gitignore

d33b119

Moving from test to tests for integration tests location

2c70a54

Basic core build

b56d520

Basic aws-sdk-cpp-core-tests build

bec56d9

Detecting CJSON directories

2620a0e

Extracting TinyXML2 as external dependency

4e50351

Adding CRT to the linkage

07f9538

Partial implementation of build for core-test

2c4fda7

Build fails from mixed use of types from CRT and SDK that was fixed after the creation of this branch. The rebase is going to be huge, so breaking the push in 2 steps befor continuing.

Adjusting paths and dependencies preparing for testing core

80f7826

Added core-tests to build

4b3b522

Introduced generated exported visibility header. Added all core-tests to the build for ctest. All tests build, not all pass, fixes coming up next.

Draft DynamoDB changes for build scripts

a992f33

Build and export header changes to use as reference for the required code-generation changes in following commits.

Moving core test and test resources to test/

ea22a72

Updating Generator Java code

b952437

Adding message about REGENERATE_CLIENT removal

dd82801

Refactor generate_sdks.py

63e159e

Updating Generator templates for 1.10

1204dc7

sdavtaker force-pushed the build-dev-1.10-pr16 branch from c447f4f to 1204dc7 Compare August 26, 2022 19:38

sdavtaker marked this pull request as ready for review August 26, 2022 19:41

sbiscigl approved these changes Aug 26, 2022

View reviewed changes

SergeyRyabinin reviewed Aug 26, 2022

View reviewed changes

sdavtaker marked this pull request as draft August 26, 2022 20:25

sdavtaker force-pushed the build-dev-1.10 branch from ea22a72 to 2420fad Compare September 2, 2022 19:11

sdavtaker changed the title ~~Generation support for 1.10 - build dev 1.10 pr16~~ Generation support for 1.11 - build dev 1.10 pr16 Nov 28, 2022

sdavtaker marked this pull request as ready for review February 15, 2023 18:21

jmklix closed this Jan 26, 2024

jmklix deleted the build-dev-1.10-pr16 branch January 26, 2024 18:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Generation support for 1.11 - build dev 1.10 pr16 #2060

Generation support for 1.11 - build dev 1.10 pr16 #2060

Uh oh!

sdavtaker commented Aug 26, 2022 •

edited

Loading

Uh oh!

sbiscigl Aug 26, 2022

Uh oh!

sbiscigl Aug 26, 2022

Uh oh!

sbiscigl Aug 26, 2022

Uh oh!

sbiscigl Aug 26, 2022

Uh oh!

SergeyRyabinin Aug 26, 2022

Uh oh!

sdavtaker Aug 26, 2022

Uh oh!

Uh oh!

		@@ -1,152 +1,190 @@
		# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Generation support for 1.11 - build dev 1.10 pr16 #2060

Generation support for 1.11 - build dev 1.10 pr16 #2060

Uh oh!

Conversation

sdavtaker commented Aug 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbiscigl Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

sbiscigl Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

sbiscigl Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

sbiscigl Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

SergeyRyabinin Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

sdavtaker Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sdavtaker commented Aug 26, 2022 •

edited

Loading