Skip to content

DDB Enhanced: Support arbitrary map fields #1902

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
1 task
PatrickBuTaxdoo opened this issue Jun 17, 2020 · 6 comments
Closed
1 task

DDB Enhanced: Support arbitrary map fields #1902

PatrickBuTaxdoo opened this issue Jun 17, 2020 · 6 comments
Labels
feature-request A feature should be added or improved.

Comments

@PatrickBuTaxdoo
Copy link

Describe the Feature

If I have a class that I want to have mapped via the DynamoDB Enhanced client, I have to create a class like this:

@DynamoDbBean
public class Entry {
    String ID;
}

This works fine, as long as the data structure is known.

I have a case where I need to support arbitrary Maps (or JSON objects). The key will always be a String, but the value type is unknown, so I use Object. I would like to have that mapped to a Map field in the DynamoDB item. My bean class now looks like this:

@DynamoDbBean
public class Entry {
    String ID;
    Map<String, Object> variables;
}

Trying to run TableSchema.fromBean(Entry.class) now yields this error:

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0
	at java.base/java.util.Collections$EmptyList.get(Collections.java:4483)
	at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.createMapConverter(DefaultAttributeConverterProvider.java:179)
	at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverter(DefaultAttributeConverterProvider.java:149)
	at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.createMapConverter(DefaultAttributeConverterProvider.java:183)
	at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverter(DefaultAttributeConverterProvider.java:149)
	at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.converterFor(DefaultAttributeConverterProvider.java:133)
	at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttribute.converterFrom(StaticAttribute.java:161)
	at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttribute.resolve(StaticAttribute.java:157)
	at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema.lambda$new$0(StaticTableSchema.java:89)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
	at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema.<init>(StaticTableSchema.java:94)
	at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema.<init>(StaticTableSchema.java:73)
	at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema$Builder.build(StaticTableSchema.java:335)
	at software.amazon.awssdk.enhanced.dynamodb.mapper.BeanTableSchema.createStaticTableSchema(BeanTableSchema.java:225)
	at software.amazon.awssdk.enhanced.dynamodb.mapper.BeanTableSchema.create(BeanTableSchema.java:120)
	at software.amazon.awssdk.enhanced.dynamodb.TableSchema.fromBean(TableSchema.java:55)

The DDB Enhanced client seems to try to use a Map for the object type, and fails to get the class parameters.

Using just Object as seen here

@DynamoDbBean
public class Entry {
    String ID;
    Object variables;
}

yields the same error.

If there is another possibility to achieve that, please let me know.

Is your Feature Request related to a problem?

I want to store arbitrarily structured data in a map field, which is not possible by now.

Proposed Solution

Create a mapper for Objects, and use instanceof to determine the type converter.

Describe alternatives you've considered

JSON-encoding the data and store as string, but using this approach I would sacrifice searchability of the nested attributes.

Additional Context

Storing data with unknown structure, and still keep it searchable (via scan) would make a great addition!

  • I may be able to implement this feature request

Your Environment

  • AWS Java SDK version used: 2.13.33
  • JDK version used: 11
  • Operating System and version: Windows 10
@PatrickBuTaxdoo PatrickBuTaxdoo added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Jun 17, 2020
@debora-ito
Copy link
Member

Hi @PatrickBuTaxdoo, DynamoDB Enhanced client won't work with Objects and won't infer types at runtime. You need to write a custom AttributeConverter for your attribute so DynamoDB Enhanced knows how to convert to and from DynamoDB supported types.

You can see some examples in the DynamoDB README page and in the DynamoDB Enhanced test folder.

@debora-ito debora-ito removed the needs-triage This issue or PR still needs to be triaged. label Jun 27, 2020
@debora-ito
Copy link
Member

I'll close this, feel free to reach out if you have any other question.

@PatrickBuTaxdoo
Copy link
Author

I implemented a custom AttributeConverter that solved my issue without reflection. Its behaviour is similar to deserializing JSON using Gson to a Map (and back). Let me know if that could be a useful addition to this library.

@bigunyak
Copy link

@PatrickBuTaxdoo , there is a request for something like that in #2162.
Maybe you could contribute at least with an example? :)

@ankitakundra
Copy link

@PatrickBuTaxdoo Can you please provide an example for the custom AttributeConverter that you wrote

@PatrickBuTaxdoo
Copy link
Author

This attribute converter does not support all data types, but it was sufficient for my use case. Serialization from POJOs is not supported, only primitives, List and Map are supported. For attributes with known structure, I used POJOs and the capabilities of the DynamoDB enhanced client, for attributes with arbitrary structure, I used the following attribute converter. The code as of now only supports attributes that are at least on the top-level a map in DynamoDB (hence the type parameter of AttributeConverter).

import software.amazon.awssdk.core.BytesWrapper;
import software.amazon.awssdk.enhanced.dynamodb.AttributeConverter;
import software.amazon.awssdk.enhanced.dynamodb.AttributeValueType;
import software.amazon.awssdk.enhanced.dynamodb.EnhancedType;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;

import java.math.BigDecimal;
import java.util.*;
import java.util.stream.Collectors;

public class JsonAttributeConverter implements AttributeConverter<Map<String, Object>> {

    @Override
    public AttributeValue transformFrom(Map<String, Object> input) {
        return transformToAttributeValue(input);
    }

    @Override
    public Map<String, Object> transformTo(AttributeValue input) {
        return input.m().entrySet().stream()
                .collect(Collectors.toMap(Map.Entry::getKey, e -> transformToObject(e.getValue())));
    }

    @Override
    public EnhancedType<Map<String, Object>> type() {
        return EnhancedType.mapOf(String.class, Object.class);
    }

    @Override
    public AttributeValueType attributeValueType() {
        return AttributeValueType.M;
    }

    /** Transforms the given object to an {@link AttributeValue} for DynamoDB. */
    private static AttributeValue transformToAttributeValue(Object input) {
        if (input instanceof Map) {
            Map<String, Object> map = (Map<String, Object>) input;
            return AttributeValue.builder()
                    .m(
                            map.entrySet().stream()
                                    .collect(
                                            Collectors.toMap(
                                                    Map.Entry::getKey,
                                                    e -> transformToAttributeValue(e.getValue()))))
                    .build();
        }
        if (input instanceof List) {
            List<Object> input1 = (List<Object>) input;
            List<AttributeValue> converted =
                    input1.stream()
                            .map(JsonAttributeConverter::transformToAttributeValue)
                            .collect(Collectors.toList());

            return AttributeValue.builder().l(converted).build();
        }
        if (input instanceof Boolean) {
            return AttributeValue.builder().bool((Boolean) input).build();
        }
        if (input instanceof String) {
            return AttributeValue.builder().s((String) input).build();
        }
        if (input instanceof Number) {
            return AttributeValue.builder().n(String.valueOf(input)).build();
        }
        throw new IllegalArgumentException("cannot serialize the given AttributeValue");
    }

    private static Object transformToObject(AttributeValue input) {
        // map
        if (input.hasM()) {
            return input.m().entrySet().stream()
                    .collect(
                            Collectors.toMap(
                                    Map.Entry::getKey, e -> transformToObject(e.getValue())));
        }
        // list
        if (input.hasL()) {
            return input.l().stream()
                    .map(JsonAttributeConverter::transformToObject)
                    .collect(Collectors.toList());
        }
        // string set
        if (input.hasSs()) {
            return input.ss();
        }
        // convert number sets to list of BigDecimals
        if (input.hasNs()) {
            return input.ns().stream().map(BigDecimal::new).collect(Collectors.toList());
        }
        // convert binary sets to list of byte arrays
        if (input.hasBs()) {
            return input.bs().stream().map(BytesWrapper::asByteArray).collect(Collectors.toList());
        }
        // string
        if (input.s() != null) {
            return input.s();
        }
        // convert number to BigDecimal
        if (input.n() != null) {
            return new BigDecimal(input.n());
        }
        // boolean
        if (input.bool() != null) {
            return input.bool();
        }
        // binary
        if (input.b() != null) {
            return input.b().asByteArray();
        }

        throw new IllegalArgumentException("cannot deserialize the given AttributeValue");
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request A feature should be added or improved.
Projects
None yet
Development

No branches or pull requests

4 participants