java.lang.ArrayIndexOutOfBoundsException when using Streams sorting

[pminearo]

We are having an issue in our production environment. I have a work around to it, but figured I would submit this to the group, just in case it is an actual bug in OpenJDK.

Has anyone seen this problem?

The following code:

public Object createSomeObject(final Map<Object, Object> map) {
		....
		....
		....
		
        final Set<Object> keySet = map.keySet();
        // This is the line of code that is throwing the exception
        // We need to sort the keys for a reason. And we are not
        // guaranteed to have a SortedMap, 
        final List<Object> keys = keySet.stream().sorted().collect(Collectors.toList());
        ....
        ....
        ....
}

Is throwing the following exception:

java.lang.ArrayIndexOutOfBoundsException: Index 9 out of bounds for length 9
        at java.base/java.util.stream.SortedOps$SizedRefSortingSink.accept(SortedOps.java:369)
        at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at our.name.space.OurClass.createSomeObject(....)

I can not reproduce this in a test. And, because our production environment can have different types of Map objects with different types of keys; I am not sure which type of Map, or key, is being passed in the method.
With that said, I am pretty sure streams should not be throwing ArrayIndexOutOfBoundsException. Part of the reason for streams is not have to worry about this type of issue.

Also, there is no concurrency issues. Usually, you get a ConcurrentModificationException when a collection is modified by 2 different threads, at least we do.

I switched the code to what’s below and it works fine.

        ....
        ....
        ....
        final Set<?> keySet = map.keySet();
        final List<Object> keys = new ArrayList<>(keySet);

        keys.sort((o1, o2) -> {

            if (o1 == null && o2 == null) {
                return 0;
            } else if (o1 != null && o2 == null) {
                return 1;
            } else if (o1 == null) {
                return -1;
            } else if (o1 instanceof final String string1 && o2 instanceof final String string2) {
                return string1.compareTo(string2);
            } else if (o1 instanceof final String string1) {
                final String string2 = o2.toString();
                return string1.compareTo(string2);
            } else if (o2 instanceof final String string2) {
                final String string1 = o1.toString();
                return string1.compareTo(string2);
            }

            final String string1 = o1.toString();
            final String string2 = o2.toString();

            return string1.compareTo(string2);
        });

        ....
        ....
        ....

Using the List.sort() works, but sorting via Streams didn’t. This is an odd one.

My environment:
GraalVM version 22.2.0
CE or EE: CE
JDK version: JDK17
OS and OS Version: CentOS Linux release 7.9.2009
Architecture: amd64

$ java -version
openjdk version "17.0.4" 2022-07-19
OpenJDK Runtime Environment GraalVM CE 22.2.0 (build 17.0.4+8-jvmci-22.2-b06)
OpenJDK 64-Bit Server VM GraalVM CE 22.2.0 (build 17.0.4+8-jvmci-22.2-b06, mixed mode, sharing)

$ java -Xinternalversion
OpenJDK 64-Bit Server VM (17.0.4+8-jvmci-22.2-b06) for linux-amd64 JRE (17.0.4+8-jvmci-22.2-b06), built on Jul 20 2022 18:51:07 by "buildslave" with gcc 10.3.0

[wirthi]

HI @pminearo

to verify, this is happening in JIT mode, you are not using native-image, correct?

Can you reproduce the same problem on actual OpenJDK, without any GraalVM involved?

Christian

[fernando-valdez]

closed due to a lack of response from the reporter.

[aukevanleeuwen]

Wanted to note that I've seen the same behaviour (not using Graal at all, actually using Amazon Corretto (17.0.8-amzn from SKDman).

Although I do not understand the reason for it failing exactly, I had an obvious bug in my code where I forgot to change a HashMap to a ConcurrentHashMap when trying to optimize stuff over multiple threads.

The code that broke was literally:

    private final Map<String, String> calculatedHashes = new HashMap<>(100_000);
    
    // some code to precalculate the map with using `.parallel()`

    log.error("SIZE: {}", calculatedHashes.size());
    log.error("KEYS: {}", calculatedHashes.keySet().stream().sorted().toList()); // ← fails here with:

08:32:38.588 [main] ERROR nl.some.company.package.AccessCodeDataGenerator -- SIZE: 99970

java.lang.ArrayIndexOutOfBoundsException: Index 99970 out of bounds for length 99970

	at java.base/java.util.stream.SortedOps$SizedRefSortingSink.accept(SortedOps.java:369)
	at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
	at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
	at nl.some.company.package.AccessCodeDataGenerator.preCalculateHashes(AccessCodeDataGenerator.java:193)
	at nl.some.company.package.AccessCodeDataGenerator.generateUserAccessCodeEntities(AccessCodeDataGenerator.java:105)
	at nl.some.company.package.AccessCodeDataGenerator.generate_data_access_code_data(AccessCodeDataGenerator.java:94)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

I do not understand exactly why it would break at this point to be honest. I would guess that even though functionally it would be broken due to the HashMap not being thread safe, I didn't expect that getting the key set would be in this kind of 'inconsistent' state? Anyway: I cannot reproduce anymore with a ConcurrentHashMap, but maybe this comment helps somebody in the future to point out the problem.