memory leak in class org.springframework.core.ResolvableType [SPR-11394]

[spring-projects-issues]

oaktree opened SPR-11394 and commented

Anayzling heap dump shows:

6,454 instances of "java.lang.Class", loaded by "<system class loader>" occupy 6,081,912 (11.44%) bytes. 

Biggest instances:
•class org.springframework.core.ResolvableType @ 0x7833d2c50 - 720,944 (1.36%) bytes. 
*********************************************************************************

Class Name                                                          | Shallow Heap | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------
class org.springframework.core.ResolvableType @ 0x7833d2c50         |           16 |       720,944 |      1.36%
|- org.springframework.util.ConcurrentReferenceHashMap @ 0x7833d2d00|           40 |       720,880 |      1.36%
|- org.springframework.core.ResolvableType @ 0x7833d2ce0            |           32 |            32 |      0.00%
|- org.springframework.core.ResolvableType[0] @ 0x7833d2cd0         |           16 |            16 |      0.00%
----------------------------------------------------------------------------------------------------------------

---

Affects: 4.0.1

Issue Links:

• Memory leak in ConcurrentReferenceHashMap [SPR-11440] #16066 Memory leak in ConcurrentReferenceHashMap
• Duplicate entries in ResolvableType cache [SPR-12275] #16880 Duplicate entries in ResolvableType cache
• Performance issue on ResolvableType cache [SPR-12122] #16738 Performance issue on ResolvableType cache
• memory leak in class ConcurrentReferenceHashMap [SPR-16699] #21240 memory leak in class ConcurrentReferenceHashMap

2 votes, 12 watchers

[spring-projects-issues]

Juergen Hoeller commented

Phil, since ConcurrentReferenceHashMap is using soft references, it shouldn't really cause a memory leak... I guess we could switch to weak references there for more aggressive cleanup though?

Juergen

[spring-projects-issues]

Phil Webb commented

Yes, ConcurrentReferenceHashMap should be using soft references, unless we have some bug in that or ResolvableType that is somehow stopping GC.

ConcurrentReferenceHashMap already support weak references. It should just be a case of changing to a different constructor.

[spring-projects-issues]

Juergen Hoeller commented

Hmm, the actual leak might be that ResolvableType refers to its own instances in the static cache there, so that the ResolvableType class itself can't get garbage-collected... in which case weak references would indeed help.

However, with weak references, the entries would disappear almost immediately at the next GC run, since we use ResolvableType instances as keys here, and those typically aren't held as a strong reference anywhere...

As a consequence, I wonder whether we should switch the cache to use the original Type as the key - a reference that is strongly held by the ClassLoader. Once the ClassLoader disappears, the cache would be cleared.

Juergen

[spring-projects-issues]

Juergen Hoeller commented

We need further feedback on how exactly a leak shows up there. According to our analysis, the GC should release those soft references once it actually requires memory, and we don't see anything on our side that would prevent that...

Jeurgen

[spring-projects-issues]

David Geary commented

We saw a problem in Spring XD M4 (which uses Spring 4.0.0.RC1) that may be related to this - running on Java 7 with the following options: -d64 -Xms2G -Xmx2G -XX:+UseG1GC

It gets to the point where it doesn't actually throw out of memory errors, but starts to run very slowly as the majority of time is spent in garbage collection -
but no SoftReferences are being cleared.

From a heap dump we are seeing:

Class Name 									Instances		Size
org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference		29642514 (97.4%)	2015690952 (88.2%)

Stack trace 

"task-scheduler-8" prio=5 tid=78 RUNNABLE
	at java.lang.Object.hashCode(Native Method)
	at java.util.HashMap.hash(HashMap.java:366)
	at java.util.HashMap.getEntry(HashMap.java:466)
	   Local Variable: java.util.HashMap#5563
	at java.util.HashMap.containsKey(HashMap.java:453)
	at java.util.HashSet.contains(HashSet.java:201)
	at org.springframework.util.ConcurrentReferenceHashMap$Segment.restructureIfNecessary(ConcurrentReferenceHashMap.java:544)
	   Local Variable: org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference#10694956
	   Local Variable: java.util.HashSet#2033
	   Local Variable: org.springframework.util.ConcurrentReferenceHashMap$Reference[]#43
	at org.springframework.util.ConcurrentReferenceHashMap$Segment.getReference(ConcurrentReferenceHashMap.java:430)
	   Local Variable: org.springframework.util.ConcurrentReferenceHashMap$Segment#48
	at org.springframework.util.ConcurrentReferenceHashMap.getReference(ConcurrentReferenceHashMap.java:238)
	at org.springframework.util.ConcurrentReferenceHashMap.get(ConcurrentReferenceHashMap.java:217)
	at org.springframework.core.ResolvableType.forType(ResolvableType.java:1017)
	   Local Variable: org.springframework.core.ResolvableType#113
	at org.springframework.core.ResolvableType.forType(ResolvableType.java:997)
	at org.springframework.core.ResolvableType.forType(ResolvableType.java:970)
	at org.springframework.core.ResolvableType.forClass(ResolvableType.java:739)
	at org.springframework.core.convert.support.GenericConversionService.getRequiredTypeInfo(GenericConversionService.java:271)
	   Local Variable: class org.springframework.core.convert.converter.ConverterFactory
	at org.springframework.core.convert.support.GenericConversionService.addConverterFactory(GenericConversionService.java:105)
	   Local Variable: org.springframework.xd.tuple.LocaleAwareStringToNumberConverterFactory#1
	   Local Variable: org.springframework.xd.tuple.DefaultTupleConversionService#1
	at org.springframework.xd.tuple.TupleBuilder.setNumberFormatFromLocale(TupleBuilder.java:122)
	at org.springframework.xd.tuple.TupleBuilder.tuple(TupleBuilder.java:63)
	   Local Variable: org.springframework.xd.tuple.TupleBuilder#1
	at org.springframework.xd.tuple.integration.JsonToTupleTransformer.transformPayload(JsonToTupleTransformer.java:57)
	   Local Variable: java.util.ArrayList#7899
	   Local Variable: java.util.ArrayList#7898
	at org.springframework.xd.tuple.integration.JsonToTupleTransformer.transformPayload(JsonToTupleTransformer.java:36)
	at org.springframework.integration.transformer.AbstractPayloadTransformer.doTransform(AbstractPayloadTransformer.java:33)

Class Name								Retained Size

org.springframework.util.ConcurrentReferenceHashMap#3 			1,950,497,260
org.springframework.util.ConcurrentReferenceHashMap$Segment[]#3		1,950,497,196
org.springframework.util.ConcurrentReferenceHashMap$Reference[]#43	258,126,716
org.springframework.util.ConcurrentReferenceHashMap$Segment#38		207,236,560
org.springframework.util.ConcurrentReferenceHashMap$Reference[]#48	207,235,856
org.springframework.util.ConcurrentReferenceHashMap$Segment#33		192,809,688
org.springframework.util.ConcurrentReferenceHashMap$Reference[]#42	192,801,776
org.springframework.util.ConcurrentReferenceHashMap$Segment#41		191,239,220
org.springframework.util.ConcurrentReferenceHashMap$Reference[]#47	191,236,204
org.springframework.util.ConcurrentReferenceHashMap$Segment#45		177,357,020
org.springframework.util.ConcurrentReferenceHashMap$Reference[]#44	177,356,724
org.springframework.util.ConcurrentReferenceHashMap$Segment#36		170,526,836
org.springframework.util.ConcurrentReferenceHashMap$Reference[]#34	170,526,540
org.springframework.util.ConcurrentReferenceHashMap$Segment#43		139,098,052
org.springframework.util.ConcurrentReferenceHashMap$Reference[]#39	139,097,416
org.springframework.util.ConcurrentReferenceHashMap$Segment#39		133,248,148
org.springframework.util.ConcurrentReferenceHashMap$Reference[]#35	133,247,852
org.springframework.util.ConcurrentReferenceHashMap$Segment#44		117,149,284
org.springframework.util.ConcurrentReferenceHashMap$Reference[]#40	117,148,580
org.springframework.util.ConcurrentReferenceHashMap$Segment#46		109,010,628

[spring-projects-issues]

Phil Webb commented

I have found at least one issue with the underlying ConcurrentReferenceHashMap (see #16066) which has now been fixed. I have also added some more aggressive cleaning for the cache.

Could you try the latest snapshot an let me know if you are still seeing issues?

[spring-projects-issues]

Charles O'Farrell commented

Apologies if this is unrelated, but we just hit this in Atlassian Stash when a customer was running Java 1.7.0_51 (see #15959). The problem turned out that the equals method wasn't being called on proxied objects (in some cases) and resulted in the ConcurrentReferenceHashMap containing the same value multiple times (which in our cases resulted in serious memory/performance issues). Just mentioning this just in case it helps someone.

[spring-projects-issues]

Holger Brandl commented

I'm having a similar problem when using 4.1.0.RELEASE along with Tomcat8. When checking memory after undeployment of my webapp, many DAOs and Controller-classes are retained.

I've tried to trace it down using YourKit and it seems to be related to this issue:
my.company.controller.someclass
<Retained from several objects simultaneously (don't have a dominator)>
org.springframework.web.method.support.InvocableHandlerMethod
org.springframework.web.method.HandlerMethod$HandlerMethodParameter
org.springframework.core.SerializableTypeWrapper$MethodParameterTypeProvider
org.springframework.core.ResolvableType
org.springframework.util.ConcurrentReferenceHashMap$Entry
org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference
org.springframework.util.ConcurrentReferenceHashMap$Reference[]
org.springframework.util.ConcurrentReferenceHashMap$Segment
org.springframework.util.ConcurrentReferenceHashMap$Segment[]
org.springframework.util.ConcurrentReferenceHashMap
statics or constant pool of org.springframework.core.ResolvableType
java.lang.Object[]
java.util.Vector
org.apache.catalina.loader.WebappClassLoader

As a workaround I'm using reflection now to clean it up:
@Override
public void destroy() throws Exception {
Field cache = ResolvableType.class.getDeclaredField("cache");
cache.setAccessible(true);
ConcurrentReferenceHashMap concurrentReferenceHashMap = (ConcurrentReferenceHashMap) cache.get(null);
concurrentReferenceHashMap.clear();

    Field entrySet= ConcurrentReferenceHashMap.class.getDeclaredField("entrySet");
    entrySet.setAccessible(true);
    ((Set)entrySet.get(concurrentReferenceHashMap)).clear();
}

Maybe this also helps to track down the problem: it turned out that the ResolvableType.cache contained around 10k entries including many duplicates of "java.lang.String=java.lang.String" or "boolean=boolean".

[spring-projects-issues]

Juergen Hoeller commented

We're addressing #16880 - unnecessary duplicate entries - for tomorrow's Spring Framework 4.1.1 release. Please give it a try once available and let us know whether it improves the situation for you.

Note that it is normal for the ResolvableType cache to contain entries with empty references even on shutdown: Those would usually be empty SoftReferences for which the garbage collector cleared the references - however, ResolvableType's cache will only purge those entries on next access, so they might still show up in a debugging session on shutdown. In any case, those are not a leak, just a temporary pre-cleanup situation.

If a ClassLoader leak or out-of-memory scenario happens on repeated restart of an application, let us know, since that would clearly be a bug.

Juergen

[spring-projects-issues]

Stéphane Nicoll commented

I have tried to reproduce that problem and I couldn't. I can see those items in the cache but they are all marked as weak references. I can also see the kind of back trace in yourkit but they are not really surprising as the JVM is free to reclaim those any time it wants to. If you have enough memory it can just sit there.

Do you have a concrete OutOfMemory and a way to reproduce this issue? That would be very helpful.

Investigating this issue also helped to find a related issue (see #16880). It should be available in 4.1.1.BUILD-SNAPSHOT in a couple of hours. Can you give this a try and let us know how it goes?

[spring-projects-issues]

Holger Brandl commented

@Stephane: I don't have a concrete OutOfMemory. Using 4.1.1.BUILD-SNAPSHOT, yourkit did not show any ResolveType related memory problems.

You're comments helped me to understand the problem better. Thank you for your help.

[spring-projects-issues]

Juergen Hoeller commented

Alright, let's consider this as resolved for 4.1.1 for the time being. To be reopened if any problems remain.

Juergen