diff --git a/build-tools/xamarin-android-docimporter-ng/.gitignore b/build-tools/xamarin-android-docimporter-ng/.gitignore new file mode 100644 index 000000000..c65b3afe8 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/.gitignore @@ -0,0 +1,7 @@ +bin +obj +packages +*.userprefs +*.csproj.user +api-*.params.txt +api-*.params.xml diff --git a/build-tools/xamarin-android-docimporter-ng/DealingWithJavaApiDescription.md b/build-tools/xamarin-android-docimporter-ng/DealingWithJavaApiDescription.md new file mode 100644 index 000000000..cf64ab9f0 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/DealingWithJavaApiDescription.md @@ -0,0 +1,103 @@ + +# Dealing with Java API description + +## Java API XML description files: how it exists nowadays + +Historically, Google used to ship Android Framework API information as in XML format, and to build Mono.Android.dll we parsed and interpreted these description files. Those files defined most of our API XML description format that we use nowadays. + +As of Android 3.0, Google had not shipped the source code including these XML description files for long time, so we had been unable to build new API. Therefore we ended up to change our approach so that we built API description XML from android.jar by ourselves. At that time, the API extraction tool "jar2xml" was written in Java using Java reflection API (java.lang.reflect) and bytecode manipulator (asm). + +And that turned to become the first core part of our Java Bindings support. + +Java reflection API dependency caused several problems, especially on that we cannot extract correct java.* framework API (because with reflection API they always report java.* structure from the Java framework that the tool executes). Thus at some stage we ended up to change our strategy again and implemented bytecode parser "class-parse". + +Unfortunately, class-parse was written in the way that it only reflects bytecode information, which means the outcome was totally different from the API information that Java reflection API offered. There is no type hierarchy, therefore no method overrides are resolved (which caused a lot of differences between the old and the new API information). + +Therefore we needed to "adjust" the API description to match the old one, otherwise our C# binding generator ("generator.exe") and Metadata.xml don't work at all. This part is called "api-xml-adjuster" (implemented as Xamarin.Android.Tools.ApiXmlAdjuster.dll). + + +## Java API parsers + +We have many Java parsers (unfortunately) within xamarin-android SDK and its own build system. They exist for different purposes and reasons. + +(1) class-parse + api-xml-adjuster (Xamarin.Android.Tools.ApiXmlAdjuster) + +class-parse is the bytecode parser tool (I wrote "the", which means that it is the only implementation) that parses class library jars into api.xml.class-parse. + +api-xml-adjuster exists for the reason we described earlier. It has no other reason to exist. Since it also needs to retrieve Java API information from reference binding DLLs, it is tied to binding generator's type system. + +(2) JavaDocScraper + +Java bytecode does not contain method parameter names by design. We can still bind Java API without them, but the resulting API becomes awkward (a set of methods with parameter names "p0", "p1", ...). + +Our solution for that is JavaDoc parsers. + +JavaDoc is not great for retrieving information because it is not well structured. And it can be actually in any format, depending on "doclet"; Google indeed offers its own Android API Reference with its own "DroidDoc" doclet. But there is no other way, so we have different parser implementations for each "standard" doclet. + +Doclet had changed in each Java releases: Java6, Java7 and Java8 had each doclet format that are different enough to confuse our document scraper. To make things worse, DroidDoc, the Google doclet, had also changed for each big API releases. We have three different JavaDoc scrapers up to 8 (note that it does not even include Java9 as of writing time) and two different DroidDoc scrapers (which may become three, depending on the upcoming work). + +DroidDoc parsers are used for two different purposes: + +* Build Mono.Android.dll: this requires from the ancient API documentation parser up to the latest parser, because we unify all the support API levels. +* Build Android support libraries: their API information is offered through developer.android.com and the docs are based on DroidDoc. For this purpose we don't need older API parsers. + +JavaDoc parsers are used for generic Java Binding projects. + +(We should probably support external doclet parsers so that users can provide and specify any kind of doclets, but we have heard of any requests for that. There is no plan to stabilize JavaDoc scraper API either.) + +(JavaDocScraper in this context had been implemented in Java in jar2xml before, and now it is rewritten in C# in class-parse.) + +(3) JavaDoc to C# Documentation converter (javadoc-to-mdoc) + +Apart from JavaDocScraper for parameter names, we still have another reason to parse JavaDocs and DroidDocs. Bindings need API documentation, and they should be in our .NET API manner. It helps IDEs provide API information. + +That means, we need almost entire API information including details. + +It has been implemented in mono/mcs/tools/javadoc-to-mdoc. And it had been used only to generate Xamarin.Android API documentation i.e. it supported only DroidDocs. It was extended to support JavaDocs when we brought in this feature to xamarin-android to support any Java Binding projects. Yet, that is limited to the standard doclets for exactly the same reason as JavaDocScraper for method parameter names. + +(Nowadays there should be almost no reason to have different JavaDoc scrapers, but as explained above, JavaDocScraper used to be Java, while javadoc-to-mdoc has been C# since its beginning.) + +(4) DroidDoc parser for parameter names + +JavaDocScraper for parameter names is problematic, not only because it always needs to renew the implementation whenever DroidDocs are updated, but also because it is not efficient to parse HTML docs every time we build the bindings. And that annoyed our Components team because unlike Xamarin.Android itself, they have to run JavaDoc Scraper every time (we don't run class-parse and api-xml-adjuster every time; we generate api-XY.xml.in only when new APIs get released). + +Since the only information we/they need is method parameter names, they could be generated only once whenever new versions of the components get released. So the team had implemented support for "parameter names only" XML description format in class-parse. + +It was part of Xamarin private repo, but now it is extracted at https://github.com/atsushieno/xamarin-android-docimporter (which was expected to be forked under xamarin, but it did not happen yet). + +It is limited to DroidDoc as it was only for Android Components (support libraries and Google Play services). And it's not ready for xamarin-android that needs to build and run on Linux (this doesn't). + +(5) Java stub API source parser for parameter names + +DroidDoc support has been getting more and more problematic as Google stopped shipping "docs" SDK components anymore, and new API documentations are available only via the web. + +Since Google ships "stub sources" in each platform (API Level) package, it is possible to parse Java sources and extract parameter names from each type. So as a replacement to DroidDoc parameter name parser, we now have a prototype of Java source parser implementation that exactly does this job. + +It is part of xamarin-android-docimporter-ng in Java.Interop. + +The stub sources are expected to have full type names almost everywhere so that we don't have to "resolve" type names (although we have detected that "@Deprecated" is used without "java.lang." which smells... hopefully not any more). It is safer scheme that we use it only to generate parameter names, not the entire API structure. That task can still be done by class-parse. + +(6) DroidDoc parser for parameter names, reimplemented + +The implementation at (4) was quite incomplete and did not try to parse all Android API, which exposes various issues. Thus it was rewritten to try to do better thing. The new DroidDoc scraper at https://github.com/atsushieno/xamarin-android-docimporter-ng generates the same parameter name description file as Java stub parser. It only targets Android API up to 23 because (5) covers the rest. + +However this exposes further issues - some API documentations cannot be parsed because of Google's buggy HTML generation (e.g. android/content/ContentProvider.html significantly breaks its documentation structure. We parse the docs in extraneous way (e.g. inspecting descendants of certain elements instead of just children). + +It is part of xamarin-android-docimporter-ng in Java.Interop. + + + +*** List of parsers and their roles + +| parser | info to generate | target libs | target versions | how often | +|-----------------------------|-----------------------|-----|---|---| +|class-parse+api-xml-adjuster | entire API definition | all | all ages | every build +|JavaDocScraper | parameter names | all | all ages | android.jar - every API release +|JavaDocScraper | parameter names | all | all ages | others - every build +|javadoc-to-mdoc | docs | all | latest | android.jar - every API release +|javadoc-to-mdoc | docs | all | latest | others - every release build +|xamarin-android-docimporter | parameter names | support/GPS | latest | every components release +|java-stub-parser | parameter names | android.jar | API 24 or later | every API release +|new DroidDoc parser | parameter names | android.jar | API 23 or earlier | only once, or every parser bugfixes + + diff --git a/build-tools/xamarin-android-docimporter-ng/README.md b/build-tools/xamarin-android-docimporter-ng/README.md new file mode 100644 index 000000000..666a67e6a --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/README.md @@ -0,0 +1,8 @@ +New droiddoc importer that works not only on OSX but works everywhere. + +HtmlLoader.cs and those HTML DTD resources are from +https://github.com/xamarin/xamarin-android/tree/master/src/Xamarin.Android.Tools.JavadocImporter + +JavaApi.XmlModel.cs is from here, with some minor changes: +https://github.com/xamarin/java.interop/blob/master/src/Xamarin.Android.Tools.ApiXmlAdjuster/JavaApi.XmlModel.cs + diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/ImporterOptions.cs b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/ImporterOptions.cs new file mode 100644 index 000000000..3cdc44090 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/ImporterOptions.cs @@ -0,0 +1,15 @@ +using System; +using System.IO; + +namespace Xamarin.Android.ApiTools +{ + public class ImporterOptions + { + public string InputZipArchive { get; set; } + public string DocumentDirectory { get; set; } + public string OutputTextFile { get; set; } + public string OutputXmlFile { get; set; } + public TextWriter DiagnosticWriter { get; set; } + public bool FrameworkOnly { get; set; } + } +} diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/JavaApi.XmlModel.cs b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/JavaApi.XmlModel.cs new file mode 100644 index 000000000..3f838518a --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/JavaApi.XmlModel.cs @@ -0,0 +1,355 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Xml; + +namespace Xamarin.Android.Tools.ApiXmlAdjuster +{ + public partial class JavaApi + { + public JavaApi () + { + Packages = new List (); + } + + public string ExtendedApiSource { get; set; } + public IList Packages { get; set; } + } + + public partial class JavaPackage + { + public JavaPackage (JavaApi parent) + { + Parent = parent; + + Types = new List (); + } + + public JavaApi Parent { get; private set; } + + public string Name { get; set; } + public IList Types { get; set; } + + // Content of this value is not stable. + public override string ToString () + { + return string.Format ("[Package] " + Name); + } + } + + public abstract partial class JavaType + { + protected JavaType (JavaPackage parent) + { + Parent = parent; + + Implements = new List (); + Members = new List (); + } + + public JavaPackage Parent { get; private set; } + + public bool IsReferenceOnly { get; set; } + + public bool Abstract { get; set; } + public string Deprecated { get; set; } + public bool Final { get; set; } + public string Name { get; set; } + public bool Static { get; set; } + public string Visibility { get; set; } + + public string ExtendedJniSignature { get; set; } + + public IList Implements { get; set; } + public JavaTypeParameters TypeParameters { get; set; } + public IList Members { get; set; } + + public string FullName { + get { return Parent.Name + (Parent.Name.Length > 0 ? "." : string.Empty) + Name; } + } + + // Content of this value is not stable. + public string ToStringHelper () + { + // FIXME: add type attributes. + return Parent.Name + "." + Name; + } + } + + public partial class JavaInterface : JavaType + { + public JavaInterface (JavaPackage parent) + : base (parent) + { + } + + // Content of this value is not stable. + public override string ToString () + { + return "[Interface] " + ToStringHelper (); + } + } + + public partial class JavaClass : JavaType + { + public JavaClass (JavaPackage parent) + : base (parent) + { + } + + public string Extends { get; set; } + public string ExtendsGeneric { get; set; } + public string ExtendedJniExtends { get; set; } + + // Content of this value is not stable. + public override string ToString () + { + return "[Class] " + ToStringHelper (); + } + } + + + class ManagedType : JavaType + { + static JavaPackage dummy_system_package, dummy_system_io_package, dummy_system_xml_package; + static JavaType system_object, system_exception, system_io_stream, system_xml_xmlreader; + + static ManagedType () + { + dummy_system_package = new JavaPackage (null) { Name = "System" }; + system_object = new ManagedType (dummy_system_package) { Name = "Object" }; + system_exception = new ManagedType (dummy_system_package) { Name = "Exception" }; + dummy_system_package.Types.Add (system_object); + dummy_system_package.Types.Add (system_exception); + dummy_system_io_package = new JavaPackage (null) { Name = "System.IO" }; + system_io_stream = new ManagedType (dummy_system_io_package) { Name = "Stream" }; + dummy_system_io_package.Types.Add (system_io_stream); + dummy_system_xml_package = new JavaPackage (null) { Name = "System.Xml" }; + system_xml_xmlreader = new ManagedType (dummy_system_xml_package) { Name = "XmlReader" }; + dummy_system_io_package.Types.Add (system_xml_xmlreader); + } + + public static IEnumerable DummyManagedPackages { + get { + yield return dummy_system_package; + yield return dummy_system_io_package; + yield return dummy_system_xml_package; + } + } + + public ManagedType (JavaPackage package) : base (package) + { + } + } + + + public partial class JavaImplements + { + public string Name { get; set; } + public string NameGeneric { get; set; } + + public string ExtendedJniType { get; set; } + } + + public partial class JavaMember + { + protected JavaMember (JavaType parent) + { + Parent = parent; + } + + public JavaType Parent { get; private set; } + + public string Deprecated { get; set; } + public bool Final { get; set; } + public string Name { get; set; } + public bool Static { get; set; } + public string Visibility { get; set; } + } + + public partial class JavaField : JavaMember + { + public JavaField (JavaType parent) + : base (parent) + { + } + + public bool Transient { get; set; } + public string Type { get; set; } + public string TypeGeneric { get; set; } + public string Value { get; set; } + public bool Volatile { get; set; } + + // Content of this value is not stable. + public override string ToString () + { + return "[Field] " + TypeGeneric + " " + Name; + } + } + + public partial class JavaMethodBase : JavaMember + { + protected JavaMethodBase (JavaType parent) + : base (parent) + { + Parameters = new List (); + Exceptions = new List (); + } + + public IList Parameters { get; set; } + public IList Exceptions { get; set; } + public JavaTypeParameters TypeParameters { get; set; } + + public bool ExtendedBridge { get; set; } + public string ExtendedJniReturn { get; set; } + public bool ExtendedSynthetic { get; set; } + // We cannot get 'ExtendedJniSignature' from DLLs, so we don't have it here. + + // Content of this value is not stable. + public string ToStringHelper (string returnType, string name, JavaTypeParameters typeParameters) + { + return string.Format ("{0}{1}{2}{3}{4}{5}({6})", + returnType, + returnType == null ? null : " ", + Name, + typeParameters == null ? null : "<", + typeParameters == null ? null : string.Join (", ", typeParameters.TypeParameters), + typeParameters == null ? null : ">", + string.Join (", ", Parameters)); + } + } + + public partial class JavaConstructor : JavaMethodBase + { + public JavaConstructor (JavaType parent) + : base (parent) + { + } + + // it was required in the original API XML, but removed in class-parsed... + public string Type { get; set; } + + // Content of this value is not stable. + public override string ToString () + { + return "[Constructor] " + ToStringHelper (null, Parent.Name, null); + } + } + + public partial class JavaMethod : JavaMethodBase + { + public JavaMethod (JavaType parent) + : base (parent) + { + } + + public bool Abstract { get; set; } + public bool Native { get; set; } + public string Return { get; set; } + public bool Synchronized { get; set; } + + // Content of this value is not stable. + public override string ToString () + { + return "[Method] " + ToStringHelper (Return, Name, TypeParameters); + } + } + + public partial class JavaParameter + { + public JavaParameter (JavaMethodBase parent) + { + Parent = parent; + } + + public JavaMethodBase Parent { get; private set; } + public string Name { get; set; } + public string Type { get; set; } + + // Content of this value is not stable. + public override string ToString () + { + return Type + " " + Name; + } + } + + public partial class JavaException + { + public string Name { get; set; } + public string Type { get; set; } + } + + public partial class JavaTypeParameters + { + public JavaTypeParameters (JavaType parent) + { + ParentType = parent; + TypeParameters = new List (); + } + + public JavaTypeParameters (JavaMethodBase parent) + { + ParentMethod = parent; + TypeParameters = new List (); + } + + public JavaType ParentType { get; set; } + public JavaMethodBase ParentMethod { get; set; } + + public IList TypeParameters { get; set; } + } + + public partial class JavaTypeParameter + { + public JavaTypeParameter (JavaTypeParameters parent) + { + Parent = parent; + } + + public JavaTypeParameters Parent { get; set; } + + public string Name { get; set; } + + public string ExtendedJniClassBound { get; set; } + public string ExtendedClassBound { get; set; } + public string ExtendedInterfaceBounds { get; set; } + public string ExtendedJniInterfaceBounds { get; set; } + + public JavaGenericConstraints GenericConstraints { get; set; } + + public override string ToString () + { + return Name; + } + } + + public partial class JavaGenericConstraints + { + public JavaGenericConstraints () + { + GenericConstraints = new List (); + } + + public string BoundsType { get; set; } // extends / super + + public IList GenericConstraints { get; set; } + + public override string ToString () + { + string csts = string.Join (" & ", GenericConstraints); + if (csts == "java.lang.Object") + return string.Empty; + return " " + (BoundsType ?? "extends") + " " + csts; + } + } + + public partial class JavaGenericConstraint + { + public string Type { get; set; } + + public override string ToString () + { + return Type; + } + } +} diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/JavaApiParameterNamesExporter.cs b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/JavaApiParameterNamesExporter.cs new file mode 100644 index 000000000..d44a102f3 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/JavaApiParameterNamesExporter.cs @@ -0,0 +1,143 @@ +using System; +using System.IO; +using System.Linq; +using System.Xml; +using Xamarin.Android.Tools.ApiXmlAdjuster; + +namespace Xamarin.Android.Tools.ApiXmlAdjuster +{ + public static class JavaApiParameterNamesExporter + { + public static void WriteParameterNamesXml (this JavaApi api, string file) + { + using (var xw = XmlWriter.Create (file, new XmlWriterSettings { Indent = true })) + WriteParameterNamesXml (api, xw); + } + + public static void WriteParameterNamesXml (this JavaApi api, XmlWriter writer) + { + writer.WriteStartElement ("parameter-names"); + + Action writeTypeParameters = tps => { + if (tps != null && tps.TypeParameters.Any ()) { + writer.WriteStartElement ("type-parameters"); + foreach (var gt in tps.TypeParameters) { + writer.WriteStartElement ("type-parameter"); + writer.WriteAttributeString ("name", gt.Name); + // no need to supply constraints. + writer.WriteEndElement (); + } + writer.WriteEndElement (); + } + }; + + foreach (var package in api.Packages) { + writer.WriteStartElement ("package"); + writer.WriteAttributeString ("name", package.Name); + + foreach (var type in package.Types) { + if (!type.Members.OfType ().Any (m => m.Parameters.Any ())) + continue; // we care only about types that has any methods that have parameters. + + writer.WriteStartElement (type is JavaClass ? "class" : "interface"); + writer.WriteAttributeString ("name", type.Name); + + writeTypeParameters (type.TypeParameters); + + // we care only about methods that have parameters. + foreach (var mb in type.Members.OfType ().Where (m => m.Parameters.Any ())) { + if (mb is JavaConstructor) + writer.WriteStartElement ("constructor"); + else { + writer.WriteStartElement ("method"); + writer.WriteAttributeString ("name", mb.Name); + } + + writeTypeParameters (mb.TypeParameters); + + foreach (var para in mb.Parameters) { + writer.WriteStartElement ("parameter"); + // For possible generic instances in parameter type, we replace all ", " with "," to ease parsing. + writer.WriteAttributeString ("type", para.Type.Replace (", ", ",")); + writer.WriteAttributeString ("name", para.Name); + writer.WriteEndElement (); + } + + writer.WriteEndElement (); + } + + writer.WriteEndElement (); + } + + writer.WriteEndElement (); + } + + writer.WriteEndElement (); + } + + +/* + * The Text Format is: + * + * package {packagename} + * #--------------------------------------- + * interface {interfacename}{optional_type_parameters} -or- + * class {classname}{optional_type_parameters} + * {optional_type_parameters}{methodname}({parameters}) + * + * Anything after # is treated as comment. + * + * optional_type_parameters: "" -or- "" (no constraints allowed) + * parameters: type1 p0, type2 p1 (pairs of {type} {name}, joined by ", ") + * + * It is with strict indentations. two spaces for types, four spaces for methods. + * + * Constructors are named as "#ctor". + * + * Commas are used by both parameter types and parameter separators, + * but only parameter separators can be followed by a whitespace. + * It is useful when writing text parsers for this format. + * + * Type names may contain whitespaces in case it is with generic constraints (e.g. "? extends FooBar"), + * so when parsing a parameter type-name pair, the only trustworthy whitespace for tokenizing name is the *last* one. + */ + + public static void WriteParameterNamesText (this JavaApi api, string file) + { + using (var sw = new StreamWriter (file)) + WriteParameterNamesText (api, sw); + } + + public static void WriteParameterNamesText (this JavaApi api, TextWriter writer) + { + Action writeTypeParameters = (indent, tps) => { + if (tps != null && tps.TypeParameters.Any ()) + writer.Write ($"{indent}<{string.Join (",", tps.TypeParameters.Select (p => p.Name))}>"); + }; + + foreach (var package in api.Packages) { + writer.WriteLine (); + writer.WriteLine ($"package {package.Name}"); + writer.WriteLine ("#---------------------------------------"); + + foreach (var type in package.Types) { + if (!type.Members.OfType ().Any (m => m.Parameters.Any ())) + continue; // we care only about types that has any methods that have parameters. + writer.Write (type is JavaClass ? " class " : " interface "); + writer.Write (type.Name); + writeTypeParameters ("", type.TypeParameters); + writer.WriteLine (); + + // we care only about methods that have parameters. + foreach (var mb in type.Members.OfType ().Where (m => m.Parameters.Any ())) { + writer.Write (" "); + writeTypeParameters (" ", mb.TypeParameters); + var name = mb is JavaConstructor ? "#ctor" : mb.Name; + // For possible generic instances in parameter type, we replace all ", " with "," to ease parsing. + writer.WriteLine ($" {name}({string.Join (", ", mb.Parameters.Select (p => p.Type.Replace (", ", ",") + ' ' + p.Name))})"); + } + } + } + } + } +} diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Properties/AssemblyInfo.cs b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Properties/AssemblyInfo.cs new file mode 100644 index 000000000..acfa580db --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Properties/AssemblyInfo.cs @@ -0,0 +1,26 @@ +using System.Reflection; +using System.Runtime.CompilerServices; + +// Information about this assembly is defined by the following attributes. +// Change them to the values specific to your project. + +[assembly: AssemblyTitle ("Xamarin.Android.ApiTools.ParameterNameExtractor")] +[assembly: AssemblyDescription ("")] +[assembly: AssemblyConfiguration ("")] +[assembly: AssemblyCompany ("")] +[assembly: AssemblyProduct ("")] +[assembly: AssemblyCopyright ("${AuthorCopyright}")] +[assembly: AssemblyTrademark ("")] +[assembly: AssemblyCulture ("")] + +// The assembly version has the format "{Major}.{Minor}.{Build}.{Revision}". +// The form "{Major}.{Minor}.*" will automatically update the build and revision, +// and "{Major}.{Minor}.{Build}.*" will update just the revision. + +[assembly: AssemblyVersion ("1.0.*")] + +// The following attributes are used to specify the signing key for the assembly, +// if desired. See the Mono documentation for more information about signing. + +//[assembly: AssemblyDelaySign(false)] +//[assembly: AssemblyKeyFile("")] diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/DroidDocImporter.cs b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/DroidDocImporter.cs new file mode 100644 index 000000000..8e4a92953 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/DroidDocImporter.cs @@ -0,0 +1,222 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Xml; +using System.Xml.Linq; +using System.Xml.XPath; +using Xamarin.Android.Tools.ApiXmlAdjuster; + +namespace Xamarin.Android.ApiTools.DroidDocImporter +{ + public class DroidDocScrapingImporter + { + static bool ClassContains (XElement e, string cls) + { + return e.Attribute ("class")?.Value?.Split (' ')?.Contains (cls) == true; + } + + static readonly string [] excludes = new string [] { + "classes.html", + "hierarchy.html", + "index.html", + "package-summary.html", + "packages-wearable-support.html", + "packages-wearable-support.html", + }; + static readonly string [] non_frameworks = new string [] { + "android.support.", + "com.google.android.gms.", + "renderscript." + }; + + /* + + The DroidDoc format from API Level 16 to 23, the format is: + + - All pages have ToC links and body (unlike standard JavaDoc which is based on HTML frames). + - The actual doc section is a div element whose id is "doc-col". + - The "doc-col" div element has a section div element whose id is "jd-header" and another one with id "jd-content". + - "jd-header" div element contains the type signature (modifiers, name, and inheritance). + - Here we care only about type name and kind (whether it is a class or interface). + - Generic arguments are insignificant. + - In the following terms I explain the "correct" (or "expected") document structure, but in fact + Google completely broke it and it is impossible to retrieve the document tree like this. + We workaround this issue by changing the strategy "iterate children of 'jd-content'" + with "iterate descendants of 'jd-content'"... It occurs only in API Level 15 or later. + - "jd-content" div element contains a collection of sections. Each section consists of: + - an "h2" element whose value text indicates the section name ("Public Constructors", "Protected Methods" etc.) + - There was an issue in javax/xml/validation/SchemaFactory.html in API Level 15 that the method details contain + "h2" and confuses the parser. To workaround this, we accept only limited kind of values. + - the content, which follows the h2 element. + - The section content is a collection of members. Each member consists of: + - an anchor ("A") element with "name" attribute, and + - a div element which contains an h4 child element whose class contains "jd-details-title". + - The h4 element contains the member signature. We parse it and retrieve the method name and list of parameters. + - Parameters are tokenized by ", ". + - Note that the splitter contains a white space which disambiguates any use of generic arguments (we don't want to split "Foo bar" as "Foo bar") + + API Level 10 to 15 has slightly different format: + + - There is no "doc-col" element. But "jd-header" and "jd-content" are still alive. + + */ + public void Import (ImporterOptions options) + { + options.DiagnosticWriter.WriteLine (options.DocumentDirectory); + + string referenceDocsTopDir = Path.Combine (options.DocumentDirectory, "reference"); + var htmlFiles = Directory.GetDirectories (referenceDocsTopDir).SelectMany (d => Directory.GetFiles (d, "*.html", SearchOption.AllDirectories)); + + var api = new JavaApi (); + + foreach (var htmlFile in htmlFiles) { + + // skip irrelevant files. + if (excludes.Any (x => htmlFile.EndsWith (x, StringComparison.OrdinalIgnoreCase))) + continue; + var packageName = Path.GetDirectoryName (htmlFile).Substring (referenceDocsTopDir.Length + 1).Replace ('/', '.'); + if (options.FrameworkOnly && non_frameworks.Any (n => packageName.StartsWith (n, StringComparison.Ordinal))) + continue; + + options.DiagnosticWriter.WriteLine ("-- " + htmlFile); + + var doc = new HtmlLoader ().GetJavaDocFile (htmlFile); + + var header = doc.Descendants ().FirstOrDefault (e => e.Attribute ("id")?.Value == "jd-header"); + var content = doc.Descendants ().FirstOrDefault (e => e.Attribute ("id")?.Value == "jd-content"); + + if (header == null || content == null) + continue; + + var apiSignatureTokens = header.Value.Replace ('\r', ' ').Replace ('\n', ' ').Replace ('\t', ' ').Trim (); + if (apiSignatureTokens.Contains ("extends ")) + apiSignatureTokens = apiSignatureTokens.Substring (0, apiSignatureTokens.IndexOf ("extends ", StringComparison.Ordinal)).Trim (); + if (apiSignatureTokens.Contains ("implements ")) + apiSignatureTokens = apiSignatureTokens.Substring (0, apiSignatureTokens.IndexOf ("implements ", StringComparison.Ordinal)).Trim (); + bool isClass = apiSignatureTokens.Contains ("class"); + options.DiagnosticWriter.WriteLine (apiSignatureTokens); + + var javaPackage = api.Packages.FirstOrDefault (p => p.Name == packageName); + if (javaPackage == null) { + javaPackage = new JavaPackage (api) { Name = packageName }; + api.Packages.Add (javaPackage); + } + + var javaType = isClass ? (JavaType)new JavaClass (javaPackage) : new JavaInterface (javaPackage); + javaType.Name = apiSignatureTokens.Substring (apiSignatureTokens.LastIndexOf (' ') + 1); + javaPackage.Types.Add (javaType); + + string sectionType = null; + var sep = new string [] { ", " }; + var ssep = new char [] { ' ' }; + foreach (var child in content.Descendants ()) { + if (child.Name == "h2") { + var value = child.Value; + switch (value) { + case "Public Constructors": + case "Protected Constructors": + case "Public Methods": + case "Protected Methods": + sectionType = value; + break; + } + continue; + } + + if (sectionType == null) + continue; + + if (child.Name != "a" || child.Attribute ("name") == null) + continue; + + var h4 = child.XPathSelectElement ("following-sibling::div/h4[contains(@class, 'jd-details-title')]"); + if (h4 == null) + continue; + + string sigTypeOnly = child.Attribute ("name").Value; + string sigTypeAndName = h4.Value.Replace ('\n', ' ').Replace ('\r', ' ').Trim (); + if (!sigTypeAndName.Contains ('(')) + continue; + JavaMethodBase javaMethod = null; + string name = sigTypeAndName.Substring (0, sigTypeAndName.IndexOf ('(')).Split (ssep, StringSplitOptions.RemoveEmptyEntries).Last (); + switch (sectionType) { + case "Public Constructors": + case "Protected Constructors": + javaMethod = new JavaConstructor (javaType) { Name = name }; + break; + case "Public Methods": + case "Protected Methods": + string mname = sigTypeAndName.Substring (0, sigTypeAndName.IndexOf ('(')); + javaMethod = new JavaMethod (javaType) { Name = name }; + break; + } + javaType.Members.Add (javaMethod); + + var paramTypes = SplitTypes (sigTypeOnly.Substring (sigTypeOnly.IndexOf ('(') + 1).TrimEnd (')'), 0).ToArray (); + var parameters = sigTypeAndName.Substring (sigTypeAndName.IndexOf ('(') + 1).TrimEnd (')') + .Split (sep, StringSplitOptions.RemoveEmptyEntries) + .Select (s => s.Trim ()) + .ToArray (); + foreach (var p in paramTypes.Zip (parameters, (to, tn) => new { Type = to, TypeAndName = tn }) + .Select (pp => new { Type = pp.Type, Name = pp.TypeAndName.Split (' ') [1] })) + javaMethod.Parameters.Add (new JavaParameter (javaMethod) { Name = p.Name, Type = p.Type }); + } + javaType.Members = javaType.Members.OfType () + .OrderBy (m => m.Name + "(" + string.Join (",", m.Parameters.Select (p => p.Type)) + ")") + .ToArray (); + } + foreach (var pkg in api.Packages) + pkg.Types = pkg.Types.OrderBy (t => t.Name).ToArray (); + api.Packages = api.Packages.OrderBy (p => p.Name).ToArray (); + + if (options.OutputTextFile != null) + api.WriteParameterNamesText (options.OutputTextFile); + if (options.OutputXmlFile != null) + api.WriteParameterNamesXml (options.OutputXmlFile); + } + + IEnumerable SplitTypes (string types, int start) + { + if (start == types.Length) + yield break; + var p2 = types.IndexOf (',', start); + if (p2 < 0) { + yield return types.Substring (start); + yield break; + } + + var p1 = types.IndexOf ('<', start); + if (p1 < 0 || p2 < p1) { + yield return types.Substring (start, p2 - start); + foreach (var s in SplitTypes (types, p2 + ", ".Length)) + yield return s; + yield break; + } + + int open = 1; + for (int i = p1 + 1; i < types.Length; i++) { + switch (types [i]) { + case '<': + open++; + break; + case '>': + open--; + if (open == 0) { + i++; // will be positioned either at ',' or at the end. + yield return types.Substring (start, i - start); + if (i + 2 < types.Length && types [i] == ',' && types [i + 1] == ' ') // skip ", " + i += 2; + if (i < types.Length) + foreach (var s in SplitTypes (types, i)) + yield return s; + yield break; + } + break; + } + } + throw new ArgumentException ("Unexpected list of parameters: " + types); + } + } + +} diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HTMLlat1.ent b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HTMLlat1.ent new file mode 100644 index 000000000..c64448604 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HTMLlat1.ent @@ -0,0 +1,194 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HTMLspecial.ent b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HTMLspecial.ent new file mode 100644 index 000000000..5ce5c6ad0 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HTMLspecial.ent @@ -0,0 +1,77 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HTMLsymbol.ent b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HTMLsymbol.ent new file mode 100644 index 000000000..524bfe11b --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HTMLsymbol.ent @@ -0,0 +1,241 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HtmlLoader.cs b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HtmlLoader.cs new file mode 100644 index 000000000..2e690f6f2 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/HtmlLoader.cs @@ -0,0 +1,209 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; +using System.Xml; +using System.Xml.Linq; +using Sgml; + +namespace Xamarin.Android.ApiTools.DroidDocImporter +{ + + class HtmlLoader + { + // okay, SgmlReader itself has something similar, but I need something that makes sure to resolve only embedded resources. + class EmbeddedResourceEntityResolver : Sgml.IEntityResolver + { + public IEntityContent GetContent (Uri uri) + { + return new EmbeddedResourceEntityContent (uri.LocalPath); + } + + class EmbeddedResourceEntityContent : Sgml.IEntityContent + { + public EmbeddedResourceEntityContent (string name) + { + if (name == null) + throw new ArgumentNullException (nameof (name)); + this.name = Path.GetFileName (name); + } + + string name; + + public Encoding Encoding { + get { return Encoding.UTF8; } + } + + public string MimeType { + get { return "text/plain"; } + } + + public Uri Redirect { + get { return new Uri ("file:///" + this.name); } + } + + public Stream Open () + { + return typeof (HtmlLoader).Assembly.GetManifestResourceStream (name); + } + } + } + + enum JavaDocKind + { + DroidDoc, + DroidDoc2, + Java6, + Java7, + Java8 + } + + Sgml.SgmlDtd HtmlDtd; + + public HtmlLoader () + { + HtmlDtd = LoadHtmlDtd (); + } + + Sgml.SgmlDtd LoadHtmlDtd () + { + return Sgml.SgmlDtd.Parse (new Uri ("file:///"), + "HTML", + "-//W3C//DTD HTML 4.01//EN", + "file:///strict.dtd", + string.Empty, new NameTable (), new EmbeddedResourceEntityResolver ()); + } + + public XElement GetJavaDocFile (string path) + { + JavaDocKind kind; + return GetJavaDocFile (path, out kind); + } + + XElement GetJavaDocFile (string path, out JavaDocKind kind) + { + kind = JavaDocKind.DroidDoc; + string rawHTML = ReadAndSanitizeHtmlFile (path); + if (rawHTML.Substring (0, 500).IndexOf ("Generated by javadoc (build 1.6", StringComparison.Ordinal) > 0) + kind = JavaDocKind.Java6; + if (rawHTML.Substring (0, 500).IndexOf ("Generated by javadoc (version 1.7", StringComparison.Ordinal) > 0) + kind = JavaDocKind.Java7; + if (rawHTML.Substring (0, 500).IndexOf ("Generated by javadoc (1.8", StringComparison.Ordinal) > 0) + kind = JavaDocKind.Java8; + var html = new Sgml.SgmlReader () { + InputStream = new StringReader (rawHTML), + CaseFolding = Sgml.CaseFolding.ToLower, + Dtd = HtmlDtd + }; + var doc = XDocument.Load (html, LoadOptions.SetLineInfo | LoadOptions.SetBaseUri); + return doc.Root; + } + + string ReadAndSanitizeHtmlFile (string path) + { + var info = new FileInfo (path); + var contents = new StringBuilder (checked((int)info.Length)); + bool veryFirstChar = true; + bool in_tag = false; + char in_quote = '\0'; + using (var r = info.OpenText ()) { + + int ch; + while ((ch = r.Read ()) >= 0) { + if (ch == '<') { + // Some of the HTML files are invalid, containing constructs such as + // 'foo <0', which should be 'foo <0'. + // There are also which needs to be replaced by <?>. + // + // We check the next char to '<' so that those with only valid NCName are treated as start tag. + char next = (char)r.Peek (); + if (!in_tag && (next == '/' || next == '!' || XmlConvert.IsStartNCNameChar (next))) { + in_tag = true; + contents.Append ('<'); + } else { + contents.Append ("<"); + } + } else if (ch == '>' && in_tag) { + if (in_quote != '\0') + contents.Append (">"); + else { + in_tag = false; + contents.Append ('>'); + } + } else if (in_tag && (ch == '"' || ch == '\'')) { + if (in_quote == ch) + in_quote = '\0'; + else if (in_quote == '\0') + in_quote = (char) ch; + contents.Append ((char) ch); + } else if (ch == '&') { + var b = new List (); + while ((ch = r.Read ()) >= 0 && ch != ';' && ch != ' ') // sometimes the input HTML contains '&' as a standalone character (i.e. Google emits invalid HTML... android/support/test/espresso/idling/CountingIdlingResource.html has that problem.) + b.Add ((char)ch); + var entity = new string (b.ToArray ()); + switch (entity) { + case "#124": + contents.Append ("|"); + break; + case "#160": + case "#xA0": + case "nbsp": + contents.Append ("\u00A0"); + break; + case "8211": + contents.Append ("\u2011"); + break; + default: + contents.Append ("&").Append (entity).Append (";"); + break; + } + } else if (ch == '\0') + contents.Append ("NUL"); + else + contents.Append ((char)ch); + if (veryFirstChar) + veryFirstChar = false; + } + } + + var firstPass = contents.ToString (); + contents.Clear (); + int open_count = 0; + bool in_quot = false, in_apos = false; + foreach (char ch in firstPass) { + if (ch == '"' && open_count > 0) { + if (in_quot) + open_count = 1; // reset. Something like <... ...="...<..." (without '>') happened + in_quot = !in_quot; + } + if (ch == '\'' && open_count > 0) { + if (in_quot) + open_count = 1; // reset. Something like <... ...='...<...' (without '>') happened + in_apos = !in_apos; + } + + if (ch == '<') { + if (open_count > 0) + contents.Append ("<"); + else + contents.Append ((char)ch); + open_count++; + } else if (ch == '>') { + if (open_count > 0) + open_count--; + if (open_count > 0) + contents.Append (">"); + else + contents.Append ((char)ch); + } else + contents.Append ((char)ch); + } + + // In some documents (e.g. java/util/prefs/Preferences.html) there are invalid !DOCTYPE in the middle, which breaks SgmlReader. Kill them. + contents.Replace ("", "<?>").ToString (); + } + } +} diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/strict.dtd b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/strict.dtd new file mode 100644 index 000000000..b27455943 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.DroidDocImporter/strict.dtd @@ -0,0 +1,870 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +%HTMLlat1; + + +%HTMLsymbol; + + +%HTMLspecial; + + + + + + + + + + + + + +]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.JavaStubImporter/JavaStubParser.cs b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.JavaStubImporter/JavaStubParser.cs new file mode 100644 index 000000000..2d01458d4 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.JavaStubImporter/JavaStubParser.cs @@ -0,0 +1,591 @@ +using System; +using System.Linq; +using Irony.Parsing; +using Irony.Ast; +using System.Collections.Generic; +using Xamarin.Android.Tools.ApiXmlAdjuster; + +namespace Xamarin.Android.ApiTools.JavaStubImporter +{ + [Language ("JavaStub", "1.0", "Java Stub grammar in Android SDK (android-stubs-src.jar)")] + public partial class JavaStubGrammar : Grammar + { + internal class NestedType : JavaMember + { + public NestedType (JavaType type) + : base (type) + { + } + + public JavaType Type { get; set; } + } + + NonTerminal DefaultNonTerminal (string label) + { + var nt = new NonTerminal (label); + nt.AstConfig.NodeCreator = delegate { throw new NotImplementedException (label); }; + return nt; + } + + KeyTerm Keyword (string label) + { + var ret = ToTerm (label); + ret.AstConfig.NodeCreator = delegate (AstContext ctx, ParseTreeNode node) { + node.AstNode = node.Token.ValueString; + }; + return ret; + } + + KeyTerm T (string term) + { + return Keyword (term); + } + + AstNodeCreator CreateArrayCreator () + { + return delegate (AstContext ctx, ParseTreeNode node) { + ProcessChildren (ctx, node); + node.AstNode = (from n in node.ChildNodes select (T)n.AstNode).ToArray (); + }; + } + + AstNodeCreator CreateStringFlattener (string separator = "", string prefix = "") + { + return delegate (AstContext ctx, ParseTreeNode node) { + ProcessChildren (ctx, node); + node.AstNode = prefix + string.Join (separator, node.ChildNodes.Select (n => n.AstNode?.ToString ())); + }; + } + + void SelectSingleChild (AstContext ctx, ParseTreeNode node) + { + ProcessChildren (ctx, node); + if (node.ChildNodes.Count == 1) + node.AstNode = node.ChildNodes.First ().AstNode; + } + + void ProcessChildren (AstContext ctx, ParseTreeNode node) + { + foreach (var cn in node.ChildNodes) { + if (cn.Term.AstConfig.NodeCreator != null) + cn.Term.AstConfig.NodeCreator (ctx, cn); + } + } + + AstNodeCreator SelectChildValueAt (int index) + { + return delegate (AstContext ctx, ParseTreeNode node) { + ProcessChildren (ctx, node); + node.AstNode = node.ChildNodes [index].AstNode; + }; + } + + void DoNothing (AstContext ctx, ParseTreeNode node) + { + ProcessChildren (ctx, node); + // do nothing except for processing children. + } + + public JavaStubGrammar () + { + CommentTerminal single_line_comment = new CommentTerminal ("SingleLineComment", "//", "\r", "\n"); + CommentTerminal delimited_comment = new CommentTerminal ("DelimitedComment", "/*", "*/"); + + NonGrammarTerminals.Add (single_line_comment); + NonGrammarTerminals.Add (delimited_comment); + + IdentifierTerminal identifier = new IdentifierTerminal ("identifier"); + + KeyTerm keyword_package = Keyword ("package"); + KeyTerm keyword_import = Keyword ("import"); + KeyTerm keyword_public = Keyword ("public"); + KeyTerm keyword_protected = Keyword ("protected"); + KeyTerm keyword_static = Keyword ("static"); + KeyTerm keyword_final = Keyword ("final"); + KeyTerm keyword_abstract = Keyword ("abstract"); + KeyTerm keyword_synchronized = Keyword ("synchronized"); + KeyTerm keyword_default = Keyword ("default"); + KeyTerm keyword_native = Keyword ("native"); + KeyTerm keyword_volatile = Keyword ("volatile"); + KeyTerm keyword_transient = Keyword ("transient"); + KeyTerm keyword_enum = Keyword ("enum"); + KeyTerm keyword_class = Keyword ("class"); + KeyTerm keyword_interface = Keyword ("interface"); + KeyTerm keyword_at_interface = Keyword ("@interface"); + KeyTerm keyword_extends = Keyword ("extends"); + KeyTerm keyword_implements = Keyword ("implements"); + KeyTerm keyword_throw = Keyword ("throw"); + KeyTerm keyword_throws = Keyword ("throws"); + KeyTerm keyword_null = Keyword ("null"); + KeyTerm keyword_super = Keyword ("super"); + KeyTerm keyword_true = Keyword ("true"); + KeyTerm keyword_false = Keyword ("false"); + KeyTerm keyword_new = Keyword ("new"); + + var compile_unit = DefaultNonTerminal ("compile_unit"); + var opt_package_decl = DefaultNonTerminal ("opt_package_declaration"); + var package_decl = DefaultNonTerminal ("package_declaration"); + var imports = DefaultNonTerminal ("imports"); + var import = DefaultNonTerminal ("import"); + var type_decls = DefaultNonTerminal ("type_decls"); + var type_decl = DefaultNonTerminal ("type_decl"); + var enum_decl = DefaultNonTerminal ("enum_decl"); + var class_decl = DefaultNonTerminal ("class_decl"); + var opt_generic_arg_decl = DefaultNonTerminal ("opt_generic_arg_decl"); + var opt_extends_decl = DefaultNonTerminal ("opt_extends_decl"); + var opt_implements_decl = DefaultNonTerminal ("opt_implements_decl"); + var implements_decl = DefaultNonTerminal ("implements_decl"); + var interface_decl = DefaultNonTerminal ("interface_decl"); + var iface_or_at_iface = DefaultNonTerminal ("iface_or_at_iface"); + var type_body = DefaultNonTerminal ("type_body"); + var type_members = DefaultNonTerminal ("type_members"); + var type_member = DefaultNonTerminal ("type_member"); + var nested_type_decl = DefaultNonTerminal ("nested_type_decl"); + var ctor_decl = DefaultNonTerminal ("ctor_decl"); + var method_decl = DefaultNonTerminal ("method_decl"); + var field_decl = DefaultNonTerminal ("field_decl"); + var opt_field_assignment = DefaultNonTerminal ("opt_field_assignment"); + var static_ctor_decl = DefaultNonTerminal ("static_ctor_decl"); + var opt_enum_member_initializers = DefaultNonTerminal ("opt_enum_member_initializers"); + var enum_member_initializers = DefaultNonTerminal ("enum_member_initializers"); + var enum_member_initializer = DefaultNonTerminal ("enum_member_initializer"); + var terminate_decl_or_body = DefaultNonTerminal ("terminate_decl_or_body"); + var assignments = DefaultNonTerminal ("assignments"); + var assignment = DefaultNonTerminal ("assignment"); + var assign_expr = DefaultNonTerminal ("assign_expr"); + var rvalue_expressions = DefaultNonTerminal ("rvalue_expressions"); + var rvalue_expression = DefaultNonTerminal ("rvalue_expression"); + var array_literal = DefaultNonTerminal ("array_literal"); + var annotations = DefaultNonTerminal ("annotations"); + var annotation = DefaultNonTerminal ("annotation"); + var opt_annotation_args = DefaultNonTerminal ("opt_annotation_args"); + var annotation_value_assignments = DefaultNonTerminal ("annotation_value_assignments"); + var annot_assign_expr = DefaultNonTerminal ("annot_assign_expr"); + var inner_annotations = DefaultNonTerminal ("inner_annotations"); + var modifiers_then_opt_generic_arg = DefaultNonTerminal ("modifiers_then_opt_generic_arg"); + var modifier_or_generic_arg = DefaultNonTerminal ("modifier_or_generic_arg"); + var modifiers = DefaultNonTerminal ("modifiers"); + var modifier = DefaultNonTerminal ("modifier"); + var argument_decls = DefaultNonTerminal ("argument_decls"); + var argument_decl = DefaultNonTerminal ("argument_decl"); + var comma_separated_types = DefaultNonTerminal ("comma_separated_types"); + var throws_decl = DefaultNonTerminal ("throws_decl"); + var opt_throws_decl = DefaultNonTerminal ("opt_throws_decl"); + var type_name = DefaultNonTerminal ("type_name"); + var dotted_identifier = DefaultNonTerminal ("dotted_identifier"); + var array_type = DefaultNonTerminal ("array_type"); + var vararg_type = DefaultNonTerminal ("vararg_type"); + var generic_type = DefaultNonTerminal ("generic_type"); + var generic_definition_arguments = DefaultNonTerminal ("generic_definition_arguments"); + var generic_definition_argument = DefaultNonTerminal ("generic_definition_argument"); + var generic_definition_constraints = DefaultNonTerminal ("generic_definition_constraints"); + var generic_definition_arguments_spec = DefaultNonTerminal ("generic_definition_arguments_spec"); + var generic_instance_arguments_spec = DefaultNonTerminal ("generic_instance_arguments_spec"); + var generic_instance_arguments = DefaultNonTerminal ("generic_instance_arguments"); + var generic_instance_argument = DefaultNonTerminal ("generic_instance_argument"); + var generic_instance_identifier_or_q = DefaultNonTerminal ("generic_instance_identifier_or_q"); + var generic_instance_constraints = DefaultNonTerminal ("generic_instance_constraints"); + var generic_instance_constraints_extends = DefaultNonTerminal ("generic_instance_constraints_extends"); + var generic_instance_constraints_super = DefaultNonTerminal ("generic_instance_constraints_super"); + var generic_instance_constraint_types = DefaultNonTerminal ("generic_instance_constraint_types"); + var impl_expressions = DefaultNonTerminal ("impl_expressions"); + var impl_expression = DefaultNonTerminal ("impl_expression"); + var call_super = DefaultNonTerminal ("call_super"); + var super_args = DefaultNonTerminal ("super_args"); + var default_value_expr = DefaultNonTerminal ("default_value_expr"); + var default_value_null_casted = DefaultNonTerminal ("default_value_null_casted"); + var default_value_literal = DefaultNonTerminal ("default_value_literal"); + var runtime_exception = DefaultNonTerminal ("runtime_exception"); + var numeric_terminal = TerminalFactory.CreateCSharpNumber ("numeric_value_literal"); + numeric_terminal.AddPrefix ("-", NumberOptions.AllowSign); + numeric_terminal.AddPrefix ("+", NumberOptions.AllowSign); + //numeric_terminal.AddSuffix ("f"); + numeric_terminal.AddSuffix ("L"); + var numeric_literal = DefaultNonTerminal ("numeric_literal"); + var string_literal = TerminalFactory.CreateCSharpString ("string_literal"); + var value_literal = DefaultNonTerminal ("value_literal"); + + // + + compile_unit.Rule = opt_package_decl + imports + type_decls; + opt_package_decl.Rule = package_decl | Empty; + package_decl.Rule = keyword_package + type_name + ";"; + imports.Rule = MakeStarRule (imports, import); + import.Rule = keyword_import + type_name + ";"; + type_decls.Rule = MakeStarRule (type_decls, type_decl); + + type_decl.Rule = class_decl | interface_decl | enum_decl; + // FIXME: those modifiers_then_opt_generic_arg should be actually just modifiers... see modifiers_then_opt_generic_arg.Rule below. + enum_decl.Rule = annotations + modifiers_then_opt_generic_arg + keyword_enum + identifier + opt_implements_decl + "{" + opt_enum_member_initializers + type_members + "}"; + class_decl.Rule = annotations + modifiers_then_opt_generic_arg + keyword_class + identifier + opt_generic_arg_decl + opt_extends_decl + opt_implements_decl + type_body; + interface_decl.Rule = annotations + modifiers_then_opt_generic_arg + iface_or_at_iface + identifier + opt_generic_arg_decl + opt_extends_decl + opt_implements_decl + type_body; + iface_or_at_iface.Rule = keyword_interface | keyword_at_interface; + + opt_generic_arg_decl.Rule = Empty | "<" + generic_definition_arguments + ">"; + opt_extends_decl.Rule = Empty | keyword_extends + implements_decl; // when it is used with an interface, it can be more than one... + opt_implements_decl.Rule = Empty | keyword_implements + implements_decl; + implements_decl.Rule = MakePlusRule (implements_decl, ToTerm (","), type_name); + type_body.Rule = T ("{") + type_members + T ("}"); + annotations.Rule = MakeStarRule (annotations, annotation); + annotation.Rule = T ("@") + type_name + opt_annotation_args; + opt_annotation_args.Rule = Empty | T ("(") + annotation_value_assignments + T (")"); + annotation_value_assignments.Rule = MakeStarRule (annotation_value_assignments, ToTerm (","), annot_assign_expr); + annot_assign_expr.Rule = assign_expr;// | identifier + "=" + "{" + inner_annotations + "}"; + inner_annotations.Rule = MakeStarRule (inner_annotations, ToTerm (","), annotations); + + // HACK: I believe this is an Irony bug that adding opt_generic_arg_decl here results in shift-reduce conflict, but it's too complicated to investigate the actual issue. + // As a workaround I add generic arguments as part of this "modifier" so that it can be safely added to a generic method declaration. + modifiers_then_opt_generic_arg.Rule = MakeStarRule (modifiers_then_opt_generic_arg, modifier_or_generic_arg); + modifiers.Rule = MakeStarRule (modifiers, modifier); + modifier_or_generic_arg.Rule = modifier | generic_definition_arguments_spec; + modifier.Rule = keyword_public | keyword_protected | keyword_final | keyword_abstract | keyword_synchronized | keyword_default | keyword_native | keyword_volatile | keyword_transient | keyword_static; + + type_members.Rule = MakeStarRule (type_members, type_member); + type_member.Rule = nested_type_decl | ctor_decl | method_decl | field_decl | static_ctor_decl; + nested_type_decl.Rule = type_decl; + opt_enum_member_initializers.Rule = Empty | enum_member_initializers + ";"; + enum_member_initializers.Rule = MakeStarRule (enum_member_initializers, ToTerm (","), enum_member_initializer); + enum_member_initializer.Rule = identifier + "(" + ")"; + static_ctor_decl.Rule = annotations + keyword_static + "{" + assignments + "}"; + assignments.Rule = MakeStarRule (assignments, assignment); + assignment.Rule = assign_expr + ";"; + assign_expr.Rule = identifier + "=" + rvalue_expression; + rvalue_expressions.Rule = MakeStarRule (rvalue_expressions, ToTerm (","), rvalue_expression); + rvalue_expression.Rule = value_literal | type_name | identifier | array_literal | annotation; + array_literal.Rule = "{" + rvalue_expressions + "}"; + + field_decl.Rule = annotations + modifiers_then_opt_generic_arg + type_name + identifier + opt_field_assignment + ";"; + opt_field_assignment.Rule = Empty | "=" + rvalue_expression; + terminate_decl_or_body.Rule = ";" | ("{" + impl_expressions + "}") | (keyword_default + default_value_literal + ";"); + + ctor_decl.Rule = annotations + modifiers_then_opt_generic_arg + identifier + "(" + argument_decls + ")" + opt_throws_decl + terminate_decl_or_body; // these Empties can make the structure common to method_decl. + + method_decl.Rule = annotations + modifiers_then_opt_generic_arg + /*opt_generic_arg_decl*/ type_name + identifier + "(" + argument_decls + ")" + opt_throws_decl + terminate_decl_or_body; + + impl_expressions.Rule = MakeStarRule (impl_expressions, impl_expression); + impl_expression.Rule = call_super | runtime_exception; + call_super.Rule = keyword_super + "(" + super_args + ")" + ";"; + super_args.Rule = MakeStarRule (super_args, ToTerm (","), default_value_expr); + default_value_expr.Rule = keyword_null | default_value_null_casted | default_value_literal; + default_value_null_casted.Rule = "(" + type_name + ")" + keyword_null; + default_value_literal.Rule = numeric_terminal | "\"\"" | "{" + "}" | keyword_true | keyword_false; + runtime_exception.Rule = keyword_throw + keyword_new + identifier + "(\"Stub!\"" + ")" + ";"; + + argument_decls.Rule = annotations | MakeStarRule (argument_decls, ToTerm (","), argument_decl); + + argument_decl.Rule = annotations + type_name + identifier; + + throws_decl.Rule = keyword_throws + comma_separated_types; + comma_separated_types.Rule = MakeStarRule (comma_separated_types, ToTerm (","), type_name); + opt_throws_decl.Rule = Empty | throws_decl; + + type_name.Rule = dotted_identifier | array_type | vararg_type | generic_type; + + vararg_type.Rule = type_name + T ("..."); + array_type.Rule = type_name + ("[") + T ("]"); + + generic_definition_arguments_spec.Rule = "<" + generic_definition_arguments + ">"; + generic_type.Rule = dotted_identifier + generic_instance_arguments_spec; + generic_instance_arguments_spec.Rule = "<" + generic_instance_arguments + ">"; + generic_definition_arguments.Rule = MakePlusRule (generic_definition_arguments, ToTerm (","), generic_definition_argument); + generic_definition_argument.Rule = identifier + generic_definition_constraints; + generic_definition_constraints.Rule = Empty | generic_instance_constraints_extends | generic_instance_constraints_super; + generic_instance_arguments.Rule = MakePlusRule (generic_instance_arguments, ToTerm (","), generic_instance_argument); + generic_instance_argument.Rule = generic_instance_identifier_or_q + generic_instance_constraints; + generic_instance_identifier_or_q.Rule = type_name | T ("?"); + generic_instance_constraints.Rule = Empty | generic_instance_constraints_extends | generic_instance_constraints_super; + generic_instance_constraints_extends.Rule = keyword_extends + generic_instance_constraint_types; + generic_instance_constraints_super.Rule = keyword_super + generic_instance_constraint_types; + generic_instance_constraint_types.Rule = MakePlusRule (generic_instance_constraint_types, ToTerm ("&"), type_name); + + dotted_identifier.Rule = MakePlusRule (dotted_identifier, ToTerm ("."), identifier); + + numeric_literal.Rule = numeric_terminal; + numeric_literal.Rule |= "(" + numeric_literal + "/" + numeric_literal + ")"; + value_literal.Rule = string_literal | numeric_literal | keyword_null; + + // Define AST node creators + + Func stripGenerics = s => s.IndexOf ('<') > 0 ? s.Substring (0, s.IndexOf ('<')) : s; + + single_line_comment.AstConfig.NodeCreator = DoNothing; + delimited_comment.AstConfig.NodeCreator = DoNothing; + identifier.AstConfig.NodeCreator = (ctx, node) => node.AstNode = node.Token.ValueString; + compile_unit.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = new JavaPackage (null) { Name = (string)node.ChildNodes [0].AstNode, Types = ((IEnumerable)node.ChildNodes [2].AstNode).ToList () }; + }; + opt_package_decl.AstConfig.NodeCreator = SelectSingleChild; + package_decl.AstConfig.NodeCreator = SelectChildValueAt (1); + imports.AstConfig.NodeCreator = CreateArrayCreator (); + import.AstConfig.NodeCreator = SelectChildValueAt (1); + type_decls.AstConfig.NodeCreator = CreateArrayCreator (); + type_decl.AstConfig.NodeCreator = SelectSingleChild; + opt_generic_arg_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = node.ChildNodes.Count == 0 ? null : node.ChildNodes [1].AstNode; + }; + opt_extends_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = node.ChildNodes.Count == 0 ? null : node.ChildNodes [1].AstNode; + }; + opt_implements_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = node.ChildNodes.Count == 0 ? null : node.ChildNodes [1].AstNode; + }; + implements_decl.AstConfig.NodeCreator = CreateArrayCreator (); + Action fillType = (node, type) => { + var modsOrTps = (IEnumerable)node.ChildNodes [1].AstNode; + var mods = modsOrTps.OfType (); + bool isEnum = node.ChildNodes [2].AstNode as string == "enum"; + type.Abstract |= mods.Contains ("abstract"); + type.Static |= mods.Contains ("static"); + type.Final |= mods.Contains ("final"); + type.Visibility = mods.FirstOrDefault (s => s == "public" || s == "protected") ?? ""; + type.Name = (string)node.ChildNodes [3].AstNode; + type.Deprecated = ((IEnumerable)node.ChildNodes [0].AstNode).Any (v => v == "java.lang.Deprecated" || v == "Deprecated") ? "deprecated" : "not deprecated"; + type.TypeParameters = isEnum ? null : (JavaTypeParameters) node.ChildNodes [4].AstNode; + type.Members = (IList) node.ChildNodes [7].AstNode; + }; + enum_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + var type = new JavaClass (null) { Extends = "java.lang.Enum", Final = true }; + var methods = new JavaMember [] { + new JavaMethod (null) { + Deprecated = "not deprecated", + Name = "valueOf", + // Return needs to be filled later, with full package name. + Static = true, + Visibility = "public", + Parameters = new JavaParameter [] { new JavaParameter (null) { Name = "name", Type = "java.lang.String" }}, + }, + new JavaMethod (null) { + Deprecated = "not deprecated", + Name = "values", + // Return needs to be filled later, with full package name. + Static = true, + Visibility = "public", + Parameters = new JavaParameter [0], + } + }; + fillType (node, type); + var ml = new List (methods); + if (node.ChildNodes [6].ChildNodes.Count > 0) + ml.AddRange ( + ((IEnumerable)node.ChildNodes [6].AstNode) + .Select (s => new JavaField (null) { + Name = s, + Final = true, + Deprecated = "not deprecated", + Static = true, + // Type needs to be filled later, with full package name. + Visibility = "public" + }) + .Concat (type.Members) + ); + type.Members = ml; + node.AstNode = type; + }; + class_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + var exts = ((IEnumerable) node.ChildNodes [5].AstNode) ?? Enumerable.Empty (); + var impls = ((IEnumerable) node.ChildNodes [6].AstNode) ?? Enumerable.Empty (); + var ext = exts.FirstOrDefault () ?? "java.lang.Object"; + var type = new JavaClass (null) { + Extends = stripGenerics (ext), + ExtendsGeneric = ext, + Implements = impls.Select (s => new JavaImplements { Name = stripGenerics (s), NameGeneric = s }).ToArray (), + }; + fillType (node, type); + node.AstNode = type; + }; + interface_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + bool annot = node.ChildNodes [2].AstNode as string == "@interface"; + var exts = ((IEnumerable)node.ChildNodes [5].AstNode) ?? Enumerable.Empty (); + var impls = ((IEnumerable)node.ChildNodes [6].AstNode) ?? Enumerable.Empty (); + var type = new JavaInterface (null) { + Implements = exts.Concat (impls).Select (s => new JavaImplements { Name = stripGenerics (s), NameGeneric = s }).ToList (), + }; + if (annot) + type.Implements.Add (new JavaImplements { Name = "java.lang.annotation.Annotation", NameGeneric = "java.lang.annotation.Annotation" }); + fillType (node, type); + node.AstNode = type; + }; + iface_or_at_iface.AstConfig.NodeCreator = SelectSingleChild; + type_body.AstConfig.NodeCreator = SelectChildValueAt (1); + type_members.AstConfig.NodeCreator = CreateArrayCreator (); + type_member.AstConfig.NodeCreator = SelectSingleChild; + nested_type_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = new NestedType (null) { Type = (JavaType) node.ChildNodes [0].AstNode }; + }; + Action fillMethodBase = (node, method) => { + bool ctor = node.ChildNodes.Count == 8; + var modsOrTps = (IEnumerable)node.ChildNodes [1].AstNode; + var mods = modsOrTps.OfType (); + method.Static = mods.Contains ("static"); + method.Visibility = mods.FirstOrDefault (s => s == "public" || s == "protected") ?? ""; + method.Name = (string)node.ChildNodes [ctor ? 2 : 3].AstNode; + method.Parameters = ((IEnumerable)node.ChildNodes [ctor ? 4 : 5].AstNode).ToArray (); + method.ExtendedSynthetic = mods.Contains ("synthetic"); + // HACK: Exception "name" can be inconsistent for nested types, and this nested type detection is hacky. + Func stripPackage = s => { + var packageTokens = s.Split ('.').TakeWhile (t => !t.Any (c => Char.IsUpper (c))); + return s.Substring (Enumerable.Sum (packageTokens.Select (t => t.Length)) + packageTokens.Count ()); + }; + method.Exceptions = ((IEnumerable)node.ChildNodes [ctor ? 6 : 7].AstNode)?.Select (s => new JavaException { Type = s, Name = stripPackage (s) })?.ToArray (); + method.Deprecated = ((IEnumerable)node.ChildNodes [0].AstNode).Any (v => v == "java.lang.Deprecated" || v == "Deprecated") ? "deprecated" : "not deprecated"; + method.Final = mods.Contains ("final"); + method.TypeParameters = modsOrTps.OfType ().FirstOrDefault (); + }; + ctor_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + var annots = node.ChildNodes [0].AstNode; + var modsOrTps = (IEnumerable)node.ChildNodes [1].AstNode; + var mods = modsOrTps.OfType (); + var ctor = new JavaConstructor (null); + fillMethodBase (node, ctor); + node.AstNode = ctor; + }; + method_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + var annots = node.ChildNodes [0].AstNode; + var modsOrTps = (IEnumerable)node.ChildNodes [1].AstNode; + var mods = modsOrTps.OfType (); + var method = new JavaMethod (null) { + Return = (string)node.ChildNodes [2].AstNode, + Abstract = mods.Contains ("abstract"), + Native = mods.Contains ("native"), + Synchronized = mods.Contains ("synchronized"), + ExtendedSynthetic = mods.Contains ("synthetic"), + }; + fillMethodBase (node, method); + node.AstNode = method; + }; + field_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + var annots = node.ChildNodes [0].AstNode; + var modsOrTps = (IEnumerable)node.ChildNodes [1].AstNode; + var mods = modsOrTps.OfType (); + var value = node.ChildNodes [4].AstNode?.ToString (); + var type = (string)node.ChildNodes [2].AstNode; + node.AstNode = new JavaField (null) { + Static = mods.Contains ("static"), + Visibility = mods.FirstOrDefault (s => s == "public" || s == "protected") ?? "", + Type = stripGenerics (type), + TypeGeneric = type, + Name = (string) node.ChildNodes [3].AstNode, + Deprecated = ((IEnumerable) node.ChildNodes [0].AstNode).Any (v => v == "java.lang.Deprecated" || v == "Deprecated") ? "deprecated" : "not deprecated", + Value = value == "null" ? null : value, // null will not be explicitly written. + Volatile = mods.Contains ("volatile"), + Final = mods.Contains ("final"), + Transient = mods.Contains ("transient"), + }; + }; + opt_field_assignment.AstConfig.NodeCreator = (ctx, node) => node.AstNode = node.ChildNodes.Count > 0 ? node.ChildNodes [1].AstNode : null; + static_ctor_decl.AstConfig.NodeCreator = DoNothing; // static constructors are ignorable. + opt_enum_member_initializers.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + if (node.ChildNodes.Count > 0) + node.AstNode = node.ChildNodes [0].AstNode; + }; + enum_member_initializers.AstConfig.NodeCreator = CreateArrayCreator (); + enum_member_initializer.AstConfig.NodeCreator = SelectChildValueAt (0); + terminate_decl_or_body.AstConfig.NodeCreator = DoNothing; // method/ctor body doesn't matter. + assignments.AstConfig.NodeCreator = CreateArrayCreator (); + assignment.AstConfig.NodeCreator = SelectChildValueAt (0); + assign_expr.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = new KeyValuePair ((string)node.ChildNodes [0].AstNode, node.ChildNodes [2].AstNode?.ToString ()); + }; + rvalue_expressions.AstConfig.NodeCreator = CreateArrayCreator (); + rvalue_expression.AstConfig.NodeCreator = SelectSingleChild; + array_literal.AstConfig.NodeCreator = CreateStringFlattener (); + annotations.AstConfig.NodeCreator = CreateArrayCreator (); + annotation.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node.ChildNodes [1]); // we only care about name. + node.AstNode = node.ChildNodes [1].AstNode; + }; + opt_annotation_args.AstConfig.NodeCreator = DoNothing; + annotation_value_assignments.AstConfig.NodeCreator = DoNothing; + annot_assign_expr.AstConfig.NodeCreator = DoNothing; + modifiers_then_opt_generic_arg.AstConfig.NodeCreator = CreateArrayCreator (); + modifier_or_generic_arg.AstConfig.NodeCreator = SelectSingleChild; + modifiers.AstConfig.NodeCreator = CreateArrayCreator (); + modifier.AstConfig.NodeCreator = CreateStringFlattener (); + argument_decls.AstConfig.NodeCreator = CreateArrayCreator (); + argument_decl.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = new JavaParameter (null) { Type = (string) node.ChildNodes [1].AstNode, Name = (string) node.ChildNodes [2].AstNode }; + }; + opt_throws_decl.AstConfig.NodeCreator = SelectSingleChild; + throws_decl.AstConfig.NodeCreator = SelectChildValueAt (1); + comma_separated_types.AstConfig.NodeCreator = CreateArrayCreator (); + type_name.AstConfig.NodeCreator = SelectSingleChild; + dotted_identifier.AstConfig.NodeCreator = CreateStringFlattener ("."); + array_type.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = node.ChildNodes [0].AstNode + "[]"; + }; + vararg_type.AstConfig.NodeCreator = CreateStringFlattener (); + generic_type.AstConfig.NodeCreator = CreateStringFlattener (); + generic_definition_arguments_spec.AstConfig.NodeCreator = SelectChildValueAt (1); + generic_instance_arguments_spec.AstConfig.NodeCreator = (ctx, node) => { + // It is distinct from generic type parameters definition. + ProcessChildren (ctx, node); + node.AstNode = "<" + string.Join (", ", (IEnumerable) node.ChildNodes [1].AstNode) + ">"; + }; + generic_definition_arguments.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = new JavaTypeParameters ((JavaMethod) null) { TypeParameters = node.ChildNodes.Select (c => c.AstNode).Cast ().ToList () }; + }; + generic_definition_argument.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + node.AstNode = new JavaTypeParameter (null) { + Name = (string) node.ChildNodes [0].AstNode, + GenericConstraints = (JavaGenericConstraints) node.ChildNodes [1].AstNode + }; + }; + generic_definition_constraints.AstConfig.NodeCreator = SelectSingleChild; + generic_instance_arguments.AstConfig.NodeCreator = CreateArrayCreator (); + generic_instance_argument.AstConfig.NodeCreator = CreateStringFlattener (); + generic_instance_identifier_or_q.AstConfig.NodeCreator = SelectSingleChild; + generic_instance_constraints.AstConfig.NodeCreator = (ctx, node) => { + ProcessChildren (ctx, node); + var c = (JavaGenericConstraints) node.ChildNodes.FirstOrDefault ()?.AstNode; + if (c != null) + node.AstNode = " " + c.BoundsType + " " + string.Join (" & ", c.GenericConstraints.Select (cc => cc.Type)); + }; + AstNodeCreator createGenericConstaints = (ctx, node) => { + ProcessChildren (ctx, node); + var cl = ((IEnumerable) node.ChildNodes [1].AstNode).Select (s => new JavaGenericConstraint { Type = s }); + node.AstNode = new JavaGenericConstraints () { + BoundsType = (string) node.ChildNodes [0].AstNode, + GenericConstraints = cl.Any () ? cl.ToArray () : null, + }; + }; + generic_instance_constraints_extends.AstConfig.NodeCreator = createGenericConstaints; + generic_instance_constraints_super.AstConfig.NodeCreator = createGenericConstaints; + generic_instance_constraint_types.AstConfig.NodeCreator = CreateArrayCreator (); + impl_expressions.AstConfig.NodeCreator = CreateArrayCreator (); + impl_expression.AstConfig.NodeCreator = SelectSingleChild; + // each expression item is not seriously processed. + // They are insignificant except for consts, and for consts they are just string values. + call_super.AstConfig.NodeCreator = CreateStringFlattener (); + super_args.AstConfig.NodeCreator = CreateStringFlattener (); + default_value_expr.AstConfig.NodeCreator = CreateStringFlattener (); + default_value_null_casted.AstConfig.NodeCreator = CreateStringFlattener (); + default_value_literal.AstConfig.NodeCreator = CreateStringFlattener (); + runtime_exception.AstConfig.NodeCreator = CreateStringFlattener (); + Func stripTail = (s, t) => s.EndsWith (t, StringComparison.Ordinal) ? s.Substring (0, s.Length - t.Length) : s; + numeric_terminal.AstConfig.NodeCreator = (ctx, node) => node.AstNode = stripTail (stripTail (node.Token.Text, "L"), "f"); + numeric_literal.AstConfig.NodeCreator = CreateStringFlattener (); + string_literal.AstConfig.NodeCreator = (ctx, node) => node.AstNode = '"' + node.Token.ValueString + '"'; + value_literal.AstConfig.NodeCreator = SelectSingleChild; + + this.Root = compile_unit; + } + } +} + diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.JavaStubImporter/JavaStubSourceImporter.cs b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.JavaStubImporter/JavaStubSourceImporter.cs new file mode 100644 index 000000000..90877e46f --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.JavaStubImporter/JavaStubSourceImporter.cs @@ -0,0 +1,108 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.IO.Compression; +using System.Linq; +using Irony.Parsing; +using Xamarin.Android.Tools.ApiXmlAdjuster; + +namespace Xamarin.Android.ApiTools.JavaStubImporter +{ + public class JavaStubSourceImporter + { + public void Import (ImporterOptions options) + { + ZipArchive zip; + using (var stream = File.OpenRead (options.InputZipArchive)) { + zip = new ZipArchive (stream); + foreach (var ent in zip.Entries) { + options.DiagnosticWriter.WriteLine (ent.FullName); + if (!ent.Name.EndsWith (".java", StringComparison.OrdinalIgnoreCase)) + continue; + var java = new StreamReader (ent.Open ()).ReadToEnd (); + if (!ParseJava (java)) + break; + } + } + foreach (var pkg in api.Packages) { + foreach (var t in pkg.Types) { + // Our API definitions don't contain non-public members, so remove those (but it does contain non-public types). + t.Members = t.Members.Where (m => m != null && m.Visibility != "").ToList (); + // Constructor "type" is the full name of the class. + foreach (var c in t.Members.OfType ()) + c.Type = (pkg.Name.Length > 0 ? (pkg.Name + '.') : string.Empty) + t.Name; + // Pupulated enum fields need type to be filled + var cls = t as JavaClass; + if (cls != null && cls.Extends == "java.lang.Enum") { + cls.ExtendsGeneric = "java.lang.Enum<" + pkg.Name + "." + t.Name + ">"; + foreach (var m in cls.Members.OfType ()) { + if (m.Type == null) { + m.Type = pkg.Name + "." + t.Name; + m.TypeGeneric = pkg.Name + "." + t.Name; + } + } + foreach (var m in cls.Members.OfType ()) { + if (m.Name == "valueOf") + m.Return = pkg.Name + "." + t.Name; + else if (m.Name == "values") + m.Return = pkg.Name + "." + t.Name + "[]"; + } + } + t.Members = t.Members.OfType () + .OrderBy (m => m.Name + "(" + string.Join (",", m.Parameters.Select (p => p.Type)) + ")") + .ToArray (); + } + pkg.Types = pkg.Types.OrderBy (t => t.Name).ToArray (); + } + api.Packages = api.Packages.OrderBy (p => p.Name).ToArray (); + + if (options.OutputTextFile != null) + api.WriteParameterNamesText (options.OutputTextFile); + if (options.OutputXmlFile != null) + api.WriteParameterNamesXml (options.OutputXmlFile); + } + + JavaStubGrammar grammar = new JavaStubGrammar () { LanguageFlags = LanguageFlags.Default | LanguageFlags.CreateAst }; + JavaApi api = new JavaApi (); + + bool ParseJava (string javaSourceText) + { + var parser = new Irony.Parsing.Parser (grammar); + var result = parser.Parse (javaSourceText); + foreach (var m in result.ParserMessages) + Console.WriteLine ($"{m.Level} {m.Location} {m.Message}"); + if (result.HasErrors ()) + return false; + var parsedPackage = (JavaPackage)result.Root.AstNode; + FlattenNestedTypes (parsedPackage); + var pkg = api.Packages.FirstOrDefault (p => p.Name == parsedPackage.Name); + if (pkg == null) { + api.Packages.Add (parsedPackage); + pkg = parsedPackage; + } else + foreach (var t in parsedPackage.Types) + pkg.Types.Add (t); + pkg.Types = pkg.Types.OrderBy (t => t.Name).ToList (); + return true; + } + + void FlattenNestedTypes (JavaPackage package) + { + Action,JavaType> flatten = null; + flatten = (list, t) => { + list.Add (t); + foreach (var nt in t.Members.OfType ()) { + nt.Type.Name = t.Name + '.' + nt.Type.Name; + foreach (var nc in nt.Type.Members.OfType ()) + nc.Name = nt.Type.Name; + flatten (list, nt.Type); + } + t.Members = t.Members.Where (_ => !(_ is JavaStubGrammar.NestedType)).ToArray (); + }; + var results = new List (); + foreach (var t in package.Types) + flatten (results, t); + package.Types = results.ToList (); + } + } +} diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.ParameterNameExtractor.csproj b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.ParameterNameExtractor.csproj new file mode 100644 index 000000000..92a7536cc --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/Xamarin.Android.ApiTools.ParameterNameExtractor.csproj @@ -0,0 +1,76 @@ + + + + Debug + AnyCPU + {A7F30A05-820C-4580-8246-B84D55147408} + Library + Xamarin.Android.ApiTools.ParameterNameExtractor + Xamarin.Android.ApiTools.ParameterNameExtractor + v4.6.1 + + + true + full + false + bin\Debug + DEBUG; + prompt + 4 + false + + + true + bin\Release + prompt + 4 + false + + + + + ..\packages\Mono.Options.5.3.0.1\lib\net4-client\Mono.Options.dll + + + ..\packages\Irony.0.9.1\lib\net40\Irony.dll + + + ..\packages\Microsoft.Xml.SgmlReader.1.8.14\lib\net45\SgmlReaderDll.dll + + + + + + + + + + + + + + + + + + + + + + + + + HTMLlat1.ent + + + HTMLspecial.ent + + + HTMLsymbol.ent + + + strict.dtd + + + + \ No newline at end of file diff --git a/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/packages.config b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/packages.config new file mode 100644 index 000000000..6b5459731 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/Xamarin.Android.ApiTools.ParameterNameExtractor/packages.config @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file diff --git a/build-tools/xamarin-android-docimporter-ng/generate.sh b/build-tools/xamarin-android-docimporter-ng/generate.sh new file mode 100755 index 000000000..57da7eb78 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/generate.sh @@ -0,0 +1,19 @@ +#!/bin/bash + +ANDROID_TOOLCHAIN=~/android-toolchain +ANDROID_SDK_PATH=~/android-sdk-linux + +DROID_DOC_API_LEVELS="10 15 16 17 18 19 20 21 22 23" +JAVA_STUB_API_LEVELS="24 25 26 27" + + +for n in $JAVA_STUB_API_LEVELS +do + time mono --debug xamarin-android-docimporter-ng/bin/Debug/xamarin-android-docimporter-ng.exe -source-stub-zip=$ANDROID_SDK_PATH/platforms/android-$n/android-stubs-src.jar -output-text=api-$n.params.txt -output-xml=api-$n.params.xml -verbose -framework-only +done + +for API_LEVEL in $DROID_DOC_API_LEVELS +do + time mono --debug xamarin-android-docimporter-ng/bin/Debug/xamarin-android-docimporter-ng.exe -droiddoc=$ANDROID_TOOLCHAIN/docs/docs-api-$API_LEVEL/ -output-text=api-$API_LEVEL.params.txt -output-xml=api-$API_LEVEL.params.xml -verbose -framework-only +done + diff --git a/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng.sln b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng.sln new file mode 100644 index 000000000..f287f376f --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng.sln @@ -0,0 +1,23 @@ + +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio 2012 +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "xamarin-android-docimporter-ng", "xamarin-android-docimporter-ng\xamarin-android-docimporter-ng.csproj", "{A9D8ADB5-8312-4977-A9EC-C1635F542535}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Xamarin.Android.ApiTools.ParameterNameExtractor", "Xamarin.Android.ApiTools.ParameterNameExtractor\Xamarin.Android.ApiTools.ParameterNameExtractor.csproj", "{A7F30A05-820C-4580-8246-B84D55147408}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|x86 = Debug|x86 + Release|x86 = Release|x86 + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {A9D8ADB5-8312-4977-A9EC-C1635F542535}.Debug|x86.ActiveCfg = Debug|x86 + {A9D8ADB5-8312-4977-A9EC-C1635F542535}.Debug|x86.Build.0 = Debug|x86 + {A9D8ADB5-8312-4977-A9EC-C1635F542535}.Release|x86.ActiveCfg = Release|x86 + {A9D8ADB5-8312-4977-A9EC-C1635F542535}.Release|x86.Build.0 = Release|x86 + {A7F30A05-820C-4580-8246-B84D55147408}.Debug|x86.ActiveCfg = Debug|Any CPU + {A7F30A05-820C-4580-8246-B84D55147408}.Debug|x86.Build.0 = Debug|Any CPU + {A7F30A05-820C-4580-8246-B84D55147408}.Release|x86.ActiveCfg = Release|Any CPU + {A7F30A05-820C-4580-8246-B84D55147408}.Release|x86.Build.0 = Release|Any CPU + EndGlobalSection +EndGlobal diff --git a/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/Program.cs b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/Program.cs new file mode 100644 index 000000000..d50fd6002 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/Program.cs @@ -0,0 +1,40 @@ +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Xml; +using System.Xml.Linq; +using System.Xml.XPath; +using Mono.Options; +using Xamarin.Android.ApiTools.DroidDocImporter; +using Xamarin.Android.ApiTools.JavaStubImporter; + +namespace Xamarin.Android.ApiTools +{ + public class Driver + { + public static void Main (string [] args) + { + var options = CreateOptions (args); + if (options.DocumentDirectory != null) + new DroidDocScrapingImporter ().Import (options); + if (options.InputZipArchive != null) + new JavaStubSourceImporter ().Import (options); + } + + static ImporterOptions CreateOptions (string [] args) + { + var ret = new ImporterOptions (); + var options = new OptionSet { + {"droiddoc=", v => ret.DocumentDirectory = v }, + {"source-stub-zip=", v => ret.InputZipArchive = v }, + {"output-text=", v => ret.OutputTextFile = v }, + {"output-xml=", v => ret.OutputXmlFile = v }, + {"verbose", v => ret.DiagnosticWriter = Console.Error }, + {"framework-only", v => ret.FrameworkOnly = true }, + }; + options.Parse (args); + return ret; + } + } +} diff --git a/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/Properties/AssemblyInfo.cs b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/Properties/AssemblyInfo.cs new file mode 100644 index 000000000..e49f339d9 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/Properties/AssemblyInfo.cs @@ -0,0 +1,26 @@ +using System.Reflection; +using System.Runtime.CompilerServices; + +// Information about this assembly is defined by the following attributes. +// Change them to the values specific to your project. + +[assembly: AssemblyTitle ("xamarin-android-docimporter-ng")] +[assembly: AssemblyDescription ("")] +[assembly: AssemblyConfiguration ("")] +[assembly: AssemblyCompany ("")] +[assembly: AssemblyProduct ("")] +[assembly: AssemblyCopyright ("${AuthorCopyright}")] +[assembly: AssemblyTrademark ("")] +[assembly: AssemblyCulture ("")] + +// The assembly version has the format "{Major}.{Minor}.{Build}.{Revision}". +// The form "{Major}.{Minor}.*" will automatically update the build and revision, +// and "{Major}.{Minor}.{Build}.*" will update just the revision. + +[assembly: AssemblyVersion ("1.0.*")] + +// The following attributes are used to specify the signing key for the assembly, +// if desired. See the Mono documentation for more information about signing. + +//[assembly: AssemblyDelaySign(false)] +//[assembly: AssemblyKeyFile("")] diff --git a/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/packages.config b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/packages.config new file mode 100644 index 000000000..2deaf39ec --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/packages.config @@ -0,0 +1,4 @@ + + + + \ No newline at end of file diff --git a/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng.csproj b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng.csproj new file mode 100644 index 000000000..c2abd0073 --- /dev/null +++ b/build-tools/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng/xamarin-android-docimporter-ng.csproj @@ -0,0 +1,54 @@ + + + + Debug + x86 + {A9D8ADB5-8312-4977-A9EC-C1635F542535} + Exe + xamarinandroiddocimporterng + xamarin-android-docimporter-ng + v4.6.1 + + + true + full + false + bin\Debug + DEBUG; + prompt + 4 + true + x86 + + + true + bin\Release + prompt + 4 + true + x86 + + + + + ..\packages\Mono.Options.5.3.0.1\lib\net4-client\Mono.Options.dll + + + + + + + + + + + + + + + {A7F30A05-820C-4580-8246-B84D55147408} + Xamarin.Android.ApiTools.ParameterNameExtractor + + + + \ No newline at end of file