5.21 source-tree — a project's source in any of fourteen languages

A source-tree producer reads a project’s source files in TypeScript (and TSX), JavaScript (and JSX), Python, Go, Rust, C, C++, C#, Ruby, PHP, Kotlin, Scala, Swift and Java, parsing each with that language’s grammar when a query reaches it. Nothing is ingested. Java has emits of its own besides, which name types by binary name and say what the file alone shows about types (below), and the kind also reads the source a jar was compiled from (from: jar), with the rows of §5.19.

- { name: pyDeclarations, kind: source-tree, emit: declarations, languages: [python] }
- { name: allFiles,       kind: source-tree, emit: files }                 # every language
- { name: goCalls,        kind: source-tree, emit: calls, languages: [go], maxFileKb: 256 }
- { name: javaMetrics,    kind: source-tree, emit: captures, languages: [java], query: queries/java-metrics.scm }

// Which handler serves a route, whatever the language: decorators, annotations and attributes alike
MATCH (p:CodeProject {key: 'https://github.com/acme/shop'})-[:DECLARES]->(d:SourceDeclaration)
WHERE any(a IN d.annotationText WHERE a CONTAINS '/items/{id}')
RETURN d.language, d.owner, d.name, d.annotationText, d.path, d.fromLine

The contract:

• The key is a project’s location: an https git URL, optionally @ref (a commit, tag or branch; the default branch without one), read at the commit it resolves to; or a folder under the world’s data/ tree, an operator-trusted root, or the folder the operator mounted for project keys. A git project is read once per commit and kept; a folder is read afresh for each fetch. A tag or branch, the default branch included, is resolved to its commit at most once a minute, so a pushed commit is read by the first fetch a minute after it.

• Five emits, each row in the columns of a standard-model parent (and a sixth, captures, and a seventh, version, below), so a realm’s language children of those parents (PythonDeclaration under SourceDeclaration) carry them under the same names as Java’s:

  • files (Source): root, path, language, lines, sha256, commit (git), url (on a host with a file view), parsed, and errors — the number of syntax errors the file’s tree holds. Error recovery keeps everything around a broken region, so a file with errors still answers.
  • declarations (SourceDeclaration): each class, interface, struct, enum, trait, module, namespace, impl, function, method, constructor, field, variable and constant the language has: kind, name, owner (the enclosing types, outermost first, .-joined; a Go receiver or C++’s Order::total names its type explicitly), fromLine / toLine, signature (up to the body, on one line, without annotations), javadoc (the doc comment above it, or a docstring), the numbered text of a non-type declaration (at most 400 lines), annotations (the names of its decorators, annotations or attributes) and annotationText (each in full).
  • imports (SourceImport): imported (what it imports from: a module, package, path or header), names (what it brings in, in source order; empty for a whole module), alias, isStatic (false).
  • calls (CallSite): method (the called name), receiver (what it is called on, as written), arguments (each as written), enclosing (the function or method it is in — for a call inside a decorator or attribute, the declaration it decorates), receiverType and argumentTypes (empty: syntax does not say) and resolution, always name.
  • references (SourceReference): each name used where a type goes — a parameter’s, field’s or return type, a base class or interface, a generic argument, an annotation, a constructor call — as name (the last segment, as written), enclosing (the innermost declaration it is used in: a field’s type is used in the field; empty at the top level) and resolution, always name. A declaration’s own name is not a reference (class Order does not use Order), nor is a generic parameter inside the declaration that declares it (T in class Box<T>). A built-in type (string, int) is a reference like any other; join to the project’s declarations to keep the project’s own. Every row carries id, sourceId (its file’s id), path, language, line, url, project and anchorKey.

• version says whether the project changed, without reading it. One row per project key: project (as read), commit (a repository’s, the default branch or @ref resolved as above) and version — the commit, or for a folder a digest of every file’s path, size and modification time. No file is read or parsed, so it is cheap enough to ask every minute; it is what a documents source watches (README, sync.watch) to re-read a project only when a commit lands. A folder’s version moves when a file is added, removed or rewritten, even with the same bytes.

• id names a row where it was read, not what it is. It holds the project as read (host/repo@<commit> for a repository) and the row’s line and column, so it changes with every commit and whenever the code above it moves. It is the key to join rows of one read (enclosingId = d.id). To keep anything about a declaration from one read to the next — a document named after it, a stored note — name it by the project key, path, kind, owner and name, numbering the second and later of one name in a file by line; and leave text’s line numbers out of anything compared across reads.

• A call or a reference is matched by name. resolution: name says the callee or type is only known by its name: any function or type of that name may be the one meant. Present such a row as “calls something named save” or “uses a type named Order”, never as a resolved edge; where two of the project’s types share a name, say so. Java’s rows say exact (from bytecode, §5.18) or type (the receiver’s static type is known, §5.19).

• languages narrows what is read to the named languages (empty reads them all). A name no language has is refused with a PRODUCER_ERROR warning listing the languages there are.

• captures runs the realm’s own query. query: names a tree-sitter query file (.scm) in the realm, relative to its root, and languages: names the one language whose grammar it is written against. Each node the query captures is one row: capture (the capture’s name, without @), nodeType (the grammar’s name for the node, cast_expression), text (the node’s text, whitespace collapsed, at most 200 characters), line / column / toLine / toColumn (where its first and last characters are), match (an id shared by the captures of one match, so a cast and its @cast.type are read together), pattern (the pattern’s index in the query, from 0), and the innermost declaration it is in: enclosingId (that declaration’s declarations row id, so WHERE c.enclosingId = d.id joins them), enclosing (its name) and enclosingKind (empty at the top level). A node two patterns capture under one name is one row. Text predicates (#eq?, #not-eq?, #match?, #any-of?) are applied.

  ; queries/java-metrics.scm
  (if_statement) @if
  [(for_statement) (enhanced_for_statement) (while_statement) (do_statement)] @loop
  (cast_expression type: (_) @cast.type) @cast

  // Ifs per method, longest first; commented-out code is a comment node, so it never counts
  MATCH (p:CodeProject {key: $project})-[:CAPTURES]->(c:JavaCapture {capture: 'if'})
  RETURN c.path, c.enclosing, count(*) AS ifs ORDER BY ifs DESC

  Refused with a PRODUCER_ERROR warning and no rows: a query file the realm does not have, one with a character outside ASCII (tree-sitter would silently lose its end; the warning names the line), and one that does not compile against the grammar (the warning gives tree-sitter’s row and column). Refused when the realm is loaded: query: on another emit, captures without query:, a query: that leaves the realm (.., a leading /), and captures naming other than one language. A file capturing more than maxCaptures nodes (default 10000, at most 100000) answers its first maxCaptures and is reported. Captures are not kept: each fetch parses the files again.

• Each file is bounded. A file larger than maxFileKb (default 512, at most 4096) is not read; a file that takes longer than parseTimeoutMs (default 2000, at most 60000) to parse is abandoned; past maxFiles (default 20000) source files the rest are not read. Each is a PRODUCER_ERROR warning naming the file, never a half-read file. A key that names no readable project is a warning with the reason.

• What is not read. Directories a build or package manager writes (node_modules, target, build, out, .git, …) are skipped. Generated files are read like any other.

• Java’s own emits, by binary name. A name is resolved as the file resolves it, from the file alone: written qualified, a type the file declares (pkg.Outer$Inner), a single-type import, java.lang, or (with no on-demand import that could supply it) the file’s own package. Where an on-demand import could, the name stays as written and the row says it is unresolved.

  • types: each class, interface, enum, record and annotation type a project declares outside a method or anonymous class — name (binary), simpleName, packageName, kind, extends and implements (resolved), annotations (resolved), annotationText, modifiers, isTopLevel, line, lastLine, module (the Maven module whose pom.xml directory holds the file; '' for the root), link.
  • members: each field (one per variable), constructor and method of those types — owner (binary), name, kind, signature, type (a field’s element type, resolved), returnType, parameters, parameterTypes (resolved), annotations, modifiers, and the declaration columns (fromLine, toLine, text, javadoc, annotationText). Id: <project>!<owner>#<name>(<parameter types>).
  • annotations: each annotation written on a type, member or parameter, with its attributes as written — name, qualifiedName (null where the file cannot say), attributes (name=value), value/values, text, targetKind, targetId, targetName, owner.
  • imports of a Java file carry, beside every language’s columns, name, isStatic, isWildcard and packageName.
  • configs: every key of the project’s .properties and YAML files, flattened to a dotted path (server.ssl.enabled, servers[0].url), with value as written, document (of a multi-document YAML file, from 0), fileName, module and line. YAML is composed, never constructed.
  • projectFiles: every file of the project, whatever its language — path, fileName, extension, module, link.

• from: jar: the source a jar was compiled from. The key is what a jar row (§5.18) carries: a sourceKey (<jar path>!com/x/Cache.java) or a spanKey (…#<line>#<name>). The file is looked for beside the jar as a -sources.jar, then under roots: in order, as §5.19 says, with §5.19’s emits, columns and ids: files, spans (context, default 6), declarations, calls (the calls on the key’s line, with receiverType, argumentTypes, argumentTypesResolved and resolution: type where the receiver’s static type is known) and locals. maxFileKb defaults to 1024 for jar keys. A file in another language answers files and spans, with a warning for the rest; a file with syntax errors answers around them.

  - { name: javaSourceSpan,  kind: source-tree, from: jar, emit: spans, roots: [src], context: 6 }   # key: a spanKey
  - { name: javaCallSitesAt, kind: source-tree, from: jar, emit: calls, roots: [src] }               # key: a spanKey
  - { name: upstream,        kind: source-tree, from: jar, emit: declarations,
      roots: [ { git: "https://github.com/acme/acme", ref: "v{version}" } ] }

  An argument’s type is what the file shows: new X(…), a literal, a cast, a parameter (a method’s, a typed lambda’s, a catch clause’s), a local declared before the call, a field of an enclosing type. A method’s return value, an inherited field, a parenthesized expression and a lambda’s inferred parameter are unknown (""), never guessed. Expressions (arguments, receiver, initializer, a call’s text) are as written, whitespace collapsed.

Cost: the first query of a git project pays for a shallow, blob-less fetch of the commit and one fetch of the language files’ contents; each file is parsed once, the first time an emit needs it, and kept with the commit. A folder is listed and its files parsed again on each fetch. A repository of a few hundred files is read, from a cold git cache, in about ten seconds.