Skip to content

Commit 9abc2f1

Browse files
dacharycdacharyc
committed
Add a tool to write GitHub metrics to CSV
1 parent 443c183 commit 9abc2f1

8 files changed

Lines changed: 792 additions & 0 deletions

File tree

‎github-metrics/package-lock.json‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎github-reporting/README.md‎

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# GitHub Reporting
2+
3+
This tool reads GitHub metrics stored in MongoDB Atlas (written by the [github-metrics](../github-metrics/README.md) tool) and exports them to
4+
CSV files for reporting to external stakeholders.
5+
6+
## Overview
7+
8+
The tool exports metrics to three separate CSV files for easy import into Google Sheets:
9+
10+
- **summary.csv** - Core metrics per date/repo (clones, views, stars, forks, watchers)
11+
- **referrals.csv** - One row per referrer per date/repo
12+
- **top-paths.csv** - One row per path per date/repo
13+
14+
The Date, Owner, and Repository columns in each file allow you to join/link data across sheets for analysis.
15+
16+
## Prerequisites
17+
18+
**Atlas**:
19+
20+
- An Atlas Database User with read permissions for the **Developer Docs** -> **Project Metrics** project.
21+
- A valid connection string for the cluster above.
22+
23+
Contact a member of the Developer Docs team to be added to this project and get the connection string.
24+
25+
**System**:
26+
27+
- Node.js/npm installed
28+
29+
## Setup
30+
31+
1. **Create a `.env` file**
32+
33+
Create a `.env` file that contains the following:
34+
35+
```
36+
ATLAS_CONNECTION_STRING="yourConnectionString"
37+
```
38+
39+
Replace the placeholder value with your connection string.
40+
41+
> Note: The `.env` file is in the `.gitignore`, so no worries about accidentally committing credentials.
42+
43+
2. **Install the dependencies**
44+
45+
From the root of the directory, run:
46+
47+
```
48+
npm install
49+
```
50+
51+
## Usage
52+
53+
The tool supports two invocation methods: direct command-line arguments or a configuration file.
54+
55+
### Method 1: Direct Command-Line Arguments
56+
57+
Use the `export` command with options:
58+
59+
```bash
60+
node --env-file=.env index.js export [options]
61+
```
62+
63+
**Options:**
64+
65+
| Option | Description |
66+
|--------|--------------------------------------------------------------------------------------------------|
67+
| `-s, --start-date <date>` | Start date for the report (ISO format, e.g., 2024-01-01) |
68+
| `-e, --end-date <date>` | End date for the report (ISO format, e.g., 2024-12-31) |
69+
| `-p, --projects <projects...>` | Space-separated list of owner/repo projects (e.g., mongodb/docs mongodb/sample-app-nodejs-mflix) |
70+
| `-o, --output <path>` | Output directory for CSV files |
71+
72+
**Examples:**
73+
74+
```bash
75+
# Export all metrics from all projects
76+
node --env-file=.env index.js export -o my-report
77+
78+
# Export metrics for a specific date range
79+
node --env-file=.env index.js export -s 2024-01-01 -e 2024-12-31 -o q4-report
80+
81+
# Export metrics for specific projects
82+
node --env-file=.env index.js export -p mongodb/docs mongodb/docs-notebooks -o docs-report
83+
84+
# Combine all options
85+
node --env-file=.env index.js export -s 2024-01-01 -e 2024-03-31 -p mongodb/docs -o q1-docs-report
86+
```
87+
88+
### Method 2: Configuration File
89+
90+
Use the `export-config` command with a JSON configuration file:
91+
92+
```bash
93+
node --env-file=.env index.js export-config <config-file> [options]
94+
```
95+
96+
**Options:**
97+
98+
| Option | Description |
99+
|--------|-------------|
100+
| `-o, --output <path>` | Output directory for CSV files (overrides config file) |
101+
102+
**Example configuration file (`config.json`):**
103+
104+
```json
105+
{
106+
"startDate": "2025-01-01",
107+
"endDate": "2025-12-31",
108+
"projects": [
109+
{ "owner": "mongodb", "repo": "docs" },
110+
{ "owner": "mongodb", "repo": "docs-notebooks" }
111+
],
112+
"output": "annual-report"
113+
}
114+
```
115+
116+
**Run with config file:**
117+
118+
```bash
119+
node --env-file=.env index.js export-config config.json
120+
```
121+
122+
**Override output directory:**
123+
124+
```bash
125+
node --env-file=.env index.js export-config config.json -o different-output
126+
```
127+
128+
## Output
129+
130+
The tool creates a directory containing three CSV files:
131+
132+
```
133+
my-report/
134+
├── summary.csv
135+
├── referrals.csv
136+
└── top-paths.csv
137+
```
138+
139+
### summary.csv
140+
141+
| Column | Description |
142+
|--------|-------------------------------------------------|
143+
| Date | ISO timestamp of when metrics were collected |
144+
| Owner | GitHub organization/owner |
145+
| Repository | Repository name |
146+
| Clones | Number of clones in the last 14 days |
147+
| Page Views | Total page views in the last 14 days |
148+
| Unique Views | Unique visitors in the last 14 days |
149+
| Stars | Star count (cumulative total, current count) |
150+
| Forks | Fork count (cumulative total, current count) |
151+
| Watchers | Watcher count (cumulative total, current count) |
152+
153+
### referrals.csv
154+
155+
| Column | Description |
156+
|--------|-------------|
157+
| Date | ISO timestamp of when metrics were collected |
158+
| Owner | GitHub organization/owner |
159+
| Repository | Repository name |
160+
| Referrer | Traffic source (e.g., google.com, github.com) |
161+
| Count | Total visits from this referrer |
162+
| Uniques | Unique visitors from this referrer |
163+
164+
### top-paths.csv
165+
166+
| Column | Description |
167+
|--------|-------------|
168+
| Date | ISO timestamp of when metrics were collected |
169+
| Owner | GitHub organization/owner |
170+
| Repository | Repository name |
171+
| Path | Path within the repository |
172+
| Count | Total visits to this path |
173+
| Uniques | Unique visitors to this path |
174+
175+
## Importing to Google Sheets
176+
177+
1. Create a new Google Sheet
178+
2. Go to **File** → **Import**
179+
3. Upload each CSV file as a separate sheet
180+
4. Use the Date, Owner, and Repository columns to create relationships between sheets using VLOOKUP or pivot tables

‎github-reporting/config.json.example‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"startDate": "2026-01-01",
3+
"endDate": "2026-01-13",
4+
"projects": [
5+
{ "owner": "mongodb", "repo": "sample-app-java-mflix" },
6+
{ "owner": "mongodb", "repo": "sample-app-nodejs-mflix" },
7+
{ "owner": "mongodb", "repo": "sample-app-python-mflix" }
8+
],
9+
"output": "output"
10+
}

‎github-reporting/index.js‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
#!/usr/bin/env node
2+
3+
import { Command } from 'commander';
4+
import { readFileSync } from 'fs';
5+
import { readMetricsFromAtlas } from './read-from-db.js';
6+
import { writeMetricsToCsv, generateOutputDir } from './write-csv.js';
7+
8+
const program = new Command();
9+
10+
program
11+
.name('github-reporting')
12+
.description('Read GitHub metrics from MongoDB Atlas and export to CSV')
13+
.version('1.0.0');
14+
15+
// Direct invocation with command-line arguments
16+
program
17+
.command('export')
18+
.description('Export metrics to CSV using command-line arguments')
19+
.option('-s, --start-date <date>', 'Start date for the report (ISO format, e.g., 2024-01-01)')
20+
.option('-e, --end-date <date>', 'End date for the report (ISO format, e.g., 2024-12-31)')
21+
.option('-p, --projects <projects...>', 'List of owner/repo projects (e.g., mongodb/docs realm/realm-js)')
22+
.option('-o, --output <path>', 'Output directory for CSV files')
23+
.action(async (options) => {
24+
try {
25+
const dateRange = buildDateRange(options.startDate, options.endDate);
26+
const projects = parseProjects(options.projects);
27+
const outputDir = options.output || generateOutputDir(dateRange);
28+
29+
console.log('Fetching metrics from MongoDB Atlas...');
30+
if (dateRange.startDate || dateRange.endDate) {
31+
console.log(`Date range: ${dateRange.startDate || 'beginning'} to ${dateRange.endDate || 'now'}`);
32+
}
33+
if (projects.length > 0) {
34+
console.log(`Projects: ${projects.map(p => `${p.owner}/${p.repo}`).join(', ')}`);
35+
} else {
36+
console.log('Projects: all');
37+
}
38+
39+
const metrics = await readMetricsFromAtlas(dateRange, projects);
40+
await writeMetricsToCsv(metrics, outputDir);
41+
} catch (error) {
42+
console.error('Error:', error.message);
43+
process.exit(1);
44+
}
45+
});
46+
47+
// Config file invocation
48+
program
49+
.command('export-config')
50+
.description('Export metrics to CSV using a configuration file')
51+
.argument('<config-file>', 'Path to JSON configuration file')
52+
.option('-o, --output <path>', 'Output directory for CSV files (overrides config file)')
53+
.action(async (configFile, options) => {
54+
try {
55+
const config = loadConfig(configFile);
56+
const dateRange = buildDateRange(config.startDate, config.endDate);
57+
const projects = config.projects || [];
58+
const outputDir = options.output || config.output || generateOutputDir(dateRange);
59+
60+
console.log('Fetching metrics from MongoDB Atlas...');
61+
if (dateRange.startDate || dateRange.endDate) {
62+
console.log(`Date range: ${dateRange.startDate || 'beginning'} to ${dateRange.endDate || 'now'}`);
63+
}
64+
if (projects.length > 0) {
65+
console.log(`Projects: ${projects.map(p => `${p.owner}/${p.repo}`).join(', ')}`);
66+
} else {
67+
console.log('Projects: all');
68+
}
69+
70+
const metrics = await readMetricsFromAtlas(dateRange, projects);
71+
await writeMetricsToCsv(metrics, outputDir);
72+
} catch (error) {
73+
console.error('Error:', error.message);
74+
process.exit(1);
75+
}
76+
});
77+
78+
/**
79+
* Build a date range object from start and end date strings.
80+
*/
81+
function buildDateRange(startDate, endDate) {
82+
const dateRange = {};
83+
if (startDate) {
84+
dateRange.startDate = startDate;
85+
}
86+
if (endDate) {
87+
dateRange.endDate = endDate;
88+
}
89+
return dateRange;
90+
}
91+
92+
/**
93+
* Parse project strings (owner/repo format) into project objects.
94+
*/
95+
function parseProjects(projectStrings) {
96+
if (!projectStrings || projectStrings.length === 0) {
97+
return [];
98+
}
99+
100+
return projectStrings.map(projectStr => {
101+
const parts = projectStr.split('/');
102+
if (parts.length !== 2) {
103+
throw new Error(`Invalid project format: "${projectStr}". Expected format: owner/repo`);
104+
}
105+
return { owner: parts[0], repo: parts[1] };
106+
});
107+
}
108+
109+
/**
110+
* Load and parse a JSON configuration file.
111+
*/
112+
function loadConfig(configPath) {
113+
try {
114+
const content = readFileSync(configPath, 'utf-8');
115+
return JSON.parse(content);
116+
} catch (error) {
117+
if (error.code === 'ENOENT') {
118+
throw new Error(`Configuration file not found: ${configPath}`);
119+
}
120+
if (error instanceof SyntaxError) {
121+
throw new Error(`Invalid JSON in configuration file: ${error.message}`);
122+
}
123+
throw error;
124+
}
125+
}
126+
127+
program.parse();

0 commit comments

Comments
 (0)
⚡