Understanding the Limitations of Zenn's Dashboard
Zenn's dashboard offers a straightforward display of metrics such as page views and likes. However, it lacks a public stats endpoint and an export button, which can make data extraction cumbersome. Developers often resort to manual checking, leading to inefficiencies and potential oversights. This limitation creates a need for a more automated and reliable approach to collect and monitor these metrics.
For users who require consistent analytics tracking, these dashboard constraints can be frustrating. The absence of a direct export function means that data must be either scraped or manually recorded. Both methods are error-prone and demand significant time and effort, especially if changes occur in the dashboard's structure or CSS.
Why JSON Endpoints are Preferable to Scraping HTML
Scraping HTML might seem like a quick solution to extract data, but it is fraught with challenges. For instance, the initial approach of parsing Zenn's public article pages using BeautifulSoup proved unreliable. A simple CSS update by Zenn rendered the scraper ineffective, causing the collection process to fail unnoticed for days. This underscores the inherent fragility of relying on HTML scraping techniques.
An alternative and more robust method involves using Zenns private JSON endpoint. By observing the network requests made by the Zenn dashboard, it becomes clear that the endpoint provides structured data, including page views, likes, and comments. Accessing this endpoint requires a session cookie, which can be securely stored as an environment variable. This approach eliminates dependency on HTML structure and ensures greater reliability.
Designing a Resilient Data Collection Probe
The probe used for data collection is designed to fail fast and fail clean. Unlike conventional scripts that log warnings and continue operating under erroneous conditions, this probe raises an exception whenever something goes wrong. Whether its a missing field, an empty response, or an HTTP 401 error, the probe stops immediately. This ensures no faulty data compromises the dataset.
The key principle here is to delegate recovery to the scheduler rather than attempting to handle errors within the probe itself. By marking the GitHub Actions job as failed, the next scheduled run will automatically retry from a clean slate. This design choice makes the system more reliable over time.
Using SQLite for Efficient Data Storage
SQLite serves as the backbone for data storage in this setup. Each run of the probe writes a new row for every article, capturing details like page views and timestamps. The database is lightweight, portable, and well-suited for handling the time-series data generated by this process.
By committing data to SQLite after validating the response, the system ensures that only accurate and complete information is stored. If the response shape deviates from the expected format, the probe will terminate, preventing the insertion of any incorrect data. This practice enhances the integrity of the analytics data collected.
Automating with GitHub Actions
GitHub Actions provides a seamless way to schedule and automate the execution of the probe. By configuring a cron job, the probe can run at hourly intervals, ensuring up-to-date metrics. In the event of a failure, the next scheduled run will act as a clean retry, eliminating the need for manual intervention.
This automation framework not only saves time but also ensures consistent data collection. By integrating the probe with GitHub Actions, developers can focus on analyzing the data instead of worrying about its collection. This setup is both scalable and reliable, making it an ideal solution for tracking Zenn dashboard metrics.