From testing to monitoring: Datafold's unified data quality platform

We talk to data teams every day that have trouble ensuring data integrity across their entire data stack, scaling pipelines, preventing data quality issues, and keeping everything in sync across complex systems. Despite the proliferation of data quality vendors, it’s still too hard for these teams to build data quality programs that support every stage of the data development lifecycle.

Datafold is tackling this issue head on. Today we’re releasing Monitors, a powerful product that gives data teams real-time visibility into data issues across their entire stack. With this release, Datafold evolves into the only unified platform for proactive data quality, giving you more control and confidence at every stage of your data workflow.

Whether you’re tasked with streamlining CI/CD processes, handling large data migrations, or monitoring critical data pipelines, Datafold now offers powerful new tools to maintain data quality for every part of your data stack.

The new Datafold: Unified, proactive, powerful

The problem is (relatively) simple: Data teams today rely on multiple disjointed tools—or no tooling at all—to manage data quality, leaving major gaps in their data quality management. For data teams, the result of this disjointed tooling and lack of automated processes is often slow development time, increasing data quality incidents (in both volume and severity), poor decision-making, and reduced trust with the business.

We also know that data quality has traditionally been a reactive process—issues are often only discovered and resolved once they’ve hit production (or a stakeholder’s hands). Datafold's focus is on tooling that shifts data quality to the left—catching issues as early as possible, or even preventing them from happening in the first place.

With the introduction of Monitors, we further enable this approach by offering data monitoring that spans across your entire stack—from “from reconciling source-target replication upstream to detecting anomalies in the transformation layer.

Datafold now offers automated reconciliation for data migrations, comprehensive CI/CD testing, and proactive data observability—all in one unified platform.

• Data Migration Acceleration: For teams migrating to new databases or transformation workflows, our AI-powered Data Migration Agent does the heavy lifting for you to automatically convert SQL scripts and validate cross-database parity.
• CI/CD Testing: We believe integrating data quality into CI/CD pipelines is critical for catching issues early in the development and deployment cycle. Datafold automates comprehensive testing within the CI/CD process for any code-based transformation workflow (e.g., dbt, stored procedures, Airflow), ensuring that your data transformation code is rigorously tested before deploying to production.
• Data Monitoring and Observability: Monitors provide automated, real-time tracking of your data, detecting issues like upstream data replication issue, schema changes, and anomalies in key metrics. With customizable monitors and alerts, be the first to detect, investigate, and resolve data quality issues.

To learn more about Monitors and the new Datafold experience, please join us at our upcoming webinar on October 24th at 1pm EST.

Introducing Monitors: Real-time data monitoring across your entire stack

With real-time Monitors, your team gains better control of data quality across upstream sources and production environments, and the detailed information needed to resolve data quality issues quickly.

We now support the following monitor types:

• Data Diffs: Instantly detect value-level differences between source and target databases, ensuring accuracy of mission-critical data replication.
• Metrics: Use machine learning to detect anomalies in key metrics like row count, data freshness, or any metric defined in SQL.
• Data Tests: Automatically validate data using business rules to identify records that don’t meet your criteria.
• Schema Change Alerts: Receive alerts when any table structure changes or column types change, preventing silent failures in the data you don’t control.

With Monitors, get notified right in Slack, Teams, or PagerDuty so your team never misses an issue. And because Monitors are built for data engineers, teams can manage them through both the UI and as code—giving you flexibility in how you create and maintain your monitoring workflows at scale.

‍

What we’re focused on next

Just like software quality, data quality is a complex, multifaceted challenge that requires targeted tools, proper processes, and decision support for practitioners. As we expand our use cases, we will continue to focus on:

1. Expanding integrations across the data stack to provide deeper insights into potential sources and impacts of data incidents.
2. Harnessing valuable metadata on data changes and lineage to enhance root cause analysis and reduce incident triage and response times.
3. Developing comprehensive how-tos and tutorials for various platforms, enabling teams to implement proactive testing capabilities regardless of their chosen orchestrator or framework.

There's still significant manual effort in the data engineer's workflow that we're eager to automate, and our focus remains on helping our users deliver high-quality data faster and more efficiently.

To learn more about Monitors and the new Datafold experience, please join us at our upcoming webinar on October 24th at 1pm EST.